Automatic laser emission direction adjusting method and system based on unmanned aerial vehicle

Abstract

The invention discloses an unmanned aerial vehicle-based laser direction automatic adjustment method and system, and relates to the technical field of unmanned aerial vehicles, and the method comprises the steps: collecting IMU, GPS, laser radar and visible light camera data in real time, and carrying out the primary processing; fusing the processed data through a Kalman filtering algorithm, and estimating the attitude of the unmanned aerial vehicle in real time; constructing a deep learning model, and automatically identifying the contour and direction of the target object; calculating the offset distance from the central point of the image to the target object according to the contour and direction of the target object; an environment model is established, and compensation parameters of the laser emission direction are calculated; dynamically adjusting the laser direction according to the attitude estimation result, the offset distance from the central point of the image to the target object and the compensation parameter of the laser direction; the center point coordinate of the target object in the image is determined according to the contour and direction information of the target object, the laser emitting direction is adjusted in real time, and accurate pointing of the laser can be kept even if the unmanned aerial vehicle moves or the target object moves.

Description

Technical Field

[0001] The invention relates to the technical field of unmanned aerial vehicles, and in particular to an automatic laser direction adjustment method and system based on unmanned aerial vehicles. Background Art

[0002] In recent years, with the rapid development and widespread application of drone technology, drones are increasingly used in agriculture, surveying and mapping, security, logistics and other fields. Especially in the fields of precision agriculture, environmental monitoring and emergency rescue, the autonomous navigation and target recognition capabilities of drones have become a hot topic of research. Traditional drone navigation and target recognition mainly rely on a single sensor, which performs relatively stably in static or simple environments, but in complex dynamic environments, it is difficult to achieve high-precision navigation and target recognition due to signal interference, data delay and other issues.

[0003] Although multi-sensor fusion technology and deep learning have made significant progress in UAV navigation and target recognition, existing technical solutions still have shortcomings in some aspects. First, when processing multi-sensor data, the traditional Kalman filter algorithm often assumes that the sensor error is Gaussian distributed, which is not always true in practical applications, especially in complex dynamic environments. The nonlinearity and uncertainty of sensor data will lead to deviations in the filtering results. Secondly, although the existing target recognition methods can achieve high-precision object recognition, their real-time performance in dynamic environments still needs to be improved. In addition, the existing laser direction adjustment methods mainly rely on static environmental models, lack adaptability to dynamic environments, and cannot maintain the precise pointing of the laser when the UAV moves or the target object moves. Summary of the invention

[0004] In view of the above existing problems, the present invention is proposed.

[0005] Therefore, the present invention provides a method and system for automatically adjusting the laser direction of a UAV based on the UAV to solve the problems of low laser direction accuracy and insufficient real-time performance of UAVs in complex dynamic environments in the prior art.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] In a first aspect, an embodiment of the present invention provides a method for automatically adjusting the laser direction based on a drone, which includes real-time collection of IMU, GPS, laser radar and visible light camera data, and preliminary processing;

[0008] The processed data is fused through the Kalman filter algorithm to estimate the attitude of the drone in real time;

[0009] Build a deep learning model to automatically identify the contour and orientation of target objects;

[0010] According to the contour and direction of the target object, calculate the offset distance from the center point of the image to the target object;

[0011] Establish an environmental model and calculate the compensation parameters of the laser direction;

[0012] The laser direction is dynamically adjusted according to the posture estimation result, the offset distance from the image center to the target object and the compensation parameters of the laser direction.

[0013] As a preferred solution of the automatic adjustment method of laser direction based on drone of the present invention, the real-time collection of IMU, GPS, laser radar and visible light camera data and preliminary processing include the following steps:

[0014] Plan the flight path of the drone and set the latitude, longitude and altitude of the target point on the ground station;

[0015] Start the drone and perform preliminary positioning through the inertial navigation system and GPS;

[0016] When the UAV reaches the target area, the IMU is activated to collect the angular velocity and acceleration data of the UAV, the GPS is activated to collect the latitude and longitude coordinates and altitude data of the UAV, the laser radar is activated to collect the three-dimensional point cloud data around the target point, and the visible light camera is activated to collect the image data of the target point.

[0017] The collected data are cleaned, filtered, noise suppressed and aligned.

[0018] As a preferred solution of the automatic adjustment method of laser direction based on UAV of the present invention, the processed data is fused by Kalman filter algorithm, and the attitude of the UAV is estimated in real time, including the following steps:

[0019] Initialize the state vector and covariance matrix;

[0020] Use GPS data and IMU data to update the state vector and covariance matrix to obtain a preliminary estimate of the drone’s attitude;

[0021] Use the ICP algorithm to align the point cloud data collected by the lidar with the map point cloud to obtain the relative pose change;

[0022] The relative pose change obtained by point cloud registration is applied to the updated state vector of the Kalman filter to obtain the corrected pose estimate of the UAV.

[0023] As a preferred solution of the automatic adjustment method of laser direction based on drone of the present invention, the construction of deep learning model and automatic recognition of the outline and direction of the target object includes the following steps:

[0024] Collect a large amount of image data containing target objects, annotate the contour and orientation information of the target objects, and build a training data set;

[0025] Use the labeled dataset to train the convolutional neural network model and use the loss function to optimize the bounding box regression and orientation angle classification of the target object;

[0026] During the flight of the drone, image data is captured in real time and input into the trained convolutional neural network model for target recognition to extract the contour and direction information of the target object. The expression is:

[0027]

[0028] Among them, O is the contour and direction information of the target object, C is the contour of the target object, I is the input image, α is the direction angle of the target object, i is the contour index of the target object, N is the total number of points on the contour of the target object, and x i and i are the horizontal and vertical coordinates of the i-th point on the contour of the target object, respectively. and are the average abscissa and ordinate of the contour points, respectively.

[0029] As a preferred solution of the automatic adjustment method of laser direction based on drone of the present invention, wherein: according to the contour and direction of the target object, calculating the offset distance from the center point of the image to the target object includes the following steps:

[0030] According to the contour and direction information of the target object, determine the coordinates of the center point of the target object in the image;

[0031] Use the camera's intrinsic parameters to convert the center point coordinates of the target object in the image into the distance in the physical world, and calculate the offset of the target object in the physical world. The expression is:

[0032]

[0033] Among them, ΔX is the horizontal offset of the target object in the physical world, ΔY is the vertical offset of the target object in the physical world, w is the width of the image, h is the height of the image, Z is the distance from the target object to the camera, and f x is the horizontal focal length of the camera, f y is the vertical focal length of the camera.

[0034] As a preferred solution of the automatic adjustment method of laser direction based on UAV of the present invention, the steps of establishing an environmental model and calculating the compensation parameters of the laser direction include the following steps:

[0035] The UAV attitude and position information estimated by the Kalman filter is used to transform the point cloud data collected by the lidar into a global coordinate system to form a three-dimensional point cloud map;

[0036] Use meshing methods to convert point cloud data into a three-dimensional model of the surrounding environment;

[0037] Determine the position of the target point that the laser is directed to in the three-dimensional model;

[0038] According to the position of the laser to the target point, the position and distance of the obstacles on the laser path are calculated. The expression is:

[0039]

[0040] Where d is the distance to the nearest obstacle on the laser path, P global is the coordinate of the point cloud data in the global coordinate system, p is a point in the point cloud data, and p tatget is the coordinate of the target point in the global coordinate system;

[0041] According to the position and distance of obstacles on the laser path, the compensation parameters of the laser direction are designed, and the expression is:

[0042]

[0043] Among them, Δθ is the pitch angle compensation parameter of the laser direction, Δφ is the yaw angle compensation parameter of the laser direction, and k d is the proportionality coefficient, and D is the maximum effective distance of the laser.

[0044] As a preferred solution of the automatic laser direction adjustment method based on drone of the present invention, the laser direction is dynamically adjusted according to the attitude estimation result, the offset distance from the image center point to the target object and the compensation parameter of the laser direction, including the following steps:

[0045] Design a control algorithm that takes the attitude estimation result, the offset distance from the image center to the target object, and the compensation parameters of the laser direction as input, and outputs the laser angle adjustment instruction so that the laser always points to the target object. The expression is:

[0046]

[0047] Among them, θ cmd is the pitch angle adjustment instruction of the laser, φ cmd is the yaw angle adjustment instruction of the laser, φ is the pitch angle of the drone, θ is the roll angle of the drone, k X is the proportional coefficient of the horizontal offset, k Y is the scaling factor for the vertical offset.

[0048] In a second aspect, the present invention provides a laser direction automatic adjustment system based on an unmanned aerial vehicle, comprising:

[0049] Data acquisition module, which collects IMU, GPS, lidar and visible light camera data in real time and performs preliminary processing;

[0050] The data fusion module fuses the processed data through the Kalman filter algorithm to estimate the attitude of the UAV in real time;

[0051] The target recognition module builds a deep learning model to automatically identify the outline and direction of the target object;

[0052] An offset calculation module calculates the offset distance from the center point of the image to the target object according to the contour and direction of the target object;

[0053] Environmental modeling module, which builds environmental model and calculates compensation parameters of laser direction;

[0054] The laser direction adjustment module dynamically adjusts the laser direction according to the posture estimation result, the offset distance from the image center point to the target object and the compensation parameters of the laser direction.

[0055] In a third aspect, an embodiment of the present invention provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program is executed by the processor, any step of the method for automatic adjustment of laser direction based on a drone as described in the first aspect of the present invention is implemented.

[0056] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, any step of the method for automatic adjustment of laser direction based on a drone as described in the first aspect of the present invention is implemented.

[0057] The beneficial effects of the present invention are: real-time collection of IMU, GPS, lidar and visible light camera data for preliminary processing; fusing the processed data through the Kalman filter algorithm to estimate the attitude of the drone in real time; building a deep learning model to automatically identify the contour and direction of the target object; calculating the offset distance from the center point of the image to the target object according to the contour and direction of the target object; establishing an environmental model to calculate the compensation parameters of the laser direction; dynamically adjusting the laser direction according to the attitude estimation result, the offset distance from the center point of the image to the target object and the compensation parameters of the laser direction; by determining the center point coordinates of the target object in the image according to the contour and direction information of the target object, an accurate reference point is provided for subsequent offset distance calculation, and the system can adjust the laser direction in real time, and can maintain the precise pointing of the laser even when the drone moves or the target object moves. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0059] Figure 1 This is a flow chart of the automatic adjustment method of laser direction based on a drone in Example 1.

[0060] Figure 2 This is a flow chart of the automatic laser direction adjustment system of the UAV in Example 1.

[0061] Figure 3 Computer equipment diagram for sequential acquisition method of multi-resource inertia and primary frequency regulation auxiliary services. DETAILED DESCRIPTION

[0062] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0063] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0064] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0065] Example 1, reference Figure 1-Figure 3 , which is the first embodiment of the present invention, provides a method for automatically adjusting the laser direction based on a drone, comprising the following steps:

[0066] S100 collects IMU, GPS, LiDAR and visible light camera data in real time and performs preliminary processing including the following steps:

[0067] Plan the flight path of the UAV and set the latitude and longitude coordinates and altitude of the target point on the ground station; start the UAV and perform preliminary positioning through the inertial navigation system and GPS; when the UAV reaches the target area, start the IMU to collect the UAV's angular velocity and acceleration data, the GPS to collect the UAV's latitude and longitude coordinates and altitude data, the lidar to collect the three-dimensional point cloud data around the target point, and the visible light camera to collect the image data of the target point; perform data cleaning, data filtering, noise suppression and data alignment on the collected data.

[0068] It should be noted that by planning the flight path of the UAV and setting the latitude and longitude coordinates and altitude of the target point on the ground station, accurate planning of the UAV mission is achieved, ensuring that the UAV can fly according to the predetermined path, improving the efficiency and safety of mission execution; by starting the UAV and performing preliminary positioning through the inertial navigation system and GPS, the initial positioning of the UAV is achieved, laying the foundation for subsequent precise navigation and control, and ensuring the stability and reliability of the UAV during the take-off phase; when the UAV reaches the target area, the IMU is started to collect the angular velocity and acceleration data of the UAV, the GPS collects the latitude and longitude coordinates and altitude data of the UAV, the lidar collects the three-dimensional point cloud data around the target point, and the visible light camera collects the image data of the target point, realizing the real-time collection of multi-sensor data, providing rich environmental information, and providing basic data support for subsequent data processing and fusion; the collected data is cleaned, filtered, noise suppressed and aligned, realizing data preprocessing, removing noise and outliers, improving the quality and reliability of the data, and providing high-quality data input for subsequent data fusion and target recognition.

[0069] S200, fusing the processed data through the Kalman filter algorithm, and estimating the attitude of the drone in real time includes the following steps:

[0070] The initialization state vector includes the attitude (pitch angle, roll angle and yaw angle) and covariance matrix of the UAV; the state vector and covariance matrix are updated using GPS data and IMU data to obtain a preliminary attitude estimate of the UAV; the point cloud data collected by the lidar is aligned with the map point cloud using the ICP algorithm (iterative closest point algorithm) to obtain relative posture changes; the relative posture changes obtained by point cloud alignment are applied to the state vector updated by the Kalman filter to obtain a corrected attitude estimate of the UAV.

[0071] It should be noted that by initializing the state vector and covariance matrix, the initialization of the Kalman filter is achieved, which provides initial conditions for subsequent state estimation and ensures the stability and convergence of the filter; by using GPS data and IMU data to update the state vector and covariance matrix, a preliminary attitude estimation of the UAV is obtained, and the preliminary fusion of multi-sensor data is achieved, which improves the accuracy and robustness of attitude estimation and reduces the error influence of a single sensor; by using the ICP algorithm to align the point cloud data collected by the lidar with the map point cloud to obtain relative posture changes, the accurate alignment of the lidar data is achieved, high-precision relative posture information is provided, and the accuracy of attitude estimation is further improved; by applying the relative posture changes obtained by point cloud alignment to the updated state vector of the Kalman filter, the corrected attitude estimation of the UAV is obtained, and the deep fusion of multi-sensor data is achieved, which further improves the accuracy and robustness of attitude estimation and ensures the stability and reliability of the UAV in complex dynamic environments. S300, building a deep learning model, automatically identifying the contour and direction of the target object includes the following steps,

[0072] Collect a large amount of image data containing target objects, annotate the contour and orientation information of the target objects, and build a training data set; use the annotated data set to train the convolutional neural network model, and use the loss function to optimize the bounding box regression and orientation angle classification of the target object; during the flight of the drone, capture image data in real time, input the trained convolutional neural network model for target recognition, and extract the contour and orientation information of the target object. The expression is:

[0073]

[0074] Among them, O is the contour and direction information of the target object, C is the contour of the target object, I is the input image, α is the direction angle of the target object, i is the contour index of the target object, N is the total number of points on the contour of the target object, and x i and i are the horizontal and vertical coordinates of the i-th point on the contour of the target object, respectively. and are the average abscissa and ordinate of the contour points, respectively.

[0075] It should be noted that by collecting a large amount of image data containing target objects, marking the contour and direction information of the target objects, and constructing a training data set, a high-quality training data set is constructed, which provides sufficient data support for the training of the deep learning model and improves the generalization ability and recognition accuracy of the model; by using the labeled data set to train the convolutional neural network model, and using the loss function to optimize the bounding box regression and direction angle classification of the target object, the training and optimization of the deep learning model are realized, the recognition accuracy of the target object and the accuracy of direction estimation are improved, and reliable recognition results are provided for subsequent target tracking and laser direction adjustment; by capturing image data in real time during the flight of the UAV, inputting the trained convolutional neural network model for target recognition, and extracting the contour and direction information of the target object, real-time recognition and direction estimation of the target object are realized, the real-time performance of the system is improved, and precise operation in a dynamic environment is ensured.

[0076] S400, calculating the offset distance from the center point of the image to the target object according to the contour and direction of the target object includes the following steps:

[0077] According to the contour and direction information of the target object, the coordinates of the center point of the target object in the image are determined; the coordinates of the center point of the target object in the image are converted into the distance in the physical world using the intrinsic parameters of the camera, and the offset of the target object in the physical world is calculated. The expression is:

[0078]

[0079] Among them, ΔX is the horizontal offset of the target object in the physical world, ΔY is the vertical offset of the target object in the physical world, w is the width of the image, h is the height of the image, Z is the distance from the target object to the camera, and f x is the horizontal focal length of the camera, f y is the vertical focal length of the camera.

[0080] It should be noted that by determining the coordinates of the center point of the target object in the image based on the contour and direction information of the target object, the target object is accurately positioned, providing an accurate reference point for the subsequent offset distance calculation; by using the camera's internal parameters to convert the coordinates of the center point of the target object in the image into the distance in the physical world, the offset of the target object in the physical world is calculated, thus achieving accurate conversion from image coordinates to the physical world, providing high-precision offset distance information, and providing an accurate basis for adjusting the laser direction.

[0081] S500, establishing an environment model, and calculating compensation parameters of the laser direction include the following steps:

[0082] Using the Kalman filter to estimate the drone's attitude and position information, the point cloud data collected by the lidar is converted into a global coordinate system to form a three-dimensional point cloud map; using the gridding method, the point cloud data is converted into a three-dimensional model of the surrounding environment; the position of the target point that the laser is shooting at in the three-dimensional model is determined; according to the position of the target point that the laser is shooting at, the position and distance of the obstacles on the laser shooting path are calculated, and the expression is:

[0083]

[0084] Where d is the distance to the nearest obstacle on the laser path, P global is the coordinate of the point cloud data in the global coordinate system, p is a point in the point cloud data, and p target is the coordinate of the target point in the global coordinate system;

[0085] According to the position and distance of obstacles on the laser path, the compensation parameters of the laser direction are designed, and the expression is:

[0086]

[0087] Among them, Δθ is the pitch angle compensation parameter of the laser direction, Δφ is the yaw angle compensation parameter of the laser direction, and k d is the proportionality coefficient, and D is the maximum effective distance of the laser.

[0088] It should be noted that by using the UAV attitude and position information estimated by the Kalman filter, the point cloud data collected by the lidar is converted into a global coordinate system to form a three-dimensional point cloud map, realizing the global coordinate conversion of the point cloud data and providing accurate three-dimensional data support for environmental modeling; by using the gridding method, the point cloud data is converted into a three-dimensional model of the surrounding environment, the construction of the environmental model is realized, and detailed environmental information is provided, which provides a basis for the path planning and obstacle detection of the laser shooting; by determining the position of the target point of the laser shooting in the three-dimensional model and calculating the position and distance of the obstacles on the laser shooting path, the accurate planning of the laser shooting path is realized, the basis for obstacle detection and obstacle avoidance is provided, and the safety and effectiveness of the laser shooting is ensured; by designing the compensation parameters of the laser shooting according to the position and distance of the obstacles on the laser shooting path, the dynamic adjustment of the laser shooting is realized, the accuracy and robustness of the laser shooting is improved, and the precise operation in complex environments is ensured.

[0089] S600, dynamically adjusting the laser direction according to the posture estimation result, the offset distance from the image center point to the target object and the compensation parameter of the laser direction includes the following steps:

[0090] Design a control algorithm that takes the attitude estimation result, the offset distance from the image center to the target object, and the compensation parameters of the laser direction as input, and outputs the laser angle adjustment instruction so that the laser always points to the target object. The expression is:

[0091]

[0092] Among them, θ cmd is the pitch angle adjustment instruction of the laser, φ cmd is the yaw angle adjustment instruction of the laser, φ is the pitch angle of the drone, θ is the roll angle of the drone, k X is the proportional coefficient of the horizontal offset, k Y is the scaling factor for the vertical offset.

[0093] Furthermore, θ cmd and φ cmd The value range of is [-π, π], which represents the pitch angle and yaw angle adjustment instructions of the laser;

[0094] When θ cmd A positive value indicates an upward adjustment, and a negative value indicates a downward adjustment; when φ cmd A positive value indicates adjustment to the right, and a negative value indicates adjustment to the left.

[0095] It should be noted that by designing a control algorithm, taking the attitude estimation result, the offset distance from the image center point to the target object, and the compensation parameters of the laser direction as input, and outputting the angle adjustment instruction of the laser, dynamic adjustment of the laser direction is achieved, ensuring that the laser always points to the target object, and can remain accurate even when the drone or the target object moves; by taking the attitude estimation result, the offset distance from the image center point to the target object, and the compensation parameters of the laser direction as input, and outputting the angle adjustment instruction of the laser, comprehensive consideration of multiple factors is achieved, the accuracy and robustness of the laser direction are improved, and precise operation in complex dynamic environments is ensured.

[0096] This embodiment also provides a laser direction automatic adjustment system based on a drone, comprising:

[0097] The data acquisition module collects IMU (inertial measurement unit), GPS, lidar and visible light camera data in real time and performs preliminary processing; the data fusion module fuses the processed data through the Kalman filter algorithm to estimate the drone's attitude in real time; the target recognition module builds a deep learning model to automatically identify the contour and direction of the target object; the offset calculation module calculates the offset distance from the center point of the image to the target object based on the contour and direction of the target object; the environmental modeling module establishes an environmental model and calculates the compensation parameters of the laser direction; the laser direction adjustment module dynamically adjusts the laser direction based on the attitude estimation result, the offset distance from the center point of the image to the target object and the compensation parameters of the laser direction.

[0098] This embodiment also provides a computer device, which is suitable for the automatic adjustment method of laser direction based on a drone, including: a memory and a processor; the memory is used to store computer executable instructions, and the processor is used to execute computer executable instructions to implement the automatic adjustment method of laser direction based on a drone as proposed in the above embodiment.

[0099] The computer device may be a terminal, and the computer device includes a processor, a memory, a communication interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covering the display screen, or a key, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

[0100] The present embodiment also provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for automatically adjusting the laser direction based on an unmanned aerial vehicle as proposed in the above embodiment; the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic memory, a flash memory, a disk or an optical disk.

[0101] In summary, the present invention achieves the following by: collecting IMU, GPS, lidar and visible light camera data in real time and performing preliminary processing; fusing the processed data through the Kalman filter algorithm and estimating the attitude of the drone in real time; building a deep learning model to automatically identify the contour and direction of the target object; calculating the offset distance from the center point of the image to the target object according to the contour and direction of the target object; establishing an environmental model to calculate the compensation parameters of the laser direction; dynamically adjusting the laser direction according to the attitude estimation result, the offset distance from the center point of the image to the target object and the compensation parameters of the laser direction; determining the center point coordinates of the target object in the image according to the contour and direction information of the target object, providing an accurate reference point for subsequent offset distance calculation, and the system can adjust the laser direction in real time to maintain the precise pointing of the laser even when the drone moves or the target object moves.

[0102] Example 2, referring to Table 1, is the second example of the present invention. To further verify the technical solution of the present invention, experimental simulation data of the automatic adjustment method of the laser direction based on the UAV is provided.

[0103] The latitude and longitude coordinates of the target point are set on the ground station as (116.4074, 39.9042) and the altitude is 100 meters. A straight line flight path is planned from the starting point (116.4070, 39.9040, altitude 50 meters) to the target point.

[0104] Start the drone and perform preliminary positioning through the inertial navigation system (INS) and global positioning system (GPS). After the drone takes off, it performs preliminary positioning through GPS data to ensure that the drone is at the predetermined starting point.

[0105] When the drone reaches the target area, the inertial measurement unit (IMU) is activated to collect the drone's angular velocity and acceleration data, the GPS collects the drone's latitude and longitude coordinates and altitude data, the lidar collects the three-dimensional point cloud data around the target point, and the visible light camera collects the image data of the target point. The collected data is cleaned, filtered, noise suppressed, and aligned to ensure the accuracy and integrity of the data.

[0106] Initialize the state vector and covariance matrix, and use GPS data and IMU data to update the state vector and covariance matrix to obtain a preliminary estimate of the drone's attitude.

[0107] The point cloud data collected by the lidar is registered with the map point cloud using the iterative closest point (ICP) algorithm to obtain the relative pose change.

[0108] The relative pose change obtained by point cloud registration is applied to the updated state vector of the Kalman filter to obtain the corrected UAV pose estimate.

[0109] Collect a large amount of image data containing target objects, annotate the contour and direction information of the target objects, and build a training data set.

[0110] The convolutional neural network (CNN) model is trained using the labeled dataset, and the loss function is used to optimize the bounding box regression and orientation angle classification of the target object.

[0111] During the flight of the drone, image data is captured in real time and input into the trained CNN model for target recognition to extract the contour and direction information of the target object.

[0112] According to the contour and direction information of the target object, the coordinates of the center point of the target object in the image are determined.

[0113] Use the camera's intrinsic parameters to convert the coordinates of the target object's center point in the image into the distance in the physical world, and calculate the horizontal and vertical offsets of the target object in the physical world.

[0114] The UAV attitude and position information estimated by the Kalman filter is used to transform the point cloud data collected by the lidar into the global coordinate system to form a three-dimensional point cloud map.

[0115] Using meshing methods, the point cloud data is converted into a 3D model of the surrounding environment.

[0116] Determine the position of the target point that the laser is shooting at in the three-dimensional model, and calculate the position and distance of obstacles on the laser shooting path.

[0117] According to the position and distance of obstacles on the laser shooting path, the compensation parameters of the laser shooting are designed.

[0118] A control algorithm is designed to take the attitude estimation results, the offset distance from the image center to the target object, and the compensation parameters of the laser direction as input, and output the laser angle adjustment instructions so that the laser always points to the target object and remains accurate even when the drone moves or the target object moves.

[0119] The details are shown in Table 1 below:

[0120] Table 1 Experimental record data table

[0121]

[0122]

[0123] Through the comparative analysis of the above table data, it can be clearly seen that the attitude estimation accuracy of the prior art 1 and 2 is 0.5 meters and 0.6 meters respectively, while the attitude estimation accuracy of the schemes 1 to 4 of the present invention is all below 0.2 meters, and the highest reaches 0.1 meters. This shows that the present invention significantly improves the attitude estimation accuracy of the UAV through the Kalman filter algorithm and multi-sensor data fusion, and reduces the error influence of a single sensor.

[0124] The target recognition accuracy of the prior arts 1 and 2 is 10 pixels and 12 pixels respectively, while the target recognition accuracy of the schemes 1 to 4 of the present invention is all below 6 pixels, with the lowest reaching 4 pixels. This shows that the present invention significantly improves the accuracy and robustness of target recognition by constructing a deep learning model to automatically recognize the contour and direction of the target object.

[0125] The offset distance calculation errors of prior arts 1 and 2 are 0.3 meters and 0.4 meters respectively, while the offset distance calculation errors of schemes 1 to 4 of the present invention are all below 0.12 meters, with the lowest reaching 0.05 meters. This shows that the present invention significantly improves the accuracy of offset distance calculation through precise image processing and camera internal reference correction.

[0126] The laser direction accuracy of prior art 1 and 2 is 0.4m and 0.5m respectively, while the laser direction accuracy of solutions 1 to 4 of the present invention is all below 0.15m, with the lowest reaching 0.05m. This shows that the present invention ensures that the laser always points to the target object by dynamically adjusting the laser direction, and can maintain accuracy even when the drone moves or the target object moves.

[0127] The environmental modeling errors of prior art 1 and 2 are 0.2 meters and 0.3 meters respectively, while the environmental modeling errors of solutions 1 to 4 of the present invention are all below 0.12 meters, with the lowest reaching 0.05 meters. This shows that the present invention significantly improves the accuracy of the environmental model through accurate registration of laser radar data and environmental modeling.

[0128] The overall effect scores of prior arts 1 and 2 are 6 and 5 respectively, while the overall effect scores of solutions 1 to 4 of the present invention are all above 8, with the highest reaching 10. This shows that the present invention exhibits higher comprehensive performance and reliability in practical applications, and can better adapt to task requirements in complex dynamic environments.

[0129] In summary, the present invention significantly improves the autonomous navigation and target recognition capabilities of the UAV through multi-sensor data fusion, deep learning models and environmental modeling technology, especially in the automatic adjustment of laser direction, showing higher accuracy.

[0130] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A method for automatically adjusting the laser direction based on an unmanned aerial vehicle, characterized in that: include, Collect IMU, GPS, LiDAR and visible light camera data in real time and perform preliminary processing; The processed data is fused through the Kalman filter algorithm to estimate the attitude of the drone in real time; Build a deep learning model to automatically identify the contour and orientation of target objects; According to the contour and direction of the target object, calculate the offset distance from the center point of the image to the target object; Establish an environmental model and calculate the compensation parameters of the laser direction; The laser direction is dynamically adjusted according to the posture estimation result, the offset distance from the image center to the target object and the compensation parameters of the laser direction.

2. The method for automatically adjusting the laser direction based on a drone as claimed in claim 1, characterized in that: The real-time collection of IMU, GPS, LiDAR and visible light camera data and preliminary processing include the following steps: Plan the flight path of the drone and set the latitude, longitude and altitude of the target point on the ground station; Start the drone and perform preliminary positioning through the inertial navigation system and GPS; When the UAV reaches the target area, the IMU is activated to collect the angular velocity and acceleration data of the UAV, the GPS is activated to collect the latitude and longitude coordinates and altitude data of the UAV, the laser radar is activated to collect the three-dimensional point cloud data around the target point, and the visible light camera is activated to collect the image data of the target point. The collected data are cleaned, filtered, noise suppressed and aligned.

3. The method for automatically adjusting the laser direction based on an unmanned aerial vehicle according to claim 2, characterized in that: The processed data is fused through the Kalman filter algorithm, and the real-time estimation of the drone's attitude includes the following steps: Initialize the state vector and covariance matrix; Use GPS data and IMU data to update the state vector and covariance matrix to obtain a preliminary estimate of the drone’s attitude; Use the ICP algorithm to align the point cloud data collected by the lidar with the map point cloud to obtain the relative pose change; The relative pose change obtained by point cloud registration is applied to the updated state vector of the Kalman filter to obtain the corrected pose estimate of the UAV.

4. The method for automatically adjusting the laser direction based on an unmanned aerial vehicle according to claim 3, characterized in that: Building a deep learning model to automatically identify the contour and orientation of a target object includes the following steps: Collect a large amount of image data containing target objects, annotate the contour and orientation information of the target objects, and build a training data set; Use the labeled dataset to train the convolutional neural network model and use the loss function to optimize the bounding box regression and orientation angle classification of the target object; During the flight of the drone, image data is captured in real time and input into the trained convolutional neural network model for target recognition to extract the contour and direction information of the target object. The expression is: Among them, O is the contour and direction information of the target object, C is the contour of the target object, I is the input image, α is the direction angle of the target object, i is the contour index of the target object, N is the total number of points on the contour of the target object, and x i and i are the horizontal and vertical coordinates of the i-th point on the contour of the target object, respectively. and are the average abscissa and ordinate of the contour points, respectively.

5. The method for automatically adjusting the laser direction based on an unmanned aerial vehicle according to claim 4, characterized in that: According to the contour and direction of the target object, calculating the offset distance from the center point of the image to the target object includes the following steps: According to the contour and direction information of the target object, determine the coordinates of the center point of the target object in the image; Use the camera's intrinsic parameters to convert the center point coordinates of the target object in the image into the distance in the physical world, and calculate the offset of the target object in the physical world. The expression is: Among them, ΔX is the horizontal offset of the target object in the physical world, ΔY is the vertical offset of the target object in the physical world, w is the width of the image, h is the height of the image, Z is the distance from the target object to the camera, and f x is the horizontal focal length of the camera, f y is the vertical focal length of the camera.

6. The method for automatically adjusting the laser direction based on an unmanned aerial vehicle according to claim 5, characterized in that: Establishing the environment model and calculating the compensation parameters of the laser direction include the following steps: The UAV attitude and position information estimated by the Kalman filter is used to transform the point cloud data collected by the lidar into a global coordinate system to form a three-dimensional point cloud map; Use meshing methods to convert point cloud data into a three-dimensional model of the surrounding environment; Determine the position of the target point that the laser is directed to in the three-dimensional model; According to the position of the laser to the target point, the position and distance of the obstacles on the laser path are calculated. The expression is: Where d is the distance to the nearest obstacle on the laser path, P global is the coordinate of the point cloud data in the global coordinate system, p is a point in the point cloud data, and p target is the coordinate of the target point in the global coordinate system; According to the position and distance of obstacles on the laser path, the compensation parameters of the laser direction are designed, and the expression is: Among them, Δθ is the pitch angle compensation parameter of the laser direction, Δφ is the yaw angle compensation parameter of the laser direction, and k d is the proportionality coefficient, and D is the maximum effective distance of the laser.

7. The method for automatically adjusting the laser direction based on an unmanned aerial vehicle according to claim 6, characterized in that: According to the attitude estimation result, the offset distance from the center point of the image to the target object and the compensation parameters of the laser direction, the dynamic adjustment of the laser direction includes the following steps: Design a control algorithm that takes the attitude estimation result, the offset distance from the image center to the target object, and the compensation parameters of the laser direction as input, and outputs the angle adjustment instruction of the laser. The expression is: Among them, θ cmd is the pitch angle adjustment instruction of the laser, φ cmd is the yaw angle adjustment instruction of the laser, φ is the pitch angle of the drone, θ is the roll angle of the drone, k X is the proportional coefficient of the horizontal offset, k Y is the scaling factor for the vertical offset.

8. A laser direction automatic adjustment system based on a drone, based on the laser direction automatic adjustment method based on a drone according to any one of claims 1 to 7, characterized in that: include, Data acquisition module, which collects IMU, GPS, lidar and visible light camera data in real time and performs preliminary processing; The data fusion module fuses the processed data through the Kalman filter algorithm to estimate the attitude of the drone in real time; The target recognition module builds a deep learning model to automatically identify the outline and direction of the target object; An offset calculation module calculates the offset distance from the center point of the image to the target object according to the contour and direction of the target object; Environmental modeling module, which builds environmental model and calculates compensation parameters of laser direction; The laser direction adjustment module dynamically adjusts the laser direction according to the posture estimation result, the offset distance from the image center point to the target object and the compensation parameters of the laser direction.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the automatic adjustment method of laser direction based on a drone according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the automatic laser direction adjustment method based on a drone according to any one of claims 1 to 7 are implemented.

Images