A UAV control system with multi-dimensional perception compensation and its control method

Abstract

The present invention discloses a multi-dimensional perception compensation unmanned aerial vehicle control system and its control method, which relates to the technical field of unmanned aerial vehicle control, and includes the following steps: capturing real-time image data of the current flight path of the unmanned aerial vehicle through the camera of the vision sensor, and monitoring the flight environment in real time and detecting surrounding changes through continuously acquired image data; comparing and analyzing two consecutive frames of images to calculate the difference between them. By capturing image data in real time through the vision sensor and performing intelligent analysis on the differences between images, the present invention can accurately identify and predict dynamic changes in the environment. In a high-dynamic scene, the laser beam density of the lidar can be increased intelligently, thereby improving the accuracy of point cloud data, ensuring that the unmanned aerial vehicle can perform accurate obstacle avoidance and flight path planning in a rapidly changing environment, enhancing flight safety and the accuracy of path planning, and ensuring that the unmanned aerial vehicle can complete tasks stably and efficiently.

Description

Technical Field

[0001] The present invention relates to the technical field of UAV control, and particularly to a UAV control system with multi-dimensional perception compensation and its control method. Background Art

[0002] A UAV control system with multi-dimensional perception compensation is a UAV control system integrating multiple sensors and feedback mechanisms, aiming to improve the flight stability and accuracy of UAVs in complex environments. By combining various perception devices such as visual sensors, lidar (LiDAR), inertial measurement units (IMUs), and barometers, the system can real-time obtain multi-dimensional information such as the attitude, position, speed, and external environment changes of the UAV. On this basis, the control system compensates for sensor errors, external disturbances (such as wind or airflow changes), and the dynamic characteristics of the aircraft itself through dynamic compensation algorithms, so as to achieve more accurate heading control, path planning, and stable flight. Especially in adverse weather or complex terrain conditions, it can ensure the flight safety of the UAV and the high efficiency of mission execution.

[0003] During the UAV control process, visual sensors and lidar play crucial roles. They jointly provide accurate environmental perception data for the flight control system, thus supporting real-time dynamic compensation. The visual sensor can detect obstacles, ground features or changes in the flight environment by capturing images or video streams, and provide accurate position information for correcting the heading and positioning. The lidar scans the environment through laser beams and accurately obtains the distance information around the aircraft. Especially in low light or poor visual environment, it provides more stable and reliable three-dimensional depth perception than traditional visual sensors. The complementarity of these two sensors enables the system to comprehensively process different types of perception errors (such as the errors of visual sensors in weak light or strong light conditions, and the limitations of lidar in complex terrains), and optimize flight control through data fusion and compensation algorithms to ensure the stable flight and efficient navigation of the UAV in complex and dynamic environments.

[0004] The existing technologies have the following deficiencies:

[0005] When visual sensors and lidar jointly provide environmental perception data for the flight control system, the lidar emits laser beams and measures the return time to obtain the distance and position of the target object, generating a high-density three-dimensional point cloud. The UAV uses this point cloud data to real-time construct an accurate map of the surrounding environment. Especially in complex terrains, low light or adverse weather conditions, the lidar can effectively make up for the limitations of visual sensors.

[0006] However, the density of the laser beams emitted by lidar in the prior art usually cannot be intelligently adjusted. When the vision sensor captures a scene with fast movement or dynamic changes (such as a fast-moving object or wind-blown leaves), the image may become blurred or distorted, resulting in the inability to accurately obtain environmental details. In this case, if the density of the lidar cannot be dynamically increased, it may not be able to provide sufficiently accurate real-time data, thus affecting the prediction of the flight path and increasing the collision risk.

[0007] The above information disclosed in the background art section is only used to enhance the understanding of the background of the present disclosure. Therefore, it may include information that does not constitute the prior art known to those of ordinary skill in the art. Summary of the Invention

[0008] The object of the present invention is to provide a multi-dimensional perception compensation unmanned aerial vehicle control system and its control method. By using a vision sensor to capture image data in real time and intelligently analyze the differences between images, it can accurately identify and predict the dynamic changes of the environment, provide an accurate basis for adjusting the laser beam density of the lidar. In a high-dynamic scene, the laser beam density of the lidar can be intelligently increased, thereby improving the accuracy of point cloud data, ensuring that the unmanned aerial vehicle can perform accurate obstacle avoidance and flight path planning in a rapidly changing environment, enhancing flight safety and the accuracy of path planning, and ensuring that the unmanned aerial vehicle can complete tasks stably and efficiently, so as to solve the problems in the above background art.

[0009] To achieve the above object, the present invention provides the following technical solution: A multi-dimensional perception compensation unmanned aerial vehicle control method, including the following steps:

[0010] Capture real-time image data of the current flight path of the unmanned aerial vehicle through the camera of the vision sensor, and monitor the flight environment in real time by continuously acquiring the image data to detect the surrounding changes;

[0011] Compare and analyze two consecutive frames of images, calculate the differences between them, and form a difference set of the differences between the images to comprehensively understand the changes in the flight environment;

[0012] Extract the key features reflecting the high-speed dynamic changes of the shooting scene from the difference set, and perform feature engineering processing on the extracted key features to provide input for subsequent intelligent prediction and improve the accuracy of subsequent prediction;

[0013] Use the data after feature engineering processing as input and transfer it to a pre-trained convolutional neural network to intelligently analyze and predict the difference degree between the images, and feedback the dynamic change degree of the shooting scene based on the prediction result;

[0014] Set an initial low laser beam emission density for the lidar and configure it to meet the flight shooting requirements in a low-dynamic scene;

[0015] In a high-dynamic scene, based on the prediction result of the image difference degree by a convolutional neural network, the laser beam density of a lidar is intelligently adjusted to provide point cloud data with higher precision, ensuring precise obstacle avoidance and flight path planning of an unmanned aerial vehicle in a complex environment.

[0016] Preferably, the image comparison is achieved through image difference metrics, where the image difference metrics include pixel-level differences, edge changes, or motion detection algorithms.

[0017] Preferably, key features reflecting the high-speed dynamic changes of the shooting scene are extracted from the difference set, where the extracted features include the degree of expansion and contraction of the edge region in the image and the vector distribution of object motion in the image. After feature engineering processing of the degree of expansion and contraction of the edge region in the image and the vector distribution of object motion in the image, an edge dynamic expansion factor and a motion vector dispersion factor are generated respectively. The speed, direction change of object motion in the shooting scene and the dynamic change degree of the edge region in the environment are quantified through the edge dynamic expansion factor and the motion vector dispersion factor, comprehensively reflecting the high-speed dynamic changes of the shooting scene.

[0018] Preferably, the edge dynamic expansion factor and the motion vector dispersion factor after feature engineering processing are used as inputs and passed to a pre-trained convolutional neural network. Based on the convolutional neural network, a difference evaluation coefficient is generated. The difference degree between images is intelligently analyzed and predicted through the difference evaluation coefficient, and the dynamic change degree of the shooting scene is fed back.

[0019] Preferably, the steps for dividing the low-dynamic scene and the high-dynamic scene are as follows:

[0020] The difference evaluation coefficient generated when the difference degree between images is intelligently analyzed and predicted by a pre-trained convolutional neural network is compared and analyzed with a pre-set difference evaluation coefficient reference threshold to divide the shooting scene. The specific division process is as follows: If the difference evaluation coefficient is greater than the difference evaluation coefficient reference threshold, the shooting scene is divided into a high-dynamic scene; if the difference evaluation coefficient is less than or equal to the difference evaluation coefficient reference threshold, the shooting scene is divided into a low-dynamic scene.

[0021] Preferably, for two consecutive frames of images, the specific steps for generating an edge dynamic expansion factor after feature engineering processing of the degree of expansion and contraction of the edge region in the image are as follows:

[0022] First, the two consecutive frames of images are processed through an edge detection algorithm to extract the edge region in each frame.

[0023] Then, calculate the relative motion of each edge point in the image, and use the edge difference matrix to quantify the change of edge points in two frames of images. The quantization expression of the edge difference matrix is:

[0024] ;

[0025] Among them, and represent the edge intensity at position . The difference matrix represents the edge change amplitude at position ;

[0026] Extract the edge expansion and contraction regions through the edge difference matrix, quantify the dynamic degree of the edge expansion and contraction regions, and use the relative area change rate of the expansion and contraction regions to define the edge dynamic expansion factor. The defined expression is:

[0027] ;

[0028] Among them: is the set of all pixel points in the edge region of the image, represents the edge dynamic expansion factor, represents the relative area change rate at position . Calculate the total area change of edge expansion and contraction in the image through weighted summation. The specific calculation expression is:

[0029] ;

[0030] Among them: represents the weight of the image region at position ;

[0031] Preferably, for two consecutive frames of images, the specific steps to generate the motion vector dispersion factor after performing feature engineering processing on the vector distribution of the object motion in the images are as follows:

[0032] First, calculate the motion vector for each pixel in two consecutive frames of images. The motion vector represents the displacement of each pixel from the first frame to the second frame, and is calculated by the optical flow method or sparse feature matching; for each pixel point The displacement in the image, calculate its motion vector through the following formula, and the calculation expression is:

[0033] ;

[0034] Among them: is the position of pixel in the first frame, is the position of this pixel in the second frame, Represents a pixel Motion vector;

[0035] After calculating the motion vector of each pixel, the dispersion degree of the motion vector is quantified, and a motion vector dispersion factor is generated. The quantization expression of the motion vector dispersion factor is:

[0036] ;

[0037] Where: Represents the motion vector dispersion factor, Represents the magnitude of the motion vector of the th pixel, Is the th pixel's motion vector direction angle, And Are adjustable exponential parameters that control the influence of motion magnitude and direction, N Is the total number of pixels in the image.

[0038] Preferably, in a high-dynamic scene, based on the prediction result of the image difference degree by the convolutional neural network, the laser beam density of the lidar is intelligently adjusted. The specific steps are as follows:

[0039] In a high-dynamic scene, first, the difference evaluation coefficient generated by the convolutional neural network is used to quantify the dynamic change degree of the scene. In order to dynamically adjust the laser beam density, a laser beam density adjustment coefficient is introduced. The calculation expression of the laser beam density adjustment coefficient is:

[0040] ;

[0041] Where: Is the laser beam density adjustment coefficient, Is the adjustment coefficient used to control the influence sensitivity of the difference evaluation coefficient on the laser beam density; Is the difference evaluation coefficient;

[0042] After determining the high-dynamic scene and calculating the laser beam density adjustment coefficient, the initial low emission laser beam density of the lidar is intelligently adjusted according to the laser beam density adjustment coefficient. The adjustment formula of the laser beam density is as follows:

[0043] ;

[0044] Where: Is the preset initial low emission laser beam density, Is the adjusted actual emission laser beam density, representing the density required in a high-dynamic scene.

[0045] A multi-dimensional perception compensation unmanned aerial vehicle (UAV) control system, including an image data acquisition and environment monitoring module, an image difference comparison and analysis module, a feature extraction and processing module, an intelligent prediction and analysis module, a lidar initial configuration module, and a lidar dynamic adjustment module;

[0046] The image data acquisition and environment monitoring module captures real-time image data of the current flight path of the UAV through the camera of the vision sensor, and monitors the flight environment in real time through the continuously acquired image data to detect changes in the surroundings;

[0047] The image difference comparison and analysis module compares and analyzes two consecutive frames of images, calculates the difference between them, forms a difference set of the differences between the images, and comprehensively understands the changes in the flight environment;

[0048] The feature extraction and processing module extracts key features reflecting the high-speed dynamic changes of the shooting scene from the difference set, performs feature engineering processing on the extracted key features, provides input for subsequent intelligent prediction, and improves the accuracy of subsequent prediction;

[0049] The intelligent prediction and analysis module takes the data processed by feature engineering as input, transmits it to a pre-trained convolutional neural network, intelligently analyzes and predicts the difference degree between the images, and feeds back the dynamic change degree of the shooting scene based on the prediction result;

[0050] The lidar initial configuration module sets an initial low laser beam density for the lidar and configures it to meet the flight shooting requirements in a low-dynamic scene;

[0051] The lidar dynamic adjustment module, in a high-dynamic scene, intelligently adjusts the laser beam density of the lidar based on the prediction result of the difference degree of the images by the convolutional neural network, provides higher-precision point cloud data, and ensures precise obstacle avoidance and flight path planning of the UAV in a complex environment.

[0052] In the above technical solution, the technical effects and advantages provided by the present invention:

[0053] The present invention can accurately identify and predict the dynamic changes of the environment by capturing image data in real time through the vision sensor and intelligently analyzing the differences between the images, provides an accurate basis for adjusting the laser beam density of the lidar. In a high-dynamic scene, the laser beam density of the lidar can be intelligently increased, thereby improving the accuracy of the point cloud data, ensuring that the UAV can perform precise obstacle avoidance and flight path planning in a rapidly changing environment, significantly improving the flight safety and the accuracy of path planning. Especially in a complex dynamic environment, it can better respond to external changes and ensure that the UAV can complete tasks stably and efficiently. Brief Description of the Drawings

[0054] To more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings.

[0055] Figure 1 It is a method flowchart of a multi - dimensional perception compensation UAV control method of the present invention.

[0056] Figure 2 It is a module schematic diagram of a multi - dimensional perception compensation UAV control system of the present invention. Detailed implementation manners

[0057] Now, the exemplary embodiments will be described more fully with reference to the accompanying drawings. However, the exemplary embodiments can be implemented in various forms and should not be construed as limited to the examples set forth herein; rather, these exemplary embodiments are provided so that the present disclosure will be more thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art.

[0058] The present invention provides a Figure 1 UAV control method with multi - dimensional perception compensation as shown below, including the following steps:

[0059] Capture real - time image data of the current flight path of the UAV through the camera of the visual sensor. By continuously acquiring the image data, the flight environment is monitored in real - time to detect changes in the surroundings.

[0060] Through the continuously acquired image data, the system can understand the obstacles, terrain features, and potential dynamic changes (such as fast - moving objects or weather changes) in the environment in real - time. This process provides basic data for subsequent dynamic scene analysis, flight path optimization, and obstacle avoidance decision - making, ensuring the safety and flight stability of the UAV in complex environments.

[0061] Compare and analyze two consecutive frames of images, calculate the difference between them, and form a difference set of the image differences to comprehensively understand the changes in the flight environment.

[0062] These are usually areas or objects with relatively large changes, which may involve fast - moving objects or environmental changes, such as leaves blown by the wind, flight obstacles, etc. In this way, it can be determined whether there are significant dynamic changes in the current scene.

[0063] Generally, image comparison can be achieved through image difference metrics (such as pixel - level differences, edge changes, or motion detection algorithms).

[0064] 1. Pixel-level difference: Pixel-level difference refers to comparing the color values or brightness values of each pixel point in two consecutive frames of images to calculate the subtle differences between the images. This method determines the degree of difference between two frames of images at the same position by comparing the pixels one by one. If the value of a certain pixel changes significantly, it indicates that the environment in this area has changed (for example, the movement of an object or the appearance of a new object). Pixel-level difference is suitable for detecting relatively simple and local changes in the environment, but it may not be robust enough when dealing with noise or large-scale changes.

[0065] 2. Edge change: Edge change is to compare the differences between two frames of images by extracting the edge information in the images. Edges usually represent the contours and shapes of objects in the images, so edge change can more accurately reflect the changes in the positions of objects or the changes in the environmental structure. Through edge detection algorithms (such as Canny edge detection), it is possible to effectively identify which areas in the image have undergone morphological changes, such as the movement, addition, or disappearance of objects. The edge change method is more suitable for dealing with large-scale environmental changes, especially in complex backgrounds.

[0066] 3. Motion detection algorithm: The motion detection algorithm identifies dynamically moving objects in the scene by analyzing the changes between image frames. This method not only focuses on the static differences in the images but also determines the motion trajectories of objects through motion estimation algorithms (such as optical flow method). By comparing the motion information of consecutive frames, the system can detect changes in objects such as people, vehicles, or other dynamic objects and determine whether these objects affect the flight path or potential flight risks. Motion detection is more suitable for dynamic scenes, such as tracking fast-moving objects or environments with large changes.

[0067] Extract key features that reflect the high-speed dynamic changes of the shooting scene from the difference set, perform feature engineering on the extracted key features, provide input for subsequent intelligent prediction, and improve the accuracy of subsequent prediction;

[0068] Extract key features that reflect the high-speed dynamic changes of the shooting scene from the difference set. Among them, the extracted features include the degree of expansion and contraction of the edge area in the image and the vector distribution of the object movement in the image. After performing feature engineering on the degree of expansion and contraction of the edge area in the image and the vector distribution of the object movement in the image, an edge dynamic expansion factor and a motion vector dispersion factor are generated respectively. The speed, direction change of the object movement in the shooting scene and the dynamic change degree of the edge area in the environment are quantified through the edge dynamic expansion factor and the motion vector dispersion factor, comprehensively reflecting the high-speed dynamic changes of the shooting scene.

[0069] For the shooting scene, two consecutive frames of images are compared and analyzed. The greater the change in the degree of expansion and contraction of the edge region in the images, the greater the dynamic change of the shooting scene usually indicates. The edge region represents the contour and structure of the object. Between two consecutive frames of images, if there is a significant expansion or contraction in the edge region, it means that there have been drastic changes in the object or environment in the image, such as the movement of the object, the change in shape, or the appearance of a new object. Especially in a high-speed dynamic scene, such edge changes are usually accompanied by fast-moving objects or drastic environmental changes. Therefore, the degree of expansion and contraction of the edge region is an important indicator for measuring the dynamic change of the scene.

[0070] For two consecutive frames of images, the specific steps for generating the edge dynamic expansion factor after performing feature engineering on the degree of expansion and contraction of the edge region in the images are as follows:

[0071] First, process two consecutive frames of images through an edge detection algorithm (such as Canny edge detection or Sobel operator) to extract the edge region in each frame. Then, calculate the relative movement of each edge point in the image, and use the edge difference matrix to quantify the change of edge points in the two frames of images. The quantification expression of the edge difference matrix is:

[0072] ;

[0073] Among them, and represent the edge intensity at position , and the difference matrix represents the edge change amplitude at position ;

[0074] The function of this step is to accurately extract the dynamic region in the image through edge change detection, especially the fast-changing edge part.

[0075] Quantify the dynamic degree of the edge expansion and contraction regions extracted through the edge difference matrix, and define the edge dynamic expansion factor using the relative area change rate of the expansion and contraction regions. The defined expression is:

[0076] ;

[0077] Among them: is the set of all pixel points in the edge region of the image, represents the edge dynamic expansion factor, represents the relative area change rate at position , and calculate the total area change of the edge expansion and contraction in the image through weighted summation. The specific calculation expression is:

[0078] ;

[0079] Wherein: represents the weight of the image area at the position (usually determined depending on the spatial distribution density of edge points);

[0080] The function of this step is to generate an edge dynamic expansion factor by quantifying the expansion and contraction changes of the edge region, which is used to quantify the high-speed dynamic change degree of the scene.

[0081] It can be seen from the edge dynamic expansion factor that for two consecutive frames of images, the larger the value of the edge dynamic expansion factor generated after feature engineering processing of the expansion and contraction degree of the edge region in the image, the greater the dynamic change of the shooting scene usually indicates. The edge dynamic expansion factor reflects the rapid changes in the image by quantifying the expansion and contraction degree of the edge region in two consecutive frames of images. If the edge region undergoes significant expansion or contraction, it means that the objects in the scene have undergone large movements or morphological changes. Therefore, the edge dynamic expansion factor will increase, indicating that the dynamic change of the scene is relatively intense. On the contrary, if the change in the edge region is small, it means that the change in the scene is relatively gentle or static, and the value of the edge dynamic expansion factor will be small. Therefore, as a quantization index, the edge dynamic expansion factor can effectively reflect the dynamic degree of object movement and environmental changes in the image.

[0082] For the shooting scene, by comparing and analyzing two consecutive frames of images, a large vector distribution of object movement in the image usually indicates a large dynamic change in the shooting scene. The vector distribution of object movement reflects the movement direction and speed of the object at each pixel point in the image. When the objects in the scene undergo rapid or large-scale movements, the distribution of vectors will become more extensive and dispersed, meaning that the relative movement speed of the objects is faster and covers a larger spatial range. At this time, not only the number of motion vectors in the image increases, but also the distribution becomes more irregular, showing obvious dynamic changes. By analyzing the distribution of these motion vectors, it is possible to effectively detect whether there are high-speed dynamic changes or complex motion patterns in the scene, such as fast-moving objects or sudden events (such as the rapid movement of aircraft, vehicles or animals).

[0083] For two consecutive frames of images, the specific steps for generating a motion vector dispersion factor after feature engineering processing of the vector distribution of object movement in the image are as follows:

[0084] First, calculate the motion vector for each pixel in two consecutive frames of images. The motion vector represents the displacement of each pixel from the first frame to the second frame, and is usually calculated by the optical flow method or sparse feature matching (such as SIFT, SURF). For each pixel point For the displacement in the image, its motion vector can be calculated by the following formula, and the calculation expression is:

[0085] ;

[0086] Where: is the position of the pixel in the first frame, is the position of the pixel in the second frame, represents the pixel motion vector;

[0087] This process can reflect the motion of an object by calculating the displacement of each pixel in the image. Its role is to generate corresponding motion vectors for each pixel, thus providing basic data for the subsequent calculation of the motion vector dispersion factor.

[0088] After calculating the motion vectors of each pixel, the next step is to quantify the dispersion degree of these motion vectors to generate the motion vector dispersion factor. The motion vector dispersion factor measures the distribution breadth and dispersion of all motion vectors in the image. To accurately reflect this, the following formula is used for quantification:

[0089] ;

[0090] Where: represents the motion vector dispersion factor, represents the magnitude (i.e., displacement) of the motion vector of the th pixel, is the direction angle of the motion vector of the and are adjustable exponential parameters that control the influence of motion magnitude and direction, N is the total number of pixels in the image;

[0091] This step measures the distribution degree of all motion vectors in the image by combining the dispersion of displacement and direction, and then generates the motion vector dispersion factor. If the motion of the object in the image is relatively intense or extensive, the motion vector dispersion factor will increase, indicating that the dynamic change of the scene is relatively intense. The role of this step is to comprehensively quantify the distribution of motion vectors in the image through the synthesized motion vector dispersion factor and reflect the degree of dynamic change in the scene.

[0092] It can be known from the motion vector dispersion factor that for two consecutive frames of images, the larger the performance value of the motion vector dispersion factor generated after performing feature engineering on the vector distribution of the object motion in the images, generally means that the dynamic changes in the shooting scene are greater. Because the motion vector dispersion factor reflects the extensiveness and distribution range of the object motion in the image. When the object moves quickly or over a large range in the scene, the distribution of the motion vectors will become more scattered and complex, resulting in an increase in the motion vector dispersion factor. This indicates that significant dynamic changes have occurred in the scene, which may include the rapid movement of multiple objects or complex motion patterns. On the contrary, if the motion vector dispersion factor is small, it indicates that there is less object motion or less scene change, the scene is relatively static, and the dynamic changes are not obvious. Therefore, by quantifying the distribution degree of the motion vectors, the motion vector dispersion factor can effectively reflect the dynamic change degree of the shooting scene.

[0093] The data after feature engineering processing will be used as input and passed to a pre-trained convolutional neural network to intelligently analyze and predict the difference degree between the images, and feedback the dynamic change degree of the shooting scene based on the prediction results;

[0094] The edge dynamic expansion factor and the motion vector dispersion factor after feature engineering processing are used as input and passed to a pre-trained convolutional neural network. Based on the convolutional neural network, a difference evaluation coefficient is generated, and the difference degree between the images is intelligently analyzed and predicted through the difference evaluation coefficient, and the dynamic change degree of the shooting scene is feedback.

[0095] The pre-trained convolutional neural network refers to a deep learning model that has been trained with a large amount of historical data and labeled samples. The convolutional neural network is a deep learning model that can automatically learn features from image data. It extracts the spatial features and complex hierarchical relationships in the image through multiple convolutional layers, pooling layers, and fully connected layers. In this scenario, the pre-trained convolutional neural network is not trained from scratch, but has already learned through a training set containing a large number of image differences and dynamic change samples, and its parameters have been optimized during this training process. Through this training, the network can identify subtle changes and different types of dynamic patterns in the image, including object motion, scene change, etc. Specifically, the goal of the network is to learn from features such as the edge dynamic expansion factor and the motion vector dispersion factor of the image how to intelligently evaluate the dynamic change degree of the scene, and output a difference evaluation coefficient to quantify the difference degree between the images.

[0096] This "pre-training" process is crucial because it ensures that the network can perform effective reasoning based on previous learning experiences when new edge extension data and motion vector data are input. The training process typically involves a supervised learning process, in which each training sample contains the input image features (such as edge extension and motion vector data) and the corresponding "label", that is, the degree of dynamic change of the scene. In this way, the convolutional neural network can "learn" the relationship between different features in the image and the actual dynamic changes during the training phase. These features include but are not limited to the motion speed in the image, the change of object position, the scene dynamics, etc. The network will abstract these features layer by layer and finally output a numerical value through the fully connected layer. This numerical value is the difference evaluation coefficient of the scene. The difference evaluation coefficient reflects the degree of difference between images and can tell the system the degree of dynamic change of the current scene.

[0097] The advantage of using such a pre-trained convolutional neural network is that it can quickly and accurately perform intelligent analysis on new image data without having to retrain from scratch. This not only greatly improves the processing efficiency but also the prediction accuracy of the optimized model is higher than that of traditional rule-based methods. Especially when faced with a large amount of dynamically changing image data, the convolutional neural network can capture more complex and detailed scene change patterns through deep learning methods. It can process high-dimensional and non-linear image data features, automatically extract important information. In this way, the system can quickly respond to various changes in the flight environment in practical applications and provide real-time feedback. By generating the difference evaluation coefficient, the system can intelligently analyze the dynamic differences between images and then precisely adjust the flight path to ensure the safe flight and efficient navigation of the drone in complex or rapidly changing scenarios.

[0098] The convolutional neural network is not specifically limited here, as long as it can realize the comprehensive analysis of the edge dynamic expansion factor and the motion vector dispersion factor to generate the difference evaluation coefficient is acceptable. To implement the technical solution of the present invention, the present invention provides a specific implementation method; the calculation formula for generating the difference evaluation coefficient is: , where in the formula, , are the preset proportionality coefficients of the edge dynamic expansion factor and the motion vector dispersion factor respectively, and , are both greater than 0. The preset proportionality coefficients ( and ) are used to adjust the edge dynamic expansion factor and the motion vector dispersion factor Parameters of the weights in the calculation of the difference evaluation coefficient. The role of these proportional coefficients is to balance the contribution degrees of these two features to the final difference evaluation coefficient, that is, how to synthesize these two features to obtain a more accurate measure of dynamic change. By adjusting and and values, the system can pay more attention to the influence of edge dynamic changes or the spread of motion vectors, thereby optimizing the calculation of the difference evaluation coefficient to better reflect the dynamic changes of the scene.

[0099] It can be seen from the difference evaluation coefficient that for two consecutive frames of images, the larger the performance value of the edge dynamic expansion factor generated after feature engineering processing of the expansion and contraction degree of the edge region in the image, and for two consecutive frames of images, the larger the performance value of the motion vector spread factor generated after feature engineering processing of the vector distribution of object motion in the image, that is, the larger the performance value of the difference evaluation coefficient generated when the difference degree between images is intelligently analyzed and predicted through a pre-trained convolutional neural network, the greater the dynamic change of the shooting scene, and vice versa, the smaller the dynamic change of the shooting scene.

[0100] Set an initial low laser beam emission density for the lidar and configure it to meet the flight shooting requirements in a low-dynamic scene;

[0101] In a high-dynamic scene, based on the prediction result of the convolutional neural network for the image difference degree, intelligently adjust the laser beam density of the lidar to provide higher-precision point cloud data to ensure precise obstacle avoidance and flight path planning of the drone in a complex environment;

[0102] In this step, in order to save energy and improve calculation efficiency, the system first configures the initial low laser beam emission density of the lidar. A lower laser density is sufficient to meet the flight requirements in most cases, especially in a low-dynamic environment. The low-density laser beam can cover a larger area but has lower accuracy. Therefore, in a scene with less dynamic change, this configuration is sufficient. This step provides a baseline value for subsequent dynamic adjustment and is a preset response to environmental changes.

[0103] The steps for dividing the low-dynamic scene and the high-dynamic scene are as follows:

[0104] Compare and analyze the difference evaluation coefficient generated when the difference degree between images is intelligently analyzed and predicted through a pre-trained convolutional neural network with a preset difference evaluation coefficient reference threshold to divide the shooting scene. The specific division process is: if the difference evaluation coefficient is greater than the difference evaluation coefficient reference threshold, divide the shooting scene into a high-dynamic scene; if the difference evaluation coefficient is less than or equal to the difference evaluation coefficient reference threshold, divide the shooting scene into a low-dynamic scene.

[0105] In a high-dynamic range (HDR) scenario, based on the prediction result of the image difference degree by a convolutional neural network, the laser beam density of a lidar is intelligently adjusted. The specific steps are as follows:

[0106] In a high-dynamic range (HDR) scenario, first, the difference evaluation coefficient generated by the convolutional neural network is used to quantify the dynamic change degree of the scenario. To dynamically adjust the laser beam density, a laser beam density adjustment coefficient is introduced. The calculation expression of the laser beam density adjustment coefficient is:

[0107] ;

[0108] Where: is the laser beam density adjustment coefficient; is the difference evaluation coefficient, is the adjustment coefficient, which is used to control the influence sensitivity of the difference evaluation coefficient on the laser beam density. Usually, a relatively large value is taken to strengthen the influence of dynamic changes;

[0109] The function of this step is to determine the dynamic change degree of the scenario based on the image difference prediction result and calculate the laser beam density adjustment coefficient, providing a preliminary direction for the adjustment of the lidar laser beam density.

[0110] After determining the high-dynamic range (HDR) scenario and calculating the laser beam density adjustment coefficient, next, the initial low emission laser beam density of the lidar is intelligently adjusted according to the laser beam density adjustment coefficient. The adjustment formula of the laser beam density is as follows:

[0111] ;

[0112] Where: is the preset initial low emission laser beam density, which is usually applicable to low-dynamic range (LDR) scenarios, is the actual emission laser beam density after adjustment, representing the density required in the high-dynamic range (HDR) scenario;

[0113] In this way, when the difference evaluation coefficient is large, the laser beam density will increase accordingly to provide more detailed point cloud data to meet the high-precision requirements in the high-dynamic range (HDR) environment. The function of this step is to intelligently adjust the laser beam density of the lidar so that it can adapt to the dynamic changes of the scenario, improve the environmental perception ability, and ensure the stable and precise flight of the unmanned aerial vehicle in a complex environment.

[0114] In a high-dynamic scenario, by intelligently adjusting the laser beam density of a lidar based on the prediction results of the convolutional neural network for image difference degree, it is possible to respond in real time to the complexity of dynamic changes in the scenario and provide point cloud data with higher accuracy. This precise data supports the drone to more finely perceive the surrounding environment, especially in the case of fast-moving obstacles or large environmental changes, ensuring that the drone can perform accurate obstacle avoidance and effective flight path planning. By increasing the laser beam density, the system can more accurately capture the minute changes in the environment, enabling the drone to maintain a stable flight ability in complex and dynamic environments, avoiding collision risks, and ensuring the efficient execution of flight missions.

[0115] Through the above-mentioned drone control method with multi-dimensional perception compensation, it is possible to effectively solve the problem in the prior art that the emission density of lidar cannot be intelligently adjusted, especially in complex and dynamically changing flight environments. By using a vision sensor to capture image data in real time and intelligently analyze the differences between images, it is possible to accurately identify and predict the dynamic changes in the environment, providing an accurate basis for adjusting the laser beam density of the lidar. In a high-dynamic scenario, the laser beam density of the lidar can be intelligently increased, thereby improving the accuracy of point cloud data and ensuring that the drone can perform accurate obstacle avoidance and flight path planning in a rapidly changing environment. This solution significantly improves flight safety and the accuracy of path planning, especially in complex dynamic environments, and can better respond to external changes, ensuring the stable and efficient completion of tasks by the drone.

[0116] The present invention provides a Figure 2 drone control system with multi-dimensional perception compensation as shown, including an image data acquisition and environment monitoring module, an image difference comparison and analysis module, a feature extraction and processing module, an intelligent prediction and analysis module, a lidar initial configuration module, and a lidar dynamic adjustment module;

[0117] The image data acquisition and environment monitoring module captures real-time image data of the current flight path of the drone through the camera of the vision sensor, and monitors the flight environment in real time through the continuously acquired image data to detect the surrounding changes;

[0118] The image difference comparison and analysis module compares and analyzes two consecutive frames of images, calculates the difference between them, and forms a difference set of the image differences to comprehensively understand the changes in the flight environment;

[0119] The feature extraction and processing module extracts key features reflecting the high-speed dynamic changes of the shooting scene from the difference set, and performs feature engineering processing on the extracted key features to provide input for subsequent intelligent prediction and improve the accuracy of subsequent prediction;

[0120] The intelligent prediction and analysis module takes the data processed by feature engineering as input and passes it to a pre-trained convolutional neural network to intelligently analyze and predict the difference degree between images, and feeds back the dynamic change degree of the shooting scene based on the prediction results;

[0121] The lidar initial configuration module sets the initial low laser beam density for the lidar and configures it to meet the flight shooting requirements in low-dynamic scenarios;

[0122] The lidar dynamic adjustment module intelligently adjusts the laser beam density of the lidar based on the prediction results of the image difference degree by the convolutional neural network in high-dynamic scenarios, provides higher-precision point cloud data, and ensures the precise obstacle avoidance and flight path planning of the drone in complex environments.

[0123] A drone control method with multi-dimensional perception compensation provided by an embodiment of the present invention is implemented through the above-mentioned drone control system with multi-dimensional perception compensation. The specific methods and processes of the drone control system with multi-dimensional perception compensation are detailed in the embodiments of the above-mentioned drone control method with multi-dimensional perception compensation, and will not be elaborated here.

[0124] The above formulas are all dimensionless and take their numerical calculations. The formulas are obtained by collecting a large amount of data for software simulation to obtain a formula closest to the actual situation. The preset parameters in the formulas are set by those skilled in the art according to the actual situation.

[0125] As mentioned above, only the specific implementation manners of the present application are described, but the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present application can easily think of changes or substitutions, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0126] Only some exemplary embodiments of the present invention have been described by way of illustration above. Undoubtedly, for those of ordinary skill in the art, the described embodiments can be modified in various different ways without departing from the spirit and scope of the present invention. Therefore, the above drawings and descriptions are illustrative in nature and should not be construed as limiting the protection scope of the claims of the present invention.

Claims

1. A method for controlling an unmanned aerial vehicle with multi-dimensional perception compensation, characterized in that The steps include: The camera of the visual sensor captures the real-time image data of the current flight path of the drone. Through the continuously acquired image data, the flight environment is monitored in real time to detect the surrounding changes. Compare and analyze two consecutive frames of images, calculate the differences between them, and form a difference set of the image differences to comprehensively understand the changes in the flight environment. Extract the key features reflecting the high-speed dynamic changes of the shooting scene from the difference set, and perform feature engineering processing on the extracted key features to provide input for subsequent intelligent prediction and improve the accuracy of subsequent prediction. Use the data after feature engineering processing as input and transfer it to a pre-trained convolutional neural network to perform intelligent analysis and prediction on the difference degree between images, and feedback the dynamic change degree of the shooting scene based on the prediction results. Set the initial low laser beam density for the lidar and configure it to meet the flight shooting requirements in the low-dynamic scene. In the high-dynamic scene, based on the prediction results of the convolutional neural network on the image difference degree, intelligently adjust the laser beam density of the lidar to provide higher-precision point cloud data to ensure the precise obstacle avoidance and flight path planning of the drone in a complex environment. In the high-dynamic scene, based on the prediction results of the convolutional neural network on the image difference degree, the steps for intelligently adjusting the laser beam density of the lidar are as follows: In the high-dynamic scene, use the difference evaluation coefficient generated by the convolutional neural network to quantify the dynamic change degree of the scene, and introduce a laser beam density adjustment coefficient. The calculation expression of the laser beam density adjustment coefficient is: ； Wherein: is the laser beam density adjustment coefficient, is the adjustment coefficient for controlling the influence sensitivity of the difference evaluation coefficient on the laser beam density; is the difference evaluation coefficient; Intelligently adjust the initial low laser beam density of the lidar according to the laser beam density adjustment coefficient. The adjustment formula of the laser beam density is as follows: ； Wherein: is a preset initial low-emission laser beam density is the adjusted actual emission laser beam density, representing the density required in a high-dynamic scene.

2. The drone control method with multi-dimensional perception compensation according to claim 1, wherein The image comparison is realized through image difference measurement. Among them, the image difference measurement includes pixel-level difference, edge change or motion detection algorithm.

3. A method for controlling a drone with multi-dimensional perception compensation according to claim 1, characterized in that, Extract the key features reflecting the high-speed dynamic changes of the shooting scene from the difference set. Among them, the extracted features include the expansion and contraction degree of the edge area in the image and the vector distribution of the object movement in the image. After performing feature engineering processing on the expansion and contraction degree of the edge area in the image and the vector distribution of the object movement in the image, an edge dynamic expansion factor and a motion vector dispersion factor are generated respectively. The speed, direction change of the object movement in the shooting scene and the dynamic change degree of the edge area in the environment are quantified through the edge dynamic expansion factor and the motion vector dispersion factor, comprehensively reflecting the high-speed dynamic changes of the shooting scene.

4. A method for controlling a drone with multi-dimensional perception compensation according to claim 3, characterized in that, Use the edge dynamic expansion factor and the motion vector dispersion factor after feature engineering processing as input and transfer them to a pre-trained convolutional neural network. Based on the convolutional neural network, a difference evaluation coefficient is generated. Through the difference evaluation coefficient, the difference degree between images is intelligently analyzed and predicted, and the dynamic change degree of the shooting scene is feedback.

5. A method for controlling a drone with multi-dimensional perception compensation according to claim 4, characterized in that, The steps for dividing the low-dynamic scene and the high-dynamic scene are as follows: Compare the difference evaluation coefficient generated when the intelligent analysis and prediction of the difference degree between images is performed through a pre-trained convolutional neural network with the pre-set reference threshold of the difference evaluation coefficient, and divide the shooting scene. The specific division process is as follows: If the difference evaluation coefficient is greater than the reference threshold of the difference evaluation coefficient, the shooting scene is divided into a high-dynamic scene; if the difference evaluation coefficient is less than or equal to the reference threshold of the difference evaluation coefficient, the shooting scene is divided into a low-dynamic scene.

6. The method for controlling a drone with multi-dimensional perception compensation according to claim 3, wherein For two consecutive frames of images, the specific steps for generating the edge dynamic expansion factor after performing feature engineering processing on the expansion and contraction degrees of the edge regions in the images are as follows: First, process the two consecutive frames of images through an edge detection algorithm to extract the edge regions in each frame; Then, calculate the relative motion of each edge point in the image, and use the edge difference matrix to quantify the changes of the edge points in the two frames of images. The quantization expression of the edge difference matrix is: ； Among them, and represent the edge strength at position ; the difference matrix represents the edge change amplitude at position . Extract the edge expansion and contraction regions through the edge difference matrix, quantify the dynamic degree of the edge expansion and contraction regions, and define the edge dynamic expansion factor using the relative area change rate of the expansion and contraction regions. The defined expression is: ； Wherein: is the set of all pixel points in the edge region of the image, represents the edge dynamic expansion factor, represents at the position the relative area change rate, and the total area change of edge expansion and contraction in the image is calculated by weighted summation. The specific calculation expression is: ； Wherein: represents the weight of the image region at the position.

7. A method for controlling a drone with multi-dimensional perception compensation according to claim 3, characterized in that For two consecutive frames of images, the specific steps for generating the motion vector dispersion factor after performing feature engineering processing on the vector distribution of object motion in the images are as follows: First, calculate the motion vector for each pixel in two consecutive frames. The motion vector represents the displacement of each pixel from the first frame to the second frame, which is calculated by the optical flow method or sparse feature matching; for each pixel point For the displacement in the image, calculate its motion vector through the following formula, and the calculation expression is: ； Wherein: is the position of the pixel in the first frame, is the position of the pixel in the second frame, represents the pixel motion vector; After calculating the motion vector of each pixel, quantify the dispersion degree of the motion vector and generate the motion vector dispersion factor. The quantization expression of the motion vector dispersion factor is: ； Wherein: represents the motion vector dispersion factor, represents the magnitude of the motion vector of the th pixel, is the direction angle of the motion vector of the th pixel, and are adjustable exponential parameters that control the influence of motion magnitude and direction, N is the total number of pixels in the image.

8. A multi-dimensional perception compensation UAV control system for implementing the multi-dimensional perception compensation UAV control method described in any one of the above claims 1-7, characterized in that, Including an image data acquisition and environment monitoring module, an image difference comparison and analysis module, a feature extraction and processing module, an intelligent prediction and analysis module, a lidar initial configuration module, and a lidar dynamic adjustment module; The image data acquisition and environment monitoring module captures real-time image data of the current flight path of the drone through the camera of the vision sensor, and monitors the flight environment in real time through the continuously acquired image data to detect changes in the surroundings; The image difference comparison and analysis module compares and analyzes two consecutive frames of images, calculates the difference between them, forms a difference set of the differences between the images, and comprehensively understands the changes in the flight environment; The feature extraction and processing module extracts the key features reflecting the high-speed dynamic changes of the shooting scene from the difference set, and performs feature engineering processing on the extracted key features to provide input for subsequent intelligent prediction and improve the accuracy of subsequent prediction; The intelligent prediction and analysis module takes the data processed by feature engineering as input, transfers it to a pre-trained convolutional neural network, performs intelligent analysis and prediction on the difference degree between images, and feeds back the dynamic change degree of the shooting scene based on the prediction result; The lidar initial configuration module sets the initial low laser beam density for the lidar and configures it to meet the flight shooting requirements in a low-dynamic scene; The lidar dynamic adjustment module, in a high-dynamic scenario, based on the prediction result of the convolutional neural network for image difference degree, intelligently adjusts the laser beam density of the lidar to provide higher-precision point cloud data, ensuring precise obstacle avoidance and flight path planning of the drone in a complex environment; In a high-dynamic scenario, based on the prediction result of the convolutional neural network for image difference degree, intelligently adjusts the laser beam density of the lidar to provide higher-precision point cloud data, ensuring precise obstacle avoidance and flight path planning of the drone in a complex environment; In a high-dynamic scenario, based on the prediction result of the convolutional neural network for image difference degree, the specific steps for intelligently adjusting the laser beam density of the lidar are as follows: In a high-dynamic scenario, use the difference evaluation coefficient generated by the convolutional neural network to quantify the dynamic change degree of the scenario, and introduce a laser beam density adjustment coefficient. The calculation expression of the laser beam density adjustment coefficient is: ； Wherein: is the laser beam density adjustment coefficient, is the adjustment coefficient used to control the influence sensitivity of the difference evaluation coefficient on the laser beam density; is the difference evaluation coefficient; Intelligently adjust the initial low emission laser beam density of the lidar according to the laser beam density adjustment coefficient. The adjustment formula for the laser beam density is as follows: ； Wherein: is a preset initial low-emission laser beam density, is the adjusted actual emission laser beam density, representing the density required in a high-dynamic scene.

Images