Multi-dimensional perception compensation unmanned aerial vehicle control system and control method thereof

Abstract

The invention discloses a multi-dimensional perception compensation unmanned aerial vehicle control system and a control method thereof, and relates to the technical field of unmanned aerial vehicle control, and the method comprises the following steps: capturing the real-time image data of the current flight path of an unmanned aerial vehicle through a camera of a visual sensor, monitoring the flight environment in real time through the continuously obtained image data, and obtaining the real-time image data of the current flight path of the unmanned aerial vehicle. Detecting a surrounding change condition; and comparing and analyzing two continuous frames of images, and calculating the difference between the two continuous frames of images. According to the invention, the visual sensor captures the image data in real time and intelligently analyzes the difference between the images, so that the dynamic change of the environment can be accurately identified and predicted, and the laser beam density of the laser radar can be intelligently increased in a high-dynamic scene, thereby improving the precision of the point cloud data, and reducing the cost. It is ensured that the unmanned aerial vehicle can perform accurate obstacle avoidance and flight path planning in a rapidly changing environment, the flight safety and the precision of path planning are improved, and it is ensured that the unmanned aerial vehicle stably and efficiently completes tasks.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicle control, and in particular to a multi-dimensional perception compensation unmanned aerial vehicle control system and a control method thereof. Background Art

[0002] The multi-dimensional perception compensation drone control system is a drone control system that integrates multiple sensors and feedback mechanisms, aiming to improve the flight stability and accuracy of drones in complex environments. By combining multiple perception devices such as visual sensors, laser radar (LiDAR), inertial measurement unit (IMU), barometer, etc., the system can obtain multi-dimensional information such as the drone's attitude, position, speed, external environment changes, etc. in real time. On this basis, the control system uses a dynamic compensation algorithm to compensate for sensor errors, external disturbances (such as wind or airflow changes), and the aircraft's own dynamic characteristics, thereby achieving more accurate heading control, path planning, and stable flight, especially in severe weather or complex terrain conditions, to ensure the flight safety of the drone and the efficiency of mission execution.

[0003] In the process of drone control, visual sensors and lidar play a vital role. Together, they provide accurate environmental perception data for the flight control system, thus supporting real-time dynamic compensation. By capturing images or video streams, visual sensors can detect obstacles, ground features or changes in the flight environment, and provide accurate location information for heading and positioning corrections. Lidar scans the environment with laser beams to accurately obtain distance information around the aircraft, especially in low light or poor visual environments, providing more stable and reliable three-dimensional depth perception than traditional visual sensors. The complementarity of these two sensors allows the system to comprehensively process different types of perception errors (such as the errors of visual sensors in low or strong light conditions, and the limitations of lidar in complex terrain), optimize flight control through data fusion and compensation algorithms, and ensure stable flight and efficient navigation of drones in complex and dynamic environments.

[0004] The prior art has the following deficiencies: When visual sensors and lidar work together to provide environmental perception data for the flight control system, lidar generates a high-density three-dimensional point cloud by emitting laser beams and measuring the return time to obtain the distance and position of the target object. The drone uses this point cloud data to build an accurate map of the surrounding environment in real time, especially in complex terrain, low light or bad weather conditions. Lidar can effectively make up for the limitations of visual sensors.

[0005] However, the density of laser beams emitted by LiDAR in existing technologies cannot usually be adjusted intelligently. When visual sensors capture fast-moving or dynamically changing scenes (such as fast-moving objects or leaves blown by the wind), the image may appear blurred or distorted, resulting in the inability to accurately obtain environmental details. In this case, if the density of LiDAR cannot be increased dynamically, it may not be able to provide sufficiently accurate real-time data, thereby affecting the prediction of the flight path and increasing the risk of collision.

[0006] The above information disclosed in this Background section is only for enhancement of understanding of the background of the present disclosure and therefore it may contain information that does not constitute the prior art that is already known to one of ordinary skill in the art. Summary of the invention

[0007] The purpose of the present invention is to provide a multi-dimensional perception and compensation UAV control system and a control method thereof. By using a visual sensor to capture image data in real time and perform intelligent analysis on the differences between images, the present invention can accurately identify and predict the dynamic changes of the environment, and provide a precise basis for adjusting the laser beam density for the laser radar. In high dynamic scenes, the laser beam density of the laser radar can be intelligently increased, thereby improving the accuracy of the point cloud data, ensuring that the UAV can accurately avoid obstacles and plan flight paths in rapidly changing environments, improving flight safety and the accuracy of path planning, and ensuring that the UAV can complete its tasks stably and efficiently, so as to solve the problems in the above-mentioned background technology.

[0008] In order to achieve the above object, the present invention provides the following technical solution: a multi-dimensional perception compensation drone control method, comprising the following steps: The camera of the visual sensor captures real-time image data of the current flight path of the drone. Through the continuously acquired image data, the flight environment is monitored in real time and changes in the surroundings are detected; Compare and analyze two consecutive frames of images, calculate the difference between them, and form a difference set of the differences between the images to fully understand the changes in the flight environment; Extract key features that reflect the high-speed dynamic changes of the shooting scene from the difference set, perform feature engineering on the extracted key features, provide input for subsequent intelligent prediction, and improve the accuracy of subsequent predictions; The data processed by feature engineering will be passed as input to the pre-trained convolutional neural network to intelligently analyze and predict the differences between images, and feedback the degree of dynamic changes in the shooting scene based on the prediction results; Set an initial low emission laser beam density for the LiDAR and configure it to meet the flight photography requirements in low-dynamic scenes; In high-dynamic scenarios, based on the prediction results of image differences by the convolutional neural network, the laser beam density of the lidar is intelligently adjusted to provide higher-precision point cloud data, ensuring accurate obstacle avoidance and flight path planning for drones in complex environments.

[0009] Preferably, the image comparison is achieved by image difference metrics, wherein the image difference metrics include pixel level differences, edge changes or motion detection algorithms.

[0010] Preferably, key features reflecting the high-speed dynamic changes of the shooting scene are extracted from the difference set, wherein the extracted features include the expansion and contraction degree of the edge area in the image and the vector distribution of the object motion in the image, and after feature engineering processing of the expansion and contraction degree of the edge area in the image and the vector distribution of the object motion in the image, an edge dynamic expansion factor and a motion vector scatter factor are generated respectively, and the speed and direction change of the object motion in the shooting scene and the degree of dynamic change of the edge area in the environment are quantified by the edge dynamic expansion factor and the motion vector scatter factor, so as to fully reflect the high-speed dynamic changes of the shooting scene.

[0011] Preferably, the edge dynamic expansion factor and motion vector scatter factor after feature engineering are used as input and passed to a pre-trained convolutional neural network, and a difference evaluation coefficient is generated based on the convolutional neural network. The difference between images is intelligently analyzed and predicted through the difference evaluation coefficient, and the dynamic change degree of the shooting scene is fed back.

[0012] Preferably, the steps of dividing the low dynamic scene and the high dynamic scene are as follows: The difference evaluation coefficient generated by the pre-trained convolutional neural network when performing intelligent analysis and prediction of the differences between images is compared and analyzed with the pre-set difference evaluation coefficient reference threshold, and the shooting scene is divided. The specific division process is: if the difference evaluation coefficient is greater than the difference evaluation coefficient reference threshold, the shooting scene is divided into a high-dynamic scene; if the difference evaluation coefficient is less than or equal to the difference evaluation coefficient reference threshold, the shooting scene is divided into a low-dynamic scene.

[0013] Preferably, for two consecutive frames of images, the specific steps of generating the edge dynamic expansion factor after feature engineering processing of the expansion and contraction degree of the edge area in the image are as follows: First, two consecutive frames of images are processed using an edge detection algorithm to extract the edge area in each frame; Then, the relative motion of each edge point in the image is calculated, and the edge difference matrix is ​​used to quantify the change of edge points in the two frames of images. The quantization expression of the edge difference matrix is: ; in, and Indicates at location The edge strength at , the difference matrix Indicates at location The margin variation of The edge expansion and contraction areas extracted by the edge difference matrix are used to quantify the dynamic degree of the edge expansion and contraction areas. The edge dynamic expansion factor is defined by the relative area change rate of the expansion and contraction areas. The definition expression is: ; in: is the set of all pixels in the edge area of ​​the image. represents the edge dynamic expansion factor, Indicates at location The relative area change rate at the position is calculated by weighted summation to calculate the total area change of edge expansion and contraction in the image. The specific calculation expression is: ; in: Indicates location The weight of the image region at .

[0014] Preferably, for two consecutive frames of images, the specific steps of generating a motion vector scatter factor after feature engineering processing is performed on the vector distribution of the object motion in the image are as follows: First, the motion vector of each pixel in two consecutive frames is calculated. The motion vector represents the displacement of each pixel from the first frame to the second frame, which is calculated by optical flow method or sparse feature matching. The displacement in the image is calculated by the following formula to calculate its motion vector: ; in: It's pixels The position in the first frame, Is the pixel The position in the second frame, Represents pixels Motion vector; After calculating the motion vector of each pixel, the dispersion of the motion vector is quantified and a motion vector dispersion factor is generated. The quantization expression of the motion vector dispersion factor is: ; in: represents the motion vector spread factor, Indicates The size of the motion vector of pixels, It is The direction angle of the motion vector of each pixel, and is an adjustable exponential parameter that controls the magnitude and direction of the movement. N is the total number of pixels in the image.

[0015] Preferably, in a high dynamic scene, based on the prediction result of the image difference by the convolutional neural network, the laser beam density of the laser radar is intelligently adjusted, and the specific steps are as follows: In high-dynamic scenes, the difference evaluation coefficient generated by the convolutional neural network is first used to quantify the dynamic change degree of the scene. In order to dynamically adjust the laser beam density, the laser beam density adjustment coefficient is introduced. The calculation expression of the laser beam density adjustment coefficient is: ; in: is the laser beam density adjustment factor, is the adjustment factor used to control the sensitivity of the difference evaluation coefficient to the laser beam density; After determining the high dynamic scene and calculating the laser beam density adjustment coefficient, the initial low emission laser beam density of the lidar is intelligently adjusted according to the laser beam density adjustment coefficient. The adjustment formula of the laser beam density is as follows: ; in: is the preset initial low emission laser beam density, is the actual emitted laser beam density after adjustment, indicating the density required in high dynamic scenes.

[0016] A multi-dimensional perception compensation UAV control system, including an image data acquisition and environment monitoring module, an image difference comparison and analysis module, a feature extraction and processing module, an intelligent prediction and analysis module, a laser radar initial configuration module, and a laser radar dynamic adjustment module; The image data acquisition and environment monitoring module captures the real-time image data of the current flight path of the UAV through the camera of the visual sensor. Through the continuously acquired image data, the flight environment is monitored in real time and the changes in the surroundings are detected; The image difference comparison and analysis module compares and analyzes two consecutive frames of images, calculates the difference between them, and forms a difference set of the differences between the images to fully understand the changes in the flight environment; The feature extraction and processing module extracts key features that reflect the high-speed dynamic changes of the shooting scene from the difference set, performs feature engineering on the extracted key features, provides input for subsequent intelligent prediction, and improves the accuracy of subsequent predictions; The intelligent prediction and analysis module uses the data processed by feature engineering as input and passes it to the pre-trained convolutional neural network to intelligently analyze and predict the differences between images, and feedback the degree of dynamic changes in the shooting scene based on the prediction results; The LiDAR initial configuration module sets the initial low emission laser beam density for the LiDAR and configures it to meet the flight shooting requirements in low-dynamic scenes; The dynamic adjustment module of the LiDAR intelligently adjusts the laser beam density of the LiDAR in high-dynamic scenarios based on the prediction results of image differences by the convolutional neural network, providing higher-precision point cloud data to ensure accurate obstacle avoidance and flight path planning for drones in complex environments.

[0017] In the above technical solution, the technical effects and advantages provided by the present invention are: The present invention uses visual sensors to capture image data in real time and perform intelligent analysis on the differences between images. It can accurately identify and predict dynamic changes in the environment, and provide accurate laser beam density adjustment basis for the lidar. In high-dynamic scenarios, the laser beam density of the lidar can be intelligently increased, thereby improving the accuracy of point cloud data, ensuring that the UAV can accurately avoid obstacles and plan flight paths in rapidly changing environments, significantly improving flight safety and path planning accuracy, especially in complex dynamic environments, and being able to better respond to external changes, ensuring that the UAV can complete its tasks stably and efficiently. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0019] Figure 1 This is a method flow chart of a multi-dimensional perception compensation drone control method of the present invention.

[0020] Figure 2 This is a module schematic diagram of a multi-dimensional perception and compensation UAV control system of the present invention. DETAILED DESCRIPTION

[0021] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these example embodiments are provided so that the description of the present disclosure will be more comprehensive and complete, and the concept of the example embodiments will be fully conveyed to those skilled in the art.

[0022] The present invention provides Figure 1 A multi-dimensional perception compensation UAV control method shown includes the following steps: The camera of the visual sensor captures real-time image data of the current flight path of the drone. Through the continuously acquired image data, the flight environment is monitored in real time and changes in the surroundings are detected; Through the continuous acquisition of image data, the system can understand obstacles, terrain features and potential dynamic changes in the environment (such as fast-moving objects or weather changes) in real time. This process provides basic data for subsequent dynamic scene analysis, flight path optimization and obstacle avoidance decisions, ensuring the safety and flight stability of the drone in complex environments.

[0023] Compare and analyze two consecutive frames of images, calculate the difference between them, and form a difference set of the differences between the images to fully understand the changes in the flight environment; These are usually areas or objects with large changes, which may involve fast-moving objects or environmental changes, such as leaves blown by the wind, flying obstacles, etc. In this way, it can be determined whether there are large dynamic changes in the current scene.

[0024] Typically, image comparison can be achieved using image difference metrics such as pixel-level differences, edge changes, or motion detection algorithms.

[0025] 1. Pixel-level difference: Pixel-level difference refers to comparing the color value or brightness value of each pixel in two consecutive frames of images to calculate the subtle differences between the images. This method determines the degree of difference between the two frames of images at the same position by comparing the pixels at the same position in the image one by one. If the value of a pixel changes significantly, it means that the environment in the area has changed (for example, the movement of an object or the appearance of a new object). Pixel-level difference is suitable for detecting simpler, local changes in the environment, but may not be robust enough when dealing with noise or large-scale changes.

[0026] 2. Edge change: Edge change compares the difference between two frames by extracting edge information in the image. Edges usually represent the outline and shape of objects in the image, so edge changes can more accurately reflect changes in the position of objects or changes in environmental structures. Through edge detection algorithms (such as Canny edge detection), it is possible to effectively identify which areas in the image have undergone morphological changes, such as movement, addition, or disappearance of objects. The edge change method is more suitable for processing large-scale environmental changes than pixel-level differences, especially in complex backgrounds.

[0027] 3. Motion detection algorithm: Motion detection algorithm identifies dynamically moving objects in the scene by analyzing the changes between image frames. This method not only focuses on static differences in the image, but also determines the movement trajectory of the object through motion estimation algorithms (such as optical flow). By comparing the motion information of consecutive frames, the system can detect changes such as people, vehicles or other dynamic objects, and determine whether these objects affect the flight path or potential flight risks. Motion detection is more suitable for dynamic scenes, such as tracking fast-moving objects or environments with large changes.

[0028] Extract key features that reflect the high-speed dynamic changes of the shooting scene from the difference set, perform feature engineering on the extracted key features, provide input for subsequent intelligent prediction, and improve the accuracy of subsequent predictions; Key features that reflect the high-speed dynamic changes of the shooting scene are extracted from the difference set, where the extracted features include the expansion and contraction degree of the edge area in the image and the vector distribution of the object motion in the image. After feature engineering processing of the expansion and contraction degree of the edge area in the image and the vector distribution of the object motion in the image, an edge dynamic expansion factor and a motion vector scatter factor are generated respectively. The edge dynamic expansion factor and the motion vector scatter factor are used to quantify the speed and direction change of the object motion in the shooting scene and the degree of dynamic change of the edge area in the environment, thereby comprehensively reflecting the high-speed dynamic changes of the shooting scene.

[0029] For the shooting scene, two consecutive frames of images are compared and analyzed. The greater the change in the expansion and contraction of the edge area in the image, the greater the dynamic change of the shooting scene. The edge area represents the outline and structure of the object. If the edge area expands or contracts significantly between two consecutive frames, it means that the object or environment in the image has changed dramatically, such as the movement of the object, the change of shape, or the appearance of a new object. Especially in high-speed dynamic scenes, such edge changes are usually accompanied by fast-moving objects or drastic environmental changes. Therefore, the expansion and contraction of the edge area is an important indicator to measure the dynamic change of the scene.

[0030] For two consecutive frames of images, the specific steps of generating the edge dynamic expansion factor after feature engineering processing of the expansion and contraction degree of the edge area in the image are as follows: First, two consecutive frames of images are processed by edge detection algorithms (such as Canny edge detection or Sobel operator) to extract the edge area in each frame. Then, the relative motion of each edge point in the image is calculated, and the edge difference matrix is ​​used to quantify the change of edge points in the two frames of images. The quantization expression of the edge difference matrix is: ; in, and Indicates at location The edge strength at , the difference matrix Indicates at location The margin variation of The purpose of this step is to accurately extract dynamic areas in the image, especially rapidly changing edge parts, through edge change detection.

[0031] The edge expansion and contraction areas extracted by the edge difference matrix are used to quantify the dynamic degree of the edge expansion and contraction areas. The edge dynamic expansion factor is defined by the relative area change rate of the expansion and contraction areas. The definition expression is: ; in: is the set of all pixels in the edge area of ​​the image. represents the edge dynamic expansion factor, Indicates at location The relative area change rate at the position is calculated by weighted summation to calculate the total area change of edge expansion and contraction in the image. The specific calculation expression is: ; in: Indicates location The weight of the image area at (usually determined by the spatial distribution density of edge points); The purpose of this step is to generate an edge dynamic expansion factor by quantifying the expansion and contraction changes of the edge area, which is used to quantify the degree of high-speed dynamic changes of the scene.

[0032] It can be seen from the edge dynamic expansion factor that for two consecutive frames of images, the larger the performance value of the edge dynamic expansion factor generated after feature engineering processing of the expansion and contraction degree of the edge area in the image, the greater the dynamic change of the shooting scene. The edge dynamic expansion factor reflects the rapid changes in the image by quantifying the expansion and contraction degree of the edge area in two consecutive frames of images. If the edge area expands or contracts significantly, it means that the objects in the scene have undergone large movements or morphological changes, so the edge dynamic expansion factor will increase, indicating that the dynamic changes of the scene are more drastic. Conversely, if the changes in the edge area are small, it means that the changes in the scene are relatively gentle or static, and the value of the edge dynamic expansion factor will be smaller. Therefore, as a quantitative indicator, the edge dynamic expansion factor can effectively reflect the dynamic degree of object movement and environmental changes in the image.

[0033] For the shooting scene, two consecutive frames of images are compared and analyzed. The vector distribution of the object's motion in the image is large, which usually indicates that the dynamic changes of the shooting scene are large. The vector distribution of the object's motion reflects the direction and speed of the object's motion at each pixel in the image. When the objects in the scene move quickly or over a large range, the distribution of the vectors becomes more extensive and dispersed, which means that the relative motion speed of the objects is faster and covers a larger spatial range. At this time, the number of motion vectors in the image not only increases, but also becomes more irregularly distributed, showing obvious dynamic changes. By analyzing the distribution of these motion vectors, it is possible to effectively detect whether there are high-speed dynamic changes or complex motion patterns in the scene, such as fast-moving objects or emergencies (such as the rapid movement of aircraft, vehicles or animals).

[0034] For two consecutive frames of images, the specific steps of generating the motion vector scatter factor after feature engineering processing of the vector distribution of the object motion in the image are as follows: First, the motion vector of each pixel in two consecutive frames is calculated. The motion vector represents the displacement of each pixel from the first frame to the second frame. It is usually calculated by optical flow method or sparse feature matching (such as SIFT, SURF). The displacement in the image is calculated by the following formula to calculate its motion vector: ; in: It's pixels The position in the first frame, Is the pixel The position in the second frame, Represents pixels Motion vector; This process can reflect the movement of the object by calculating the displacement of each pixel in the image. Its function is to generate a corresponding motion vector for each pixel, thereby providing basic data for the subsequent motion vector scatter factor calculation.

[0035] After calculating the motion vector of each pixel, the next step is to quantify the dispersion of these motion vectors and generate a motion vector dispersion factor, which measures the distribution breadth and dispersion of all motion vectors in the image. In order to accurately reflect this, the following formula is used for quantification: ; in: represents the motion vector spread factor, Indicates The size of the motion vector of pixels, It is The direction angle of the motion vector of each pixel, and is an adjustable exponential parameter that controls the magnitude and direction of the movement. N is the total number of pixels in the image; This step measures the distribution of all motion vectors in the image by combining the distribution of displacement and direction, and then generates a motion vector scatter factor. If the motion of the objects in the image is more intense or extensive, the motion vector scatter factor will increase, indicating that the dynamic changes in the scene are more intense. The purpose of this step is to comprehensively quantify the distribution of motion vectors in the image through the synthesized motion vector scatter factor, reflecting the degree of dynamic changes in the scene.

[0036] From the motion vector scatter factor, it can be seen that for two consecutive frames of images, the larger the performance value of the motion vector scatter factor generated after feature engineering of the vector distribution of the object movement in the image, the greater the dynamic change in the shooting scene. Because the motion vector scatter factor reflects the extensiveness and distribution range of the object movement in the image, when the object moves quickly or over a large range in the scene, the distribution of the motion vector will become more dispersed and complex, resulting in an increase in the motion vector scatter factor. This indicates that significant dynamic changes have occurred in the scene, which may include rapid movement of multiple objects or complex motion patterns. Conversely, if the motion vector scatter factor is small, it means that the object moves less or the scene changes less, the scene is more static, and the dynamic changes are not obvious. Therefore, the motion vector scatter factor can effectively reflect the degree of dynamic change in the shooting scene by quantifying the distribution degree of the motion vector.

[0037] The data processed by feature engineering will be passed as input to the pre-trained convolutional neural network to intelligently analyze and predict the differences between images, and feedback the degree of dynamic changes in the shooting scene based on the prediction results; The edge dynamic expansion factor and motion vector scatter factor after feature engineering are used as input and passed to the pre-trained convolutional neural network. The difference evaluation coefficient is generated based on the convolutional neural network. The difference between images is intelligently analyzed and predicted through the difference evaluation coefficient, and the dynamic change degree of the shooting scene is fed back.

[0038] A pre-trained convolutional neural network refers to a deep learning model that has been trained with a large amount of historical data and annotated samples. A convolutional neural network is a deep learning model that can automatically learn features from image data. It extracts spatial features and complex hierarchical relationships in images through multiple layers of convolutional layers, pooling layers, and fully connected layers. In this scenario, the pre-trained convolutional neural network is not trained from scratch, but has been learned through a training set containing a large number of image differences and dynamic change samples, and its parameters are optimized during this training process. Through this training, the network can recognize subtle changes in images and different types of dynamic patterns, including object motion, scene changes, etc. Specifically, the goal of the network is to learn how to intelligently evaluate the degree of dynamic change of the scene from features such as the edge dynamic expansion factor and the motion vector spread factor of the image, and output a difference evaluation coefficient to quantify the difference between images.

[0039] This "pre-training" process is key because it ensures that the network can make effective inferences based on previous learning experience when new edge extension data and motion vector data are input. The training process usually involves a supervised learning process, in which each training sample contains the input image features (such as edge extension and motion vector data) and the corresponding "label", that is, the degree of dynamic change of the scene. In this way, the convolutional neural network can "learn" the relationship between different features in the image and the actual dynamic changes during the training phase. These features include but are not limited to the speed of movement in the image, the change in the position of objects, the dynamics of the scene, etc. The network will abstract these features layer by layer, and finally output a value through the fully connected layer, which is the difference evaluation coefficient of the scene. The difference evaluation coefficient reflects the difference between images and can tell the system the degree of dynamic change of the current scene.

[0040] The advantage of using such a pre-trained convolutional neural network is that it can quickly and accurately perform intelligent analysis on new image data without retraining from scratch. This not only greatly improves processing efficiency, but also allows the prediction accuracy of the optimized model to be higher than that of traditional rule-based methods. Especially when faced with a large amount of dynamically changing image data, the convolutional neural network can capture more complex and detailed scene change patterns through deep learning methods. It can process high-dimensional, nonlinear image data features and automatically extract important information, so that the system can quickly respond to changes in various flight environments in practical applications and provide real-time feedback. By generating a difference evaluation coefficient, the system can intelligently analyze the dynamic differences between images, and then accurately adjust the flight path to ensure the safe flight and efficient navigation of the drone in complex or high-speed changing scenes.

[0041] The convolutional neural network is not specifically limited here, and can achieve the dynamic expansion factor of the edge and motion vector spread factor Perform comprehensive analysis to generate differential assessment coefficients In order to realize the technical solution of the present invention, the present invention provides a specific implementation method; the difference evaluation coefficient The resulting calculation formula is: , where The edge dynamic expansion factor and motion vector spread factor The preset scaling factor of All are greater than 0. Preset proportional coefficient ( and ) is used to adjust the edge dynamic expansion factor and motion vector spread factor Evaluating coefficients in differences The weight parameter in the calculation. The role of these proportional coefficients is to balance the contribution of these two features to the final difference evaluation coefficient, that is, how to combine these two features to obtain a more accurate dynamic change measurement. and The value of can make the system pay more attention to the impact of edge dynamic changes or motion vector dispersion, and then optimize the calculation of the difference evaluation coefficient to better reflect the dynamic changes of the scene.

[0042] It can be seen from the difference evaluation coefficient that, for two consecutive frames of images, the larger the performance value of the edge dynamic expansion factor generated after feature engineering processing of the expansion and contraction degree of the edge area in the image, the larger the performance value of the motion vector scatter factor generated after feature engineering processing of the vector distribution of the motion of the object in the image for two consecutive frames of images, that is, the larger the performance value of the difference evaluation coefficient generated when the difference between images is intelligently analyzed and predicted by the pre-trained convolutional neural network, the greater the dynamic change of the shooting scene, and vice versa.

[0043] Set an initial low emission laser beam density for the LiDAR and configure it to meet the flight photography requirements in low-dynamic scenes; In high-dynamic scenes, based on the prediction results of image differences by convolutional neural networks, the laser beam density of the lidar is intelligently adjusted to provide more accurate point cloud data, ensuring accurate obstacle avoidance and flight path planning for drones in complex environments. In this step, in order to save energy and improve computing efficiency, the system first configures the initial low emission laser beam density of the lidar. The lower laser density is sufficient to meet the flight needs in most cases, especially in low-dynamic environments. Low-density laser beams can cover a larger area, but with lower accuracy. Therefore, this configuration is sufficient in scenes with small dynamic changes. This step provides a baseline value for subsequent dynamic adjustments and is a preset response to environmental changes.

[0044] The steps for dividing low dynamic scenes and high dynamic scenes are as follows: The difference evaluation coefficient generated by the pre-trained convolutional neural network when performing intelligent analysis and prediction of the differences between images is compared and analyzed with the pre-set difference evaluation coefficient reference threshold, and the shooting scene is divided. The specific division process is: if the difference evaluation coefficient is greater than the difference evaluation coefficient reference threshold, the shooting scene is divided into a high-dynamic scene; if the difference evaluation coefficient is less than or equal to the difference evaluation coefficient reference threshold, the shooting scene is divided into a low-dynamic scene.

[0045] In high-dynamic scenes, based on the prediction results of image differences by the convolutional neural network, the laser beam density of the lidar is intelligently adjusted. The specific steps are as follows: In high-dynamic scenes, the difference evaluation coefficient generated by the convolutional neural network is first used to quantify the dynamic change degree of the scene. In order to dynamically adjust the laser beam density, the laser beam density adjustment coefficient is introduced. The calculation expression of the laser beam density adjustment coefficient is: ; in: is the laser beam density adjustment factor, It is the adjustment coefficient, which is used to control the sensitivity of the difference evaluation coefficient to the laser beam density. It is usually set to a larger value to enhance the impact of dynamic changes. The purpose of this step is to determine the degree of dynamic change of the scene based on the image difference prediction results, and calculate the laser beam density adjustment coefficient to provide preliminary direction for adjusting the laser beam density of the lidar.

[0046] After determining the high dynamic scene and calculating the laser beam density adjustment coefficient, the initial low emission laser beam density of the lidar is intelligently adjusted according to the laser beam density adjustment coefficient. The adjustment formula of the laser beam density is as follows: ; in: is the preset initial low emission laser beam density, is the actual emitted laser beam density after adjustment, indicating the required density in a high dynamic scene; In this way, when the difference evaluation coefficient is large, the laser beam density will increase accordingly to provide more refined point cloud data to meet the high-precision requirements in high-dynamic environments. The role of this step is to intelligently adjust the laser beam density of the lidar so that it can adapt to the dynamic changes of the scene and improve the environmental perception ability, ensuring that the drone can fly stably and accurately in complex environments.

[0047] In highly dynamic scenes, the laser beam density of the LiDAR is intelligently adjusted based on the prediction results of the image difference by the convolutional neural network, which can respond to the complexity of dynamic changes in the scene in real time and provide more accurate point cloud data. This precise data supports the UAV to perceive the surrounding environment more carefully, especially when obstacles move quickly or the environment changes greatly, ensuring that the UAV can accurately avoid obstacles and effectively plan flight paths. By increasing the density of the laser beam, the system can more accurately capture small changes in the environment, allowing the UAV to always maintain stable flight capabilities in complex and dynamic environments, avoid collision risks, and ensure the efficient execution of flight missions.

[0048] The above-mentioned multi-dimensional perception compensation drone control method can effectively solve the problem that the laser radar emission density cannot be intelligently adjusted in the existing technology, especially in complex and dynamically changing flight environments. By capturing image data in real time through visual sensors and intelligently analyzing the differences between images, it is possible to accurately identify and predict the dynamic changes of the environment, providing the laser radar with an accurate basis for adjusting the laser beam density. In high-dynamic scenarios, the laser beam density of the laser radar can be intelligently increased, thereby improving the accuracy of the point cloud data and ensuring that the drone can accurately avoid obstacles and plan flight paths in rapidly changing environments. This solution significantly improves flight safety and path planning accuracy, especially in complex dynamic environments, and can better respond to external changes, ensuring that the drone can complete its tasks stably and efficiently.

[0049] The present invention provides Figure 2 A multi-dimensional perception compensation UAV control system shown includes an image data acquisition and environment monitoring module, an image difference comparison and analysis module, a feature extraction and processing module, an intelligent prediction and analysis module, a laser radar initial configuration module, and a laser radar dynamic adjustment module; The image data acquisition and environment monitoring module captures the real-time image data of the current flight path of the UAV through the camera of the visual sensor. Through the continuously acquired image data, the flight environment is monitored in real time and the changes in the surroundings are detected; The image difference comparison and analysis module compares and analyzes two consecutive frames of images, calculates the difference between them, and forms a difference set of the differences between the images to fully understand the changes in the flight environment; The feature extraction and processing module extracts key features that reflect the high-speed dynamic changes of the shooting scene from the difference set, performs feature engineering on the extracted key features, provides input for subsequent intelligent prediction, and improves the accuracy of subsequent predictions; The intelligent prediction and analysis module uses the data processed by feature engineering as input and passes it to the pre-trained convolutional neural network to intelligently analyze and predict the differences between images, and feedback the degree of dynamic changes in the shooting scene based on the prediction results; The LiDAR initial configuration module sets the initial low emission laser beam density for the LiDAR and configures it to meet the flight shooting requirements in low-dynamic scenes; The dynamic adjustment module of the LiDAR intelligently adjusts the laser beam density of the LiDAR in high-dynamic scenarios based on the prediction results of image differences by the convolutional neural network, providing higher-precision point cloud data to ensure accurate obstacle avoidance and flight path planning for drones in complex environments.

[0050] An embodiment of the present invention provides a multi-dimensional perception compensation UAV control method, which is realized by the above-mentioned multi-dimensional perception compensation UAV control system. The specific method and process of a multi-dimensional perception compensation UAV control system are detailed in the embodiment of the above-mentioned multi-dimensional perception compensation UAV control method, which will not be repeated here.

[0051] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters in the formula are set by technicians in this field according to actual conditions.

[0052] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with the technical field within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

[0053] The above description is only by way of illustration of certain exemplary embodiments of the present invention. It is undoubted that those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the above drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

Claims

1. A multi-dimensional perception compensation drone control method, characterized in that: The following steps are involved: The camera of the visual sensor captures real-time image data of the current flight path of the drone. Through the continuously acquired image data, the flight environment is monitored in real time and changes in the surroundings are detected; Compare and analyze two consecutive frames of images, calculate the difference between them, and form a difference set of the differences between the images to fully understand the changes in the flight environment; Extract key features that reflect the high-speed dynamic changes of the shooting scene from the difference set, perform feature engineering on the extracted key features, provide input for subsequent intelligent prediction, and improve the accuracy of subsequent predictions; The data processed by feature engineering will be passed as input to the pre-trained convolutional neural network to intelligently analyze and predict the differences between images, and feedback the degree of dynamic changes in the shooting scene based on the prediction results; Set an initial low emission laser beam density for the LiDAR and configure it to meet the flight photography requirements in low-dynamic scenes; In high-dynamic scenarios, based on the prediction results of image differences by the convolutional neural network, the laser beam density of the lidar is intelligently adjusted to provide higher-precision point cloud data, ensuring accurate obstacle avoidance and flight path planning for drones in complex environments.

2. The multi-dimensional perception compensation drone control method according to claim 1, characterized in that: Image comparison is achieved through image difference metrics, where the image difference metrics include pixel-level differences, edge changes, or motion detection algorithms.

3. The multi-dimensional perception compensation drone control method according to claim 1, characterized in that: Key features that reflect the high-speed dynamic changes of the shooting scene are extracted from the difference set, where the extracted features include the expansion and contraction degree of the edge area in the image and the vector distribution of the object motion in the image. After feature engineering processing of the expansion and contraction degree of the edge area in the image and the vector distribution of the object motion in the image, an edge dynamic expansion factor and a motion vector scatter factor are generated respectively. The edge dynamic expansion factor and the motion vector scatter factor are used to quantify the speed and direction change of the object motion in the shooting scene and the degree of dynamic change of the edge area in the environment, thereby comprehensively reflecting the high-speed dynamic changes of the shooting scene.

4. The multi-dimensional perception compensation drone control method according to claim 3 is characterized in that: The edge dynamic expansion factor and motion vector scatter factor after feature engineering are used as input and passed to the pre-trained convolutional neural network. The difference evaluation coefficient is generated based on the convolutional neural network. The difference between images is intelligently analyzed and predicted through the difference evaluation coefficient, and the dynamic change degree of the shooting scene is fed back.

5. The multi-dimensional perception compensation drone control method according to claim 4, characterized in that: The steps for dividing low dynamic scenes and high dynamic scenes are as follows: The difference evaluation coefficient generated by the pre-trained convolutional neural network when performing intelligent analysis and prediction of the differences between images is compared and analyzed with the pre-set difference evaluation coefficient reference threshold, and the shooting scene is divided. The specific division process is: if the difference evaluation coefficient is greater than the difference evaluation coefficient reference threshold, the shooting scene is divided into a high-dynamic scene; if the difference evaluation coefficient is less than or equal to the difference evaluation coefficient reference threshold, the shooting scene is divided into a low-dynamic scene.

6. The multi-dimensional perception compensation drone control method according to claim 3, characterized in that: For two consecutive frames of images, the specific steps of generating the edge dynamic expansion factor after feature engineering processing of the expansion and contraction degree of the edge area in the image are as follows: First, two consecutive frames of images are processed using an edge detection algorithm to extract the edge area in each frame; Then, the relative motion of each edge point in the image is calculated, and the edge difference matrix is ​​used to quantify the change of edge points in the two frames of images. The quantization expression of the edge difference matrix is: ； in, and Indicates at location The edge strength at , the difference matrix Indicates at location The margin variation of The edge expansion and contraction areas extracted by the edge difference matrix are used to quantify the dynamic degree of the edge expansion and contraction areas. The edge dynamic expansion factor is defined by the relative area change rate of the expansion and contraction areas. The definition expression is: ； in: is the set of all pixels in the edge area of ​​the image. represents the edge dynamic expansion factor, Indicates at location The relative area change rate at the position is calculated by weighted summation to calculate the total area change of edge expansion and contraction in the image. The specific calculation expression is: ； in: Indicates location The weight of the image region at .

7. The multi-dimensional perception compensation drone control method according to claim 3, characterized in that: For two consecutive frames of images, the specific steps of generating the motion vector scatter factor after feature engineering processing of the vector distribution of the object motion in the image are as follows: First, the motion vector of each pixel in two consecutive frames is calculated. The motion vector represents the displacement of each pixel from the first frame to the second frame, which is calculated by optical flow method or sparse feature matching. The displacement in the image is calculated by the following formula to calculate its motion vector: ； in: It's pixels The position in the first frame, Is the pixel The position in the second frame, Represents pixels Motion vector; After calculating the motion vector of each pixel, the dispersion of the motion vector is quantified and a motion vector dispersion factor is generated. The quantization expression of the motion vector dispersion factor is: ； in: represents the motion vector spread factor, Indicates The size of the motion vector of pixels, It is The direction angle of the motion vector of each pixel, and is an adjustable exponential parameter that controls the magnitude and direction of the movement. N is the total number of pixels in the image.

8. The multi-dimensional perception compensation drone control method according to claim 5, characterized in that: In high-dynamic scenes, based on the prediction results of image differences by the convolutional neural network, the laser beam density of the lidar is intelligently adjusted. The specific steps are as follows: In high-dynamic scenes, the difference evaluation coefficient generated by the convolutional neural network is first used to quantify the dynamic change degree of the scene. In order to dynamically adjust the laser beam density, the laser beam density adjustment coefficient is introduced. The calculation expression of the laser beam density adjustment coefficient is: ； in: is the laser beam density adjustment factor, is the adjustment factor used to control the sensitivity of the difference evaluation coefficient to the laser beam density; After determining the high dynamic scene and calculating the laser beam density adjustment coefficient, the initial low emission laser beam density of the lidar is intelligently adjusted according to the laser beam density adjustment coefficient. The adjustment formula of the laser beam density is as follows: ； in: is the preset initial low emission laser beam density, is the actual emitted laser beam density after adjustment, indicating the density required in high dynamic scenes.

9. A multi-dimensional perception compensation UAV control system, used to implement the multi-dimensional perception compensation UAV control method described in any one of claims 1 to 8, characterized in that: It includes image data acquisition and environment monitoring module, image difference comparison and analysis module, feature extraction and processing module, intelligent prediction and analysis module, laser radar initial configuration module and laser radar dynamic adjustment module; The image data acquisition and environment monitoring module captures the real-time image data of the current flight path of the UAV through the camera of the visual sensor. Through the continuously acquired image data, the flight environment is monitored in real time and the changes in the surroundings are detected; The image difference comparison and analysis module compares and analyzes two consecutive frames of images, calculates the difference between them, and forms a difference set of the differences between the images to fully understand the changes in the flight environment; The feature extraction and processing module extracts key features that reflect the high-speed dynamic changes of the shooting scene from the difference set, performs feature engineering on the extracted key features, provides input for subsequent intelligent prediction, and improves the accuracy of subsequent predictions; The intelligent prediction and analysis module uses the data processed by feature engineering as input and passes it to the pre-trained convolutional neural network to intelligently analyze and predict the differences between images, and feedback the degree of dynamic changes in the shooting scene based on the prediction results; The LiDAR initial configuration module sets the initial low emission laser beam density for the LiDAR and configures it to meet the flight shooting requirements in low-dynamic scenes; The dynamic adjustment module of the LiDAR intelligently adjusts the laser beam density of the LiDAR in high-dynamic scenarios based on the prediction results of image differences by the convolutional neural network, providing higher-precision point cloud data to ensure accurate obstacle avoidance and flight path planning for drones in complex environments.

Images