Unmanned aerial vehicle image-assisted path tracking and obstacle avoidance method and system

Abstract

The invention discloses an unmanned aerial vehicle image-assisted path tracking and obstacle avoidance method and system, and the method comprises the following steps: calling a multi-modal sensor group deployed on an unmanned aerial vehicle to collect environment data in response to a flight task instruction of the unmanned aerial vehicle, and carrying out the cross-modal data fusion of the environment data through an adaptive time synchronization algorithm; and calling a preset generative adversarial network model, performing image generation and illumination compensation on low-light image data in the fused environment data, and generating a target enhanced image close to a visible light environment. According to the invention, through cross-modal learning and self-supervised learning technologies, the perception and data synchronization capability of the unmanned aerial vehicle in a complex environment is improved. According to the invention, the generative adversarial network is adopted for image enhancement, and especially in a low-light environment, the accuracy of obstacle detection is improved; real-time obstacle detection and three-dimensional environment modeling are performed in combination with a deep learning model, and path planning and obstacle avoidance control are optimized through deep reinforcement learning.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of unmanned aerial vehicle flight control and autonomous navigation, in particular to an unmanned aerial vehicle image-assisted path tracking and obstacle avoidance method and system. BACKGROUND

[0002] With the rapid development of unmanned aerial vehicle technology, its application scenarios have expanded from the initial aerial photography entertainment to complex fields such as logistics distribution, power inspection, emergency rescue, and urban security. In these applications, the autonomous flight capability of unmanned aerial vehicles, especially the path tracking and obstacle avoidance capability, is the core key to ensuring task success and flight safety.

[0003] Path tracking requires the unmanned aerial vehicle to fly along the preset route with high precision, while obstacle avoidance requires the unmanned aerial vehicle to perceive obstacles in the environment in real time and take evasive action. Existing unmanned aerial vehicle navigation systems usually rely on the Global Positioning System (GPS) for global path planning, and combine ultrasonic, single-line laser radar and other sensors for simple close-range obstacle avoidance. In recent years, with the progress of computer vision technology, image-assisted navigation based on visual sensors (such as monocular or binocular cameras) has gradually become the mainstream research direction due to its large amount of information, low cost, and small size.

[0004] However, although existing image-assisted path tracking and obstacle avoidance technology has made some progress, when applied in actual complex dynamic environments, it still faces many insurmountable technical bottlenecks: First, image quality is severely affected by environmental lighting, and all-weather adaptability is poor. Existing visual navigation systems highly depend on environmental light. In low light or complex weather conditions such as night, fog, rain, or strong backlight, the images captured by RGB cameras often have problems such as many noise points, low contrast, and feature loss. Although some systems have introduced infrared cameras, single-mode sensors still have blind spots in certain environments, leading to a decrease in obstacle recognition rate and an increased risk of collision accidents.

[0005] Second, multi-modal sensor data fusion is difficult, and time and space alignment is not accurate. In order to compensate for the shortcomings of single visual sensors, high-end unmanned aerial vehicles have begun to carry cameras, laser radars, and millimeter wave radars. However, the working mechanisms, sampling frequencies, and field angles of different sensors are different. Existing technologies often lack efficient adaptive synchronization and calibration mechanisms when dealing with these multi-source heterogeneous data. Time delays or spatial calibration errors between sensors can cause image data and depth information to be mismatched, resulting in the construction of an incorrect surrounding environment model and misleading flight decisions.

[0006] Secondly, there is a contradiction between limited onboard computing resources and the demand for high real-time processing. While deep learning algorithms (such as large convolutional neural networks) achieve high accuracy in target detection, their computational demands are enormous. Due to limitations in size and battery capacity, the computing power of onboard processors in drones is extremely limited. During high-speed flight, existing technologies struggle to complete complex image denoising, target recognition, and path replanning within milliseconds, resulting in delayed obstacle avoidance responses and failing to meet the real-time requirements of dynamic environments.

[0007] Furthermore, path tracking and obstacle avoidance control lack an effective dynamic coordination mechanism. Traditional control strategies often treat path tracking and obstacle avoidance separately: when obstacle avoidance is triggered, the drone often deviates significantly from the preset path, and even after obstacle avoidance, it is difficult to smoothly return to the original path, resulting in flight trajectory oscillations. At the same time, when facing moving obstacles (such as birds and vehicles), existing algorithms are mostly based on static assumptions and lack the ability to predict the trajectory of obstacles, making it difficult to achieve agile dynamic avoidance.

[0008] Finally, the control system lacks robustness and online adaptive capability. Existing flight control parameters (such as PID parameters) are usually calibrated offline and cannot be adjusted in real time according to environmental changes during flight (such as sudden gusts or load changes), resulting in poor flight stability of UAVs under complex interference. To address this, an image-assisted path tracking and obstacle avoidance method and system for UAVs is proposed. Summary of the Invention

[0009] The purpose of this invention is to provide an image-assisted path tracking and obstacle avoidance method and system for unmanned aerial vehicles (UAVs). By fusing multimodal sensor data (including cameras, LiDAR, etc.) and combining advanced image enhancement technology, the invention enhances the UAV's perception capabilities in complex environments, ensuring path tracking and obstacle avoidance accuracy, and solving one of the problems existing in the prior art.

[0010] Firstly, to solve the aforementioned technical problems, this application adopts a technical solution: an image-assisted path tracking and obstacle avoidance method for unmanned aerial vehicles (UAVs), comprising the following steps: In response to the flight mission command of the UAV, the multimodal sensor group deployed on the UAV is invoked to collect environmental data, and the environmental data is fused across modes using an adaptive time synchronization algorithm; A preset generative adversarial network model is invoked to generate images and compensate for illumination in the low-light image data in the fused environmental data, thereby generating a target enhancement image that approximates the visible light environment. The target enhancement image and LiDAR point cloud data are input into a deep learning detection model to perform real-time obstacle detection and classification, and a three-dimensional environment model is constructed based on the detection results by calling a deep convolutional generative adversarial network. Based on the three-dimensional environment model, a dynamic flight path is generated using the A* algorithm combined with deep reinforcement learning, and a local obstacle avoidance algorithm is invoked to correct the dynamic flight path when an obstacle is detected. Based on the corrected dynamic flight path, the flight control commands are calculated by a PID controller combined with a linear quadratic regulator algorithm to control the UAV to perform path tracking and obstacle avoidance. Based on real-time flight feedback, the flight control strategy is optimized online through self-supervised learning.

[0011] In one possible implementation, the invocation of a multimodal sensor array deployed on the UAV to collect environmental data, and the use of an adaptive time synchronization algorithm to perform cross-modal data fusion on the environmental data, specifically includes: The system acquires multi-source heterogeneous data collected by the multimodal sensor group, which includes an RGB camera, an infrared camera, a lidar, and a millimeter-wave radar. Features are extracted from unlabeled data using a self-supervised learning algorithm, and the multi-source heterogeneous data is preprocessed for time synchronization and spatial calibration. A deep cross-modal adversarial network is used to map calibrated RGB images, infrared images and lidar point cloud data to a unified feature space for fusion, so as to complement and enhance environmental features.

[0012] In one possible implementation, the step of invoking a preset generative adversarial network model to perform image generation and illumination compensation on the low-light image data in the fused environmental data specifically includes: Construct a generative adversarial network that includes a generator and a discriminator; The generator inputs low-light environment images or haze images into the generator and simulates real light distribution through adversarial training to generate a visual effect image of clear visible light image. The generated visual effect image is used to restore the environmental texture and obstacle outline details, which are then used as input data for subsequent obstacle detection.

[0013] In one possible implementation, the step of inputting the enhanced target image and LiDAR point cloud data into a deep learning detection model for real-time obstacle detection and classification specifically includes: Low-level features of the target enhancement image and LiDAR data are extracted using a convolutional neural network. The extracted features are input into the YOLOv4 object detection model to identify and classify dynamic and static obstacles within the field of view; By combining the Kalman filter, the motion trajectory of the identified dynamic obstacles is tracked, and their spatial position at the next moment is predicted.

[0014] In one possible implementation, the step of constructing a 3D environment model by invoking a deep convolutional generative adversarial network based on the detection results specifically includes: The fused image surface feature data and LiDAR depth data are input into a deep convolutional generative adversarial network; The deep convolutional generative adversarial network is used to reconstruct the three-dimensional spatial structure of the flight environment and generate a virtual obstacle model that includes the location, shape and spatial relationship of obstacles. The virtual obstacle model is updated in real time in response to environmental changes to reflect the environmental state in dynamic scenes.

[0015] In one possible implementation, the generation of a dynamic flight path based on the three-dimensional environment model using the A* algorithm combined with deep reinforcement learning specifically includes: Based on the aforementioned three-dimensional environment model, the A* algorithm is used to plan the globally optimal path from the current position to the target position; Real-time environmental data is input into a deep reinforcement learning model, and the global optimal path is dynamically adjusted based on a trial-and-error learning mechanism to output a local path decision that conforms to the current flight state. The deep reinforcement learning model is configured to predict obstacles that may appear within a preset time period and output path adjustment parameters in advance. The generative adversarial network is used to generate virtual obstacle images and synthesize them into the three-dimensional environment model; The obstacle avoidance strategy of the UAV is trained based on the synthesized virtual obstacle image in a simulation environment, and the trained strategy parameters are transferred to the actual flight control.

[0016] In one possible implementation, the step of invoking a local obstacle avoidance algorithm to correct the dynamic flight path when an obstacle is detected specifically includes: In response to the detection of obstacles on the path, a series of candidate trajectories based on the current velocity and acceleration are calculated using the dynamic window method, and the optimal trajectory is selected for execution; Alternatively, an artificial potential field method can be used, where the target point is set as a gravitational field and the obstacles are set as repulsive fields. The flight trajectory can be corrected in real time according to the direction of the resultant force on the UAV to avoid multiple target obstacles.

[0017] In one possible implementation, the calculation of flight control commands using a PID controller combined with a linear quadratic regulator algorithm specifically includes: The first control variable is calculated based on the current attitude error using a PID controller to eliminate short-term flight deviations; The LQR algorithm is used to construct a cost function, and the second control variable is calculated based on the position, velocity and acceleration state of the UAV to optimize global flight energy efficiency and eliminate long-term accumulated errors. The first control quantity and the second control quantity are combined to generate the final motor speed control command, which adjusts the flight attitude of the UAV.

[0018] In one possible implementation, the online optimization of the flight control strategy based on real-time flight feedback through self-supervised learning specifically includes: Image data, lidar data, and flight status data are collected in real time during flight as unlabeled training samples; The training samples are analyzed using a self-supervised learning model, and the parameters of the control algorithm are updated online to adapt to environmental conditions such as changes in wind speed or light intensity. By accumulating flight experience through reinforcement learning mechanisms, the system can automatically adjust its flight strategies to enhance its robustness.

[0019] Secondly, to solve the above-mentioned technical problems, another technical solution adopted in this application is: a UAV image-assisted path tracking and obstacle avoidance method, the method comprising the following steps: Step 1: In response to the flight mission command of the UAV, the multimodal perception system deployed on the UAV is invoked to collect real-time environmental image data, and the real-time environmental image data is synchronized and initially fused with multi-sensor data; Step 2: Call the preset image enhancement model to perform denoising and illumination compensation on the fused image data to generate an enhanced environment image, and perform weighted fusion of the visible light image and infrared image based on deep learning methods; Step 3: Input the fused image data into the preset obstacle detection model for real-time obstacle recognition, and combine the global path planning algorithm and the local obstacle avoidance algorithm to generate the target flight path; Step 4: In response to the generated target flight path, the UAV is controlled by the flight controller to perform path tracking, and adaptive optimization and adjustment are performed using deep learning algorithms based on feedback data during the flight process.

[0020] In one possible implementation, the invocation of a multimodal perception system deployed on the UAV to acquire real-time environmental image data, and the multi-sensor data synchronization and preliminary fusion of the real-time environmental image data, includes: The system uses an RGB camera to capture visible light images, an infrared camera to capture infrared images, and a lidar and ultrasonic sensor to collect obstacle distance information. A sensor fusion algorithm is used to perform time synchronization and preliminary fusion of the collected multi-source data, wherein the sensor fusion algorithm includes a Kalman filter; The weighted fusion of visible light images and infrared images based on deep learning methods includes: using a self-attention mechanism to perform feature-weighted fusion of image data acquired by the RGB camera and image data acquired by the infrared camera.

[0021] In one possible implementation, the step of calling a preset image enhancement model to perform denoising and illumination compensation processing on the fused image data to generate an enhanced environment image includes: A convolutional neural network (CNN) is invoked to remove noise from the images captured by the RGB camera, and the brightness and contrast of the images are adjusted according to the ambient lighting conditions. When the current environment is detected to be low light or completely dark, a generative adversarial network (GAN) is invoked to enhance the low-resolution or high-noise infrared image captured by the infrared camera, and a high-definition enhanced infrared image is generated through adversarial training.

[0022] In one possible implementation, the step of inputting the fused image data into a preset obstacle detection model for real-time obstacle recognition, and generating a target flight path by combining a global path planning algorithm and a local obstacle avoidance algorithm, includes: Real-time detection of static and dynamic obstacles in the environment is achieved using deep convolutional neural networks (CNN) or YOLO models. Use the A* algorithm or the Quick Random Tree (RRT) algorithm to compute the globally optimal path from the starting point to the ending point; When an obstacle is detected in the local environment, the Dynamic Window Method (DWA) is used to perform real-time obstacle avoidance planning without changing the global optimal path; The Deep Reinforcement Learning (DRL) algorithm is used to dynamically adjust the obstacle avoidance strategy according to the real-time environment and automatically correct the flight path when dynamic obstacles are detected.

[0023] In one possible implementation, in response to the generated target flight path, the flight controller controls the UAV to perform path tracking, and adaptive optimization is performed using a deep learning algorithm based on feedback data during flight, including: The UAV is used for path tracking control using a linear quadratic control (LQR) controller or a PID controller, and its flight altitude, speed and direction are dynamically adjusted in combination with the enhanced environmental image and the obstacle distance information. Acquire feedback data during flight, including image quality, obstacle detection success rate, and flight stability; The feedback data is analyzed using reinforcement learning algorithms, and the image enhancement and obstacle avoidance strategies are adaptively adjusted to adapt to different lighting or weather conditions.

[0024] In one possible implementation, a generative adversarial network (GAN) is used to enhance infrared images in low-light environments, thereby improving image quality and addressing the problem of insufficient image recognition capabilities during low-light and nighttime flight.

[0025] Multimodal sensor fusion technology: By fusing data from multimodal sensors (RGB, infrared, lidar), the perception accuracy and stability in complex environments are improved, especially in the detection and obstacle avoidance capabilities of dynamic obstacles.

[0026] Combining deep learning and reinforcement learning: Deep learning optimizes image processing and obstacle recognition, while reinforcement learning is used to adaptively adjust the flight strategy, solving the problem of maintaining path tracking and obstacle avoidance accuracy in complex environments.

[0027] Thirdly, to solve the aforementioned technical problems, another technical solution adopted in this application is: an image-assisted path tracking and obstacle avoidance system for unmanned aerial vehicles, comprising: The multimodal perception module is configured to drive RGB cameras, infrared cameras, LiDAR and millimeter-wave radar deployed on the drone to collect environmental data and perform data synchronization and fusion based on self-supervised learning; The image enhancement processing module is configured to call a generative adversarial network model to perform illumination compensation and denoising on low-light images, and generate target enhanced images. The environment modeling and detection module is configured to use the YOLOv4 model to detect and classify obstacles, and to use a deep convolutional generative adversarial network to build a real-time updated 3D environment model based on the detection results. The path planning module is configured to combine the A* algorithm with deep reinforcement learning to plan dynamic flight paths and perform local obstacle avoidance using the dynamic window method or the artificial potential field method. The flight control module is configured to output flight control commands using a combination of a PID controller and an LQR algorithm, and to optimize the control strategy online based on self-supervised learning to drive the UAV to perform path tracking and obstacle avoidance.

[0028] Fourthly, to solve the above-mentioned technical problems, another technical solution adopted in this application is: an electronic device, including a processor, a memory and a communication interface, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the steps of the above-mentioned UAV image-assisted path tracking and obstacle avoidance method.

[0029] Fifth aspect: To solve the above-mentioned technical problems, another technical solution adopted in this application is: a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the UAV image-assisted path tracking and obstacle avoidance method as described above.

[0030] The present invention has the following beneficial effects: 1. This invention improves the perception and data synchronization capabilities of UAVs in complex environments through cross-modal learning and self-supervised learning technologies; 2. This invention uses generative adversarial networks for image enhancement, which improves the accuracy of obstacle detection, especially in low-light environments; 3. This invention combines a deep learning model for real-time obstacle detection and 3D environment modeling, and optimizes path planning and obstacle avoidance control through deep reinforcement learning; 4. Through self-supervised learning and online optimization mechanisms, this invention enables the UAV to automatically adjust its control strategy based on real-time feedback during flight, thereby adapting to different flight environments. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 This is a flowchart illustrating the UAV image-assisted path tracking and obstacle avoidance method of the present invention. Figure 2 This is a flowchart illustrating Embodiment 2 of the present invention; Figure 3 This is a block diagram of the UAV image-assisted path tracking and obstacle avoidance system according to Embodiment 3 of the present invention; Figure 4 This is a schematic diagram of the structure of the electronic device of the present invention. Detailed Implementation

[0033] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0034] Figure 1This is a flowchart illustrating the UAV image-assisted path tracking and obstacle avoidance method according to an embodiment of the present invention. It should be noted that if substantially the same result is achieved, the method of this application is not necessarily identical. Figure 1 The sequence of processes shown is limited.

[0035] Example 1: The image quality of drones is limited in low-light and complex environments (such as nighttime, fog, haze, rain, and snow), resulting in inaccurate image processing and affecting the accuracy of path tracking and obstacle avoidance. Therefore, existing technologies often perform poorly in these environments and cannot guarantee stable flight and safe obstacle avoidance for drones.

[0036] like Figure 1 The image-assisted path tracking and obstacle avoidance method for UAVs shown includes the following steps: S10. In response to the flight mission command of the UAV, the multimodal sensor group deployed on the UAV is invoked to collect environmental data, and the environmental data is fused across modes using an adaptive time synchronization algorithm. In this embodiment, specifically, the step of calling the multimodal sensor group deployed on the UAV to collect environmental data and using an adaptive time synchronization algorithm to perform cross-modal data fusion on the environmental data specifically includes: The system acquires multi-source heterogeneous data collected by the multimodal sensor group, which includes an RGB camera, an infrared camera, a lidar, and a millimeter-wave radar. Features are extracted from unlabeled data using a self-supervised learning algorithm, and the multi-source heterogeneous data is preprocessed for time synchronization and spatial calibration. A deep cross-modal adversarial network is used to map calibrated RGB images, infrared images and lidar point cloud data to a unified feature space for fusion, so as to complement and enhance environmental features; Sensor deployment and data acquisition: A multimodal sensor array is deployed at a predetermined location on the drone's fuselage, specifically including: RGB camera: Used to capture visible light images, providing rich semantic information such as texture and color, suitable for well-lit daytime environments.

[0037] Infrared cameras: used to collect thermal imaging data to identify targets with temperature characteristics (such as pedestrians and vehicle engines) in nighttime, smoke, or camouflaged environments.

[0038] LiDAR: Used to acquire high-precision sparse point clouds of the environment, providing accurate distance and depth information.

[0039] Millimeter-wave radar: used to penetrate rain, fog, dust, and snow to provide information on the distance and relative velocity of targets (such as the Doppler effect).

[0040] For example, RGB cameras provide detailed images but perform poorly in low-light environments, which infrared cameras compensate for, while LiDAR provides accurate spatial information and millimeter-wave radar provides powerful obstacle detection.

[0041] By combining data from different sensors using state-of-the-art cross-modal learning methods (such as deep cross-modal adversarial networks), complementary enhancements can be achieved. For example, precise 3D information from LiDAR data can be fused with visual information from RGB cameras to generate a clearer and more accurate environment model.

[0042] For example, suppose a drone is flying in a complex urban environment, which includes obstacles such as buildings, trees, and pedestrians. An RGB camera can clearly capture images of obstacles during the day or in good lighting conditions, but its performance is poor at night or in foggy or hazy conditions. In this situation, an infrared camera supplements the low-light environment with thermal imaging data, enabling the identification of dynamic obstacles (such as pedestrians or vehicles). LiDAR provides precise distance information to help the drone build a 3D environment model, while millimeter-wave radar ensures that dynamic obstacles ahead can still be detected in foggy weather.

[0043] Data preprocessing and adaptive synchronization: Due to the different sampling frequencies of each sensor (e.g., 10Hz for LiDAR, 30Hz for camera), direct fusion would lead to data misalignment. This embodiment uses the following method to handle this: An adaptive time synchronization algorithm is adopted, using the timestamp of high-frequency sensors (such as cameras) as a reference, and aligning the low-frequency sensor data through interpolation algorithms (such as linear interpolation or nearest neighbor interpolation) to ensure that data frames at the same time correspond.

[0044] A feature extraction network is constructed using a self-supervised learning algorithm. This network automatically learns the extrinsic parameter matrix (spatial positional relationship) between different sensors by minimizing the multi-view geometric consistency error, without requiring manual annotation, thus achieving pixel-level or feature-level spatial calibration.

[0045] For example, during drone flight, the sampling frequencies of the RGB camera and LiDAR differ. Assuming the LiDAR has a lower sampling frequency while the RGB camera captures 30 frames per second, the RGB image and LiDAR point cloud data will be out of sync. An adaptive time synchronization algorithm can adjust the data acquisition time in real time, ensuring the RGB image and LiDAR point cloud data are aligned within the same time window. A deep learning-based self-supervised synchronization network further optimizes the spatial alignment of these two data sources, preventing distortion caused by measurement errors or motion from affecting the data fusion effect.

[0046] Cross-modal data fusion: A deep cross-modal adversarial network is used for fusion. The calibrated RGB and infrared images are mapped to a two-dimensional feature space, and the LiDAR point cloud is mapped to a three-dimensional feature space. The feature transformation module in the network projects the two images into a unified common feature subspace for stitching or weighted fusion, thereby obtaining enhanced environmental features with both rich texture and accurate geometric structure.

[0047] For example, suppose a drone is flying in the early morning fog. The RGB camera captures a completely white image, but the infrared camera captures the thermal outline of a pedestrian in front, the millimeter-wave radar detects the pedestrian's speed, and the lidar measures the pedestrian's distance to the drone as 15 meters. Through the fusion algorithm in this step, the system generates a comprehensive data packet that contains both a clear outline of the pedestrian (from infrared) and precise distance (from radar / liquid), eliminating the blind spots of the RGB camera.

[0048] S20. Call the preset generative adversarial network model to perform image generation and illumination compensation on the low-light image data in the fused environmental data to generate a target enhanced image that is close to the visible light environment. In this embodiment, specifically, the step of calling a preset generative adversarial network model to perform image generation and illumination compensation on the low-light image data in the fused environmental data specifically includes: Construct a generative adversarial network that includes a generator and a discriminator; The generator inputs low-light environment images or haze images into the generator and simulates real light distribution through adversarial training to generate a visual effect image of clear visible light image. The generated visual effect image is used to restore the environmental texture and obstacle outline details, which are then used as input data for subsequent obstacle detection.

[0049] Specifically, Generative Adversarial Network (GAN) construction: Construct a GAN model that includes a generator and a discriminator. Generator: The input is a raw image acquired under low light, with noise or blur, and the output is an enhanced image under simulated standard lighting conditions.

[0050] Discriminator: Used to determine whether the input image is a real, clear daytime image or an image generated by the generator.

[0051] By employing a contrastive learning approach, the generative adversarial network (GAN) model enhances information such as texture, brightness, and contrast in the learned image features through comparison of positive and negative samples. Especially for images with low light or high noise levels, the GAN model learns how to enhance these details, remove noise, and improve image quality.

[0052] For example, traditional image denoising methods typically rely on large amounts of manually labeled data for supervised learning, while self-supervised learning does not require labeled data. Through this unlabeled training method, the model can automatically learn the structural features of images from large-scale unlabeled image data, especially how to improve perception capabilities in complex environments (such as low-light and dynamic scenes).

[0053] For low-light images (e.g., images taken at night or in hazy weather), self-supervised learning can help the model automatically extract structural information from low-light images and enhance the brightness, contrast, and texture of the image, making the image details clearer.

[0054] Image quality can be improved by using image enhancement algorithms (such as histogram equalization, brightness and contrast adjustment) combined with self-supervised learning.

[0055] For example, suppose a drone captures a low-light image while flying at night. Some obstacles (such as power lines and buildings) are blurred and difficult to identify. Using a self-supervised learning framework, the drone will automatically learn how to improve the brightness and contrast of a large number of unlabeled nighttime images. Through denoising algorithms, noise in the image will be effectively removed, and image details (such as the outlines of power lines) will become more apparent, enhancing the accuracy of subsequent obstacle detection algorithms.

[0056] Image Generation and Illumination Compensation: Generative Adversarial Network (GAN) Structure: A GAN consists of a generator and a discriminator. The generator produces images to simulate real-world images, while the discriminator determines whether an image is real. Through adversarial training, the generator learns to generate more realistic and higher-quality images.

[0057] In low-light environments (such as at night or in fog), the generator accepts low-quality images as input and generates clearer, moderately brighter images through generative adversarial training.

[0058] This process not only enhances image quality but also restores image details (such as the edges and contours of obstacles).

[0059] The generator compensates for the lighting conditions of the input image by enhancing brightness and contrast to make the image look closer to daytime conditions, thereby improving the accuracy of obstacle detection and environmental perception.

[0060] Generative adversarial networks (GANs) can be used not only to enhance low-light images but also to generate virtual obstacle models. These virtual obstacles can be synthesized from images and used to train obstacle avoidance strategies for drones.

[0061] In actual flight, generative adversarial networks can generate realistic obstacle images for drones, thereby providing training data and improving the performance of obstacle avoidance algorithms in real environments.

[0062] For example, suppose a drone flies at night and captures a low-light image with several trees in the foreground, but the light is too dim and the tree outlines are unclear. By using a generative adversarial network (GAN), the generator enhances the tree areas in the low-light image, increasing their brightness and contrast to daytime image levels, making the tree outlines and details clearer. Subsequent obstacle detection algorithms (such as YOLO, CNN, etc.) can then accurately identify the trees and plan obstacle avoidance paths.

[0063] Generative adversarial networks (GANs) can also generate virtual models of trees, buildings, or other common obstacles to train drone obstacle avoidance algorithms. These virtual obstacles allow for training in a simulated environment, optimizing obstacle avoidance strategies without relying on real-world environmental data.

[0064] S30. Input the target enhancement image and lidar point cloud data into the deep learning detection model to perform real-time obstacle detection and classification, and call the deep convolutional generative adversarial network to construct a three-dimensional environment model based on the detection results. In this embodiment, specifically, the step of inputting the enhanced target image and LiDAR point cloud data into a deep learning detection model for real-time obstacle detection and classification includes: Low-level features of the target enhancement image and LiDAR data are extracted using a convolutional neural network. The extracted features are input into the YOLOv4 object detection model to identify and classify dynamic and static obstacles within the field of view; By combining the Kalman filter to track the motion trajectory of the identified dynamic obstacles, the spatial position of the obstacles in the next moment can be predicted. The process of constructing a 3D environment model by calling a deep convolutional generative adversarial network based on the detection results specifically includes: The fused image surface feature data and LiDAR depth data are input into a deep convolutional generative adversarial network; The deep convolutional generative adversarial network is used to reconstruct the three-dimensional spatial structure of the flight environment and generate a virtual obstacle model that includes the location, shape and spatial relationship of obstacles. The virtual obstacle model is updated in real time in response to environmental changes to reflect the environmental state in dynamic scenes. Specifically, obstacle detection and classification: The enhanced image and LiDAR data generated in step S20 are input into the detection module. Feature extraction: Use convolutional neural networks to extract low-level features such as edges and corners of the image.

[0065] Object detection: Call the YOLOv4 model and output the bounding box of the object: class probability.

[0066] YOLOv4 can complete the detection with a single forward propagation, meeting the real-time requirements of high-speed drone flight.

[0067] For identified dynamic obstacles (such as birds and vehicles), the Kalman filter is used to predict the possible position of the next moment by utilizing the position and velocity of the previous moment, thereby solving the problem of temporary occlusion or frame loss detection.

[0068] For example, suppose a drone is flying and its RGB camera captures an image of a city street containing both static obstacles (such as buildings) and dynamic obstacles (such as moving cars). Simultaneously, a LiDAR scanner provides precise depth information, helping the system distinguish obstacles in front from the distant background. Using a deep CNN model that learns across modalities, the drone can combine RGB images, infrared images, and LiDAR data to identify and classify these obstacles in real time.

[0069] RGB images: Identify the appearance of static obstacles (such as buildings).

[0070] Infrared imaging: Identifies heat sources and helps detect obstacles (such as pedestrians or vehicles) at night or in low-light environments.

[0071] LiDAR data: Provides the precise three-dimensional spatial location of obstacles, helping to accurately estimate distances.

[0072] By combining YOLOv4 and CNN, obstacles in images are labeled and classified in real time, such as labeling moving cars as dynamic obstacles and static buildings as fixed obstacles.

[0073] 3D Environment Modeling: RGB images, infrared images, and LiDAR data are fused together, and deep generative adversarial networks (GANs) or deep convolutional generative adversarial networks (DCGANs) are used to reconstruct the 3D environment. Through these models, UAVs can generate 3D environment models from 2D images, accurately representing the location, shape, and spatial relationships of obstacles.

[0074] LiDAR point cloud data provides accurate depth information, while image data provides surface features (such as texture and color). Generative adversarial networks combine this information to generate more detailed environmental models, especially in dynamic environments, where they can adaptively update the position and shape of obstacles.

[0075] Dynamic Environment Modeling: During flight, the environment is dynamically changing, and the state of obstacles may change (such as vehicle movement, pedestrian movement, etc.). By using deep generative adversarial networks to update the 3D model of obstacles and the state of the environment in real time, the generated environment model can not only accurately reflect static obstacles, but also adapt to dynamic scenes in real time.

[0076] Path planning and obstacle avoidance: Based on a 3D environment model, the A* algorithm or fast random tree is used for global path planning to find the shortest path from the starting point to the target.

[0077] Local obstacle avoidance: When a drone encounters an obstacle during flight, it uses the dynamic window method for local obstacle avoidance, adjusting its flight speed, direction, and altitude to avoid collisions.

[0078] Obstacle model updates: As the environment changes, the generated 3D model is updated in real time, and the flight path is optimized. Through continuous image and LiDAR data acquisition, combined with deep learning models and generative adversarial networks, the accuracy of environment modeling is continuously improved, and the flight path can be adjusted more intelligently.

[0079] For example, suppose a drone is flying in an indoor environment. LiDAR and image data provide a preliminary 3D environment model, including multiple static obstacles (such as walls and furniture) and dynamic obstacles (such as people). A deep generative adversarial network will fuse the point cloud data from the LiDAR and the image data to generate a detailed 3D model.

[0080] LiDAR point cloud data: provides the three-dimensional position and size of static obstacles such as walls and furniture.

[0081] Image data: Provides surface features of obstacles (such as color and texture) to help the model generate more detailed information about the appearance of objects.

[0082] As the drone flies, new dynamic obstacles (such as people) appear in its field of vision. The generative adversarial network updates the environment model in real time and adjusts the flight path through local obstacle avoidance algorithms to avoid collisions.

[0083] By combining self-supervised learning and cross-modal learning, and leveraging the advantages of image, infrared, and LiDAR data, efficient and accurate obstacle detection is achieved in dynamic environments. The ability to identify both dynamic and static obstacles is enhanced through deep learning models using YOLOv4 and CNN, combined with real-time tracking algorithms. Using deep generative adversarial networks (GANs) for 3D reconstruction of environments and generation of obstacle models breaks through the accuracy limitations of traditional environment modeling. Especially in dynamic scenes, GANs can generate and update obstacle models in real time and optimize path planning. By continuously collecting images and LiDAR data, combined with a deep generative model, not only is the obstacle detection accuracy improved, but the 3D model and path planning scheme can also be adaptively updated when the environment changes, greatly enhancing the navigation capability of UAVs in complex environments.

[0084] S40. Based on the three-dimensional environment model, a dynamic flight path is generated using the A* algorithm combined with deep reinforcement learning, and a local obstacle avoidance algorithm is called to correct the dynamic flight path when an obstacle is detected. In this embodiment, specifically, the step of generating a dynamic flight path based on the three-dimensional environment model using the A* algorithm combined with deep reinforcement learning includes: Based on the aforementioned three-dimensional environment model, the A* algorithm is used to plan the globally optimal path from the current position to the target position; Real-time environmental data is input into a deep reinforcement learning model, and the global optimal path is dynamically adjusted based on a trial-and-error learning mechanism to output a local path decision that conforms to the current flight state. The deep reinforcement learning model is configured to predict obstacles that may appear within a preset time period and output path adjustment parameters in advance. The generative adversarial network is used to generate virtual obstacle images and synthesize them into the three-dimensional environment model; The obstacle avoidance strategy of the UAV is trained based on the synthesized virtual obstacle image in a simulation environment, and the trained strategy parameters are transferred to the actual flight control. The step of invoking a local obstacle avoidance algorithm to correct the dynamic flight path when an obstacle is detected specifically includes: In response to the detection of obstacles on the path, a series of candidate trajectories based on the current velocity and acceleration are calculated using the dynamic window method, and the optimal trajectory is selected for execution; Alternatively, an artificial potential field method can be used, where the target point is set as a gravitational field and the obstacle is set as a repulsive field. The flight trajectory can be corrected in real time according to the direction of the resultant force on the UAV to avoid multiple target obstacles. The combination of the A* algorithm and deep reinforcement learning: Based on a known static environment map or a large-scale 3D model constructed in step S30, the A* algorithm is used to calculate the global path from the starting point to the ending point that has the minimum cost (shortest distance or lowest energy consumption). This path consists of a series of discrete spatial waypoints.

[0085] Deep reinforcement learning: In actual flight, the environment is dynamically changing, so the global path of the A* algorithm may be affected by real-time obstacles. Therefore, deep reinforcement learning is introduced into local path planning. Based on real-time perceived environmental data (such as images, LiDAR point clouds, obstacle locations, etc.), deep reinforcement learning optimizes path decisions through trial and error. Specifically, the UAV dynamically adjusts its path based on the current flight status and perceived obstacle information, using reward mechanisms (such as avoiding obstacles, reducing flight time, etc.).

[0086] A deep reinforcement learning (DRL) model is used as the local planner, as detailed below: State space: The inputs are the current sensor data, the drone's attitude, and its relative position to the target.

[0087] Action space: The output is the flight speed vector or heading angle adjustment.

[0088] Reward function: set as "positive points for avoiding obstacles, positive points for approaching the target, and negative points for colliding with it".

[0089] Predictive capability: The DRL model is trained extensively and can identify potential risks (such as the trend of moving objects) and output path deviation instructions a few seconds in advance, rather than reacting only when it is about to collide.

[0090] By using deep reinforcement learning to optimize path planning, drones can adjust their flight trajectory based on real-time changes in obstacles, flight attitude, and speed, thereby improving flight flexibility, safety, and efficiency.

[0091] Deep learning-based path prediction and correction: Deep reinforcement learning models not only make real-time decisions during flight but also predict the future state of the flight trajectory. By training deep learning models, obstacles that may appear in the next few seconds can be predicted, and the flight path can be adjusted in advance.

[0092] For example, deep reinforcement learning models can learn the flight strategies of drones in complex environments, identify the types of obstacles that may be encountered during flight (such as buildings, pedestrians, etc.), and adjust the flight altitude, direction, and speed to avoid these obstacles.

[0093] For example, suppose a drone is flying in an urban environment, aiming to travel from one location to another. When using the A* algorithm for global path planning, the system determines a basic path for the drone. However, during flight, the planned path can be affected by dynamic obstacles (such as moving cars, pedestrians, etc.).

[0094] At this point, the deep reinforcement learning model will adjust its flight trajectory and find the optimal local obstacle avoidance path based on real-time sensor data (e.g., a camera detects a car moving ahead, or lidar data detects a protruding part of a building).

[0095] When a sudden obstacle is detected, the local obstacle avoidance algorithm is immediately triggered, as follows: Dynamic window method: Sample multiple velocity pairs in velocity and angular velocity space to simulate the trajectory of the drone over a short period of time. Trajectories that might collide with obstacles are eliminated, and the trajectory that is closest to the target direction and has the fastest speed is selected from the remaining trajectories. Specifically, the dynamic window method is an obstacle avoidance method based on control theory, used to dynamically calculate the optimal flight speed, direction, and acceleration during flight.

[0096] The dynamic window method calculates a series of possible trajectories at each moment based on the drone's current speed, direction, and the position of obstacles, and selects the optimal trajectory to execute. The dynamic window method can quickly respond to changes in obstacles in dynamic environments, thus efficiently avoiding obstacles during flight.

[0097] The advantage of the dynamic window method is that it can perform local obstacle avoidance in real time without relying on global path planning, and can also dynamically correct the flight trajectory.

[0098] Artificial potential field method: The artificial potential field method sets a repulsive field for each obstacle in the environment and a gravitational field for the target point. During flight, the drone is subjected to the gravitational force of the target point and the repulsive force of the obstacles, forming a dynamic mechanical equilibrium. This method simulates mechanical phenomena to provide the drone with natural obstacle avoidance behavior.

[0099] In the artificial potential field method, each obstacle will generate a reverse "repulsive force" based on its distance from the drone. The drone will autonomously avoid obstacles by adjusting the "force field" and fly towards the target point.

[0100] Real-time path correction and optimization: The flight path will be continuously updated based on the 3D environment model and obstacle detection results generated in step S30.

[0101] When a new obstacle is detected, the drone will adjust its current flight trajectory using dynamic windowing or artificial potential field methods to avoid a collision.

[0102] In this process, not only will the path be dynamically corrected, but flight efficiency (shortest time, lowest energy consumption, etc.) will also be considered to ensure that the drone can maintain flight efficiency while avoiding obstacles.

[0103] For example, suppose a drone is flying in a complex indoor environment. During path planning, it will encounter multiple static and dynamic obstacles (such as walls, furniture, and moving people). When the drone flies into a room, the optimal trajectory is calculated in real time using the dynamic window method to avoid walls and furniture, and the flight speed is adjusted according to the target position.

[0104] During this process, if a new obstacle (such as a person entering a room) appears in the drone's flight path, the dynamic window method will dynamically adjust the flight path to avoid collision.

[0105] Meanwhile, the artificial potential field method can further optimize the path: the drone is subjected to the repulsive force of obstacles, automatically adjusts its direction to avoid unsafe areas, and flies towards the target point.

[0106] S50. Based on the corrected dynamic flight path, the flight control command is calculated by combining the PID controller with the linear quadratic regulator algorithm, and the UAV is controlled to perform path tracking and obstacle avoidance. Based on real-time flight feedback, the flight control strategy is optimized online through self-supervised learning. In this embodiment, specifically, the step of solving the flight control commands using a PID controller combined with a linear quadratic regulator algorithm includes: The first control variable is calculated based on the current attitude error using a PID controller to eliminate short-term flight deviations; The LQR algorithm is used to construct a cost function, and the second control variable is calculated based on the position, velocity and acceleration state of the UAV to optimize global flight energy efficiency and eliminate long-term accumulated errors. The first control quantity and the second control quantity are combined to generate the final motor speed control command, which adjusts the flight attitude of the UAV.

[0107] The online optimization of flight control strategies based on real-time flight feedback and self-supervised learning specifically includes: Image data, lidar data, and flight status data are collected in real time during flight as unlabeled training samples; The training samples are analyzed using a self-supervised learning model, and the parameters of the control algorithm are updated online to adapt to environmental conditions such as changes in wind speed or light intensity. Accumulate flight experience through reinforcement learning mechanisms and automatically adjust flight strategies; Specifically, self-supervised learning and online optimization: During continuous flight, the system collects sensor data (such as images, LiDAR data, flight status, etc.) in real time and uses this data to adjust the control algorithm. In particular, the self-supervised learning model can be continuously updated during flight to adapt to changes in different flight environments.

[0108] Environmental Adaptability: The adaptive mechanism allows the control system to adjust according to different flight environments (such as weather, flight altitude, obstacle density, etc.). For example, in environments with high wind speeds, the system will automatically enhance the control response to counteract the effects of wind on flight; while in low-light environments, the system will optimize image enhancement and flight stability control to ensure smooth flight.

[0109] Real-time recording of flight logs, including raw sensor data, control commands, actual response trajectory, and environmental interference (such as wind resistance).

[0110] Reinforcement learning and adaptive control: Reinforcement learning can be combined with online optimization to gradually improve control strategies during flight. Through trial and error, drones can learn how to adjust their flight behavior in complex environments, allowing control strategies to be continuously optimized as flight experience accumulates.

[0111] The deviation between the "actual flight trajectory" and the "planned ideal trajectory" is used as the loss function. The system runs a self-supervised learning algorithm in the background, using unlabeled flight data to fine-tune the control parameters (such as the Kp / Ki / Kd of PID or the Q / R matrix weights of LQR).

[0112] Strategy adaptation: For example, when the system detects a consistency deviation caused by long-term crosswinds, the adaptive algorithm will automatically increase the integral gain of crosswind resistance, making the drone more stable when it encounters the same wind force again.

[0113] For example, a drone encounters a strong crosswind while performing a hovering mission.

[0114] PID Response: The inner loop PID instantly detects the tilt and immediately increases the speed of the motor on the windward side to resist the wind force and keep the fuselage level within a few hundred milliseconds.

[0115] LQR Response: The outer-ring LQR detected that although the drone's attitude was level, its position was deflected by the wind by 0.5 meters. Therefore, it calculated a smooth return trajectory and directed the drone to slowly fly back to its original position, rather than violently rebounding and causing oscillation.

[0116] The core of adaptive control is that the UAV can adjust its flight control algorithm based on real-time data, and can respond flexibly to changes in climate, sudden environmental changes, or changes in flight path.

[0117] For example, suppose a drone performs the same task under different weather conditions. In windy environments, the control system automatically optimizes flight stability strategies based on feedback from self-supervised learning, enhancing the response speed of the PID controller and reducing the impact of wind on flight. In low-light environments, the system increases the intensity of image enhancement to compensate for insufficient lighting and adjusts flight attitude control, resulting in more stable flight.

[0118] Example 2: like Figure 2 As shown, to solve the above-mentioned technical problems, based on Embodiment 1, another technical solution adopted in this application is: A method for image-assisted path tracking and obstacle avoidance using unmanned aerial vehicles (UAVs), the method comprising the following steps: Step 1: In response to the flight mission command of the UAV, the multimodal perception system deployed on the UAV is invoked to collect real-time environmental image data, and the real-time environmental image data is synchronized and initially fused with multi-sensor data; Step 2: Call the preset image enhancement model to perform denoising and illumination compensation on the fused image data to generate an enhanced environment image, and perform weighted fusion of the visible light image and infrared image based on deep learning methods; Step 3: Input the fused image data into the preset obstacle detection model for real-time obstacle recognition, and combine the global path planning algorithm and the local obstacle avoidance algorithm to generate the target flight path; Step 4: In response to the generated target flight path, the UAV is controlled by the flight controller to perform path tracking, and adaptive optimization and adjustment are performed using deep learning algorithms based on feedback data during the flight process.

[0119] In this embodiment, specifically, the step of calling the multimodal perception system deployed on the UAV to collect real-time environmental image data, and performing multi-sensor data synchronization and preliminary fusion of the real-time environmental image data, includes: The system uses an RGB camera to capture visible light images, an infrared camera to capture infrared images, and a lidar and ultrasonic sensor to collect obstacle distance information. A sensor fusion algorithm is used to perform time synchronization and preliminary fusion of the collected multi-source data, wherein the sensor fusion algorithm includes a Kalman filter; The weighted fusion of visible light images and infrared images based on deep learning methods includes: using a self-attention mechanism to perform feature-weighted fusion of image data acquired by the RGB camera and image data acquired by the infrared camera.

[0120] In this embodiment, specifically, the step of calling a preset image enhancement model to perform denoising and illumination compensation processing on the fused image data to generate an enhanced environment image includes: The convolutional neural network is invoked to remove noise from the images captured by the RGB camera, and the brightness and contrast of the images are adjusted according to the ambient lighting conditions. When the current environment is detected to be low light or completely dark, a generative adversarial network is invoked to enhance the low-resolution or high-noise infrared image captured by the infrared camera, and a high-definition enhanced infrared image is generated through adversarial training.

[0121] In this embodiment, specifically, the step of inputting the fused image data into a preset obstacle detection model for real-time obstacle recognition, and generating a target flight path by combining a global path planning algorithm and a local obstacle avoidance algorithm, includes: Real-time detection of static and dynamic obstacles in the environment is achieved using deep convolutional neural networks or YOLO models. Use the A* algorithm or the fast random tree algorithm to compute the globally optimal path from the starting point to the ending point; When an obstacle is detected in the local environment, a dynamic window method is used to perform real-time obstacle avoidance planning without changing the global optimal path. By using deep reinforcement learning algorithms, the obstacle avoidance strategy is dynamically adjusted according to the real-time environment, and the flight path is automatically corrected when dynamic obstacles are detected.

[0122] In this embodiment, specifically, in response to the generated target flight path, the flight controller controls the UAV to perform path tracking, and adaptive optimization is performed using deep learning algorithms based on feedback data during flight, including: The UAV is used for path tracking control using a linear quadratic control controller or a PID controller, and its flight altitude, speed and direction are dynamically adjusted in combination with the enhanced environmental image and the obstacle distance information. Acquire feedback data during flight, including image quality, obstacle detection success rate, and flight stability; The feedback data is analyzed using reinforcement learning algorithms, and the image enhancement and obstacle avoidance strategies are adaptively adjusted to adapt to different lighting or weather conditions.

[0123] In this embodiment, specifically, a generative adversarial network is used to enhance infrared images in low-light environments, thereby improving image quality and solving the problem of insufficient image recognition capabilities during low-light and nighttime flight.

[0124] Multimodal sensor fusion technology: By fusing data from multimodal sensors (including RGB, infrared and lidar sensors), the perception accuracy and stability in complex environments are improved, especially in the detection and obstacle avoidance capabilities of dynamic obstacles.

[0125] Combining deep learning and reinforcement learning: Deep learning optimizes image processing and obstacle recognition, while reinforcement learning is used to adaptively adjust the flight strategy, solving the problem of maintaining path tracking and obstacle avoidance accuracy in complex environments.

[0126] A multimodal collaborative perception system is constructed by integrating a high-definition RGB camera, an infrared imager, a lidar, and an ultrasonic sensor onto a drone carrier.

[0127] RGB cameras are used to collect image data in the visible light band, making them suitable for daytime or well-lit work environments.

[0128] Infrared imagers are used for image acquisition in low-light environments, improving environmental perception capabilities in nighttime, foggy, or low-light scenarios.

[0129] LiDAR and ultrasonic sensors work together to provide high-precision distance measurement data for obstacles, enhancing spatial perception accuracy in complex environments.

[0130] A time synchronization mechanism is used to achieve synchronous data acquisition from a multi-sensor system, ensuring the temporal synchronization and spatial consistency of multi-source data.

[0131] Sensor fusion algorithms (such as Kalman filtering) are used to perform preliminary fusion processing on multi-source raw data to compensate for the measurement bias of a single sensor and improve data reliability and the accuracy of environmental description.

[0132] A convolutional neural network model is used to suppress noise in RGB images, and the image brightness and contrast parameters are dynamically adjusted according to the ambient light intensity to compensate for image distortion in low-light environments and optimize image quality.

[0133] In low-light or no-light environments, generative adversarial networks are used to perform infrared image enhancement operations, converting low-resolution, high-noise infrared images into high-definition, high-signal-to-noise ratio image data.

[0134] By optimizing the output of the generator network through adversarial training, the image detail recognition is improved, and the visual perception capability of drones in nighttime operations is enhanced.

[0135] Based on the self-attention mechanism of deep learning, a weighted fusion operation is performed on RGB images and infrared images.

[0136] By integrating effective feature information from dual-source images, UAVs can collaboratively utilize visible light and infrared spectral information in complex environments, thereby improving the accuracy and robustness of obstacle detection.

[0137] Real-time obstacle detection is achieved by using deep convolutional neural networks or YOLO target detection models.

[0138] The model has been trained and optimized with samples from multiple scenarios, and can efficiently and accurately identify various obstacles and targets in complex scenarios such as uneven lighting, target occlusion, and dynamic interference.

[0139] By integrating lidar distance data and image target detection results, a real-time dynamic obstacle detection and obstacle avoidance decision-making framework is constructed.

[0140] By employing deep reinforcement learning algorithms, the UAV can adaptively adjust its obstacle avoidance strategy according to dynamic changes in the environment; when it detects dynamic obstacles such as pedestrians and vehicles, it automatically corrects its flight trajectory to achieve safe avoidance.

[0141] The A* algorithm or the fast random tree algorithm is used to calculate the globally optimal flight path from the starting point to the destination, ensuring the feasibility and efficiency of the path.

[0142] When sudden obstacles appear in the local working environment, the dynamic window method is used to perform real-time obstacle avoidance adjustments, so as to achieve safe avoidance of nearby obstacles without changing the global path planning goal.

[0143] A path tracking control system is constructed using a linear quadratic controller or a proportional-integral-derivative controller to ensure that the UAV flies accurately along the preset path.

[0144] By dynamically correcting flight deviations through real-time error feedback and parameter adjustment, the accuracy of path tracking is improved.

[0145] By fusing enhanced environmental feature data with lidar spatial measurement data, the flight altitude, cruising speed, and heading angle of the UAV are dynamically adjusted.

[0146] Maintaining stable flight attitude during obstacle avoidance adjustments and path tracking ensures the safety and smoothness of the operation.

[0147] Based on real-time feedback data during flight (including image quality assessment results, obstacle detection success rate, flight attitude stability parameters, etc.), deep learning algorithms (such as reinforcement learning algorithms) are used to adaptively adjust system parameters and optimize strategies.

[0148] It enables automatic iterative optimization of image enhancement parameters and obstacle avoidance decision logic, thereby improving the system's adaptability to environmental changes.

[0149] The system can automatically adjust operating parameters according to diverse operating environments (including day-night cycles, complex weather conditions, sudden changes in lighting, etc.), optimize image enhancement algorithms and obstacle avoidance strategies, and enhance the system's environmental robustness and long-term operational stability.

[0150] Example 3: like Figure 3 As shown, to solve the above-mentioned technical problems, based on Embodiment 1, another technical solution adopted in this application is: an image-assisted path tracking and obstacle avoidance system for unmanned aerial vehicles, comprising: The multimodal perception module is configured to drive RGB cameras, infrared cameras, LiDAR and millimeter-wave radar deployed on the drone to collect environmental data and perform data synchronization and fusion based on self-supervised learning; The image enhancement processing module is configured to call a generative adversarial network model to perform illumination compensation and denoising on low-light images, and generate target enhanced images. The environment modeling and detection module is configured to use the YOLOv4 model to detect and classify obstacles, and to use a deep convolutional generative adversarial network to build a real-time updated 3D environment model based on the detection results. The path planning module is configured to combine the A* algorithm with deep reinforcement learning to plan dynamic flight paths and perform local obstacle avoidance using the dynamic window method or the artificial potential field method. The flight control module is configured to output flight control commands using a combination of a PID controller and an LQR algorithm, and to optimize the control strategy online based on self-supervised learning to drive the UAV to perform path tracking and obstacle avoidance.

[0151] For other details regarding the implementation techniques of each module in the above embodiments, please refer to the description in the above embodiments of the UAV image-assisted path tracking and obstacle avoidance method, which will not be repeated here.

[0152] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system-type embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0153] Example 4: like Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. It illustrates a structural schematic diagram suitable for implementing the electronic device in the embodiment of the present disclosure. Figure 4 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0154] likeFigure 4 As shown, the electronic device includes a processor, a memory, and a communication interface. The memory stores a computer program, and when the processor executes the computer program, it implements the image-assisted path tracking and obstacle avoidance methods for unmanned aerial vehicles (UAVs) according to the embodiments of this disclosure. The electronic device can exchange data with other devices or systems through the communication interface to achieve real-time updates and sharing of drug information.

[0155] The processor in the aforementioned electronic device serves as its core, responsible for executing the computer program stored in the memory to implement various functions of the paperless conference terminal's intelligent interaction method. The processor can employ a high-performance multi-core CPU or a dedicated chip to meet the demands of complex calculations and real-time processing. The memory stores the operating system, applications, data, and computer programs. In this embodiment, the memory stores the computer program implementing the paperless conference terminal's intelligent interaction method. The memory can be RAM, ROM, Flash memory, or other types of non-volatile memory. The communication interface connects the electronic device to other devices or networks, enabling data transmission and exchange. In this embodiment, the communication interface supports multiple communication protocols and interface standards, such as Wi-Fi, Bluetooth, USB, and Ethernet, to meet communication needs in different scenarios.

[0156] For a detailed description of this embodiment, please refer to the corresponding descriptions in the foregoing embodiments, which will not be repeated here.

[0157] Example 5: According to embodiments of the present disclosure, a computer-readable storage medium stores a computer program, which, when executed by a processor, implements the functions of the aforementioned UAV image-assisted path tracking and obstacle avoidance methods of the various embodiments of the present disclosure.

[0158] The aforementioned computer-readable storage media include, but are not limited to: optical storage media (e.g., CD-ROM and DVD), magneto-optical storage media (e.g., MO), magnetic storage media (e.g., magnetic tape or portable hard drive), media with built-in rewritable non-volatile memory (e.g., memory card), and media with built-in ROM (e.g., ROM cartridge).

[0159] For a detailed description of this embodiment, please refer to the corresponding descriptions in the foregoing embodiments, which will not be repeated here.

[0160] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for image-assisted path tracking and obstacle avoidance using unmanned aerial vehicles (UAVs), characterized in that, Includes the following steps: In response to the flight mission command of the UAV, the multimodal sensor group deployed on the UAV is invoked to collect environmental data, and the environmental data is fused across modes using an adaptive time synchronization algorithm; A preset generative adversarial network model is invoked to perform image generation and illumination compensation on the low-light image data in the fused environmental data, generating a target-enhanced image for the visible light environment. The target enhancement image and LiDAR point cloud data are input into a deep learning detection model to perform real-time obstacle detection and classification, and a three-dimensional environment model is constructed based on the detection results by calling a deep convolutional generative adversarial network. Based on the three-dimensional environment model, a dynamic flight path is generated using the A* algorithm combined with deep reinforcement learning, and a local obstacle avoidance algorithm is invoked to correct the dynamic flight path when an obstacle is detected. Based on the corrected dynamic flight path, the flight control commands are calculated by a PID controller combined with a linear quadratic regulator algorithm to control the UAV to perform path tracking and obstacle avoidance. Based on real-time flight feedback, the flight control strategy is optimized online through self-supervised learning.

2. The UAV image-assisted path tracking and obstacle avoidance method according to claim 1, characterized in that, The process of invoking a multimodal sensor array deployed on the UAV to collect environmental data and then using an adaptive time synchronization algorithm to perform cross-modal data fusion on the environmental data specifically includes: The system acquires multi-source heterogeneous data collected by the multimodal sensor group, which includes an RGB camera, an infrared camera, a lidar, and a millimeter-wave radar. Features are extracted from unlabeled data using a self-supervised learning algorithm, and the multi-source heterogeneous data is preprocessed for time synchronization and spatial calibration. A deep cross-modal adversarial network is used to map calibrated RGB images, infrared images and lidar point cloud data to a unified feature space for fusion, so as to complement and enhance environmental features.

3. The UAV image-assisted path tracking and obstacle avoidance method according to claim 1, characterized in that, The step of calling a preset generative adversarial network model to perform image generation and illumination compensation on the low-light image data in the fused environmental data specifically includes: Construct a generative adversarial network that includes a generator and a discriminator; The generator inputs low-light environment images or haze images into the generator and simulates real light distribution through adversarial training to generate a visual effect image of clear visible light image. The generated visual effect image is used to restore the environmental texture and obstacle outline details, which are then used as input data for subsequent obstacle detection.

4. The UAV image-assisted path tracking and obstacle avoidance method according to claim 1, characterized in that, The step of inputting the enhanced target image and LiDAR point cloud data into a deep learning detection model for real-time obstacle detection and classification specifically includes: Low-level features of the target enhancement image and LiDAR data are extracted using a convolutional neural network. The extracted features are input into the YOLOv4 object detection model to identify and classify dynamic and static obstacles within the field of view; By combining the Kalman filter, the motion trajectory of the identified dynamic obstacles is tracked, and their spatial position at the next moment is predicted.

5. The UAV image-assisted path tracking and obstacle avoidance method according to claim 1, characterized in that, The process of constructing a 3D environment model by calling a deep convolutional generative adversarial network based on the detection results specifically includes: The fused image surface feature data and LiDAR depth data are input into a deep convolutional generative adversarial network; The deep convolutional generative adversarial network is used to reconstruct the three-dimensional spatial structure of the flight environment and generate a virtual obstacle model that includes the location, shape and spatial relationship of obstacles. The virtual obstacle model is updated in real time in response to environmental changes to reflect the environmental state in dynamic scenes.

6. The UAV image-assisted path tracking and obstacle avoidance method according to claim 5, characterized in that, The generation of dynamic flight paths based on the aforementioned 3D environment model, using the A* algorithm combined with deep reinforcement learning, specifically includes: Based on the aforementioned three-dimensional environment model, the A* algorithm is used to plan the globally optimal path from the current position to the target position; Real-time environmental data is input into a deep reinforcement learning model, and the global optimal path is dynamically adjusted based on a trial-and-error learning mechanism to output a local path decision that conforms to the current flight state. The deep reinforcement learning model is configured to predict obstacles that may appear within a preset time period and output path adjustment parameters in advance. The generative adversarial network is used to generate virtual obstacle images and synthesize them into the three-dimensional environment model; The obstacle avoidance strategy of the UAV is trained based on the synthesized virtual obstacle image in a simulation environment, and the trained strategy parameters are transferred to the actual flight control.

7. The UAV image-assisted path tracking and obstacle avoidance method according to claim 1, characterized in that, The step of invoking a local obstacle avoidance algorithm to correct the dynamic flight path when an obstacle is detected specifically includes: In response to the detection of obstacles on the path, a series of candidate trajectories based on the current velocity and acceleration are calculated using the dynamic window method, and the optimal trajectory is selected for execution; Alternatively, an artificial potential field method can be used, where the target point is set as a gravitational field and the obstacles are set as repulsive fields. The flight trajectory can be corrected in real time according to the direction of the resultant force on the UAV to avoid multiple target obstacles.

8. The UAV image-assisted path tracking and obstacle avoidance method according to claim 1, characterized in that, The process of solving flight control commands using a PID controller combined with a linear quadratic regulator algorithm specifically includes: The first control variable is calculated based on the current attitude error using a PID controller to eliminate short-term flight deviations; The LQR algorithm is used to construct a cost function, and the second control variable is calculated based on the position, velocity and acceleration state of the UAV to optimize global flight energy efficiency and eliminate long-term accumulated errors. The first control quantity and the second control quantity are combined to generate the final motor speed control command, which adjusts the flight attitude of the UAV.

9. The UAV image-assisted path tracking and obstacle avoidance method according to claim 1, characterized in that, The online optimization of flight control strategies based on real-time flight feedback and self-supervised learning specifically includes: Image data, lidar data, and flight status data are collected in real time during flight as unlabeled training samples; The training samples are analyzed using a self-supervised learning model, and the parameters of the control algorithm are updated online to adapt to environmental conditions such as changes in wind speed or light intensity. By accumulating flight experience through reinforcement learning mechanisms, flight strategies can be automatically adjusted.

10. An image-assisted path tracking and obstacle avoidance system for unmanned aerial vehicles (UAVs), applied to the image-assisted path tracking and obstacle avoidance method for UAVs as described in any one of claims 1-9, characterized in that, include: The multimodal perception module is configured to drive RGB cameras, infrared cameras, LiDAR and millimeter-wave radar deployed on the drone to collect environmental data and perform data synchronization and fusion based on self-supervised learning; The image enhancement processing module is configured to call a generative adversarial network model to perform illumination compensation and denoising on low-light images, and generate target enhanced images. The environment modeling and detection module is configured to use the YOLOv4 model to detect and classify obstacles, and to use a deep convolutional generative adversarial network to build a real-time updated 3D environment model based on the detection results. The path planning module is configured to combine the A* algorithm with deep reinforcement learning to plan dynamic flight paths and perform local obstacle avoidance using the dynamic window method or the artificial potential field method. The flight control module is configured to output flight control commands using a combination of a PID controller and an LQR algorithm, and to optimize the control strategy online based on self-supervised learning to drive the UAV to perform path tracking and obstacle avoidance.

Images