Unmanned aerial vehicle high-speed obstacle avoidance system and method based on optoelectronic end-to-end network

Abstract

An unmanned aerial vehicle high-speed obstacle avoidance system based on an optoelectronic end-to-end network. The system comprises: an image acquisition component (100), used to acquire depth images of a current environment of an unmanned aerial vehicle; and a decoder (200) and a decision generator (300) obtained on the basis of end-to-end network training, wherein the decoder (200) is used to decode the depth images to obtain depth information of the current environment, and the decision generator (300) is used to plan an obstacle avoidance path for the unmanned aerial vehicle on the basis of the depth information. Hence, the present invention solves the problem in the prior art in which unmanned aerial vehicles cannot perform high-speed obstacle avoidance during flight while ensuring accurate autonomous navigation due to delays resulting from serial processing of perception, mapping, and planning information. Further disclosed is an unmanned aerial vehicle high-speed obstacle avoidance method based on an optoelectronic end-to-end network,

Description

UAV high-speed obstacle avoidance system and method based on optoelectronic end-to-end network

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is based on the Chinese patent application with application number 202410129508.5 and application date of January 30, 2024, and claims the priority of the Chinese patent application. The entire content of the Chinese patent application is hereby introduced into this application as a reference. Technical Field

[0003] The present application relates to the field of UAV technology, and in particular to a UAV high-speed obstacle avoidance system and method based on an optoelectronic end-to-end network. Background Art

[0004] With the development and expansion of drone technology, autonomous drone patrols are becoming increasingly important in the security field. Autonomous flight enables drones to make independent decisions and plan routes, enabling them to patrol within a pre-set area, enabling all-weather, all-day monitoring and patrol missions. Furthermore, drone-mounted sensors can collect and analyze information in real time, rapidly transmitting it to ground operators or relevant departments, providing a basis for timely decision-making.

[0005] Autonomous drone flight decisions involve perception, mapping, and planning. In the perception phase, the most important information is often the depth of the surrounding environment, which is used to plan the path. However, most related technologies use traditional modeling and planning methods to obtain control parameters. This results in delays due to the serial transmission of perception, mapping, and planning information, making it impossible for drones to achieve high-speed obstacle avoidance while ensuring accurate autonomous navigation.

[0006] Summary of the Invention

[0007] The present application provides a high-speed obstacle avoidance system and method for unmanned aerial vehicles (UAVs) based on an optoelectronic end-to-end network to solve the problem in related technologies of delays caused by the serialization of perception, mapping, and planning information, which makes it impossible for UAVs to fly at high speeds and avoid obstacles while ensuring accurate autonomous navigation.

[0008] The first aspect of the present application provides a high-speed obstacle avoidance system for a UAV based on an optoelectronic end-to-end network, comprising: an image acquisition component for acquiring a depth image of the UAV's current environment; a decoder and a decision generator obtained based on end-to-end network training, wherein the decoder is used to decode the depth image to obtain depth information of the current environment; and the decision generator is used to plan the obstacle avoidance path of the UAV based on the depth information.

[0009] Optionally, the image acquisition component includes: an optical lens; an optical mask arranged on the optical lens; and a photosensitive film for collecting light passing through the optical mask and generating the depth image based on the light.

[0010] Optionally, the optical mask, the lens and the photosensitive film are all on the same optical axis.

[0011] Optionally, the design of the optical mask includes: obtaining the surface shape of the optical lens; calculating the optical performance parameters of the optical lens based on the surface shape; obtaining an optical network that can capture the depth image based on the depth image training of the target environment, and obtaining the optical mask based on the optical performance parameters and the optical network design.

[0012] Optionally, the image acquisition component and the decoder are jointly trained.

[0013] Optionally, the joint training process includes: acquiring an optical image and a first depth image of the target environment; generating a first training data set based on the optical image and the first depth image; and performing end-to-end training on the decoder using the first training data set, wherein the decoder calculates a second depth image of the target environment based on the optical image, and updates the network parameters of the optical network in the image acquisition component and the network parameters of the decoder based on the error between the first depth image and the second depth image.

[0014] Optionally, the training process of the decision generator includes: obtaining a second training data set by simulating a privileged expert model; identifying a reference planned path, depth information and flight parameters of the UAV in the second training data, and training the decision generator using the second training data set, wherein the decision generator generates a predicted planned path based on the depth information and the flight parameters of the UAV, and updates the network parameters of the decision generator based on the error between the reference planned path and the predicted planned path.

[0015] Optionally, the use of the privileged expert model to simulate and obtain the second training data set includes: simulating the depth information, flight parameters and global information of the UAV in the simulated environment; inputting the global information into the privileged expert model, and the privileged expert model outputting a reference planning path of the UAV in the privileged expert model; and generating the second training data set based on the depth information, the flight parameters and the reference planning path.

[0016] Optionally, the decoder and the decision generator are trained as a whole, wherein network parameters of the optical network of the decoder are frozen during the overall training.

[0017] The second aspect of the present application provides a method for high-speed obstacle avoidance of a UAV based on an optoelectronic end-to-end network. The method uses a high-speed obstacle avoidance system for a UAV based on an optoelectronic end-to-end network as described in any one of the above embodiments to perform high-speed obstacle avoidance, wherein the method includes the following steps: using an image acquisition component to capture a depth image of the current environment of the UAV; using a decoder to decode the depth image to obtain depth information of the current environment, and obtain the flight parameters of the UAV; and planning the obstacle avoidance path of the UAV based on the depth information and the flight parameters.

[0018] Therefore, this application has at least the following beneficial effects:

[0019] In the embodiment of the present application, an image acquisition component is used to collect a depth image of the current environment of the drone, and a decoder and a decision generator are obtained based on end-to-end network training. The decoder is used to decode the depth image to obtain depth information of the current environment, and then the decision generator plans the obstacle avoidance path of the drone based on the depth information. Thus, through the optical depth estimation front end combined with the small model decision maker at the back end, depth information can be quickly extracted from the physical environment, thereby improving the decision update rate of the drone, allowing the drone to fly at high speed with obstacle avoidance while ensuring accurate autonomous navigation. In addition, the end-to-end optoelectronic neural network design improves the overall robustness and has a wider application.

[0020] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0022] FIG1 is a block diagram of a high-speed obstacle avoidance system for a UAV based on an optoelectronic end-to-end network according to an embodiment of the present application;

[0023] FIG2 is a schematic diagram of a decision generator provided according to an embodiment of the present application;

[0024] FIG3 is a flow chart of a method for high-speed obstacle avoidance of a UAV based on an optoelectronic end-to-end network according to an embodiment of the present application;

[0025] FIG4 is a flow chart of end-to-end high-speed obstacle avoidance according to an embodiment of the present application. DETAILED DESCRIPTION

[0026] The following describes in detail embodiments of the present application, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0027] With the rise of artificial intelligence, deep neural networks have penetrated various fields, including optics, computer vision, and robotic control. In the optical field, traditional cameras are mostly used for monocular image depth estimation, which uses image depth features such as shadows and perspective relationships for calculation. However, today's optical imaging systems can optimize the optical system through the stochastic gradient descent algorithm of neural networks, directly using light information as a calculation carrier to quickly obtain depth maps. This is significantly different from traditional low-frame-rate lidar and depth cameras.

[0028] In the field of robotic control, small neural networks used in edge computing are key to the implementation of artificial intelligence technology. For example, YOLO (You Only Look Once, a deep learning-based target detection algorithm) is used for target monitoring tasks, and Mobile-Net (based on depthwise separable convolution) is used for classification tasks. As an important means of completing real-time tasks in edge computing, neural networks have the most important feature: they can be designed in an "end-to-end" manner, enabling rapid task results.

[0029] Generally speaking, autonomous flight decisions for drones include perception, mapping, and planning. In the perception stage, the most important information is often the depth information of the current environment. Most related technologies use traditional modeling and planning methods to obtain control parameters. However, due to the serialization of perception, mapping, and planning information, delays will occur, making it impossible for drones to fly at high speeds and avoid obstacles while ensuring accurate autonomous navigation.

[0030] In order to solve the problem of high-speed obstacle avoidance flight of drones, the embodiment of the present application proposes a drone high-speed obstacle avoidance system and method based on an optoelectronic end-to-end network. The following will first explain the drone high-speed obstacle avoidance system based on an optoelectronic end-to-end network.

[0031] Specifically, Figure 1 is a block diagram of a UAV high-speed obstacle avoidance system based on an optoelectronic end-to-end network provided in an embodiment of the present application.

[0032] As shown in FIG1 , the UAV high-speed obstacle avoidance system 10 based on an optoelectronic end-to-end network includes: an image acquisition component 100 , a decoder 200 and a decision generator 300 .

[0033] Among them, the image acquisition component 100 is used to collect the depth image of the current environment of the drone; the decoder 200 and the decision generator 300 are obtained based on end-to-end network training, wherein the decoder 200 is used to decode the depth image to obtain the depth information of the current environment; the decision generator 300 is used to plan the obstacle avoidance path of the drone based on the depth information.

[0034] It can be understood that the embodiment of the present application collects the depth image of the current environment of the drone through the image acquisition component, obtains the decoder and decision generator based on end-to-end network training, uses the decoder to decode the depth image to obtain the depth information of the current environment, and then the decision generator plans the obstacle avoidance path of the drone based on the depth information. Therefore, through the optical depth estimation front end combined with the small model decision maker at the back end, it is possible to quickly extract depth information from the physical environment, thereby improving the decision update rate of the drone, allowing the drone to fly at high speed with obstacle avoidance while ensuring accurate autonomous navigation, and the end-to-end optoelectronic neural network design improves the overall robustness and has a wider application.

[0035] In the embodiment of the present application, the image acquisition component 100 includes: an optical lens, an optical mask and a photosensitive film.

[0036] Among them, the optical mask is set on the optical lens; the photosensitive film is used to collect light passing through the optical mask and generate a depth image based on the light.

[0037] Among them, the optical lens, optical mask and photosensitive film are on the same optical axis.

[0038] It can be understood that the optical mask in the embodiment of the present application is close to the standard lens, and a photosensitive film is set at a suitable image distance. The depth image of the current environment of the drone is collected through the optical lens, and the light passing through the optical mask is imaged on the photosensitive film through the lens, so that the depth information of the current environment can be obtained by subsequently decoding the depth image through a decoder.

[0039] It should be noted that the optical lens of the present application can be a free-form surface lens obtained by Zernike polynomial fitting. The phase information of each light in the image is obtained by Zernike polynomial fitting, and its coefficients are optimized to accurately estimate the scene depth. The photosensitive film can adopt low-resolution, high-frequency elements to facilitate the establishment of a high-speed depth estimation system.

[0040] Specifically, Zernike polynomials are a commonly used method to describe the surface morphology of lenses, which can be used to describe the aberration characteristics and wavefront distortion of lenses. The lens thus fitted has symmetry and non-periodicity, which can reflect the symmetric characteristics and local deformation of the lens. The theoretical basis of the method of using Zernike polynomials to fit free-form surface lenses is that Zernike polynomials are orthogonal basis functions, which can be used to describe the morphological errors of optical surfaces of arbitrary shapes. Since Zernike polynomials are complete and orthogonal, they can represent optical surfaces of any shape without overlap or repetition, and a smaller number of terms can be used to describe complex surface shapes, such as fitting the morphology of free-form surface lenses. It is widely used in computer simulation and optical design. For example, in computer-generated holograms or the optimization of optical elements, Zernike polynomials can be used to represent surface shapes and to calculate optical performance parameters such as aberrations and wavefront distortion.

[0041] In one embodiment of the present application, the optical imaging principle is: convolving a point light source in the physical world with a point spread function of an optical system, wherein the calculation formula of the point spread function is as follows:

[0042] in and represents Fourier transform and inverse Fourier transform, A, t lens , t ff 、U in They represent the frequency domain expressions of pupil function, thickness profile of standard lens, arbitrary thickness profile and wavefront intensity of point light source, respectively. s represents the transfer function.

[0043] In an embodiment of the present application, the design of the optical mask includes: obtaining the surface shape of the optical lens; calculating the optical performance parameters of the optical lens based on the surface shape; obtaining an optical network that can capture depth images based on depth image training of the target environment, and obtaining the optical mask based on the optical performance parameters and the optical network design.

[0044] It can be understood that the embodiment of the present application calculates the optical performance parameters based on the surface shape of the optical lens, obtains an optical network that can collect depth images based on the depth image training of the target environment, and obtains an optical mask based on the optical performance parameters and the optical network design for processing high-resolution optical images.

[0045] It should be noted that the optical performance parameters may include aberration and wavefront distortion, without specific limitation.

[0046] In an embodiment of the present application, the image acquisition component 100 and the decoder 200 are jointly trained, wherein the joint training process includes: acquiring an optical image and a first depth image of the target environment; generating a first training data set based on the optical image and the first depth image; and performing end-to-end training on the decoder using the first training data set, wherein the decoder 200 calculates a second depth image of the target environment based on the optical image, and updates the network parameters of the optical network in the image acquisition component and the network parameters of the decoder based on the error between the first depth image and the second depth image.

[0047] It can be understood that the embodiment of the present application generates a first training data set based on the acquired optical image and first depth image of the target environment, and performs end-to-end training on the decoder, so as to obtain a more intuitive depth image, thereby facilitating the use of depth images for drone decision-making; and the decoder calculates the second depth image of the target environment based on the optical image, and based on the error between the first depth image and the second depth image, updates the network parameters of the optical network in the image acquisition component and the network parameters of the decoder, thereby improving the accuracy of the depth information of the current environment obtained by the decoder's decoded depth image, and optimizing the parameters of the entire obstacle avoidance system.

[0048] It should be noted that the decoder of the present application can be a U-net neural network. The U-net neural network has the advantages of a deep convolutional neural network and can process complex image information, which enables the U-net neural network to process high-resolution optical images in the optical system, thereby improving the analysis and design accuracy of the optical system; and the U-net neural network can be trained end-to-end, so that it can learn features directly from the original data and process the image, which makes the application of the U-net neural network in the optical system more flexible and efficient.

[0049] For example, by designing a U-net neural network behind the free-form surface lens, we can obtain real-world image information, image it on a photosensitive sheet through the lens, and then decode it through the U-net neural network to obtain a more intuitive depth image, making it easier to use depth images to make decisions for drones.

[0050] Specifically, the depth map is sampled at different depth intervals and then combined with the RGB image through the image acquisition component to generate the resulting image as the encoding result. This step simulates the imaging process of a real lens. The image is then passed through the decoder to obtain the depth map, completing the forward propagation of the entire system. The error is calculated by comparing it with the depth map and used as the loss function for backpropagation to update the optical network parameters and the decoder network parameters.

[0051] In one embodiment of the present application, a decoder is designed to be connected after the optical image acquisition component to obtain an intuitive depth image, and then the image acquisition component and the decoder are used to form a monocular depth estimation system for estimating the scene depth information based on the optical image, wherein the image acquisition component and the decoder with optimized parameters finally obtained through training constitute the monocular depth estimation system.

[0052] In an embodiment of the present application, as shown in Figure 2, the training process of the decision generator 300 includes: using the privileged expert model simulation to obtain a second training data set; identifying the reference planning path, depth information and flight parameters of the UAV in the second training data, and using the second training data set to train the decision generator, wherein the decision generator 300 generates a predicted planning path based on the depth information and the flight parameters of the UAV, and updates the network parameters of the decision generator 300 based on the error between the reference planning path and the predicted planning path.

[0053] It can be understood that the embodiment of the present application uses the privileged expert model simulation to obtain a second training data set, uses the second training data set to train the decision generator, generates a predicted planning path based on the depth information identified from the second training data and the flight parameters of the drone, and updates the network parameters of the decision generator based on the reference planning path identified from the second training data and the generated predicted planning path, so that the decision output by the decision generator is more accurate and the parameters of the entire obstacle avoidance system are optimized.

[0054] It should be noted that by using privileged experts in a simulation environment, different scenarios and conditions can be easily simulated to explore the performance of different models and algorithms. This also makes the dataset repeatable, enabling better tracking and comparison of the performance of different models and algorithms.

[0055] In an embodiment of the present application, a privileged expert model is used to simulate and obtain a second training data set, including: simulating the depth information, flight parameters and global information of the UAV in the simulated environment; inputting the global information into the privileged expert model, and the privileged expert model outputting the reference planning path of the UAV in the privileged expert model; and generating a second training data set based on the depth information, flight parameters and reference planning path.

[0056] It can be understood that the embodiment of the present application obtains the depth information, flight parameters and global information of the simulated drone in the simulated environment, inputs the global information into the privileged expert model to output the reference planning path of the drone in the privileged expert model, and generates a second training data set based on the depth information, flight parameters and reference planning path to facilitate subsequent training of the decision generator.

[0057] It should be noted that privileged learning often uses global information to obtain better solutions. The expert can obtain global information and perform well in drone crossing. The results are then used as a training set for training. The obstacle avoidance decision maker can only use onboard resources. Among them, this application uses Mobile-net as the backbone to train the obstacle avoidance decision maker. Mobile-Net is a lightweight convolutional neural network with fewer parameters and computational complexity. Using Mobile-Net as the backbone of a multi-branch network can make the entire network have a smaller model and faster inference speed. This is very useful for embedded devices and mobile applications.

[0058] Privileged learning can utilize previous experience to improve learning efficiency. The embodiments of this application mainly utilize the data set generated by the expert model. The training of the student strategy can benefit from the experience of the privileged expert model to solve the situation where global information is required in certain tasks, and can learn how to make correct decisions with limited information.

[0059] In an embodiment of the present application, the decoder and the decision generator are trained as a whole, wherein the network parameters of the optical network of the decoder are frozen during the overall training.

[0060] It can be understood that the embodiment of the present application trains the decoder and decision generator as a whole, optimizes the parameters of the entire obstacle avoidance system in an end-to-end deep learning manner, so that the entire network can converge faster and obtain better performance. During the overall training, the network parameters of the decoder's optical network are frozen, and there is no need to retrain the optical part to better generalize to new data sets, thereby minimizing the cost of training time and computing resources and obtaining better performance.

[0061] The parameters of the optical part of the network parameters of the optical network of the frozen decoder refer to the coefficients of each polynomial of the Zernike polynomial family, and are not specifically limited.

[0062] Specifically, after freezing the parameters of the Zernike polynomials in the free-form lens, the optical part and the obstacle avoidance decision maker after privileged learning are comprehensively trained to reduce training resources and perform overall end-to-end optimization. The two upstream and downstream task networks obtained through separate pre-training are combined to achieve stronger robustness and to reduce the impact of noise or intermediate results on the final task effect as a whole.

[0063] According to the high-speed obstacle avoidance system for drones based on an optoelectronic end-to-end network proposed in the embodiment of the present application, a depth image of the drone's current environment is collected through an image acquisition component, a decoder and a decision generator are obtained based on end-to-end network training, the decoder is used to decode the depth image to obtain depth information of the current environment, and then the decision generator plans the obstacle avoidance path of the drone based on the depth information. Thus, through the optical depth estimation front end combined with the small model decision maker at the back end, depth information can be quickly extracted from the physical environment, thereby improving the decision update rate of the drone, allowing the drone to fly at high speed with obstacle avoidance while ensuring accurate autonomous navigation, and the end-to-end optoelectronic neural network design improves the overall robustness and has a wider application.

[0064] The following is a detailed description of the structure and training process of the UAV high-speed obstacle avoidance system based on the optoelectronic end-to-end network, combined with Figure 3. The specific steps are as follows:

[0065] (1) Use Zernike polynomials to fit the lens, optimize the parameters of each polynomial, and install it on the optical platform of the UAV.

[0066] The phase information of each light in the image is obtained by Zernike polynomial fitting, and its coefficients are optimized to accurately estimate the scene depth information, which can better retain the scene result information and obtain higher planning accuracy and flight speed.

[0067] The free-form surface lens obtained by training the Zernike polynomial fitting and the high-frequency low-resolution photosensitive film are installed on the optical platform of the UAV to ensure that the spatial positions of the optical mask, standard lens and photosensitive film are p M ,p L ,p s They are all on the optical axis, and the distance along the optical axis conforms to the parameters designed during training. The optical mask is close to the standard lens, and the photosensitive film is set at the appropriate image distance.

[0068] (2) Connect a U-net neural network as a decoder and train it to obtain a monocular depth estimation system.

[0069] After the light in the physical world is imaged on the photosensitive sheet through the lens, the image contains the depth information in the physical world. After being decoded by the U-net neural network, a more intuitive depth image can be obtained, so that the depth image can be used to make decisions for the drone.

[0070] Specifically, as shown in Figure 2, the monocular depth estimation system is trained by taking the RGB image and the corresponding depth map as the data set, sampling the different depth intervals of the depth map, and calculating the imaging image with the RGB image through the lens as the encoding result; then the imaging image is decoded through U-net to obtain the depth map, thereby completing the forward propagation of the entire system, and the error is calculated by comparing with the depth map, and the optical network parameters and the U-net network parameters are updated by backpropagation as the loss function. The optimized system obtained by training the free-form surface lens and the U-net neural network together is the monocular depth estimation system.

[0071] (3) Design a privileged expert model in privileged learning to obtain a training dataset, and then use a convolutional neural network as the main network to train the student strategy, i.e., the decision generator.

[0072] The training dataset is obtained from the privileged expert model in privileged learning, and the obstacle avoidance decision maker is trained using the training dataset to make correct decisions under limited information.

[0073] Specifically, the privileged expert model obtains the full-image point cloud, the global state of the body, and the reference trajectory, and uses MH for adoption processing to plan the optimal trajectory; the training data set is obtained from the privileged expert model and combined with the depth map, the body speed and height, and the reference direction for imitation training to generate an obstacle avoidance decision maker, so that the control trajectory of the drone can be predicted based on limited information, and the correct decision can be made based on the optimal processing of the control trajectory.

[0074] (4) Freeze the optical parameters and train the decoder and obstacle avoidance decision maker as a whole to obtain the parameters of the final hardware. The overall optimization of the model parameters enables the drone to go directly from receiving light from the physical world to path planning and avoiding obstacles.

[0075] The following will be described using specific embodiments, which are as follows:

[0076] (1) First, a monocular depth estimation optical system is built. The free-form surface lens obtained by training the Zernike polynomial fitting and the high-frequency, low-resolution photosensitive film are installed on the optical platform of the UAV to ensure that the spatial positions of the optical mask, standard lens and photosensitive film are p M ,p L ,p s They are all on the optical axis, and the distance along the optical axis conforms to the parameters designed during training. The optical mask is close to the standard lens, and the photosensitive film is set at the appropriate image distance.

[0077] (2) The neural network is then deployed by fixing the trained network parameters on the firmware to achieve the purpose of fast forward reasoning. The position of the lens and the neural network firmware must be adjusted. The drone has limited payload space. The entire system is placed in front of the drone without affecting the center of gravity and structure of the aircraft, and the output of the photosensitive film is connected to the network firmware.

[0078] (3) Before the aircraft takes off, test whether the output of the system is correct, including whether the photosensitive film is updated at a high frequency, whether the image of the photosensitive film is correct, and whether the control parameters output by the neural network firmware are correct. Final debugging can only be carried out after the above tests are correct.

[0079] (4) Connect the output of the network firmware to the flight control module and conduct a test flight. During this process, ensure that the connection is stable and pay attention to flight safety during the test flight. Since this system is autonomous navigation, when testing its high-speed obstacle avoidance, the tester needs to set the aircraft's flight distance and stay away from the test site to avoid accidents.

[0080] FIG4 is a flow chart of a method for high-speed obstacle avoidance of a UAV based on an optoelectronic end-to-end network provided in an embodiment of the present application.

[0081] As shown in FIG4 , the high-speed obstacle avoidance method for a UAV based on an optoelectronic end-to-end network includes the following steps:

[0082] In step S101, an image acquisition component is used to acquire a depth image of the environment in which the drone is currently located.

[0083] In step S102, the depth image is decoded by a decoder to obtain depth information of the current environment and obtain flight parameters of the UAV.

[0084] In step S103, an obstacle avoidance path of the UAV is planned based on the depth information and flight parameters.

[0085] It should be noted that the above explanation of the embodiment of the UAV high-speed obstacle avoidance system based on the optoelectronic end-to-end network is also applicable to the UAV high-speed obstacle avoidance method based on the optoelectronic end-to-end network in this embodiment, and will not be repeated here.

[0086] According to the high-speed obstacle avoidance method for drones based on an optoelectronic end-to-end network proposed in the embodiment of the present application, an image acquisition component is used to collect a depth image of the drone's current environment, a decoder and a decision generator are obtained based on end-to-end network training, the decoder is used to decode the depth image to obtain depth information of the current environment, and then the decision generator plans the obstacle avoidance path of the drone based on the depth information. Thus, through the optical depth estimation front end combined with the small model decision maker at the back end, depth information can be quickly extracted from the physical environment, thereby improving the decision update rate of the drone, allowing the drone to fly at high speed with obstacle avoidance while ensuring accurate autonomous navigation, and the end-to-end optoelectronic neural network design improves the overall robustness and has a wider application.

[0087] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0088] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Thus, a feature specified as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of this application, "N" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0089] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing a custom logical function or process step, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may be performed in a different order than shown or discussed, including performing functions in a substantially simultaneous manner or in a reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present application pertain.

[0090] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiment, the N steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array, a field programmable gate array, etc.

[0091] Those skilled in the art will understand that all or part of the steps in the method of the above embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0092] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A high-speed obstacle avoidance system for unmanned aerial vehicles based on an optoelectronic end-to-end network, characterized in that: include: Image acquisition component, used to collect depth images of the drone's current environment; Based on the decoder and decision generator obtained by end-to-end network training, The decoder is used to decode the depth image to obtain depth information of the current environment; The decision generator is used to plan an obstacle avoidance path for the UAV based on the depth information.

2. The UAV high-speed obstacle avoidance system based on optoelectronic end-to-end network according to claim 1 is characterized in that: The image acquisition component includes: Optical lens; an optical mask disposed on the optical lens; The photosensitive sheet is used to collect the light passing through the optical mask and generate the depth image based on the light.

3. The UAV high-speed obstacle avoidance system based on optoelectronic end-to-end network according to claim 2 is characterized in that: The optical lens, the optical mask and the photosensitive film are on the same optical axis.

4. The UAV high-speed obstacle avoidance system based on optoelectronic end-to-end network according to claim 2 is characterized in that: The design of the optical mask includes: Obtaining the surface shape of the optical lens; Calculating optical performance parameters of the optical lens according to the surface shape; An optical network capable of collecting the depth image is obtained by training according to the depth image of the target environment, and the optical mask is obtained according to the optical performance parameters and the optical network design.

5. The UAV high-speed obstacle avoidance system based on optoelectronic end-to-end network according to claim 1 or 4, characterized in that: The image acquisition component and the decoder are jointly trained.

6. The UAV high-speed obstacle avoidance system based on optoelectronic end-to-end network according to claim 5 is characterized in that: The joint training process includes: acquiring an optical image and a first depth image of a target environment; generating a first training data set based on the optical image and the first depth image; The decoder is end-to-end trained using the first training data set, wherein the decoder calculates a second depth image of the target environment based on the optical image, and updates network parameters of the optical network in the image acquisition component and network parameters of the decoder based on an error between the first depth image and the second depth image.

7. The UAV high-speed obstacle avoidance system based on optoelectronic end-to-end network according to claim 1 is characterized in that: The training process of the decision generator includes: Using the privileged expert model simulation to obtain a second training data set; Identify a reference planned path, depth information, and flight parameters of the UAV in the second training data, and use the second training data set to train the decision generator, wherein the decision generator generates a predicted planned path based on the depth information and the flight parameters of the UAV, and updates the network parameters of the decision generator based on the error between the reference planned path and the predicted planned path.

8. The UAV high-speed obstacle avoidance system based on optoelectronic end-to-end network according to claim 7 is characterized in that: The method of obtaining a second training data set by simulating the privileged expert model includes: Simulate the depth information, flight parameters and global information of the UAV in the simulated environment; Inputting the global information into a privileged expert model, and the privileged expert model outputting a reference planning path of the UAV in the privileged expert model; The second training data set is generated according to the depth information, the flight parameters and the reference planning path.

9. The UAV high-speed obstacle avoidance system based on optoelectronic end-to-end network according to claim 1 is characterized in that: The decoder and the decision generator are trained as a whole, wherein network parameters of the optical network of the decoder are frozen during the overall training.

10. A high-speed obstacle avoidance method for UAV based on optoelectronic end-to-end network, characterized in that: The method uses the UAV high-speed obstacle avoidance system based on an optoelectronic end-to-end network according to any one of claims 1 to 9 to perform high-speed obstacle avoidance, wherein the method comprises the following steps: Use the image acquisition component to collect the depth image of the current environment of the drone; Decoding the depth image using a decoder to obtain depth information of the current environment and obtain flight parameters of the UAV; An obstacle avoidance path for the UAV is planned according to the depth information and the flight parameters.

Images