Aircraft control method, device, equipment, storage medium and product

Abstract

The present application relates to the field of visual control technology, and discloses an aircraft control method, device, equipment, storage medium and product, including: obtaining image data corresponding to the main flight direction and non-main direction of the aircraft; performing obstacle recognition on the image data corresponding to the main flight direction through an obstacle recognition model to obtain obstacle information corresponding to the main flight direction, performing obstacle recognition on the image data corresponding to the non-main direction through a visual matching algorithm to obtain obstacle information corresponding to each non-main direction; constructing a three-dimensional map based on the obstacle information; and controlling the operation of the aircraft based on the three-dimensional map. Since a pre-trained model is used to detect obstacles in the main flight direction, and an algorithm is used to detect obstacles in the non-main direction, the limited resources of the aircraft are fully utilized, the efficiency and accuracy of obstacle detection are guaranteed, and when controlling the operation of the aircraft, control is performed based on the constructed three-dimensional map to ensure that obstacles can be avoided in various environments.

Description

Technical Field

[0001] The present application relates to the field of visual control technology, and in particular to aircraft control methods, devices, equipment, storage media and products. Background Art

[0002] A drone is an unmanned aerial vehicle controlled by a radio remote control device or its own program control device. With the development of drone-related technologies, the application scenarios of drones are becoming more and more abundant, and the application environment is becoming more and more complex. The requirements for drones to perceive the surrounding environment and avoid obstacles are also becoming higher and higher.

[0003] Since the flying environment of a drone is unknown, when the drone is flying autonomously, it needs to perceive its own motion state and the surrounding environment in real time, and combine it with a path planning algorithm to reach the target point safely and without collision in an environment with obstacles, so as to avoid collision and bombing.

[0004] At present, obstacle detection of drones relies on traditional visual matching algorithms (such as binocular vision matching algorithms). The computational efficiency and accuracy are not guaranteed, which will affect the actual obstacle avoidance effect. In addition, path planning relies heavily on obstacle detection information in a single direction, and the perception of surrounding information is not obvious. If encountering special environments (such as semi-enclosed environments), it is difficult to avoid obstacles. Summary of the invention

[0005] The main purpose of this application is to provide an aircraft control method, device, equipment, storage medium and product, aiming to solve the technical problems that related technologies lack guarantee for drone obstacle avoidance and are difficult to avoid obstacles in some environments due to a single detection direction.

[0006] To achieve the above objectives, the present application proposes an aircraft control method, the method comprising:

[0007] Acquire image data corresponding to the main flight direction and non-main direction of the aircraft, wherein the image data is collected by a multi-camera device, the main flight direction is the current heading of the aircraft, and the non-main direction is other directions in the preset directions except the main flight direction;

[0008] Obstacle recognition is performed on the image data corresponding to the main flight direction by using an obstacle recognition model to obtain obstacle information corresponding to the main flight direction; obstacle recognition is performed on the image data corresponding to non-main directions by using a visual matching algorithm to obtain obstacle information corresponding to each non-main direction, wherein the obstacle recognition model is a pre-trained deep learning model;

[0009] Constructing a three-dimensional map according to obstacle information corresponding to the main flight direction and obstacle information corresponding to each non-main direction;

[0010] The aircraft operation is controlled based on the three-dimensional map.

[0011] In a possible implementation of the present application, the operation of the obstacle recognition model is bound to the hard core of the aircraft, and the operation of the visual matching algorithm is bound to the soft core of the aircraft; the obstacle recognition model periodically performs obstacle recognition at a first implementation frequency, and the visual matching algorithm periodically performs obstacle recognition at a second implementation frequency, and the first implementation frequency is greater than the second implementation frequency.

[0012] In a possible implementation of the present application, constructing a three-dimensional map according to obstacle information corresponding to the main flight direction and obstacle information corresponding to each non-main direction includes:

[0013] Constructing a two-dimensional grid map corresponding to the main flight direction according to the obstacle information corresponding to the main flight direction, and constructing two-dimensional grid maps corresponding to each non-main direction according to the obstacle information corresponding to each non-main direction, wherein a grid in the two-dimensional grid map represents an area, and a value in the grid represents a probability of the existence of an obstacle and a distance of the obstacle from the aircraft;

[0014] The two-dimensional grid map corresponding to the main flight direction and the two-dimensional grid maps corresponding to each non-main direction are spliced ​​to obtain a three-dimensional map.

[0015] In a possible implementation of the present application, the two-dimensional grid map corresponding to the main flight direction and the two-dimensional grid maps corresponding to each non-main direction are spliced ​​to obtain a three-dimensional map, including:

[0016] Splicing the two-dimensional grid map corresponding to the main flight direction and the two-dimensional grid maps corresponding to each non-main direction to obtain a spliced ​​map;

[0017] Superimposing the obstacle probabilities corresponding to the edge overlap areas in the stitched map, wherein the edge overlap areas are overlap areas in the two-dimensional grid map corresponding to two directions of adjacent spatial positions in the stitched map;

[0018] The superimposed mosaic map is used as a three-dimensional map.

[0019] In a possible implementation of the present application, before performing obstacle recognition on the image data corresponding to the main flight direction by using an obstacle recognition model to obtain obstacle information corresponding to the main flight direction, and performing obstacle recognition on the image data corresponding to the non-main direction by using a visual matching algorithm to obtain obstacle information corresponding to each non-main direction, the method further includes:

[0020] Acquire a sample image set, wherein the sample image set includes a plurality of sample images, wherein the sample images are collected multi-eye images, and laser ranging information is marked in the sample images;

[0021] Traversing the sample image set to obtain a current sample image;

[0022] Performing true value expansion processing on the current sample image to obtain a current processed image;

[0023] Using the currently processed image as a model training sample;

[0024] At the end of the traversal, a training sample set is constructed according to the at least one model training sample obtained;

[0025] The initial recognition model is trained using the training sample set to obtain an obstacle recognition model.

[0026] In a possible implementation of the present application, performing true value expansion processing on the current sample image to obtain a current processed image includes:

[0027] Splitting the current sample image with a window of preset specifications to obtain a window image, wherein the window image is divided into a plurality of image sub-regions according to the preset specifications;

[0028] Obtaining a true value average of the laser ranging true value in the window image, wherein the true value average is a ratio of the sum of the laser ranging true values ​​corresponding to each image sub-region in the window image to the total number of image sub-regions;

[0029] Filling the laser ranging true value corresponding to the null value area in the window image with the true value average, and processing the image by windowing, wherein the null value area is an image sub-area in the window image where the corresponding laser ranging true value is null;

[0030] The processed images of each window are stitched together to obtain the current processed image.

[0031] In a possible implementation of the present application, the step of using the currently processed image as a model training sample includes:

[0032] Scanning the hole area of ​​the current processed image to obtain hole area distribution information, wherein the hole area is an area in the current processed image where the corresponding laser ranging true value is empty;

[0033] If the size of the continuous hole region in the hole value distribution information is within the hole threshold interval, obtaining an adjacent region of the continuous hole region;

[0034] Filling the continuous hole area with true values ​​according to the laser ranging true values ​​corresponding to the adjacent areas to obtain a current filled image;

[0035] The current filled image is used as a model training sample.

[0036] In a possible implementation of the present application, the step of using the current filled image as a model training sample includes:

[0037] Dividing the current filled image into grids of a preset grid size;

[0038] Counting the proportion of points in the grid whose true values ​​of laser ranging are empty, to obtain the empty point proportion;

[0039] If the empty point ratio is less than a preset ratio threshold, marking the current filled image as an obstacle-free sky image;

[0040] The marked current filled image is used as a model training sample.

[0041] In a possible implementation of the present application, controlling the operation of the aircraft based on the three-dimensional map includes:

[0042] Controlling the aircraft to fly in a straight line to a flight destination;

[0043] Determine the distance between the aircraft and obstacles during flight according to the three-dimensional map;

[0044] When the interval distance is less than a preset interval threshold, controlling the aircraft to hover, and planning a flight route around obstacles according to the three-dimensional map;

[0045] After flying around obstacles according to the indicated flight route, the step of controlling the aircraft to fly in a straight line to the flight destination is returned to.

[0046] In a possible implementation of the present application, controlling the operation of the aircraft based on the three-dimensional map includes:

[0047] selecting a first flight point and a second flight point in the flight path according to the three-dimensional map;

[0048] Controlling the aircraft to fly toward the first flight point;

[0049] When flying to the first flight point, the second flight point is used as a new first flight point, and a new second flight point is selected in the flight path according to the three-dimensional map, and the step of controlling the aircraft to fly to the first flight point is returned.

[0050] In addition, to achieve the above-mentioned purpose, the present application also proposes an aircraft control device, the device comprising:

[0051] An acquisition module is used to acquire image data corresponding to the main flight direction and non-main direction of the aircraft, wherein the image data is collected by a multi-camera device, the main flight direction is the current heading of the aircraft, and the non-main direction is a direction other than the main flight direction in the preset direction;

[0052] an identification module, configured to perform obstacle identification on the image data corresponding to the main flight direction through an obstacle identification model to obtain obstacle information corresponding to the main flight direction, and perform obstacle identification on the image data corresponding to non-main directions through a visual matching algorithm to obtain obstacle information corresponding to each non-main direction, wherein the obstacle identification model is a pre-trained deep learning model;

[0053] A construction module, used to construct a three-dimensional map according to the obstacle information corresponding to the main flight direction and the obstacle information corresponding to each non-main direction;

[0054] A control module is used to control the operation of the aircraft based on the three-dimensional map.

[0055] In addition, to achieve the above-mentioned purpose, the present application also proposes an aircraft control device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the aircraft control method as described above.

[0056] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the aircraft control method as described above are implemented.

[0057] In addition, to achieve the above-mentioned purpose, the present application also proposes a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the aircraft control method described above are implemented.

[0058] One or more technical solutions proposed in this application have at least the following technical effects:

[0059] Since a pre-trained model is used to detect obstacles in the main flight direction, and an algorithm is used to detect obstacles in non-main directions, the limited resources of the aircraft are fully utilized, the efficiency and accuracy of obstacle detection are guaranteed, and when controlling the operation of the aircraft, it is controlled according to the constructed three-dimensional map to ensure obstacle avoidance in various environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0061] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0062] Figure 1 A schematic diagram of a flow chart provided for the first embodiment of the aircraft control method of the present application;

[0063] Figure 2 A schematic diagram of a system module according to an embodiment of the present application;

[0064] Figure 3 This is a schematic diagram of an obstacle detection execution process according to an embodiment of the present application;

[0065] Figure 4 A schematic diagram of a flow chart provided for the second embodiment of the aircraft control method of the present application;

[0066] Figure 5 A schematic diagram of a raster fusion process according to an embodiment of the present application;

[0067] Figure 6 A schematic diagram of a flow chart provided for the second embodiment of the aircraft control method of the present application;

[0068] Figure 7 A laser true value construction and optimization process for an embodiment of the present application;

[0069] Figure 8 This is a schematic diagram of the module structure of the aircraft control device according to an embodiment of the present application;

[0070] Fig. 9 Schematic diagram of the device structure of the hardware operating environment involved in the aircraft control method in the embodiment of the present application.

[0071] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0072] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0073] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0074] The present application provides an aircraft control method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the aircraft control method of the present application.

[0075] In this embodiment, the aircraft control method includes steps S10 to S40:

[0076] Step S10: Acquire image data corresponding to the main flight direction and non-main directions of the aircraft.

[0077] It should be noted that the executing entity of this embodiment can be the aircraft itself or an aircraft control device arranged in the aircraft. The aircraft control device can be an electronic device such as a smart computer, a microcontroller, or other devices with the same or similar functions. This embodiment does not limit this. In this embodiment and the following embodiments, the aircraft control method of this application is explained by taking the aircraft control device as an example.

[0078] It should be noted that the main flight direction can be the current heading of the aircraft, that is, the current forward direction of the aircraft. The non-main direction can be other directions in the preset direction except the main flight direction. The preset direction can be multiple directions related to the flight control of the aircraft. For example, if the operation of the aircraft is controlled in six directions, the preset directions can include the front, back, left, right, up and down of the aircraft. Of course, if necessary, the preset directions can also be set to more or fewer directions, and this embodiment is not limited to this.

[0079] In order to facilitate the acquisition of the distance of obstacles, image data in each preset direction can be collected by a multi-eye camera device. In order to facilitate the collection of image data in each preset direction, a plurality of multi-eye camera devices can be arranged on the aircraft, wherein the multi-eye camera device can be a binocular camera, a trio of cameras or more cameras. In the embodiment of the present application, a binocular camera is taken as an example for illustration.

[0080] In actual use, the aircraft may be equipped with a flight control system, and the main flight direction may be obtained through the flight control system. The flight control system may include the following sensors:

[0081] Inertial Measurement Unit (IMU): Contains a three-axis gyroscope, a three-axis accelerometer, etc., used to detect the angular velocity and linear acceleration of the aircraft;

[0082] Magnetometer / electronic compass: used to measure the direction of the aircraft relative to the earth's magnetic field, that is, heading;

[0083] Barometric altimeter: Estimates the relative altitude of an aircraft by measuring changes in atmospheric pressure;

[0084] GPS system: Receives satellite signals to determine the aircraft's precise position, speed, and time.

[0085] Specifically, flight control information can be obtained from the flight control system. The flight control information may include aircraft status (altitude above the ground, attitude, speed), timestamp, camera calibration parameters, IMU data, barometer (altitude), GPS / RTK, etc., as well as confirmation of the drone's real-time: main flight direction, downward direction, other directions, and whether it has a return mark.

[0086] Step S20: performing obstacle recognition on the image data corresponding to the main flight direction through an obstacle recognition model to obtain obstacle information corresponding to the main flight direction, and performing obstacle recognition on the image data corresponding to non-main directions through a visual matching algorithm to obtain obstacle information corresponding to each non-main direction.

[0087] It should be noted that the obstacle recognition model is a pre-trained deep learning model, such as a pre-trained binocular matching deep model. The structure of the obstacle recognition model includes but is not limited to: feature extraction, cost body construction, cost aggregation, disparity regression, disparity optimization and other structures. The sample images used when training the obstacle recognition model can be determined according to the multi-camera device actually set on the aircraft. If the multi-camera device used is a binocular camera, the sample images used can also be binocular images.

[0088] It should be noted that the main flight direction is the direction in which the aircraft is currently moving, and is the direction in which the aircraft mainly needs to avoid obstacles. Obstacle identification needs to be prioritized. Therefore, obstacle identification can be performed on the image data corresponding to the main flight direction through an obstacle identification model to obtain obstacle information corresponding to the main flight direction, thereby ensuring a short identification time and a high identification accuracy, thereby ensuring identification efficiency and accuracy.

[0089] The non-main direction is a direction of relatively low importance in obstacle avoidance, and its requirements for recognition efficiency and recognition accuracy are lower than those for the main flight direction. Therefore, a visual matching algorithm can be used to identify obstacles in the image data corresponding to the non-main direction to obtain obstacle information corresponding to each non-main direction. The visual matching algorithm used can be determined according to the multi-camera device used. If the multi-camera device used is a binocular camera, the visual matching algorithm used can be a binocular visual matching algorithm.

[0090] It should be noted that the soft core is provided in the form of source code based on hardware description language (such as Verilog or VHDL), which is suitable for application scenarios that require high design flexibility, such as interface, encoding, decoding, algorithm and channel encryption, etc. The development cost is relatively low, but the performance is not as good as the hard core;

[0091] The hard core is provided in the form of a physical layout, which is usually a complete circuit design under a specific process. It has been physically designed and optimized for layout and routing and can be used directly for manufacturing. However, once the design is completed, the function and performance of the hard core are fixed and not easy to modify. It is suitable for application scenarios that require high performance and high reliability, such as memory, analog circuits, and bus devices. The development cost of the hard core is relatively high, but the performance is stable and predictable.

[0092] If the aircraft is equipped with both hard cores (such as GPU, NPU, FPGA, etc.) and soft cores (such as ARM, CPU, etc.), in order to ensure the operation of obstacle recognition in the main flight direction, the operation of the obstacle recognition model can be bound to the aircraft's hard core, and the operation of the visual matching algorithm can be bound to the aircraft's soft core.

[0093] At the same time, the execution frequency of obstacle recognition in the main flight direction may be different from the frequency of obstacle recognition in the non-main direction. In this case, the obstacle recognition model may be made to periodically perform obstacle recognition at a first implementation frequency, and the visual matching algorithm may be made to periodically perform obstacle recognition at a second implementation frequency, wherein the first implementation frequency is different from the second implementation frequency.

[0094] Since it is necessary to prioritize obstacle recognition in the main flight direction, the recognition frequency of the main flight direction can be made higher than that of the non-main direction. In this case, the first realization frequency is greater than the second realization frequency.

[0095] For example: bind the main flight direction to a hard core (including but not limited to: GPU / NPU / FPGA, etc.), use the deep model method to implement the binocular matching algorithm, obtain the binocular image depth information, and thus perceive the obstacle information in the main flight direction, where: implementation frequency: N times / s (N can be controlled to be above 10);

[0096] The non-main direction binding soft core (including but not limited to: ARM / CPU, etc.) adopts the traditional binocular matching algorithm to obtain the binocular image depth information, so as to perceive the obstacle information in other directions, where: implementation frequency: M times / s (M ≈ N / a), a can be set in advance by the manager of the aircraft control device, a is a positive integer, such as setting a to 4.

[0097] Step S30: constructing a three-dimensional map according to the obstacle information corresponding to the main flight direction and the obstacle information corresponding to each non-main direction.

[0098] In actual use, obstacles can be marked according to the obstacle information corresponding to the main flight direction and the obstacle information corresponding to each non-main direction with the aircraft as the center, thereby generating a three-dimensional map.

[0099] Step S40: Controlling the operation of the aircraft based on the three-dimensional map.

[0100] It is understandable that after the three-dimensional map is constructed, the aircraft can be controlled to perform obstacle avoidance flight according to the obstacle information marked in the three-dimensional map until it reaches the destination.

[0101] In a possible implementation of this embodiment, the aircraft may be controlled to fly in a straight line. In this case, step S40 shown in this embodiment may include:

[0102] Controlling the aircraft to fly in a straight line to a flight destination;

[0103] Determine the distance between the aircraft and obstacles during flight according to the three-dimensional map;

[0104] When the interval distance is less than a preset interval threshold, controlling the aircraft to hover, and planning a flight route around obstacles according to the three-dimensional map;

[0105] After flying around obstacles according to the indicated flight route, the step of controlling the aircraft to fly in a straight line to the flight destination is returned to.

[0106] It should be noted that the flight destination may be a destination that the aircraft needs to reach, and may be pre-set by an administrator of the aircraft control device. The preset interval threshold may be pre-set by an administrator of the aircraft control device, wherein different preset interval thresholds may be set for different forward speeds of the aircraft.

[0107] In actual use, while the aircraft is flying in a straight line to its flight destination, the aircraft can continuously collect image data in the main flight direction and non-main directions, and continuously update the three-dimensional map after processing, and continuously detect the interval distance between the aircraft and the obstacle during the flight according to the three-dimensional map. If the interval distance is less than the preset interval threshold, it means that the aircraft is already close to the obstacle at this time. At this time, the aircraft may collide with the obstacle if it continues to fly in a straight line. Therefore, the aircraft can be controlled to hover, and then a flight route to bypass the obstacle is planned according to the three-dimensional map, and the aircraft is controlled to fly according to the flight route to bypass the obstacle, and then the step of controlling the aircraft to fly in a straight line to the flight destination is continued until the aircraft reaches the flight destination.

[0108] When planning the flight route, a common route planning algorithm, such as the A path search algorithm, may be used, and this embodiment does not impose any limitation on this.

[0109] It is understandable that this method can ensure that the aircraft is controlled to move to the flight destination along the shortest path as much as possible. However, when avoiding obstacles and replanning the route, the aircraft needs to be kept hovering.

[0110] In a possible implementation of this embodiment, the flight of the aircraft may be controlled by multiple points. In this case, step S40 shown in this embodiment may include:

[0111] selecting a first flight point and a second flight point in the flight path according to the three-dimensional map;

[0112] Controlling the aircraft to fly toward the first flight point;

[0113] When flying to the first flight point, the second flight point is used as a new first flight point, and a new second flight point is selected in the flight path according to the three-dimensional map, and the step of controlling the aircraft to fly to the first flight point is returned.

[0114] It should be noted that the flight path may be the path between the location of the aircraft and the flight destination.

[0115] In actual use, a dynamic programming algorithm can be used to select two points with the best movement on the flight path in advance, namely the first flight point and the second flight point. Then, the aircraft is controlled to fly to the first flight point. When flying to the first flight point, the second flight point can be used as the new first flight point, and a dynamic programming algorithm can be used to select a new second flight point in the flight path according to the three-dimensional map. Then, the step is returned to control the aircraft to fly to the first flight point until it reaches the flight destination.

[0116] It should be noted that during the flight of the aircraft, the collected image data can be used to determine in real time whether there are new dynamic obstacles. If a new dynamic obstacle appears within the fixed flight radius of the aircraft, the aircraft can be controlled to hover urgently and the dynamic programming algorithm can be used to re-select the new first flight point and second flight point.

[0117] It is understandable that by controlling the operation of the aircraft in this way, the moving path can be set by selecting points in advance, and the aircraft does not need to make an emergency hover unless it encounters a moving obstacle. However, this mode of operation requires a high level of computing power and is difficult to apply to aircraft with lower configurations.

[0118] For ease of understanding, now combined Figure 2 and Figure 3 This is for illustration only, but not for limitation. Figure 2 is a schematic diagram of system modules in the aircraft control device of this embodiment; Figure 3 FIG. 4 is a schematic diagram of the obstacle detection execution flow of this embodiment.

[0119] In this example, binocular recognition is used as an example. Figure 2 As shown, the system modules in the aircraft control device can mainly include four, namely system initialization update, obstacle detection, map construction and path planning module, and visual system auxiliary module.

[0120] Among them, the visual system initialization module: according to the parameters transmitted by the drone flight control system (including but not limited to: aircraft status (altitude above the ground, attitude, speed), timestamp, camera calibration parameters, IMU data, barometer (altitude), GPS / RTK, etc.), confirm the drone's real-time: flight direction, down direction, other directions, and whether it has a return mark and other information.

[0121] The above information can be passed to subsequent modules to control the operation status of subsequent modules, including:

[0122] The flight direction is passed to: obstacle detection module;

[0123] The following direction is passed to: the visual odometry module in the visual engine system auxiliary module;

[0124] The down direction and other directions are passed to: obstacle detection module;

[0125] Whether the return mark is passed to: the visual guidance landing module in the visual engine system auxiliary module;

[0126] The above flight control information is refreshed in real time and passed to the initialization module for real-time update.

[0127] Obstacle detection module: The goal is to use binocular vision matching methods to perceive the distance of obstacles. The binocular vision matching methods include: traditional binocular vision matching methods and deep model methods. In order to effectively improve the efficiency of visual matching, soft-core and hard-core separate binding strategies and methods for achieving different frequency matching in different directions are used.

[0128] Map construction and path planning module: Based on the 6-way obstacle information perceived by the obstacle detection module, a complete 3D grid map information is constructed, and then the UAV flight path is planned in real time based on the 3D map information.

[0129] Visual system auxiliary module: including but not limited to visual odometer and visual guided landing, which only implements visual odometer for the direction below the drone, obtains aircraft attitude information as auxiliary information when GPS / RTK signal is weak. When the drone return signal is received, the visual guided landing module will be activated to guide the drone to land accurately at the designated location.

[0130] When the obstacle detection module performs obstacle detection, the specific implementation process is as follows Figure 3 As shown:

[0131] Main flight direction (also referred to as flight direction): Bind hard core (including but not limited to: GPU / NPU / FPGA, etc.), use deep model method to implement binocular matching algorithm, obtain binocular image depth information (this process may involve model matching, disparity map, disparity conversion and other processes), so as to perceive obstacle information in the main flight direction, and achieve frequency: N times / s (N can be controlled to be above 10);

[0132] Other directions (i.e. non-main directions): bind a soft core (including but not limited to: ARM / CPU, etc.), use the traditional binocular matching algorithm (this process may involve matching cost calculation, cost aggregation, disparity calculation, disparity map, disparity conversion, etc.), obtain the binocular image depth information, and thus perceive the obstacle information in other directions, achieving a frequency of M times / s (M ≈ N / 4).

[0133] Among them, the above-mentioned core binding strategy and different frequency matching methods can be flexibly adjusted and are not limited to fixed binding modes. Detection in other directions can be carried out in staggered periods or simultaneously. The above-mentioned system initialization module controls in real time whether to perform detection based on the flight direction and obstacle information.

[0134] This embodiment provides an aircraft control method. Since a pre-trained model is used to detect obstacles in the main flight direction, and an algorithm is used to detect obstacles in non-main directions, the limited resources of the aircraft are fully utilized, the efficiency and accuracy of obstacle detection are guaranteed, and when controlling the operation of the aircraft, the control is performed based on a constructed three-dimensional map, ensuring that obstacles can be avoided in various environments.

[0135] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 4 , step S30, including steps S301-S302:

[0136] Step S301: constructing a two-dimensional grid map corresponding to the main flight direction according to obstacle information corresponding to the main flight direction, and constructing two-dimensional grid maps corresponding to each non-main direction according to obstacle information corresponding to each non-main direction.

[0137] It should be noted that, to construct a two-dimensional grid map corresponding to the main flight direction according to the obstacle information corresponding to the main flight direction, the space corresponding to the main flight direction can be first divided into grids to obtain a grid map, and each grid in the grid map corresponds to an area of ​​the main flight direction. After that, obstacles are marked in the grid map according to the obstacle information of the main flight direction, and the obstacle information is filled into the corresponding grid, so as to obtain a two-dimensional grid map corresponding to the main flight direction. The construction method of constructing two-dimensional grid maps corresponding to each non-main direction according to the obstacle information corresponding to each non-main direction is similar to the implementation method of constructing a two-dimensional grid map corresponding to the main flight direction, and will not be repeated here.

[0138] In the two-dimensional grid map, a grid represents an area, and the value in the grid represents the probability of the existence of an obstacle and the distance between the obstacle and the aircraft.

[0139] Step S302: splicing the two-dimensional grid map corresponding to the main flight direction and the two-dimensional grid maps corresponding to each non-main direction to obtain a three-dimensional map.

[0140] It should be noted that the three-dimensional map can be obtained by taking the aircraft as the three-dimensional center and the spatial direction as the basis, splicing the two-dimensional grid map corresponding to the main flight direction and the two-dimensional grid maps corresponding to each non-main direction.

[0141] In a possible implementation of this embodiment, if the same obstacle has different states on grid maps of different directions included in the three-dimensional map, it may cause detection errors. To avoid this situation, step S302 in this embodiment may include:

[0142] Splicing the two-dimensional grid map corresponding to the main flight direction and the two-dimensional grid maps corresponding to each non-main direction to obtain a spliced ​​map;

[0143] Superimposing the obstacle probabilities corresponding to the edge overlap areas in the spliced ​​map;

[0144] The superimposed mosaic map is used as a three-dimensional map.

[0145] It should be noted that the edge overlap region may be an overlap region in a two-dimensional grid map corresponding to two directions of adjacent spatial positions in the stitched map, and the overlap region may be pre-calibrated by a manager of the aircraft control device.

[0146] In actual use, the probability of obstacles existing in the overlapping areas of the two-dimensional grid map corresponding to two directions of adjacent spatial positions can be added.

[0147] For ease of understanding, now combined Figure 5 This is for illustration only, but not for limitation. Figure 5 Schematic diagram of the raster fusion process of this embodiment.

[0148] The two-dimensional grid map corresponding to the main flight direction and the two-dimensional grid maps corresponding to each non-main direction are spliced ​​to obtain a spliced ​​map. At this time, there are edge overlap areas in the two-dimensional grid maps with adjacent spatial positions, such as Figure 5 As shown, the spatial positions of the front direction and the left direction are adjacent, the front grid map (i.e., the two-dimensional grid map corresponding to the front direction) and the left grid map (the grid map corresponding to the left direction) have two grid position matches, the rightmost column of the left grid map corresponds one-to-one with the leftmost column of the front grid map, and they are the edge overlap areas, the second column of the left grid map from right to left corresponds one-to-one with the second column of the front grid map from left to right, and they are the edge overlap areas, before probabilistic superposition (also called matching fusion) is performed, the left grid map is as follows Figure 5 As shown in "Initial: Left Grid Map" in the figure, the front grid map is as follows Figure 5 As shown in the "Initial: Front Grid Map" in the figure, the probability superposition is performed, and the grid map on the left is obtained as follows Figure 5 As shown in "After Fusion Matching: Left" in the figure, the front grid map is as follows Figure 5 As shown in "After Matching Fusion: Front", the value in the grid indicates the probability of the existence of an obstacle. If the value is greater than 0.5, it means that there is an obstacle ( Figure 5 The shadow grid is the grid area that is identified as having obstacles, and the shadow grid can be a grid with a shadow background). Therefore, after the probability superposition, 6 grids that are identified as having obstacles are added.

[0149] This embodiment provides an aircraft control method. A two-dimensional grid map corresponding to each direction is first constructed according to obstacle information in each direction to ensure map accuracy. The two-dimensional grid map is then fused into a three-dimensional map to facilitate control of the aircraft. In the fusion process, the probability of overlapping obstacles is superimposed to ensure that the state of the same obstacle remains consistent, thereby avoiding detection errors caused by state differences.

[0150] Based on the first embodiment of the present application, in the third embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 6 , before step S20, further comprising steps S11-S16:

[0151] Step S11: Obtain a sample image set.

[0152] It should be noted that the sample image set may include multiple sample images, and the sample images may be multi-eye images collected, and laser ranging information is marked in the sample images. The sample image set may be pre-set by the administrator of the aircraft control device.

[0153] For example: using laser to collect the true value of obstacle distance, without manual calibration, it is convenient to quickly train the drone binocular matching depth model. For the laser ranging of the same sample image, you can select N consecutive accumulated laser frames to mark the laser ranging information (covering 40%-80%): the value of N can be adjusted by the manager of the drone control equipment according to actual needs. If too many laser frames are accumulated, although the coverage rate is large, it is time-consuming and the timeliness cannot be guaranteed. If the laser frames are accumulated less, the timeliness is guaranteed, but the coverage rate is not enough, and the accuracy of the trained model cannot be guaranteed. Therefore, 4 <N<10。

[0154] The above examples are only for ease of understanding and are not intended to be limiting.

[0155] Step S12: traverse the sample image set to obtain a current sample image.

[0156] It should be noted that when traversing the sample image set, it is possible, and this embodiment does not impose any limitation on this.

[0157] In actual use, the sample image set is traversed to obtain the current sample image by traversing the sample image set according to the set index order, or by performing random traversal and taking the traversed sample image as the current sample image.

[0158] Step S13: performing true value expansion processing on the current sample image to obtain a current processed image.

[0159] In the specific implementation, the laser value in the sample image is too absolute, and the area without laser detection value is marked as empty or 0. If such an image is used directly for model training, the judgment result of the trained model may be too absolute, and the actual use effect may be poor. In order to avoid this situation, the current sample image can be subjected to true value expansion processing. Through the true value expansion processing, a certain laser value is set for the area without laser detection value (the assigned value can be smaller to ensure that the model judgment is not too absolute and the model is not misjudged), and the processed current sample image is used as the current processing image.

[0160] Step S14: using the currently processed image as a model training sample.

[0161] Step S15: At the end of the traversal, a training sample set is constructed according to the at least one model training sample obtained.

[0162] Step S16: training the initial recognition model using the training sample set to obtain an obstacle recognition model.

[0163] It can be understood that after processing, a certain value has been assigned to the part without laser detection value. The overall judgment result of the model trained in this way will not be too absolute. Therefore, the current processing sample can be used as a model training sample. With the end of the traversal, multiple model training samples can be obtained. At this time, the obtained model training samples can be aggregated to obtain a training sample set. After that, the initial recognition model can be trained according to the training sample set. During the training process, the parameters in the initial recognition model are continuously optimized. After the training is completed (such as the number of training iterations reaches the preset number, or the precision / accuracy of the model reaches the preset threshold), the trained initial recognition model can be used as an obstacle recognition model.

[0164] In a possible implementation of this embodiment, in order to reasonably perform truth value expansion, step S13 in this embodiment may include:

[0165] Splitting the current sample image with a window of preset specifications to obtain a window image, wherein the window image is divided into a plurality of image sub-regions according to the preset specifications;

[0166] Get the true average value of the laser ranging true value in the window image;

[0167] Filling the laser ranging true value corresponding to the empty value area in the window image with the true value average value, and processing the image through the window;

[0168] The processed images of each window are stitched together to obtain the current processed image.

[0169] It should be noted that the preset specifications can be pre-set by the management personnel of the aircraft control equipment, and the preset specifications can be expressed as n*n, that is, a window of n rows and n columns. The average value of the true value is the ratio of the sum of the laser ranging true values ​​(also referred to as laser true values) corresponding to each image sub-region in the window image to the total number of image sub-regions. The empty value region is the image sub-region in the window image whose corresponding laser ranging true value is empty. The window processed images are spliced ​​to obtain the current processed image by reverse processing according to the splitting method, and the window processed images are spliced ​​into the current processed image.

[0170] It is understandable that there may be only a small part of the window image with the laser true value, and the average value of the true value is relatively small and uniform. In this way, the laser ranging true value corresponding to the empty area in the window image is filled with the average value of the true value, which can ensure that the part where the laser ranging true value does not exist is filled with a reasonable laser true value. Using this processed sample image as a model training sample can ensure that the model judgment is not too absolute while also avoiding misjudgment of the model.

[0171] In actual use, when the current sample image is split by a window of preset specifications, it can be directly processed after all the splits are performed. Of course, it can also be processed one by one in the form of window sliding, and this embodiment does not limit this. It is also possible not to split the image, but to directly select a part of the area from the current sample image as the window image according to the window, and directly perform the truth dilation processing on the current sample image.

[0172] To facilitate understanding, the sliding method is now described with examples, such as:

[0173] Scan the laser ranging true value in the current sample image through an n*n window sliding; calculate the summed average value -value (i.e., the true value average) of the laser ranging true value in real time within the window (i.e., within the window image); within the window, determine the points where the true value is empty (i.e., the empty value area within the window image), and fill them with the summed average value.

[0174] In a possible implementation of this embodiment, since there may be no true value of laser ranging in the window image, that is, all areas in the window image are null value areas, for the area corresponding to this window image, even after the true value expansion, the laser ranging value of this part will still be a null value. For this type of area, further processing is required to avoid affecting the model training effect. At this time, step S14 of this embodiment may include:

[0175] Scanning the hole area of ​​the current processed image to obtain hole area distribution information;

[0176] If the size of the continuous hole region in the hole value distribution information is within the hole threshold interval, obtaining an adjacent region of the continuous hole region;

[0177] Filling the continuous hole area with true values ​​according to the laser ranging true values ​​corresponding to the adjacent areas to obtain a current filled image;

[0178] The current filled image is used as a model training sample.

[0179] It should be noted that the hole area is an area in the current processed image where the corresponding laser ranging true value is empty. The size of the continuous hole area can be the area size of the connected hole area. The hole threshold interval can be pre-set by the administrator of the aircraft control device, for example: the hole threshold interval is set to be less than w / 100, where w is the image size of the current processed image.

[0180] In actual use, if the size of the continuous hole area in the hole value distribution information is within the hole threshold range, it means that the area size of the continuous hole area is small, which may be caused by laser sparseness. At this time, the continuous hole area can be filled with true values ​​according to the laser ranging true values ​​corresponding to the adjacent areas of the continuous hole area, and the current processed image after filling is used as the current filled image. After that, the current filled image can be used as a model training sample;

[0181] If the size of the continuous hole area in the hole value distribution information is not within the hole threshold range, it means that the size of the continuous hole area is large. At this time, it can be determined that there is no obstacle and no filling is performed. After that, the current processed image is directly used as a model training sample.

[0182] In a specific implementation, since there are no reflection points in the sky area, the true value of the laser cannot be obtained (but the model needs to be learned, and the sky is obstacle-free). At the same time, since the laser itself has a sparsity problem (there are obstacles, no true value, and the model should not pay attention to it), there may be no true value of laser ranging. In order to distinguish these two situations, the step of using the current filling image as a model training sample in this embodiment may include:

[0183] Dividing the current filled image into grids of a preset grid size;

[0184] Counting the proportion of points in the grid whose true values ​​of laser ranging are empty, to obtain the empty point proportion;

[0185] If the empty point ratio is less than a preset ratio threshold, marking the current filled image as an obstacle-free sky image;

[0186] The marked current filled image is used as a model training sample.

[0187] It should be noted that the preset grid specification can be pre-set by the administrator of the aircraft control device, and each point in the grid (one grid in the grid is one point) represents a part of the current sample image. The empty point ratio can be the proportion of points whose true value of laser ranging is empty in all points of the grid. The preset proportion threshold can be pre-set by the administrator of the aircraft control device.

[0188] In actual use, if the proportion of empty points is less than the preset proportion threshold, it means that the current sample image is a sky image and there are no obstacles at this time; if the proportion of empty points is greater than the preset proportion threshold, it can indicate that this is caused by laser sparsity at this time. At this time, the current filled image can be marked as not care and not added to the training as a model training sample.

[0189] Among them, the above judgment for sky images can be executed only when the laser ranging ground truth is still sparse after the ground truth dilation processing and / or the ground truth filling of the above continuous hole regions (for example, the number of non-filled laser ranging ground truths in the processed image is less than n).

[0190] For ease of understanding, it is now combined with Figure 7 for illustration, but it does not limit the present solution. Figure 7 This is the laser ground truth construction and optimization process of this embodiment.

[0191] Such as Figure 7 shown, the specific process may include:

[0192] 1. For the environment corresponding to the sample image, use lasers to collect the ground truth of the obstacle distance without manual calibration, which is convenient for quickly training the binocular matching depth model of the drone. N consecutive frames are accumulated (4 < N < 10) (which can cover 40% - 80%);

[0193] Among them, if too many laser frames are accumulated, although the coverage rate is large, it takes a long time and the timeliness cannot be guaranteed. If the number of accumulated laser frames is small, the timeliness is guaranteed, but the coverage rate is insufficient and the accuracy of the trained model cannot be guaranteed.

[0194] 2. In the laser ground truth optimization module, by means of ground truth dilation and horizontal hole prediction, try to cover the obstacle information as much as possible to solve the laser sparsity problem.

[0195] The detailed process is as follows:

[0196] Scan the initial laser ground truth through an n*n window; within the window, calculate the weighted sum average value - value of the laser ground truth in real time; within the window, for the points where the ground truth is empty, fill them with the weighted sum average value value; after the dilation process is completed, further use the method of horizontal empty map prediction to make up for the remaining sparse problems; scan the hole values horizontally row by row, and judge whether the size of the continuous hole values is within the threshold range (less than the threshold, select w / 100 of the image). If it is within the threshold range, fill it with the laser ground truths on the left and right of the continuous hole values. If it exceeds the threshold, it is considered that there are no obstacles and no processing is done;

[0197] 3. Since there are no reflection points in the sky area, the true value of the laser cannot be obtained (but the model needs to be learned, and the sky is obstacle-free). At the same time, the laser itself has a sparse problem (there are obstacles, no true value, and the model should not care);

[0198] In order for the model to effectively distinguish the above two situations (neither of which has true value), it is necessary to fill in the valid true value for the sky so that the model can learn. The detailed process is as follows:

[0199] Divide the laser image into m*n grids; determine whether the percentage of points with a true value of 0 in the grid is less than a threshold; if less than the threshold, the grid is determined to be the sky, the true value is set to no obstacles, and the training is added; if greater than the threshold, the grid is determined to be caused by laser sparsity, the true value is set to not care, and the training is not added;

[0200] 4. If there is a problem of inaccurate information when laser scanning the edge of an obstacle, a large model can be used to assist in correcting the laser true value.

[0201] 5. Through the above truth value construction and optimization methods, an effective obstacle truth value system (i.e., model training samples) is generated, and the deep model is trained to ensure the binocular matching deep model method, effectively deal with various obstacles, and maximize the accuracy of obstacle detection.

[0202] This embodiment provides an aircraft detection method, which constructs model training samples by performing multi-level processing on sample images, ensures the training depth model, and can ensure the binocular matching depth model method, effectively deal with various obstacles, and maximize the accuracy of obstacle detection.

[0203] This application also provides an aircraft control device, please refer to Figure 8 , the aircraft control device includes:

[0204] An acquisition module 10 is used to acquire image data corresponding to a main flight direction and a non-main direction of the aircraft, wherein the image data is collected by a multi-camera device, the main flight direction is the current heading of the aircraft, and the non-main direction is a direction other than the main flight direction in a preset direction;

[0205] The recognition module 20 is used to perform obstacle recognition on the image data corresponding to the main flight direction through an obstacle recognition model to obtain obstacle information corresponding to the main flight direction, and to perform obstacle recognition on the image data corresponding to non-main directions through a visual matching algorithm to obtain obstacle information corresponding to each non-main direction. The obstacle recognition model is a pre-trained deep learning model;

[0206] A construction module 30, configured to construct a three-dimensional map according to the obstacle information corresponding to the main flight direction and the obstacle information corresponding to each non-main direction;

[0207] The control module 40 is used to control the operation of the aircraft based on the three-dimensional map.

[0208] The aircraft control device provided by the present application adopts the aircraft control method in the above embodiment, which can solve the technical problems that the obstacle avoidance of drones in the related technology is lack of guarantee, and it is difficult to avoid obstacles in some environments due to the single detection direction. Compared with the related technology, the beneficial effects of the aircraft control device provided by the present application are the same as the beneficial effects of the aircraft control method provided by the above embodiment, and the other technical features in the aircraft control device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0209] The present application provides an aircraft control device, which includes: at least one processor; and a memory that is communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the aircraft control method in the above-mentioned embodiment one.

[0210] Reference below Fig. 9 , which shows a schematic diagram of the structure of an aircraft control device suitable for implementing an embodiment of the present application. The aircraft control device in the embodiment of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig. 9 The aircraft control device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0211] like Fig. 9As shown, the aircraft control device may include a processing device 1001 (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the xxx device are also stored. The processing device 1001, ROM1002 and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the aircraft control device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows an aircraft control device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0212] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0213] The aircraft control device provided by the present application adopts the aircraft control method in the above embodiment to solve the technical problem of aircraft control. Compared with the related art, the beneficial effects of the aircraft control device provided by the present application are the same as the beneficial effects of the aircraft control method provided by the above embodiment, and the other technical features in the aircraft control device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0214] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0215] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0216] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, the computer-readable program instructions being used to execute the aircraft control method in the above-mentioned embodiment.

[0217] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0218] The computer-readable storage medium may be included in the aircraft control device; or may exist independently without being assembled into the aircraft control device.

[0219] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the aircraft control device, the aircraft control device: obtains image data corresponding to the main flight direction and non-main directions of the aircraft, the image data is collected by a multi-camera device, the main flight direction is the current heading of the aircraft, and the non-main direction is other directions in the preset direction except the main flight direction; performs obstacle recognition on the image data corresponding to the main flight direction through an obstacle recognition model to obtain obstacle information corresponding to the main flight direction, and performs obstacle recognition on the image data corresponding to the non-main direction through a visual matching algorithm to obtain obstacle information corresponding to each non-main direction, the obstacle recognition model is a pre-trained deep learning model; constructs a three-dimensional map according to the obstacle information corresponding to the main flight direction and the obstacle information corresponding to each non-main direction; and controls the operation of the aircraft based on the three-dimensional map.

[0220] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0221] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0222] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0223] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned aircraft control method, and can solve the technical problems that the obstacle avoidance of drones in the related art is lack of guarantee, and it is difficult to avoid obstacles in some environments due to the single detection direction. Compared with the related art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the aircraft control method provided by the above-mentioned embodiment, and will not be repeated here.

[0224] The present application also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of the above-mentioned aircraft control method are implemented.

[0225] The computer program product provided by this application can solve the technical problems that the obstacle avoidance of drones in related technologies is not guaranteed, and that it is difficult to avoid obstacles in some environments due to the single detection direction. Compared with related technologies, the beneficial effects of the computer program product provided by this application are the same as the beneficial effects of the aircraft control method provided by the above embodiment, and will not be repeated here.

[0226] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A method for controlling an aircraft, characterized in that: The method comprises: Acquire image data corresponding to the main flight direction and non-main direction of the aircraft, wherein the image data is collected by a multi-camera device, the main flight direction is the current heading of the aircraft, and the non-main direction is other directions in the preset directions except the main flight direction; Obstacle recognition is performed on the image data corresponding to the main flight direction by using an obstacle recognition model to obtain obstacle information corresponding to the main flight direction; obstacle recognition is performed on the image data corresponding to non-main directions by using a visual matching algorithm to obtain obstacle information corresponding to each non-main direction, wherein the obstacle recognition model is a pre-trained deep learning model; Constructing a three-dimensional map according to obstacle information corresponding to the main flight direction and obstacle information corresponding to each non-main direction; Controlling the operation of the aircraft based on the three-dimensional map; The obstacle recognition model periodically recognizes obstacles at a first realization frequency, and the visual matching algorithm periodically recognizes obstacles at a second realization frequency, and the first realization frequency is greater than the second realization frequency; Wherein, constructing a three-dimensional map according to obstacle information corresponding to the main flight direction and obstacle information corresponding to each non-main direction includes: Constructing a two-dimensional grid map corresponding to the main flight direction according to the obstacle information corresponding to the main flight direction, and constructing two-dimensional grid maps corresponding to each non-main direction according to the obstacle information corresponding to each non-main direction, wherein a grid in the two-dimensional grid map represents an area, and a value in the grid represents a probability of the existence of an obstacle and a distance of the obstacle from the aircraft; The two-dimensional grid map corresponding to the main flight direction and the two-dimensional grid maps corresponding to each non-main direction are spliced ​​to obtain a three-dimensional map; The step of splicing the two-dimensional grid map corresponding to the main flight direction and the two-dimensional grid maps corresponding to each non-main direction to obtain a three-dimensional map includes: Splicing the two-dimensional grid map corresponding to the main flight direction and the two-dimensional grid maps corresponding to each non-main direction to obtain a spliced ​​map; Superimposing the obstacle probabilities corresponding to the edge overlap areas in the stitched map, wherein the edge overlap areas are overlap areas in the two-dimensional grid map corresponding to two directions of adjacent spatial positions in the stitched map; The superimposed mosaic map is used as a three-dimensional map.

2. The aircraft control method according to claim 1, characterized in that: The operation of the obstacle recognition model is bound to the hard core of the aircraft, and the operation of the visual matching algorithm is bound to the soft core of the aircraft.

3. The aircraft control method according to claim 1, characterized in that: Before performing obstacle recognition on the image data corresponding to the main flight direction by using the obstacle recognition model to obtain obstacle information corresponding to the main flight direction, and performing obstacle recognition on the image data corresponding to the non-main direction by using the visual matching algorithm to obtain obstacle information corresponding to each non-main direction, the method further includes: Acquire a sample image set, wherein the sample image set includes a plurality of sample images, wherein the sample images are collected multi-eye images, and laser ranging information is marked in the sample images; Traversing the sample image set to obtain a current sample image; Performing true value expansion processing on the current sample image to obtain a current processed image; Using the currently processed image as a model training sample; At the end of the traversal, a training sample set is constructed according to the at least one model training sample obtained; The initial recognition model is trained using the training sample set to obtain an obstacle recognition model.

4. The aircraft control method according to claim 3, characterized in that: The performing true value expansion processing on the current sample image to obtain a current processed image includes: Splitting the current sample image with a window of preset specifications to obtain a window image, wherein the window image is divided into a plurality of image sub-regions according to the preset specifications; Obtaining a true value average of the laser ranging true value in the window image, wherein the true value average is a ratio of the sum of the laser ranging true values ​​corresponding to each image sub-region in the window image to the total number of image sub-regions; Filling the laser ranging true value corresponding to the null value area in the window image with the true value average, and processing the image by windowing, wherein the null value area is an image sub-area in the window image where the corresponding laser ranging true value is null; The processed images of each window are stitched together to obtain the current processed image.

5. The aircraft control method according to claim 3, characterized in that: The step of using the currently processed image as a model training sample comprises: Scanning the hole area of ​​the current processed image to obtain hole area distribution information, wherein the hole area is an area in the current processed image where the corresponding laser ranging true value is empty; If the size of the continuous hole region in the hole region distribution information is within the hole threshold interval, obtaining an adjacent region of the continuous hole region; Filling the continuous hole area with true values ​​according to the laser ranging true values ​​corresponding to the adjacent areas to obtain a current filled image; The current filled image is used as a model training sample.

6. The aircraft control method according to claim 5, characterized in that: The step of using the current filled image as a model training sample comprises: Dividing the current filled image into grids of a preset grid size; Counting the proportion of points in the grid whose true values ​​of laser ranging are empty, to obtain the empty point proportion; If the empty point ratio is less than a preset ratio threshold, marking the current filled image as an obstacle-free sky image; The marked current filled image is used as a model training sample.

7. The aircraft control method according to any one of claims 1 to 6, characterized in that: The controlling the operation of the aircraft based on the three-dimensional map includes: Controlling the aircraft to fly in a straight line to a flight destination; Determine the distance between the aircraft and obstacles during flight according to the three-dimensional map; When the interval distance is less than a preset interval threshold, controlling the aircraft to hover, and planning a flight route around obstacles according to the three-dimensional map; After flying around obstacles according to the indicated flight route, the step of controlling the aircraft to fly in a straight line to the flight destination is returned to.

8. The aircraft control method according to any one of claims 1 to 6, characterized in that: The controlling the operation of the aircraft based on the three-dimensional map includes: selecting a first flight point and a second flight point in the flight path according to the three-dimensional map; Controlling the aircraft to fly toward the first flight point; When flying to the first flight point, the second flight point is used as a new first flight point, and a new second flight point is selected in the flight path according to the three-dimensional map, and the step of controlling the aircraft to fly to the first flight point is returned.

9. An aircraft control device, characterized in that: The device comprises: An acquisition module is used to acquire image data corresponding to the main flight direction and non-main direction of the aircraft, wherein the image data is collected by a multi-camera device, the main flight direction is the current heading of the aircraft, and the non-main direction is a direction other than the main flight direction in the preset direction; an identification module, configured to perform obstacle identification on the image data corresponding to the main flight direction through an obstacle identification model to obtain obstacle information corresponding to the main flight direction, and perform obstacle identification on the image data corresponding to non-main directions through a visual matching algorithm to obtain obstacle information corresponding to each non-main direction, wherein the obstacle identification model is a pre-trained deep learning model; A construction module, used to construct a three-dimensional map according to the obstacle information corresponding to the main flight direction and the obstacle information corresponding to each non-main direction; A control module, used for controlling the operation of the aircraft based on the three-dimensional map; The obstacle recognition model periodically recognizes obstacles at a first realization frequency, and the visual matching algorithm periodically recognizes obstacles at a second realization frequency, and the first realization frequency is greater than the second realization frequency; The construction module is further used to construct a two-dimensional grid map corresponding to the main flight direction according to the obstacle information corresponding to the main flight direction, and to construct two-dimensional grid maps corresponding to each non-main direction according to the obstacle information corresponding to each non-main direction, wherein a grid in the two-dimensional grid map represents an area, and a value in the grid represents the probability of the existence of an obstacle and the distance of the obstacle from the aircraft; the two-dimensional grid map corresponding to the main flight direction and the two-dimensional grid maps corresponding to each non-main direction are spliced ​​to obtain a three-dimensional map; The construction module is further used to splice the two-dimensional grid map corresponding to the main flight direction and the two-dimensional grid maps corresponding to each non-main direction to obtain a spliced ​​map; superimpose the obstacle probabilities corresponding to the edge overlap areas in the spliced ​​map, wherein the edge overlap areas are overlap areas in the two-dimensional grid maps corresponding to two directions with adjacent spatial positions in the spliced ​​map; and use the spliced ​​map that has been superimposed as a three-dimensional map.

10. An aircraft control device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the aircraft control method according to any one of claims 1 to 8.

11. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the aircraft control method according to any one of claims 1 to 8 are implemented.

12. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the aircraft control method according to any one of claims 1 to 8 are implemented.

Images