Interactive unmanned aerial photography method and device

Abstract

The present invention discloses an interactive unmanned aerial photography method and device, which are applied to a unmanned aerial photography execution end of an interactive unmanned aerial photography system. The method comprises: obtaining a shooting instruction given by a user at a ground station interaction end of the interactive unmanned aerial photography system; performing a dynamic path search of an aerial photography path according to the shooting instruction with the goal of minimizing the loss of shooting effect; performing a local visible area adjustment on the searched dynamic path to obtain the best observation area of ​​each search node in the dynamic path; constructing a multi-objective optimization problem in combination with the state of the unmanned aerial vehicle and the pan-tilt rotation angle, solving the multi-objective optimization problem, thereby realizing trajectory optimization; controlling the unmanned aerial vehicle to move to the end point of the trajectory according to the optimized trajectory and performing aerial photography.

Description

Technical Field

[0001] The present invention belongs to the technical field of unmanned aerial photography, and in particular relates to an interactive unmanned aerial photography method and device. Background Art

[0002] When a drone equipped with a camera and a gimbal performs aerial photography, it is necessary to ensure that at least the following three points are met: a. The flight safety of the drone itself; b. The physical limitations of the drone's own power; c. The captured image meets certain user requirements. For scenes where drones are used for aerial photography, the quality of the captured image is particularly important. In the film industry, the resulting image directly determines the final quality of the film; in scenes such as industrial inspections, adjusting to the appropriate viewing angle for shooting and reconnaissance plays a key role in the inspection of designated areas.

[0003] At present, the main methods of drone aerial photography are: a. manual operation b. autonomous tracking and shooting c. interactive aerial photography. a: It usually requires highly skilled operators, and because the drone and camera gimbal need to be controlled separately, more than one operator is often required, which consumes high labor costs and the shooting effect cannot be guaranteed. b: Some existing technologies and methods support drones to autonomously track and shoot targets, such as DJI's autonomous tracking function and cutting-edge drone autonomous tracking algorithms. These methods can better ensure the continuous shooting of drones, but due to the lack of interaction with the photographer, the shooting requirements cannot be transmitted to the drone in real time, and the final pictures and videos are somewhat different from the user's needs. c: The current research provides an interactive shooting method that allows users to customize their shooting requirements, but the existing methods are used in relatively simple scenarios and cannot be applied to the shooting of dynamic objects in complex scenes.

[0004] Regardless of the method, there are two main problems with drones in complex scenes: 1. There is a lack of accurate and effective perception of the environment outside the picture, so the safety of the drone itself cannot be guaranteed during the shooting process. 2. The drone and the camera gimbal need to work together well, otherwise it will affect the presentation of the subject in the picture, and even make the subject leave the picture under the maneuvering of the drone.

[0005] Some existing autonomous aerial photography studies aim to simplify the operation by designing appropriate interaction methods. One of the typical interactive methods is to use key frames for interaction, such as "N. Joubert, M. Roberts, A. Truong, F. Berthouz, and P. Hanrahan, "An interactive tool for designing quadrotorcamera shots," ACM Transactions on Graphics (TOG), vol. 34, no. 6, pp. 1–11, 2015" "N. Joubert, DB Goldman, F. Berthouz, M. Roberts, JALanday, P. Hanrahan, et al., "Towards a drone cinematographer: Guiding quadrotor cameras using visual composition principles," arXiv preprint arXiv: 1610.01691, 2016" "C. Gebhardt, B. Hepp, T. N¨ageli, S. Stevˇsi′c, and O. Hilliges, "Airways: Optimization-based planning of quadrotor trajectories according to high-level user goals,” in Proceedings of the 2016chi conference on human factors in computing systems, 2016, pp. 2508–2519, and C. Gebhardt, S. Stevˇsi′c, and O. Hilliges, “Optimizing for aesthetically pleasing quadrotor camera motion,” ACM Transactions on Graphics (TOG), vol. 37, no. 4, pp. 1–11, 2018. In this method, the user gives some viewpoints in advance, and then the solver generates a smooth trajectory connecting the viewpoints. Another more intuitive way is to interact by specifying the landmarks to be photographed on the map. The main disadvantage of this method is that the generated video cannot be directly presented to the audience, and the set key frames need to be adjusted multiple times to change the expression of the video, which is not user-friendly.There are some real-time interactive methods that attempt to solve this problem, such as by giving shooting instructions or curves in "Q. Galvane, J. Fleureau, F.-L. Tariolle, and P. Guillotel, "Automated cinematography with unmanned aerial vehicles," arXiv preprint arXiv:1712.04353, 2017". There are also some methods that provide interactive interfaces on the real-time images returned by the drone, which is undoubtedly more intuitive and effective. Most of the methods mentioned above are only used for shooting static known scenes, and the design of the algorithm often ignores unknown obstacles that may appear in the environment. In addition, the targets being photographed are mostly static objects, and these methods are not suitable for shooting dynamic targets.

[0006] Some existing methods can track and shoot dynamic objects. The current cutting-edge drone target tracking trajectory tracking can maintain visibility of the target in known or unknown environments, thereby increasing the robustness of tracking. However, for general tracking algorithms, these methods only maintain a suitable distance from the target, but cannot control the viewing angle with the target, the position of the target on the screen, and other factors. Therefore, this type of tracking algorithm is still a certain distance away from the needs of aerial photography.

[0007] At present, there are also some methods that add photographic elements to the real-time tracking algorithm. Most of them adopt the model predictive control (MPC) method. This method optimizes the drone state and control input in discretized time, and adds factors such as picture position, size, and viewing angle to the optimization items to calculate the optimal drone motion trajectory. However, most of these methods have one of the following limitations: 1. The modeling of the environment is too simple. Generally, obstacles in the environment are simplified as ellipsoids. This approach may be too conservative in unstructured scenarios; 2. The motion planning of the drone and the gimbal is relatively rough. For example, the inclination angle during movement is not considered, and only image-based feedback control is used. This approach will cause the model to be inaccurate when the drone is maneuvering rapidly, thereby affecting the camera's shooting effect on the target. Summary of the invention

[0008] In view of the shortcomings of the prior art, the purpose of the embodiments of the present application is to provide an interactive drone autonomous aerial photography method and device, so as to improve the interactivity and robustness of the drone aerial photography system, so that the drone can capture the images desired by the user while ensuring safety and dynamic feasibility.

[0009] According to a first aspect of an embodiment of the present application, an interactive drone autonomous aerial photography method is provided, which is applied to a drone execution end of an interactive drone autonomous aerial photography system, including:

[0010] Obtaining a shooting instruction given by a user at a ground station interactive terminal of the interactive UAV autonomous aerial photography system;

[0011] According to the shooting instruction, a dynamic path search of the aerial photography path is performed with the goal of minimizing the loss of shooting effect;

[0012] Performing a local visible area adjustment on the searched dynamic path to obtain an optimal observation area for each search node in the dynamic path;

[0013] A multi-objective optimization problem is constructed by combining the state of the drone and the gimbal rotation angle, and the multi-objective optimization problem is solved to achieve trajectory optimization;

[0014] Control the drone to move to the end point of the trajectory according to the optimized trajectory and take aerial photos.

[0015] Furthermore, the shooting instruction includes a shooting target, a shooting angle of view, a shooting distance and a transition time.

[0016] Furthermore, the hybrid A* method is used to perform dynamic path search of the aerial photography path, and the process includes:

[0017] A node is established according to the initial motion state and time of the drone, and discrete control input is applied as sampling to expand to a series of child nodes. The expanded child nodes are added to a priority queue, and each time the node with the smallest sum of the cost function and the heuristic function is selected from the priority queue as the next expanded node. When the time of the expanded node reaches time T, the search ends and a path with time information is obtained.

[0018] Furthermore, the cost function is

[0019]

[0020] Among them, d n is the distance between the search node and the target, D n is the expected distance between the search node and the target, θ n is the relative angle between the search node and the target, Θ n is the expected angle between the search node and the target, θ n is the height direction line of sight angle between the search node and the target, Θ n is the corresponding inclination angle between the search node position and the target position on the screen; d , θ , is the corresponding weight coefficient.

[0021] Furthermore, the heuristic function is

[0022] h(n)=Tt n

[0023] Where T is the total time, t n is the time of the current node.

[0024] Furthermore, the local visible area of ​​the searched dynamic path is adjusted, including:

[0025] Determine the adjustable area based on the maximum speed of the drone;

[0026] Dividing the adjustable area into a plurality of annular sector-shaped sub-areas;

[0027] Calculate the occlusion loss of each region based on the median distance and median angle of each sub-region;

[0028] According to the sub-region with the smallest occlusion loss, a visible region for constraining the position of the drone is obtained.

[0029] Furthermore, the multi-objective optimization problem is

[0030] J=Σ i J i +J smooth

[0031] J i =max(G i (c,T),0) 3

[0032] For the inequality constraint G i (c,T)<0,J i is the penalty function for each constraint violation, J smooth is the smoothness of the trajectory.

[0033] According to a second aspect of an embodiment of the present application, an interactive drone autonomous aerial photography device is provided, which is applied to a drone execution end of an interactive drone autonomous aerial photography system, including:

[0034] An acquisition module, used to acquire a shooting instruction given by a user at a ground station interaction terminal of the interactive UAV autonomous aerial photography system;

[0035] A search module, configured to perform a dynamic path search for an aerial photography path according to the shooting instruction with the goal of minimizing the loss of shooting effect;

[0036] An adjustment module, used for adjusting the local visible area of ​​the searched dynamic path;

[0037] An optimization module is used to construct a multi-objective optimization problem by combining the drone state and the gimbal rotation angle, and solve the multi-objective optimization problem to achieve trajectory optimization;

[0038] The aerial photography module is used to control the drone to move to the end point of the trajectory according to the optimized trajectory and perform aerial photography.

[0039] According to a third aspect of an embodiment of the present application, there is provided an electronic device, including:

[0040] one or more processors;

[0041] A memory for storing one or more programs;

[0042] When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in the first aspect.

[0043] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which computer instructions are stored, and when the instructions are executed by a processor, the steps of the method described in the first aspect are implemented.

[0044] The technical solution provided by the embodiments of the present application may have the following beneficial effects:

[0045] It can be seen from the above embodiments that the present application is based on real-time user interaction, an interactive interface is run on the ground-end device, images are fed back in real time through the wireless network, shooting requirements are collected through intuitive elements, and are immediately sent to the shooting drone. The drone makes real-time planning to reflect the image requirements in itself and the movement of the onboard gimbal; a hierarchical planning method is proposed, which can not only effectively ensure safety during the shooting process, but also combines the dynamic characteristics of the drone during trajectory optimization, and can keep the target stable on the shooting screen even in fast-moving scenarios.

[0046] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0048] Figure 1 The present invention is a flow chart of an interactive drone autonomous aerial photography method according to an exemplary embodiment.

[0049] Figure 2 It is a schematic diagram of a real-time interactive interface according to an exemplary embodiment.

[0050] Figure 3 It is a block diagram of an interactive drone autonomous aerial photography device according to an exemplary embodiment.

[0051] Figure 4 The diagram is a schematic diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0052] Here, exemplary embodiments are described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application.

[0053] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0054] It should be understood that although the terms first, second, third, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0055] Figure 1 is a flow chart of an interactive drone autonomous aerial photography method according to an exemplary embodiment. Figure 1 As shown, the method is applied to a drone execution end of an interactive drone autonomous aerial photography system, and may include the following steps:

[0056] Step S11: obtaining a shooting instruction given by a user at the ground station interaction terminal of the interactive UAV autonomous aerial photography system;

[0057] Step S12: According to the shooting instruction, a dynamic path search of the aerial photography path is performed with the goal of minimizing the loss of shooting effect;

[0058] Step S13: adjusting the local visible area of ​​the searched dynamic path;

[0059] Step S14: constructing a multi-objective optimization problem based on the state of the drone and the gimbal rotation angle, and solving the multi-objective optimization problem to achieve trajectory optimization;

[0060] Step S15: Control the UAV to move to the end point of the trajectory according to the optimized trajectory and take aerial photos.

[0061] It can be seen from the above embodiments that the drone in the present application can track and shoot the target in real time and safely according to the user's shooting instructions. Based on real-time user interaction, the device on the ground runs an interactive interface, and the image is fed back in real time through the wireless network. The shooting requirements are collected through intuitive elements and immediately sent to the shooting drone. The drone makes real-time planning to reflect the image requirements in itself and the movement of the onboard gimbal; a hierarchical planning method is proposed, which can not only effectively ensure the safety during the shooting process, but also combine the dynamic characteristics of the drone during trajectory optimization, so that the target can still be kept stable on the shooting screen even in fast-moving scenarios.

[0062] In the specific implementation of step S11, a shooting instruction given by a user at the ground station interaction terminal of the interactive UAV autonomous aerial photography system is obtained;

[0063] Specifically, the aerial photography system of the present invention is divided into two parts: the ground station interaction end and the drone execution end. At the interaction end, the user gives a shooting instruction for the target, and the shooting instruction includes the position of the shooting target on the image, the shooting angle, the shooting distance and the transition time. The instruction is expressed as a set of parameters, specifically the pixel coordinate position P image , angle Θ, distance D, time τ. This set of parameters is sent to the drone via a wireless network; the drone side plans the target's shooting trajectory in real time. When receiving the user's needs, the planning layer re-plans according to the changed shooting parameters and adjusts to the shooting state the user wants. The advantage of this design is that the user can intuitively give the shooting instructions on the image and achieve the desired shooting mode without touching any movement of the drone.

[0064] Specifically, the specific form of aerial photography interaction in the present invention is that the user Figure 2 The interface shown in the figure is used for operation. The left half of the interface displays the drone shooting picture in real time, and the user can drag the box on the picture. The position of the box reflects the user's desired target picture position. The button for adjusting the shooting angle is located above the right half, and the slider below gives the shooting distance and transition time respectively. When there is no new instruction, the drone plans the trajectory in real time to maintain the current shooting state of the target. When the user gives a new aerial photography instruction in the interface, the drone plans its movement to safely and smoothly transition to the desired shooting state.

[0065] The autonomous aerial photography motion planning in this application plans the movement of the drone in a hierarchical manner from coarse to fine. Its innovations mainly lie in three aspects: 1. Dynamic path search of the aerial photography path 2. Local visible area adjustment 3. Trajectory optimization combined with the drone status and gimbal angle.

[0066] In the specific implementation of step S12, according to the shooting instruction, a dynamic path search of the aerial photography path is performed with the goal of minimizing the loss of shooting effect;

[0067] The innovation of the path search method in the present invention is that, according to the shooting requirements of a given series of moments, the search is carried out with the goal of minimizing the loss of shooting effect. This method improves the previous hybrid A* method. In the algorithm, each search node records the robot state, moment and the corresponding target predicted position. Each search node is assigned a cost function g(n) to measure the overall cost and an heuristic function h(n) to estimate the remaining cost of the distance convergence condition. The cost function of each node is defined as the loss of shooting effect, which is defined in three aspects: a. Distance, which is the distance d from the search node position to the target n and the expected distance D n The difference between the search node and the target. b. The viewing angle is the relative angle θ between the search node and the target. n The angle θ with the expected n c. Inclination, which is the height line of sight inclination between the search node and the target and the desired goal Φ n The cost function is obtained by weighted addition of the above three factors.

[0068]

[0069] In the above formula, the weight value λ d , θ , Adjust the size of different elements. Generally, the coefficients are given according to the following rules: According to the visual perception of the photographer, a shooting distance of 1m is equivalent to a viewing angle of 0.3rad, which is also equivalent to an inclination of 0.1rad, so the coefficient λ is given d =1,λ θ =3.3, It can also be adjusted while in use.

[0070] Different from the general path search that uses the end point as the end of the search, the search process of the present invention stops when the time reaches the end moment. Therefore, the heuristic function h(n) is defined as the time difference between the moment corresponding to the current node and the end moment to speed up the search process.

[0071] h(n)=Tt n

[0072] Where T is the termination time, t n is the time of the current node.

[0073] The entire search process is as follows: establish nodes according to the initial motion state and time, apply discrete control input as sampling, expand to a series of child nodes, add the expanded child nodes to a priority queue, and each time select the node with the smallest sum of g(n) and h(n) in the priority queue as the next expanded node. When the time of the expanded node reaches time T, the search ends and returns a path with time information.

[0074] In the specific implementation of step S13, the local visible area adjustment is performed on the searched dynamic path;

[0075] The specific content of the visible area selection in the present invention is to search for the best observation area near each search node (i.e., the node expanded in the above search process). The best observation area improves the sampling accuracy and shooting effect in a local search manner, reduces the impact of occlusion, and constructs constraints for the next step of trajectory optimization. The aforementioned path search gives a dynamic path, but usually due to insufficient sampling, the local optimal solution cannot be obtained. In this step, local optimization is performed again, and a certain space margin is given in the form of a feasible area in the subsequent optimization. Specifically, given the initial position obtained by the path search, the adjustable area is first determined around the maximum speed of the drone. Since distance and viewing angle are two important factors affecting shooting, the adjustable area is divided into several annular fan-shaped sub-areas. Each candidate sub-area therefore represents a certain range of viewing angles and distances. The median distance and angle of each area are taken as a measure, and the difference from the expected distance and angle is also considered as the loss of the area. In addition, in order to give the drone better visibility, candidate areas closer to the occluded area are given a greater loss. Loss function Q i Defined as:

[0076]

[0077] The above formula represents the cost function of the i-th region, where θ occ represents the viewing angle blocked by obstacles. d and λ θ Same as the previous step, λ occ You can choose a larger value such as 10 to better avoid occlusion.

[0078] Finally, we can get a region with the least loss among all candidate sub-regions. In order to facilitate the representation and subsequent optimization, this method approximates the region in the form of a pentagon, thereby obtaining an optimal visible region within a local range. The significance of this step is that 1. Due to the insufficient sampling of the mixed a*, it is difficult to ensure the optimal local shooting quality. The generation of the visible region improves the quality of the trajectory 2. The spatiotemporal constraints are expressed in the form of convex polygons such as pentagons, which provides explicit constraints for trajectory optimization 3. Constraints in the form of regions provide a certain margin, which is conducive to better convergence during optimization.

[0079] In the specific implementation of step S14, a multi-objective optimization problem is constructed by combining the state of the drone and the gimbal rotation angle, and the multi-objective optimization problem is solved to achieve trajectory optimization;

[0080] The specific content of the aerial photography trajectory planning in the present invention is to define the trajectory optimization as a multi-objective optimization problem, and add the variable violation constraint as a penalty function to the objective function J for optimization.

[0081] J=J smooth +Σ i J i

[0082] J i =max(G i (c,T),0) 3

[0083] J smooth is the smoothness of the trajectory, for the inequality constraint G i (c,T)<0, calculate the penalty function J for each constraint violation i , and then sum.

[0084] This method uses a polynomial trajectory with time and space information for characterization. This method can determine a trajectory through intermediate points and time through linear complexity operations. This method uses polynomial trajectories to represent the robot's spatial position (3D) p(t) and two-axis rotation θ(t) (2D, yaw angle and gimbal angle) respectively. This method expresses the trajectory optimization problem as a multi-objective optimization problem. This problem samples in time and adds the variables that violate the constraints to the objective function in the form of a penalty function in the hope of obtaining the optimal trajectory that meets a series of constraints. The objective functions of this method are divided into five categories:

[0085] 1. The smoothness of the trajectory itself, which can be directly calculated by integrating the fourth-order derivative of position and the second-order derivative of rotation;

[0086]

[0087] 2. Dynamic constraints. According to the differential flatness of the drone (expressed as Ψ in the following formula), the state and input of the drone (r(t) in the following formula) can be obtained from the trajectory variables and their differentials, such as speed, acceleration and angular velocity. The dynamic constraints require that these states cannot exceed a certain range r max , dynamic constraint G dynamic as follows:

[0088] G dynamic =r 2 (t)-r m 2 ax

[0089] r(t)=Ψ(p(t),θ(t))

[0090] 3. Safety constraints. First, the position of the drone itself needs to be constrained within the safety convex polyhedron in space. The safety convex polyhedron can be expressed as {x∈R 3 | f xb f ≤0}, secondly, the drone needs to perceive obstacles in the environment in real time. The drone hardware design in the present invention uses the forward-connected depth camera as the data source for building the map. Therefore, in trajectory optimization, the projection of the drone's speed in the horizontal direction and the direction of the drone's depth camera are constrained to be less than a certain threshold;

[0091] G safety_ =A f p(t)-b f

[0092]

[0093] Among them, G safety_ represents the position constraint of the safe convex polyhedron, A f and b f The coefficient matrix, G, representing the safe convex polyhedron safety_ represents the constraints of the perceived environment, θ v The direction angle of the velocity;

[0094] 4. Shooting constraints. In the above visible area selection step, the shooting position of the drone at some time has been constrained. Since the visible area is a pentagon, it can also be expressed as a convex constraint {x∈R 3 | v xb v ≤0}In addition, in order to achieve the composition of the captured image, the position and speed of the target on the image are also optimized as the objective function. Using the imaging principle of the pinhole camera model, the projection position and speed of the target on the image can be obtained according to the attitude and angular velocity of the drone and the rotation angle and angular velocity of the camera gimbal. Represents the camera projection function, and the desired image position and velocity The corresponding optimization item can be obtained by performing norm operation. In practice, the present invention also smoothes this item through a sigmoid function σ().

[0095] G view =A v p(t)-b v

[0096] G image =σ(||PP image || 2 +||VV image || 2 )

[0097] Among them, G view represents the position constraint of the visible area in step S13, A v and b v The coefficient matrix of the convex polyhedron in the visible area, G image Indicates the position constraint of the target on the screen;

[0098] The multi-objective solution of this method uses L-BFGS.

[0099] In the specific implementation of step S15, the drone is controlled to move to the end point of the trajectory according to the optimized trajectory and perform aerial photography;

[0100] After obtaining the optimized trajectory, the control module of the drone reflects the trajectory information into the speed of the motor, thereby controlling the drone to the state on the trajectory in real time. Since the user's needs for static and transitional state image shooting are taken into account during trajectory optimization, the system of this method realizes the conversion from the user's shooting needs to the drone movement.

[0101] The present invention has made improvements and innovations in path search, area search and trajectory optimization. On the basis of considering the dynamic characteristics of the drone, it searches for appropriate spatiotemporal constraints according to the aerial photography requirements, and considers a more sophisticated drone kinematic model for image-level optimization.

[0102] Method 1 (the interactive drone aerial photography method disclosed in this application) is compared with a similar method 2 (《T. Nageli, J. Alonso-Mora, A. Domahidi, D. Rus, and O. Hilliges, “Real-time motion planning for aerial videography with dynamic obstacle avoidance and viewpoint optimization,” IEEE Robotics and Automation Letters, vol. 2, no. 3, pp. 1696–1703, 2017》) given the same shooting requirements. Under different numbers of obstacles, the computational efficiency of this method is higher than that of the compared methods. The robustness comparison of planning shows that this method can maintain a higher success rate, with lower target loss of vision, occlusion and collision.

[0103] Comparison of calculation time between Table 1 and Method 2 (unit: ms)

[0104]

[0105] Comparison of computational efficiency under different obstacle densities

[0106] The method of the present invention is also superior to method 2 in the property of keeping the target in the picture position.

[0107] Table 2 Robustness comparison with method 2 (unit: %)

[0108]

[0109] Corresponding to the aforementioned embodiment of the interactive drone autonomous aerial photography method, the present application also provides an embodiment of an interactive drone autonomous aerial photography device.

[0110] Figure 3 is a block diagram of an interactive drone autonomous aerial photography device according to an exemplary embodiment. Figure 3 The device is applied to a drone execution end of an interactive drone autonomous aerial photography system, and may include:

[0111] An acquisition module 21 is used to acquire a shooting instruction given by a user at a ground station interaction terminal of the interactive UAV autonomous aerial photography system;

[0112] A search module 22, configured to perform a dynamic path search for an aerial photography path according to the shooting instruction with the goal of minimizing the loss of shooting effect;

[0113] An adjustment module 23, used for adjusting the local visible area of ​​the searched dynamic path;

[0114] The optimization module 24 is used to construct a multi-objective optimization problem by combining the drone state and the gimbal rotation angle, and solve the multi-objective optimization problem to achieve trajectory optimization;

[0115] The aerial photography module 25 is used to control the UAV to move to the end point of the trajectory according to the optimized trajectory and perform aerial photography.

[0116] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0117] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application scheme. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0118] Accordingly, the present application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned interactive drone autonomous aerial photography method. Figure 4 As shown, it is a hardware structure diagram of any device with data processing capability in which a deep learning data set access system provided by an embodiment of the present invention is located, except Figure 4 In addition to the processor, memory, DMA controller, disk, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiment is located may also include other hardware, which will not be described in detail, generally based on the actual functions of the device with data processing capabilities.

[0119] Correspondingly, the present application also provides a computer-readable storage medium on which computer instructions are stored, characterized in that when the instructions are executed by the processor, the interactive drone autonomous aerial photography method as described above is implemented. The computer-readable storage medium can be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium can also be an external storage device of a wind turbine, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), an SD card, a flash card (FlashCard), etc. equipped on the device. Further, the computer-readable storage medium can also include both an internal storage unit and an external storage device of any device with data processing capabilities. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and can also be used to temporarily store data that has been output or is to be output.

[0120] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the contents disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application.

[0121] It should be understood that the present application is not limited to the exact construction that has been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof.

Claims

1. An interactive drone autonomous aerial photography method, characterized in that: The UAV execution end applied to an interactive UAV autonomous aerial photography system includes: Obtaining a shooting instruction given by a user at a ground station interactive terminal of the interactive UAV autonomous aerial photography system; According to the shooting instruction, a dynamic path search of the aerial photography path is performed with the goal of minimizing the loss of shooting effect; Performing a local visible area adjustment on the searched dynamic path to obtain an optimal observation area for each search node in the dynamic path; A multi-objective optimization problem is constructed by combining the state of the drone and the gimbal rotation angle, and the multi-objective optimization problem is solved to achieve trajectory optimization; Control the drone to move to the end of the trajectory according to the optimized trajectory and take aerial photos; The multi-objective optimization problem is J=∑ i I i +J smooth J i =max(G i (c,T),0) 3 For the inequality constraint G i (c,T)<0,J i is the penalty function for each constraint violation, J smooth is the smoothness of the trajectory; The constraints include: Dynamic Constraints r(t)=Ψ(p(t),θ(t)), Ψ is the differential flatness characteristic of the UAV, r is the state of the UAV, r max is the maximum value of r, p is the position of the drone, and θ is the rotation of the drone, i.e. the yaw angle and the gimbal angle; Safety constraints, including the position constraints G of the safety convex polyhedron safety_poly =A f p(t)-b f , sensing the constraints of the environment A f and b f The coefficient matrix, θ, representing the safe convex polyhedron v The direction angle of the velocity; Shooting constraints, including position constraints of the visible area G view =A v p(t)-b v , the position constraint G of the target on the screen image =σ(||PP image || 2 +||VV image || 2 , A v and b v Represents the coefficient matrix of the convex polyhedron in the visible area, σ() is the sigmoid function, is the desired picture position and speed, is the projection position and speed of the target on the image, Represents the camera projection function.

2. The method according to claim 1, characterized in that The shooting instruction includes a shooting target, a shooting angle of view, a shooting distance and a transition time.

3. The method according to claim 1, characterized in that The hybrid A* method is used to perform dynamic path search for aerial photography paths. The process includes: A node is established according to the initial motion state and time of the drone, and discrete control input is applied as sampling to expand to a series of child nodes. The expanded child nodes are added to a priority queue, and each time the node with the smallest sum of the cost function and the heuristic function is selected from the priority queue as the next expanded node. When the time of the expanded node reaches time T, the search ends and a path with time information is obtained.

4. The method according to claim 3, characterized in that: The cost function is Among them, d n is the distance between the search node and the target, D n is the expected distance between the search node and the target, θ n is the relative angle between the search node and the target, Θ n is the expected angle between the search node and the target, is the height direction line of sight angle between the search node and the target, Φ n is the corresponding inclination angle between the search node position and the target position on the screen; d , θ , is the corresponding weight coefficient.

5. The method according to claim 3, characterized in that: The heuristic function is h(n)=T-t n Where T is the total time, t n is the time of the current node.

6. The method according to claim 1, characterized in that Adjust the local visible area of ​​the searched dynamic path, including: Determine the adjustable area based on the maximum speed of the drone; Dividing the adjustable area into a plurality of annular sector-shaped sub-areas; Calculate the occlusion loss of each region based on the median distance and median angle of each sub-region; According to the sub-region with the smallest occlusion loss, a visible region for constraining the position of the drone is obtained.

7. An interactive drone autonomous aerial photography device, characterized in that: The UAV execution end applied to an interactive UAV autonomous aerial photography system includes: An acquisition module, used to acquire a shooting instruction given by a user at a ground station interaction terminal of the interactive UAV autonomous aerial photography system; A search module, configured to perform a dynamic path search for an aerial photography path according to the shooting instruction with the goal of minimizing the loss of shooting effect; An adjustment module, used for adjusting the local visible area of ​​the searched dynamic path; An optimization module is used to construct a multi-objective optimization problem by combining the drone state and the gimbal rotation angle, and solve the multi-objective optimization problem to achieve trajectory optimization; Aerial photography module, used to control the drone to move to the end point of the trajectory according to the optimized trajectory and take aerial photos; The multi-objective optimization problem is J=∑ i I i +J smooth J i =max(G i (c,T),0) 3 For the inequality constraint G i (c, t)<0, J i is the penalty function for each constraint violation, J smooth is the smoothness of the trajectory; The constraints include: Dynamic Constraints r(t)=Ψ(p(t),θ(t)), Ψ is the differential flatness characteristic of the UAV, r is the state of the UAV, r max is the maximum value of r, p is the position of the drone, and θ is the rotation of the drone, i.e. the yaw angle and the gimbal angle; Safety constraints, including the position constraints G of the safety convex polyhedron safety_poly =A f p(t)-b f , sensing the constraints of the environment A f and b f The coefficient matrix, θ, representing the safe convex polyhedron v The direction angle of the velocity; Shooting constraints, including position constraints of the visible area G view =A v p(t)-b v , the position constraint G of the target on the screen image =σ(||PP image || 2 +||VV image || 2 , A v and b v Represents the coefficient matrix of the convex polyhedron in the visible area, σ() is the sigmoid function, is the desired picture position and speed, is the projection position and speed of the target on the image, Represents the camera projection function.

8. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instruction is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images