UAV visual navigation strategy method, device and medium

Abstract

The present application discloses a method, device and medium for a visual navigation strategy of a drone, the method comprising: generating a navigation image data set and a control data set, collecting the navigation image data set and the control data set; calculating a multimodal objective function model according to the navigation image data set and the control data set; mapping the navigation image data set to a low-dimensional latent space through a variational encoder, updating the multimodal objective function model according to the output of a decoder; mapping the low-dimensional latent space to the control data set, and obtaining a drone control command according to the multimodal objective function model; and controlling the drone according to the drone control command. The present application can significantly improve the performance of the control strategy, reduce the impact of the difference between virtual and real environments on autonomous navigation, and improve the robustness and generalization of strategy migration.

Description

Technical Field

[0001] The present application relates to the field of autonomous navigation of unmanned aerial vehicles, and in particular to a method, device and medium for visual navigation strategy of unmanned aerial vehicles. Background Art

[0002] Drones generally fly in open airspace. When faced with perception and control problems in open airspace (such as pure visual navigation), the solutions adopted by related technologies include the use of machine learning technologies, such as imitation learning and reinforcement learning, to solve the perception and control problems in open airspace. However, the relevant technology machine learning process has great difficulties in collecting real labeled data, which makes it difficult to obtain real labeled data, unable to adapt to real scenes, and difficult to achieve safe navigation applications. In addition, due to the low sample efficiency in related technologies, it is difficult to solve tasks using only real data, and often results in the inability of strategies to handle large perception changes.

[0003] Therefore, the above-mentioned technical problems existing in the related technology need to be solved urgently. Summary of the invention

[0004] The present application aims to solve one of the technical problems in the related art. To this end, the present application embodiment provides a method, device and medium for a drone visual navigation strategy, which can realize autonomous visual navigation of a drone.

[0005] According to one aspect of the embodiment of the present application, a drone visual navigation strategy method is provided, the method comprising:

[0006] Generate a navigation image data set and a control data set, and collect the navigation image data set and the control data set;

[0007] Calculating a multimodal objective function model according to the navigation image data set and the control data set;

[0008] Mapping the navigation image dataset to a low-dimensional latent space through a variational encoder, and updating the multimodal objective function model according to the output of the decoder;

[0009] Mapping the low-dimensional latent space to the control data set, and obtaining the drone control command according to the multimodal objective function model;

[0010] The drone is controlled according to the drone control command.

[0011] In one embodiment, generating a navigation image data set and a control data set, and collecting the navigation image data set and the control data set include:

[0012] Establish the world coordinate system, NED coordinate system, body coordinate system, obstacle coordinate system and camera coordinate system;

[0013] Generate and collect image information, attitude information and kinematic information of the UAV during operation;

[0014] The collected sample set is preprocessed.

[0015] In one embodiment, generating and collecting image information, attitude information, and kinematic information of a drone during operation includes:

[0016] Randomly select the position information and posture information of the body coordinate system relative to the world coordinate system;

[0017] The posture information is converted into Euler angles, and the conversion formula is:

[0018]

[0019]

[0020]

[0021] Among them, q x is the Euler angle of the x-axis of the world coordinate system, q y is the Euler angle of the posture information on the y-axis of the world coordinate system, q z is the Euler angle of the z-axis of the world coordinate system, is the rolling angle range of the body coordinate system relative to the world coordinate system, is the pitch angle range of the body coordinate system relative to the world coordinate system, is the yaw range of the body coordinate system relative to the world coordinate system.

[0022] In one embodiment, generating and collecting image information, attitude information and kinematic information of the drone during operation also includes:

[0023] The navigation image dataset is obtained by randomly sampling the navigation image.

[0024] The rolling angle range of the door coordinate system relative to the body coordinate system is calculated using the following formula:

[0025]

[0026] Among them, the is the rolling angle range of the door coordinate system relative to the body coordinate system, α prime is the pitch angle of the door coordinate system relative to the body coordinate system, Correction c Correction parameters for the camera.

[0027] In one embodiment, calculating the multimodal objective function model according to the navigation image dataset and the control dataset includes:

[0028] The data modes of the navigation image data set and the control data are defined as the relative posture of the RGB image and the drone, wherein the RGB image is a first-person visual RGB image;

[0029] Processing the RGB data into normal RGB data by the variational encoder;

[0030] The multimodal objective function model is calculated according to the normal RGB data, and the calculation formula includes:

[0031]

[0032] Among them, D KL is the sample data, p(x) and q(x) are the encoder variables.

[0033] In one embodiment, mapping the low-dimensional latent space to the control data set and obtaining the drone control command according to the multimodal objective function model includes:

[0034] Generate collision-free paths through RTT algorithm;

[0035] Pruning the collision-free path to a minimum set of waypoints, and jointly optimizing a set of polynomial segment sequences to connect the waypoints in the minimum set of waypoints into a smooth minimum capture trajectory from a start to a target;

[0036] The UAV control command is obtained according to the minimum capture trajectory.

[0037] In one embodiment, the expression of the minimum capture trajectory is:

[0038]

[0039] Among them, p(t) is the minimum capture trajectory, p0,p1,…,p n is the trajectory parameter, p=[p0,p1,…,p n ] T is the parameter vector.

[0040] According to one aspect of an embodiment of the present application, a drone visual navigation strategy device is provided, the device comprising:

[0041] A data set module, used to generate a navigation image data set and a control data set, and collect the navigation image data set and the control data set;

[0042] A calculation module, used for calculating a multimodal objective function model according to the navigation image data set and the control data set;

[0043] A mapping module, used to map the navigation image dataset to a low-dimensional latent space through a variational encoder, and update the multimodal objective function model according to the output of the decoder;

[0044] A control module, configured to map the low-dimensional latent space to the control data set and obtain a UAV control command according to the multimodal objective function model;

[0045] The drone is controlled according to the drone control command.

[0046] According to one aspect of an embodiment of the present application, a drone visual navigation strategy device is provided, the device comprising:

[0047] at least one processor;

[0048] at least one memory, the memory being used to store at least one program;

[0049] When at least one of the programs is executed by at least one of the processors, a drone visual navigation strategy method as described in the previous embodiment is implemented.

[0050] According to one aspect of an embodiment of the present application, a storage medium is provided, wherein the storage medium stores a program executable by a processor, and when the program executable by the processor is executed by the processor, the drone visual navigation strategy method as described in the previous embodiment is implemented.

[0051] The beneficial effects of a drone visual navigation strategy method provided by an embodiment of the present application are as follows: the present application generates a navigation image dataset and a control dataset, collects the navigation image dataset and the control dataset; calculates a multimodal objective function model based on the navigation image dataset and the control dataset; maps the navigation image dataset to a low-dimensional latent space through a variational encoder, and updates the multimodal objective function model based on the output of a decoder; maps the low-dimensional latent space to the control dataset, and obtains a drone control command based on the multimodal objective function model; and controls the drone according to the drone control command, which can significantly improve the performance of the control strategy, reduce the impact of environmental differences on autonomous navigation, and improve the robustness and generalization of strategy migration.

[0052] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0054] Figure 1 A schematic diagram of the principle of a drone visual navigation strategy method provided in an embodiment of the present application;

[0055] Figure 2 A flowchart of a drone visual navigation strategy method provided in an embodiment of the present application;

[0056] Figure 3 A flow chart of a method for generating and collecting the navigation image data set and the control data set in an embodiment of the present application;

[0057] Figure 4 A schematic diagram of a drone visual navigation strategy device provided in an embodiment of the present application;

[0058] Figure 5 A schematic diagram of another drone visual navigation strategy device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0059] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0060] The terms "first", "second", "third" and "fourth" etc. in the specification, claims and drawings of the present application are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0061] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0062] Drones generally fly in open airspace. When faced with perception and control problems in open airspace (such as pure visual navigation), the solutions adopted by related technologies include the use of machine learning technologies, such as imitation learning and reinforcement learning, to solve the perception and control problems in open airspace. However, the relevant technology machine learning process has great difficulties in collecting real labeled data, which makes it difficult to obtain real labeled data, unable to adapt to real scenes, and difficult to achieve safe navigation applications. In addition, due to the low sample efficiency in related technologies, it is difficult to solve tasks using only real data, and often results in the inability of strategies to handle large perception changes.

[0063] In order to solve the above problems, the present application proposes a UAV visual navigation strategy method, device and medium, as follows.

[0064] For ease of understanding, this manual explains the professional terms and nouns that may be mentioned, as follows:

[0065] NED coordinates: NED (North East Down) coordinate system, also known as the North East Down coordinate system, referred to as the N coordinate system, also known as the navigation coordinate system, is a reference coordinate system selected for navigation solution according to the needs of the navigation system during navigation.

[0066] Euler angles: A set of three independent angular parameters used to determine the position of a fixed-point rotating rigid body, consisting of the nutation angle θ, the precession angle (i.e., the precession angle) ψ, and the rotation angle φ. They are named after Euler, who first proposed them. The three Euler angles are asymmetric and have uncertainty in several special positions (when θ=0, φ and ψ are inseparable). For different problems, it is advisable to take different axes as basic axes and measure the Euler angles in different ways.

[0067] RTT algorithm: RTT (Render to Texture), that is, rendering to texture. In the ordinary graphics rendering process, the final result is rendered to the frame buffer and finally displayed on the screen. Then the texture can be applied to the scene drawing. For example, rendering a scene A to the texture, and pasting the texture on another scene B is like playing A. The same is true for the scenes in various mirrors. Shadow mapping can also be regarded as the use of RTT.

[0068] Minimun Snap planning: When finding the polynomial parameters p1,...,pkp1,...,pk of the trajectory, the trajectory satisfies a series of constraints, such as: the position, speed or acceleration of the start and end points, the connection between adjacent trajectories is expected to be smooth (position continuity, speed continuity, etc.), the trajectory is expected to pass through certain path points, the maximum speed, maximum acceleration, etc., or even the trajectory is expected to be within a specified space (corridor), etc. Usually there are countless trajectories that meet the constraints, but in practical problems, a specific trajectory is often required, so it is necessary to construct an optimal function to find the "optimal" specific trajectory among the feasible trajectories.

[0069] This application proposes a UAV visual navigation strategy method. Figure 1 A schematic diagram of the principle of a drone visual navigation strategy method provided in an embodiment of the present application, such as Figure 1 As shown, the principle of this application is: to learn robust visual motion strategies for real-world deployment, and to use simulation data for training. Based on a cross-modal perspective, this method separates native image data and task-related states, such as gate posture data in drone competitions, into different modalities. The two data modalities are input into a network architecture, which can learn a low-dimensional joint representation through a variational autoencoder, and map the low-dimensional joint representation to drone control instructions through an imitation learning network to achieve visual autonomous navigation of the drone.

[0070] Figure 2 A flowchart of a drone visual navigation strategy method provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the UAV visual navigation strategy method proposed in this application specifically includes:

[0071] S201: Generate a navigation image data set and a control data set, and collect the navigation image data set and the control data set.

[0072] In step S201, to create an image data set, the image data set generation parameters are set as follows: the gate attitude parameter includes the self-yaw attitude range GATE_YAW_RANGE of the gate coordinate system in the world coordinate system The Euclidean distance range R_RANGE of the door coordinate system relative to the body coordinate system in the world coordinate system is The UAV attitude parameter information includes position information and attitude information. The position information includes the X-axis movement range UAV_X_RANGE of the body coordinate system relative to the world coordinate system. The Y-axis movement range of the body coordinate system relative to the world coordinate system UAV_Y_RANGE is The Z-axis movement range of the body coordinate system relative to the world coordinate system UAV_Z_RANGE is The attitude information includes the pitch angle range UAV_PITCH_RANGE of the body coordinate system relative to the world coordinate system. The roll angle range UAV_ROLL_RANGE of the body coordinate system relative to the world coordinate system is The yaw range UAV_YAW_RANGE of the body coordinate system relative to the world coordinate system is Set the camera parameters including the camera image acquisition range FOV to Degree c And camera correction parameters Correction c .

[0073] Specifically, the steps for generating image data are as follows:

[0074] Randomly generate the position and attitude information of the drone, including random sampling of the following parameters, namely: for the position information of the body coordinate system relative to the world coordinate system, randomly select For the attitude information of the body coordinate system relative to the world coordinate system, the rotation angle of the drone around the y-axis of the body coordinate system is randomly selected as The rotation angle of the drone around the x-axis of the body coordinate system is The rotation angle of the drone around the z-axis of the body coordinate system is Converting the attitude rotation information into Euler angles yields the following formula:

[0075]

[0076]

[0077]

[0078] Set the rotation order to Z, Y, X. The quaternion is derived as follows:

[0079]

[0080]

[0081]

[0082]

[0083] Get the drone's position in the world coordinate system Among them are

[0084] Generate door posture information and randomly sample some parameters, namely: the door displacement information relative to the body coordinate system, i.e., distance information, and take The angle information includes the pitch angle range relative to the body coordinate system.

[0085]

[0086] Pick is the pitch angle of the door coordinate system relative to the body coordinate system. Then the rolling angle range of the door coordinate system relative to the body coordinate system is

[0087]

[0088] Pick is the roll angle of the door coordinates relative to the body coordinate system.

[0089] S202: Calculate a multimodal objective function model according to the navigation image data set and the control data set.

[0090] Specifically, the calculation of the multimodal objective function model in step S202 specifically includes: defining the data mode as the RGB image and the relative posture of the door and the drone, such as X RGB =I t ,X G =Y i = [r, θ, φ, ψ], RGB data is obtained by encoder q RGB Processed into normal distribution N(μ t ,σ t ), press z t Sampling. Any data pattern can be used with decoder P RGB and P gate Recover from the latent space. A data sample x i (e.g. RGB data) is embedded into the latent space to obtain a vector z, which is used to reconstruct the data sample x t (e.g. RGB images or door poses). To achieve this, in the model Under this condition, we maximize our desired output mode x t The logarithmic probability of , where θ is the model parameter.

[0091] Similar to the original derivation, after transformation, we can get: log p(x t ) = ∫q(z|x i )logp(x t )dz, where ∫q(z|x i )dz=1, so it is expanded to and

[0092]

[0093] Split the above formula into integrals:

[0094]

[0095] Among them, p(x t |z) is the posterior probability, and q(z|x i )approximate.

[0096] For any distribution p,q, D KL (p(x)||q(x))≥0, we can get the final lower bound:

[0097]

[0098] Among them, q(z|x i ) is the encoder, and x i Embedded in latent space, p(x i |z) is a decoder that converts the potential sample z into the desired representation x t .

[0099] S203: Map the navigation image dataset to a low-dimensional latent space through a variational encoder, and update the multimodal objective function model according to the output of the decoder.

[0100] In step S203, the updating of the multimodal objective function model includes the following steps: generating the following reconstruction according to the encoder configuration:

[0101]

[0102] in The losses are as follows:

[0103]

[0104]

[0105] Total=Loss RGB +Loss Gate +Loss KL

[0106] The update module adjusts the model accordingly based on Total to adapt it to the current data flow environment.

[0107] S204: Map the low-dimensional latent space to the control data set, and obtain the UAV control command according to the multimodal objective function model.

[0108] In step S204, W is the world coordinate system, B is the body coordinate system, G is i The target door coordinate system, E, defines the complete environment geometry and object categories. Assuming that all doors are upright, let y i=[r,θ,φ,ψ], indicating G i Relative spherical coordinates and yaw under B.

[0109] definition As the encoding function, the image I t Mapped to a latent compressed vector z of size N t .definition For the control strategy, the current encoding state is mapped to the body speed command v B =[v x ,v y ,v z ,v ψ ] is the linear velocity and yaw velocity. Define π * For the expert control strategy, find the optimal model parameter Θ in the observable state s. * ,Φ * To minimize the expected distance D between our control policy and the expert control policy.

[0110] Through steps S201-S204, the position information of the door coordinate system in the body coordinate system can be obtained, namely:

[0111]

[0112]

[0113]

[0114] Then there is is the position vector of the origin of the door coordinate system in the body coordinate system, let is the attitude transfer matrix from the body coordinate system to the world coordinate system. Then the position vector of the origin of the gate coordinate system in the world coordinate system is

[0115]

[0116] In the gate coordinate system, the yaw angle relative to the Z axis of the gate coordinate system is set to ψ g , set the rotation according to the ZYX axis, and set the transformation matrix to Then the corresponding quaternion can be obtained as At this point, the pose information of the door coordinate system in the world coordinate system can be obtained

[0117] Optionally, this embodiment can be based on the simulation environment Set the drone’s position. Set the door posture, control the drone's FPV camera to take pictures, and at the same time Store them together to obtain an image sample set d i =(rgbi ,g i ), where rgb i represents RGB image, g i Represents the door posture, After N iterations, we can obtain a sample set D of size N. N =[d0,d1,…,d n ], this sample set is used to update the encoder model in this embodiment.

[0118] S205: Control the drone according to the drone control command.

[0119] In step S205, the obtained drone control command includes the control command of the drone flight speed, flight route and flight altitude. For example, the present application adopts the kinematics and control of a quadcopter drone for drone control. In order to accurately follow the polynomial trajectory to be generated, the differential flatness of the standard quadcopter motion equation is used, and the formula is as follows:

[0120]

[0121]

[0122] in, is the position vector of the body coordinate system in the world coordinate system, ω is the angular velocity vector in the body coordinate system, f and M are the net thrust and torque in the body coordinate system, respectively. J and m are the inertia tensor and mass of the quadcopter, respectively. B is a unit vector aligned with the four rotor axes and indicates the direction of thrust. w is a unit vector indicating the direction of gravity. For the desired speed of the quadrotor motors, there is a simple mapping relationship between f and M to the desired speed of the four motors. A polynomial trajectory segment consists of four polynomial time functions that specify the independent evolution of the so-called plane output variables x, y, z and ψ (deflection angle) between two states in the plane output space. The nonlinear controller used to track the differentiable trajectory consists of independent calculations of thrust and torque:

[0123]

[0124]

[0125] where e x ,e v ,e R ,e ω are the error matrices for position, velocity, heading and angular velocity respectively. x ,k v ,k R ,k ωRelated to the control gain. R is the rotation matrix representing the heading of the quadrotor.

[0126] It should be noted that after obtaining the control command, the flight trajectory of the UAV needs to be planned according to the control command, including: using an n-order polynomial to represent the flight trajectory of the UAV, that is,

[0127]

[0128] Among them, p0, p1, …, p n For trajectory parameters, set the parameter vector to p = [p0, p1, ..., p n ] T , then the trajectory vector is in the following form:

[0129] p(t)=[1,t,t 2 ,…,t n ]·p

[0130] By using Minimun Snap trajectory planning to generate the optimal trajectory, the drone control command is output according to the trajectory and the current state of the robot (current position, speed and acceleration), the drone is controlled to move along the trajectory, and the expert experience data set π is collected * =(X rgb ,X command ), where X rgb First-person visual RGB images collected for drones, X command is the control command output to the drone in a similar time, that is

[0131] X command =[v x ,v y ,v z ,ψ]

[0132] Figure 3 A flow chart of a method for generating and collecting the navigation image data set and the control data set in an embodiment of the present application, such as Figure 3 As shown, generating a navigation image data set and a control data set, and collecting the navigation image data set and the control data set include:

[0133] S301. Establish a coordinate system.

[0134] In step S301, the establishment of the coordinate system specifically includes: the world coordinate system, denoted as the w system: the position with the coordinates (0,0,0) in the venue is taken as the origin O w , the north direction is x w , the east direction is y w , the ground direction is z w; NED coordinate system, denoted as n system: NED coordinate system is the coordinate system used in navigation calculations. The vectors point to the north, east, and ground respectively, so the NED coordinate system is often called the "north-east-ground coordinate system"; the body coordinate system, denoted as b system: the body coordinate system is fixedly connected to the aircraft, the coordinate system conforms to the right-hand rule, and the origin is at the center of gravity of the aircraft. b , the x-axis points to the direction of the aircraft nose b , the y-axis points from the origin to the right side of the aircraft b , the z-axis direction is determined by the right-hand rule through x and y b The body coordinate system is the basic coordinate system of the drone's inertial navigation. The acceleration state information obtained from the IMU is the value in this coordinate system. When we obtain the x-axis acceleration information output by the IMU, we cannot directly apply it to the NED coordinate system; the obstacle coordinate system is recorded as the g system: the obstacle coordinate system is fixed to the obstacle, the coordinate system conforms to the right-hand rule, and the origin is at the center of gravity of the obstacle. g .. The x-axis points to due north b , the y-axis points to the east. b , the z-axis direction is determined by the right-hand rule through x and y b ; Camera coordinate system, denoted as c-system: the image principal point of the optical system is the origin O C ; When looking directly at the optical system, x c The y axis is parallel to the horizontal axis of the imaging plane coordinate system, with the left direction being positive; c The axis is parallel to the vertical axis of the imaging plane coordinate system, and is positive downward; c The x-axis points toward the observer and is c Axis and y c The axes form a right-handed coordinate system.

[0135] S302: Generate a data set.

[0136] S303: Preprocessing.

[0137] Optionally, in this embodiment, the multimodal objective function model is calculated according to the navigation image dataset and the control dataset, including: defining the data mode of the navigation image dataset and the control data as the relative posture of the RGB image and the drone, wherein the RGB image is a first-person visual RGB image; processing the RGB data into normal RGB data through the variational encoder; and calculating the multimodal objective function model according to the normal RGB data, wherein the calculation formula includes:

[0138]

[0139] Among them, D KL is the sample data, p(x) and q(x) are the encoder variables.

[0140] Optionally, in this embodiment, the low-dimensional latent space is mapped to the control data set, and the UAV control command is obtained according to the multimodal objective function model, including: generating a collision-free path through an RTT algorithm; pruning the collision-free path to a minimum path point set, and jointly optimizing a set of polynomial segment sequences, connecting the path points in the minimum path point set into a smooth minimum capture trajectory from the start to the target; and obtaining the UAV control command according to the minimum capture trajectory.

[0141] Optionally, the expression of the minimum capture trajectory in this embodiment is:

[0142]

[0143] Among them, p(t) is the minimum capture trajectory, p0,p1,…,p n is the trajectory parameter, p=[p0,p1,…,p n ] T is the parameter vector.

[0144] The beneficial effects of a drone visual navigation strategy method provided by an embodiment of the present application are: the present application can significantly improve the performance of the control strategy, reduce the impact of the difference between the virtual and real environments on autonomous navigation, and improve the robustness and generalization of strategy migration.

[0145] Figure 4 A schematic diagram of a drone visual navigation strategy device provided in an embodiment of the present application is shown in FIG. Figure 4 As shown, a drone visual navigation strategy device of the present application includes:

[0146] The data set module 401 is used to generate a navigation image data set and a control data set, and collect the navigation image data set and the control data set;

[0147] A calculation module 402, configured to calculate a multimodal objective function model according to the navigation image data set and the control data set;

[0148] A mapping module 403, used to map the navigation image dataset to a low-dimensional latent space through a variational encoder, and update the multimodal objective function model according to the output of the decoder;

[0149] The control module 404 is used to map the low-dimensional latent space to the control data set, obtain a drone control command according to the multimodal objective function model, and control the drone according to the drone control command.

[0150] It can be seen that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0151] Reference Figure 5 , the embodiment of the present application provides a drone visual navigation strategy device, including:

[0152] at least one processor 501;

[0153] At least one memory 502, used to store at least one program;

[0154] When at least one program is executed by at least one processor 501, the at least one processor 501 implements the drone visual navigation strategy method of the aforementioned embodiment.

[0155] Similarly, the contents of the above method embodiments are all applicable to the present device embodiments. The functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0156] An embodiment of the present invention further provides a storage medium, which stores a program. When the program is executed by a processor, it is used to implement the method of the above embodiment.

[0157] The contents of the above method embodiments are all applicable to the present storage medium embodiment, and the functions specifically implemented by the present storage medium embodiment are the same as those of the above method embodiments.

[0158] Similarly, the contents of the above method embodiments are all applicable to the present storage medium embodiments. The functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0159] In some selectable embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the application is provided by way of example, for the purpose of providing a more comprehensive understanding of technology. The disclosed method is not limited to the operation and logic flow presented herein. Selectable embodiments are expected, wherein the order of various operations is changed and the sub-operation of a part described as a larger operation is performed independently.

[0160] In addition, although the present application is described in the context of functional modules, it should be understood that, unless otherwise specified, one or more of the functions and / or features can be integrated into a single physical device and / or software module, or one or more functions and / or features can be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present application. More specifically, in view of the properties, functions, and internal relationships of the various functional modules in the device disclosed herein, the actual implementation of the module will be understood within the conventional techniques of the engineer. Therefore, those skilled in the art can implement the present application set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present application, which is determined by the full scope of the attached claims and their equivalents.

[0161] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program codes.

[0162] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0163] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0164] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0165] In the above description of this specification, the description with reference to the terms "one embodiment / example", "another embodiment / example" or "certain embodiments / examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0166] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.

[0167] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some of the technical features therein by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A UAV visual navigation strategy method, characterized in that: The method comprises: Generate a navigation image data set and a control data set, and collect the navigation image data set and the control data set; Calculating a multimodal objective function model according to the navigation image data set and the control data set; Mapping the navigation image dataset to a low-dimensional latent space through a variational encoder, and updating the multimodal objective function model according to the output of the decoder; Mapping the low-dimensional latent space to the control data set, and obtaining the drone control command according to the multimodal objective function model; Control the drone according to the drone control command; The calculating the multimodal objective function model according to the navigation image data set and the control data set includes: The data modes of the navigation image data set and the control data are defined as the relative posture of the RGB image and the drone, wherein the RGB image is a first-person visual RGB image; Processing the RGB data into normal RGB data by the variational encoder; The multimodal objective function model is calculated according to the normal RGB data, and the calculation formula includes: Among them, D KL is the sample data, p(x) and q(x) are encoder variables; Mapping the low-dimensional latent space to the control data set and obtaining the UAV control command according to the multimodal objective function model includes: Generate collision-free paths through RTT algorithm; Pruning the collision-free path to a minimum set of waypoints, and jointly optimizing a set of polynomial segment sequences to connect the waypoints in the minimum set of waypoints into a smooth minimum capture trajectory from a start to a target; The UAV control command is obtained according to the minimum capture trajectory.

2. The unmanned aerial vehicle visual navigation strategy method according to claim 1 is characterized in that: Generating a navigation image data set and a control data set, and collecting the navigation image data set and the control data set, including: Establish the world coordinate system, NED coordinate system, body coordinate system, obstacle coordinate system and camera coordinate system; Generate and collect image information, attitude information and kinematic information of the UAV during operation; The collected sample set is preprocessed.

3. The unmanned aerial vehicle visual navigation strategy method according to claim 2 is characterized in that: Generate and collect image information, attitude information and kinematic information of the UAV during operation, including: Randomly select the position information and posture information of the body coordinate system relative to the world coordinate system; The posture information is converted into Euler angles, and the conversion formula is: Among them, q x is the Euler angle of the x-axis of the world coordinate system, q y is the Euler angle of the posture information on the y-axis of the world coordinate system, q z is the Euler angle of the z-axis of the world coordinate system, is the rolling angle range of the body coordinate system relative to the world coordinate system, is the pitch angle range of the body coordinate system relative to the world coordinate system, is the yaw range of the body coordinate system relative to the world coordinate system.

4. The unmanned aerial vehicle visual navigation strategy method according to claim 2 is characterized in that: Generate and collect image information, attitude information and kinematic information of the UAV during operation, including: The navigation image dataset is obtained by randomly sampling the navigation image. The rolling angle range of the door coordinate system relative to the body coordinate system is calculated using the following formula: Among them, the is the rolling angle range of the door coordinate system relative to the body coordinate system, α prime is the pitch angle of the door coordinate system relative to the body coordinate system, Correction c Correction parameters for the camera.

5. The unmanned aerial vehicle visual navigation strategy method according to claim 1 is characterized in that: The expression of the minimum capture trajectory is: Among them, p(t) is the minimum capture trajectory, p0,p1,…,p n is the trajectory parameter, p=[p0,p1,…,p n ] T is the parameter vector.

6. A drone visual navigation strategy device, applied to a drone visual navigation strategy method as claimed in any one of claims 1 to 5, characterized in that: The device comprises: A data set module, used to generate a navigation image data set and a control data set, and collect the navigation image data set and the control data set; A calculation module, used for calculating a multimodal objective function model according to the navigation image data set and the control data set; A mapping module, used to map the navigation image dataset to a low-dimensional latent space through a variational encoder, and update the multimodal objective function model according to the output of the decoder; A control module, configured to map the low-dimensional latent space to the control data set and obtain a UAV control command according to the multimodal objective function model; The drone is controlled according to the drone control command.

7. A drone visual navigation strategy device, characterized in that: The device comprises: at least one processor; at least one memory, the memory being used to store at least one program; When at least one of the programs is executed by at least one of the processors, a drone visual navigation strategy method as described in any one of claims 1-5 is implemented.

8. A storage medium, characterized in that The storage medium stores a program executable by a processor, and when the program executable by the processor is executed by the processor, the unmanned aerial vehicle visual navigation strategy method according to any one of claims 1 to 5 is implemented.

Images