Method and apparatus for controlling a drone

Abstract

The application discloses a kind of method and device for controlling unmanned plane, wherein the method includes: in the stage of unmanned plane take-off, at least one reference photo is collected and saved, and the reference photo contains target mark deployed on hangar surface;In the stage of unmanned plane returning and landing, current photo is collected, and whether target mark exists in current photo is detected;In the case where target mark exists, the pose information of target mark relative to the body is determined;First control instruction is generated based on the pose information;Second control instruction is generated based on reference photo and current photo;Target control instruction is generated based on first control instruction and / or second control instruction, to control unmanned plane landing.By the present application, the problem that unmanned plane cannot land normally to hangar is solved, and the effect that unmanned plane lands accurately to hangar is achieved.

Description

Technical Field

[0001] This application relates to the field of computers, and more specifically, to a method and apparatus for controlling a drone. Background Technology

[0002] With the increasing prevalence of unmanned drone hangars, the precise landing of drones on the hangar surface has become particularly important. The diverse environments in which hangars are deployed increase the difficulty of landing; for example, sudden winds and sandstorms can prevent drones from landing properly.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This application provides a method and apparatus for controlling a drone, so as to at least solve the technical problem in the related art that drones cannot land normally in the hangar.

[0005] According to one aspect of the embodiments of this application, a method for controlling a drone is provided, comprising: during the takeoff phase of the drone, acquiring and saving at least one reference photograph, wherein the reference photograph includes a target identifier deployed on the surface of a hangar; during the return landing phase of the drone, acquiring a current photograph and detecting whether the target identifier exists in the current photograph; if the target identifier exists, determining the pose information of the target identifier relative to the drone body; generating a first control command based on the pose information; generating a second control command based on the reference photograph and the current photograph; and generating a target control command based on the first control command and / or the second control command to control the drone to land.

[0006] In an exemplary embodiment, determining the pose information of the target identifier relative to the body includes: determining the spatial positions of M target points in the target identifier in a first coordinate system, wherein the first coordinate system is the coordinate system where the target identifier is located, and M is greater than 0; determining the coordinate positions of the target points in a second coordinate system, wherein the second coordinate system is the coordinate system where the current photo is located; determining a rotation matrix and a translation vector from the first coordinate system to the second coordinate system based on the spatial position and the coordinate position; and transforming the rotation matrix and the translation vector to the third coordinate system based on a pre-calibrated pose transformation relationship between the second coordinate system and the third coordinate system to obtain a target rotation matrix and a target translation vector, wherein the pose information includes: the target rotation matrix and the target translation vector.

[0007] In an exemplary embodiment, generating a first control command based on the pose information includes: determining the position error of the UAV at the current moment based on the pose information; obtaining the current speed of the UAV; determining a proportional gain based on the position error and the current speed; and generating the first control command based on the proportional gain and preset integral and derivative gains.

[0008] In one exemplary embodiment, determining the proportional gain based on the position error and the current velocity includes: calculating a first parameter value using the following formula:

[0009] ,in, It is the value of the first parameter. It is the position error, It is a preset time interval;

[0010] The proportional gain is determined based on the relationship between the first parameter value and the reference value.

[0011] In an exemplary embodiment, determining the proportional gain based on the relationship between the first parameter value and the reference value includes: calculating the proportional gain using the following formula when the first parameter value is less than or equal to the reference value:

[0012]

[0013] When the first parameter value is greater than the reference value, the proportional gain is calculated using the following formula:

[0014]

[0015] in, It is a preset time interval. Here, v is the position error, v is the current velocity, and Pref is the reference value. , For preset coefficients, It is a symbolic function.

[0016] In an exemplary embodiment, the first control command is generated based on the proportional gain and preset integral and derivative gains, including:

[0017] The first control command is generated using the following formula:

[0018]

[0019] in, This is the first control command. is the proportional gain, Ki is the integral gain, Kd is the differential gain, t is time, and P is the position error.

[0020] In an exemplary embodiment, generating a second control command based on the reference photo and the current photo includes: the at least one reference photo includes at least: a first reference photo of the drone correctly positioned on the hangar surface, a second reference photo of the drone at a first height above the hangar surface during takeoff, and a third reference photo at a second height above the hangar surface, wherein the second height is greater than the first height; the second reference photo or the third reference photo is used as a target reference photo, and input along with the current photo and a first prompt word into a target model to generate a first set of control data for controlling the drone to return to the shooting position corresponding to the target reference photo, wherein the first prompt word is used to instruct the target model to use the target reference photo as a reference; when the numerical fluctuation of the first set of control data is less than a preset threshold, the first reference photo is used as the target reference photo, and along with the current photo and the second prompt word, is input into the target model to generate a second set of control data, wherein the second prompt word is used to instruct the target model to control the drone to land at a target landing point and make the current photo close to the first reference photo; the second set of control data is determined as the second control command.

[0021] In an exemplary embodiment, detecting whether the target identifier exists in the current photograph includes: if the current photograph is a photograph taken by a wide-angle camera, mapping the current photograph to an orthogonal photograph using a cylindrical projection model; decoding the orthogonal photograph to obtain a decoded binary sequence; determining the Hamming distance between the decoded binary sequence and a preset standard binary sequence; and determining that the target identifier exists in the current photograph if the Hamming distance satisfies a preset condition.

[0022] In an exemplary embodiment, generating a target control command based on the first control command and / or the second control command to control the drone landing includes: determining whether the first control command exists; if the first control command does not exist, determining the second control command as the target control command; if the first control command exists, fusing the first control command and the second control command to obtain the target control command.

[0023] According to another aspect of the embodiments of this application, an apparatus for controlling a drone is also provided, comprising: a first acquisition module, configured to acquire and save at least one reference photograph during the takeoff phase of the drone, wherein the reference photograph includes a target identifier deployed on the surface of a hangar; a second acquisition module, configured to acquire a current photograph during the return landing phase of the drone, and detect whether the target identifier exists in the current photograph; if the target identifier exists, determine the pose information of the target identifier relative to the drone body; a first generation module, configured to generate a first control command based on the pose information; generate a second control command based on the reference photograph and the current photograph; and a second generation module, configured to generate a target control command based on the first control command and / or the second control command to control the drone landing.

[0024] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, wherein a computer program is stored therein, wherein the computer program is configured to perform the steps in any of the above method embodiments when executed by a processor.

[0025] According to another aspect of the embodiments of this application, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, causing the computer device to perform the steps in any of the method embodiments described above.

[0026] According to another aspect of the embodiments of this application, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to perform the steps of any of the above method embodiments through the computer program.

[0027] This application achieves the following: During the UAV takeoff phase, at least one reference photograph is acquired and stored, containing a target marker deployed on the hangar surface; during the UAV return and landing phase, a current photograph is acquired, and the presence of a target marker is detected within the current photograph; if the target marker is present, its pose information relative to the UAV is determined; a first control command is generated based on the pose information; a second control command is generated based on the reference photograph and the current photograph; and a target control command is generated based on the first and / or second control commands to control the UAV's landing. Therefore, this solves the problem of UAVs failing to land properly in hangars in related technologies, achieving precise UAV landings in hangars. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of a target identifier being a QR code according to an embodiment of this application;

[0029] Figure 2 This is a flowchart illustrating an optional method for controlling a drone according to an embodiment of this application;

[0030] Figure 3 This is an optional overall process diagram according to an embodiment of this application;

[0031] Figure 4 This is a structural block diagram of an optional device for controlling a drone according to an embodiment of this application;

[0032] Figure 5 This is a computer system architecture block diagram of an optional electronic device according to an embodiment of this application. Detailed Implementation

[0033] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0034] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0035] This invention provides a combined precision landing system and method for unmanned aerial vehicles (UAVs) based on a target model and QR code guidance, applicable to precise landing control in unmanned hangar scenarios. The technical solution of this invention will be described in detail below with reference to the accompanying drawings. The accompanying drawings illustrate preferred embodiments of the invention, but the invention can be implemented in many different forms and is not limited to the embodiments described herein.

[0036] like Figure 1 As shown, the QR code is deployed on the surface of the hangar. The QR code is 9cm long and 9cm wide, and its overall shape is as follows. Figure 1As shown. Preferably, the QR code surface uses a frosted finish to reduce specular reflection; simultaneously, the hangar surface employs a hydrophobic structure to prevent water accumulation on the QR code surface from affecting the recognition effect. As a target identifier, the spatial positions of the four corner points of the QR code in the QR code coordinate system (i.e., the first coordinate system) are predetermined fixed values.

[0037] Figure 2 This is a flowchart illustrating an optional method for controlling a drone according to an embodiment of this application, as shown below. Figure 2 As shown, the process of this method may include the following steps:

[0038] Step S202: During the takeoff phase of the UAV, at least one reference photograph is collected and saved, wherein the reference photograph contains target markings deployed on the surface of the hangar;

[0039] In this embodiment, a wide-angle fisheye camera is used as the visual sensor for photo acquisition. Its horizontal and vertical field of view both exceed 180 degrees to ensure that photos taken by the drone within a certain swing range still contain QR codes.

[0040] like Figure 3 The diagram illustrates the overall process. During the drone takeoff phase, several fisheye photos containing QR codes are collected and saved as reference photos. Specifically, the reference photos include at least: a fisheye photo of the drone correctly positioned on the hangar surface (i.e., the first reference photo), a fisheye photo of the drone at a distance of 10m from the hangar surface (i.e., the first altitude) (i.e., the second reference photo), and a fisheye photo of the drone at a distance of 15m from the hangar surface (i.e., the second altitude) (i.e., the third reference photo). These reference photos are stored in the drone's storage module for use during the return flight.

[0041] Step S204: During the UAV's return landing phase, a current photo is captured, and the presence of the target identifier in the current photo is detected; if the target identifier exists, the pose information of the target identifier relative to the UAV body is determined.

[0042] The current photo was captured when the drone returned to the Home point and was preparing to land on the hangar surface. It was then captured in real time at a frequency of 25Hz until landing was completed.

[0043] When capturing reference and current photos, the exposure time of the fisheye camera is dynamically adjusted. Specifically, the exposure time is dynamically adjusted based on the pixel brightness within a W×H area in the center of the image: if the center area is dark, the exposure time is increased to ensure the brightness of the center and improve the clarity of the QR code; if the center is bright, the exposure time is appropriately reduced to prevent specular reflection and improve clarity.

[0044] W and H are calculated according to the following formula:

[0045] W = 0.5 × 100 × IW / FOV;

[0046] H = 0.5 × 100 × IH / FOV;

[0047] Where FOV is the fisheye field of view, IW is the image width resolution, and IH is the image height resolution.

[0048] In an exemplary embodiment, detecting whether the target identifier exists in the current photograph includes: if the current photograph is a photograph taken by a wide-angle camera, mapping the current photograph to an orthogonal photograph using a cylindrical projection model; decoding the orthogonal photograph to obtain a decoded binary sequence; determining the Hamming distance between the decoded binary sequence and a preset standard binary sequence; and determining that the target identifier exists in the current photograph if the Hamming distance satisfies a preset condition.

[0049] The aforementioned wide-angle camera, specifically the fisheye camera in the above embodiment, performs virtual mapping on the acquired fisheye photos to establish a virtual camera. Assuming the virtual camera's resolution is set to 512×512, and the intrinsic parameter K is:

[0050]

[0051] in, and All are set to 400. and All values ​​are set to 256. This allows the images input to subsequent QR code detection algorithms to be processed in real time.

[0052] To achieve a large field of view with controllable distortion at a relatively small pixel level, cylindrical projection is used for the downsampled image. Specifically, the pixel coordinates (u,v) of the virtual camera image are mapped to spatial points in fisheye coordinates according to the following formula. :

[0053]

[0054]

[0055]

[0056] In this system, TanA is set empirically to 0.5; u and v represent the pixel coordinates of the image obtained after virtual mapping by the virtual camera; X', Y', and Z' are spatial points in fisheye coordinates. The spatial points (X', Y', Z') are then corrected for fisheye distortion and intrinsic parameters to obtain the corresponding fisheye image pixel coordinates (u', v'). The fisheye distortion parameters and intrinsic parameters are pre-calibrated. After the above virtual mapping and cylindrical projection processing, the resulting orthogonal image has a larger field of view, which can reduce the blind spot while ensuring detection speed.

[0057] The open-source AprilTag or ARUCO algorithm is used to detect QR codes in orthogonal images. During the detection process, the success of the detection is determined by the Hamming distance, specifically whether the Hamming distance between the decoded binary sequence and the preset standard binary sequence meets a preset condition. If the condition is met, a QR code is confirmed to exist in the current image. The preset condition can be that the Hamming distance is less than or equal to a preset distance threshold, and the specific value can be determined according to the actual situation.

[0058] In an exemplary embodiment, determining the pose information of the target identifier relative to the body includes: determining the spatial positions of M target points in the target identifier in a first coordinate system, wherein the first coordinate system is the coordinate system where the target identifier is located, and M is greater than 0; determining the coordinate positions of the target points in a second coordinate system, wherein the second coordinate system is the coordinate system where the current photo is located; determining a rotation matrix and a translation vector from the first coordinate system to the second coordinate system based on the spatial position and the coordinate position; and transforming the rotation matrix and the translation vector to the third coordinate system based on a pre-calibrated pose transformation relationship between the second coordinate system and the third coordinate system to obtain a target rotation matrix and a target translation vector, wherein the pose information includes: the target rotation matrix and the target translation vector.

[0059] For example, if the QR code detection is successful, the pose calculation step is executed. Given the spatial positions Pi (i=1,2,3,4) of the four corner points of the QR code (i.e., M target points, M=4) in the QR code coordinate system (i.e., the first coordinate system), the rotation matrix R and translation vector T from the QR code coordinate system to the fisheye camera coordinate system (i.e., the second coordinate system) are calculated using the PnP (Perspective-n-Point) method, based on the two-dimensional pixel coordinates and spatial positions Pi of the four corner points in the image.

[0060] Then, based on the pre-defined pose transformation relationship from the fisheye camera to the IMU coordinate system (i.e., the third coordinate system), the rotation matrix R and translation vector T are transformed into the IMU coordinate system to obtain the target rotation matrix and target translation vector. The pose information includes this target rotation matrix and target translation vector. Since the IMU coordinate system is the reference coordinate system for the attitude and position control of the UAV, after transforming the QR code pose into the IMU coordinate system, the pose information of the QR code relative to the UAV body can be obtained, which is used for subsequent control command generation.

[0061] Step S206: Generate a first control command based on the pose information; generate a second control command based on the reference photo and the current photo;

[0062] In an exemplary embodiment, generating a first control command based on the pose information includes: determining the position error of the UAV at the current moment based on the pose information; obtaining the current speed of the UAV; determining a proportional gain based on the position error and the current speed; and generating the first control command based on the proportional gain and preset integral and derivative gains.

[0063] Specifically, the position error P of the UAV at the current moment is determined based on the target translation vector in the pose information. The position error P is the three axial components of the translation vector, which represent the position deviation of the UAV relative to the center of the QR code (i.e., the target landing point) in the X, Y, and Z directions, respectively.

[0064] After obtaining the current speed v of the drone, the proportional gain Kp of the PID control algorithm is dynamically adjusted based on the position error P and the current speed v. The specific adjustment method is as follows:

[0065] In one exemplary embodiment, determining the proportional gain based on the position error and the current velocity includes: calculating a first parameter value using the following formula:

[0066] ,in, It is the value of the first parameter. It is the position error, It is a preset time interval; the proportional gain is determined based on the relationship between the first parameter value and the reference value.

[0067] Then, based on the relationship between the first parameter value and the reference value Pref, the proportional gain Kp is determined:

[0068] Case 1: When the value of the first parameter is less than or equal to the reference value, the proportional gain is calculated using the following formula:

[0069]

[0070] Scenario 2: When the first parameter value is greater than the reference value, the proportional gain is calculated using the following formula:

[0071]

[0072] in, It is a preset time interval. Here, v is the position error, v is the current velocity, and Pref is the reference value. , For preset coefficients, It is a symbolic function.

[0073] For example, Pref is a preset position error reference value, and in this embodiment, Pref is set to 0.05m; and As a preset coefficient, in this embodiment Take 2, The value is 20; sign is the sign function, sign(v×P) means taking the sign of v×P. Specifically: when v×P>0, the value is 1, when v×P<0, the value is -1, and when v×P=0, the value is 0.

[0074] Through the above dynamic adjustment, Kp is smaller when the wind speed is low, which can ensure landing accuracy and reduce drone sway; when the wind speed suddenly increases, Kp can respond in time to pull back the position deviation and ensure landing stability.

[0075] Using a standard PID control law, a speed control command is generated as the first control command based on the adjusted proportional gain Kp and the preset integral gain Ki and derivative gain Kd.

[0076] In an exemplary embodiment, the first control command is generated based on the proportional gain and preset integral and derivative gains, including:

[0077] The first control command is generated using the following formula:

[0078]

[0079] in, This is the first control command. is the proportional gain, Ki is the integral gain, Kd is the differential gain, t is time, and P is the position error.

[0080] In the drone landing scenario, the differential gain Kd is set to 0, and the integral gain Ki is set to a constant of 0.01. These parameters can also be simulated and adjusted using tools such as Matlab to achieve better control performance.

[0081] In an exemplary embodiment, generating a second control command based on the reference photo and the current photo includes: the at least one reference photo includes at least: a first reference photo of the drone correctly positioned on the hangar surface, a second reference photo of the drone at a first height above the hangar surface during takeoff, and a third reference photo at a second height above the hangar surface, wherein the second height is greater than the first height; the second reference photo or the third reference photo is used as a target reference photo, and input along with the current photo and a first prompt word into a target model to generate a first set of control data for controlling the drone to return to the shooting position corresponding to the target reference photo, wherein the first prompt word is used to instruct the target model to use the target reference photo as a reference; when the numerical fluctuation of the first set of control data is less than a preset threshold, the first reference photo is used as the target reference photo, and along with the current photo and the second prompt word, is input into the target model to generate a second set of control data, wherein the second prompt word is used to instruct the target model to control the drone to land at a target landing point and make the current photo close to the first reference photo; the second set of control data is determined as the second control command.

[0082] The target model described above is a vision-language-action model, and its training data includes both actual collected data and simulation data.

[0083] For actual collected data, for example, actual fisheye photos, the prompt words include three types, specifically:

[0084] The control command corresponding to "Please record this photo as a reference photo" is (0,0,0,0). The control command corresponding to "Return to the location where the reference photo was taken" is obtained by calculating the command through the QR code. The control command corresponding to "Land on H and make the photo very close to the reference photo" is obtained by calculating the command through the QR code.

[0085] For the simulation data, a drone identification system is established. This system then simulates different wind levels and weather conditions such as sandstorms, heavy fog, heavy rain, and nighttime. Similar to the actual data, the system includes the three prompts mentioned above. Instructions are calculated via QR codes and then manually corrected.

[0086] After the above fine-tuning training, when the drone returns, it will undergo coarse positioning and fine descent phases:

[0087] For the coarse positioning stage: Input a fisheye photograph (second or third reference image) saved during takeoff at a position of 10m or 15m as the target reference image, and input the prompt "Please record this photo as the reference photo." Then input the current photo and input the prompt "Return to the location where the reference photo was taken." The target model outputs control commands to control the UAV to fly to the vicinity of the shooting location corresponding to the target reference photo. In this embodiment, the photo at the 15m position is used preferentially for coarse positioning; when the UAV descends from high altitude, it first uses photos from higher positions for preliminary positioning.

[0088] For the precision landing phase: when the fluctuation of the control command value output by the target model in the first phase is less than the preset level, input a fisheye photo of the UAV when it is correctly positioned (the first reference image) as the target reference image, and input the prompt "Please record this photo as the reference photo". Then input the current photo and input the prompt "Land on H and make the photo very close to the reference photo". The target model outputs control commands to control the UAV to land precisely at the target landing point.

[0089] Through the above two-stage processing, the target model first performs coarse positioning and then fine descent, effectively improving the accuracy and reliability of landing.

[0090] Step S208: Generate a target control command based on the first control command and / or the second control command to control the drone to land.

[0091] In an exemplary embodiment, generating a target control command based on the first control command and / or the second control command to control the drone landing includes: determining whether the first control command exists; if the first control command does not exist, determining the second control command as the target control command; if the first control command exists, fusing the first control command and the second control command to obtain the target control command.

[0092] The first control command and the second control command are merged to generate the final target control command.

[0093] Specifically, the first step is to determine whether a first control command exists:

[0094] Scenario 1: When the QR code detection fails or the pose calculation fails, resulting in the inability to generate the first control command, the second control command will be directly determined as the target control command output.

[0095] Scenario 2: When the first control command is successfully generated, the Bayesian fusion method is used to fuse the first control command and the second control command. The fusion weight is determined based on actual experience, and the fused control command is used as the target control command.

[0096] By using the above fusion method, on the one hand, the adaptive capability of the target model in different scenarios is fully utilized, and on the other hand, QR code guidance is used to prevent abnormal commands from the target model, ultimately achieving precise landing in various harsh environments.

[0097] Those skilled in the art should understand that various modifications can be made based on the above embodiments: the target identifier is not limited to AprilTag or ARUCO QR code, but can also use other types of QR codes or visual identifiers. The visual sensor is not limited to fisheye cameras, but can also use other types of ultra-wide-angle cameras, as long as their field of view is large enough. The target model is not limited to OPENVLA, but can also use other vision-language-action models for fine-tuning. The command fusion method is not limited to Bayesian fusion, but can also use other fusion strategies. The specific values ​​of the first altitude and the second altitude are not limited to 10m and 15m, and can be adjusted according to the actual hangar height and UAV performance.

[0098] This application has the following beneficial effects:

[0099] By proposing to deploy QR codes on the hangar surface, adding dynamic adjustment of exposure time to prevent specular reflection, and incorporating a cylindrical projection model to achieve a large field of view with controllable distortion at a small pixel level, faster QR code detection was achieved while reducing blind spots.

[0100] By leveraging the target model's powerful understanding and learning capabilities of images and language, and combining fine-tuning with application scenarios (including fine-tuning with actual data and simulation data), precise landings were achieved in various harsh environments (such as sudden winds, sandstorms, heavy fog, heavy rain, and nighttime).

[0101] By fusing the target model generation command and the QR code generation command via PID control, the control robustness is improved, ensuring that the drone can still land safely even if the target model command is abnormal or the QR code detection fails.

[0102] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0103] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory (ROM) / random access memory (RAM), magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0104] According to another aspect of the embodiments of this application, an apparatus for controlling a drone is also provided. This apparatus can be used to implement the drone control method provided in the above embodiments, and details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0105] Figure 4 This is a structural block diagram of an optional device for controlling a drone according to an embodiment of this application, such as... Figure 4 As shown, the device for controlling the drone includes:

[0106] The first acquisition module 42 is used to acquire and save at least one reference photo during the take-off phase of the UAV, wherein the reference photo includes target markings deployed on the surface of the hangar.

[0107] The second acquisition module 44 is used to acquire current photos during the UAV's return landing phase and detect whether the target identifier exists in the current photos; if the target identifier exists, it determines the pose information of the target identifier relative to the UAV body.

[0108] The first generation module 46 is used to generate a first control command based on the pose information; and to generate a second control command based on the reference photo and the current photo.

[0109] The second generation module 48 is used to generate target control commands based on the first control command and / or the second control command to control the drone to land.

[0110] In an exemplary embodiment, the above-described apparatus is further configured to: determine the spatial positions of M target points in the target identifier in a first coordinate system, wherein the first coordinate system is the coordinate system in which the target identifier is located, and M is greater than 0; determine the coordinate positions of the target points in a second coordinate system, wherein the second coordinate system is the coordinate system in which the current photograph is located; determine a rotation matrix and a translation vector from the first coordinate system to the second coordinate system based on the spatial positions and the coordinate positions; and transform the rotation matrix and the translation vector to the third coordinate system based on a pre-calibrated pose transformation relationship between the second coordinate system and the third coordinate system to obtain a target rotation matrix and a target translation vector, wherein the pose information includes: the target rotation matrix and the target translation vector.

[0111] In an exemplary embodiment, the above-described apparatus is further configured to determine the position error of the UAV at the current moment based on the pose information; obtain the current speed of the UAV; determine a proportional gain based on the position error and the current speed; and generate the first control command based on the proportional gain and preset integral and derivative gains.

[0112] In one exemplary embodiment, the above-described apparatus is further configured to calculate the first parameter value using the following formula:

[0113] ,in, It is the value of the first parameter. It is the position error, It is a preset time interval;

[0114] The proportional gain is determined based on the relationship between the first parameter value and the reference value.

[0115] In an exemplary embodiment, the above-described apparatus is further configured to calculate the proportional gain using the following formula when the first parameter value is less than or equal to a reference value:

[0116]

[0117] When the first parameter value is greater than the reference value, the proportional gain is calculated using the following formula:

[0118]

[0119] in, It is a preset time interval. Here, v is the position error, v is the current velocity, and Pref is the reference value. , For preset coefficients, It is a symbolic function.

[0120] In one exemplary embodiment, the above-described apparatus is further configured to generate the first control command using the following formula:

[0121]

[0122] in, This is the first control command. is the proportional gain, Ki is the integral gain, Kd is the differential gain, t is time, and P is the position error.

[0123] In an exemplary embodiment, the at least one reference photograph includes at least: a first reference photograph of the drone correctly positioned on the hangar surface, a second reference photograph of the drone at a first height above the hangar surface upon takeoff, and a third reference photograph at a second height above the hangar surface, wherein the second height is greater than the first height; the device is further configured to input the second or third reference photograph as a target reference photograph, along with the current photograph and a first prompt word, into a target model to generate a first set of control data for controlling the drone to return to the shooting position corresponding to the target reference photograph, wherein the first prompt word is used to instruct the target model to use the target reference photograph as a reference; when the numerical fluctuation of the first set of control data is less than a preset threshold, the first reference photograph is used as the target reference photograph, along with the current photograph and a second prompt word, into the target model to generate a second set of control data, wherein the second prompt word is used to instruct the target model to control the drone to land at a target landing point and to make the current photograph closer to the first reference photograph; and the second set of control data is determined to be the second control command.

[0124] In an exemplary embodiment, the above-described apparatus is further configured to, when the current photograph is a photograph taken by a wide-angle camera, map the current photograph to an orthogonal photograph using a cylindrical projection model; decode the orthogonal photograph to obtain a decoded binary sequence; determine the Hamming distance between the decoded binary sequence and a preset standard binary sequence; and, when the Hamming distance satisfies a preset condition, determine that the target identifier exists in the current photograph.

[0125] In an exemplary embodiment, the apparatus is further configured to determine whether the first control instruction exists; if the first control instruction does not exist, determine the second control instruction as the target control instruction; if the first control instruction exists, fuse the first control instruction and the second control instruction to obtain the target control instruction.

[0126] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0127] According to another aspect of the embodiments of this application, a computer-readable storage medium is provided, the computer-readable storage medium including a stored program, wherein the program executes the steps in any of the above method embodiments when it is run.

[0128] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as USB flash drives, ROMs, RAMs, portable hard drives, magnetic disks, or optical disks.

[0129] According to another aspect of the embodiments of this application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor is configured to perform the steps of any of the method embodiments described above via the computer program. In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0130] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0131] According to another aspect of the embodiments of this application, a computer program product is also provided, comprising a computer program / instructions containing program code for performing the methods shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via communication section 509, and / or installed from removable medium 511. When the computer program is executed by central processing unit 501, it performs various functions provided in the embodiments of this application. The sequence numbers of the embodiments of this application above are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0132] Figure 5 A schematic block diagram of a computer system architecture for implementing embodiments of the present application is shown. Figure 5As shown, the computer system 500 includes a Central Processing Unit (CPU) 501, which can perform various appropriate actions and processes based on programs stored in ROM 502 or programs loaded into RAM 503 from storage section 508. Random access memory 503 also stores various programs and data required for system operation. The CPU 501, ROM 502, and RAM 503 are interconnected via bus 504. Input / output (I / O) interface 505 is also connected to bus 504.

[0133] The following components are connected to I / O interface 505: input section 506 including keyboard, mouse, etc.; output section 507 including cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; storage section 508 including hard disk, etc.; and communication section 509 including network interface card, modem, etc. Communication section 509 performs communication processing via a network such as the Internet. Drive 510 is also connected to I / O interface 505 as needed. Removable media 511, such as disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 510 as needed so that computer programs read from them can be installed into storage section 508 as needed.

[0134] Specifically, according to embodiments of this application, the processes described in the various method flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 509, and / or installed from removable medium 511. When the computer program is executed by central processing unit 501, it performs various functions defined in the system of this application.

[0135] It should be noted that, Figure 5 The computer system 500 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0136] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those described herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0137] The above are merely preferred embodiments of this application and are not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.

Claims

1. A method for controlling a drone, characterized in that, The method includes: During the takeoff phase of the drone, at least one reference photograph is collected and saved, wherein the reference photograph contains target markings deployed on the surface of the hangar; During the UAV's return landing phase, a current photo is captured, and the presence of the target identifier in the current photo is detected. If the target identifier is present, the pose information of the target identifier relative to the UAV body is determined. A first control command is generated based on the pose information; a second control command is generated based on the reference photo and the current photo; Target control commands are generated based on the first control command and / or the second control command to control the drone to land.

2. The method according to claim 1, characterized in that, Determining the pose information of the target identifier relative to the machine body includes: Determine the spatial positions of M target points in the target identifier in a first coordinate system, wherein the first coordinate system is the coordinate system in which the target identifier is located, and M is greater than 0; Determine the coordinate position of the target point in a second coordinate system, wherein the second coordinate system is the coordinate system in which the current photo is located; Based on the spatial position and the coordinate position, determine the rotation matrix and translation vector from the first coordinate system to the second coordinate system; Based on the pre-calibrated pose transformation relationship between the second and third coordinate systems, the rotation matrix and translation vector are transformed to the third coordinate system to obtain the target rotation matrix and the target translation vector. The pose information includes the target rotation matrix and the target translation vector.

3. The method according to claim 1 or 2, characterized in that, Generate a first control command based on the pose information, including: The position error of the UAV at the current moment is determined based on the pose information; Obtain the current speed of the drone; The proportional gain is determined based on the position error and the current velocity; The first control command is generated based on the proportional gain and the preset integral and derivative gains.

4. The method according to claim 3, characterized in that, Determining the proportional gain based on the position error and the current velocity includes: The value of the first parameter is calculated using the following formula: ,in, It is the value of the first parameter. It is the position error, It is a preset time interval; The proportional gain is determined based on the relationship between the first parameter value and the reference value.

5. The method according to claim 4, characterized in that, Determining the proportional gain based on the relationship between the first parameter value and the reference value includes: When the first parameter value is less than or equal to the reference value, the proportional gain is calculated using the following formula: When the first parameter value is greater than the reference value, the proportional gain is calculated using the following formula: in, It is a preset time interval. Where v is the position error, v is the current velocity, and Pref is the reference value. , For preset coefficients, It is a symbolic function.

6. The method according to claim 3, characterized in that, Based on the proportional gain and preset integral and derivative gains, the first control command is generated, including: The first control command is generated using the following formula: in, This is the first control command. is the proportional gain, Ki is the integral gain, Kd is the differential gain, t is time, and P is the position error.

7. The method according to claim 1, characterized in that, A second control command is generated based on the reference photo and the current photo, including: The at least one reference photograph includes at least: a first reference photograph of the drone being correctly positioned on the hangar surface, a second reference photograph of the drone taking off at a first height above the hangar surface, and a third reference photograph at a second height above the hangar surface, wherein the second height is greater than the first height; The second or third reference photo is used as the target reference photo. The current photo and the first prompt word are input into the target model to generate a first set of control data for controlling the drone to return to the shooting position corresponding to the target reference photo. The first prompt word is used to instruct the target model to use the target reference photo as a reference. When the numerical fluctuation of the first set of control data is less than a preset threshold, the first reference photo is used as the target reference photo, and the current photo and the second prompt word are input into the target model to generate the second set of control data. The second prompt word is used to instruct the target model to control the drone to land at the target landing point and make the current photo close to the first reference photo. The second set of control data is identified as the second control command.

8. The method according to claim 1, characterized in that, Detecting whether the target identifier exists in the current photo includes: In the case where the current photo is taken with a wide-angle camera, the current photo is mapped to an orthogonal photo using a cylindrical projection model; The orthogonal photograph is decoded to obtain a decoded binary sequence; Determine the Hamming distance between the decoded binary sequence and the preset standard binary sequence; If the Hamming distance meets the preset conditions, it is determined that the target identifier exists in the current photo.

9. The method according to claim 1, characterized in that, Generating target control commands based on the first control command and / or the second control command to control the drone's landing, including: Determine whether the first control command exists; In the absence of the first control instruction, the second control instruction is determined as the target control instruction; In the presence of the first control instruction, the first control instruction and the second control instruction are fused together to obtain the target control instruction.

10. A device for controlling a drone, characterized in that, include: The first acquisition module is used to acquire and save at least one reference photo during the take-off phase of the UAV, wherein the reference photo contains target markings deployed on the surface of the hangar; The second acquisition module is used to acquire current photos during the UAV's return landing phase and detect whether the target identifier exists in the current photos; if the target identifier exists, it determines the pose information of the target identifier relative to the UAV body. The first generation module is used to generate a first control command based on the pose information; and to generate a second control command based on the reference photo and the current photo. The second generation module is used to generate target control commands based on the first control command and / or the second control command to control the drone to land.

11. A computer program product, comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the method according to any one of claims 1 to 9.

13. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 9.

Images