Visual guidance landing method and device of unmanned aerial vehicle and electronic equipment

Abstract

The invention discloses a visual guidance landing method and device for an unmanned aerial vehicle and electronic equipment. The method comprises the steps that when a first feature object of an unmanned aerial vehicle hangar is detected, first image information of the first feature object is acquired, and space coordinate information of a target unmanned aerial vehicle relative to the first feature object is determined according to the acquired first image information of the first feature object; when a second feature object of the unmanned aerial vehicle hangar is detected, determining a landing direction angle of the course angle of the target unmanned aerial vehicle relative to the second feature object according to the obtained feature image information of the second feature object, and determining a target course deviation of the target unmanned aerial vehicle according to the landing direction angle and a preset direction angle; and controlling the target unmanned aerial vehicle to land in the unmanned aerial vehicle hangar according to the space coordinate information and the target course deviation. The flight parameters of the target unmanned aerial vehicle are adjusted according to the space coordinate information of the target unmanned aerial vehicle relative to the first feature object and the target course deviation, and high-precision landing of the unmanned aerial vehicle is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicles, and in particular to a method, device and electronic equipment for visually guided landing of unmanned aerial vehicles. Background Art

[0002] Autonomous landing technology of UAVs is an important part of realizing automated control of UAVs, and precision landing technology is a technology with higher accuracy and technical content among all landing technologies.

[0003] Currently, drones rely on GPS signals and inertial navigation systems for positioning and trajectory planning during landing. However, GPS signals are easily interfered with in complex environments. For example, in urban canyons, forests, indoor environments, etc., GPS signals may be blocked or fail, affecting the precise landing of drones. In addition, the landing method based on inertial navigation has limited accuracy, especially in scenarios that require precise landing, and it is difficult to meet high-precision requirements. Summary of the invention

[0004] The present invention provides a method, device and electronic equipment for visually guiding the landing of an unmanned aerial vehicle (UAV) to achieve high-precision landing of the UAV.

[0005] According to one aspect of the present invention, a method for visually guiding landing of an unmanned aerial vehicle is provided, the method comprising:

[0006] When a first feature object of the drone hangar is detected, first image information of the first feature object is obtained, and spatial coordinate information of the target drone relative to the first feature object is determined according to the first image information; the first feature object is the outer contour of the drone hangar;

[0007] When a second characteristic object of the drone hangar is detected, characteristic image information of the second characteristic object is obtained, a heading angle of the target drone relative to a landing direction angle of the second characteristic object is determined according to the characteristic image information, and a target heading deviation of the target drone is determined according to the landing direction angle and a preset direction angle; the second characteristic object is located at a first preset position of the outer contour of the drone hangar;

[0008] The target UAV is controlled to land in a UAV hangar according to the spatial coordinate information and the target heading deviation.

[0009] According to another aspect of the present invention, a vision-guided landing device for an unmanned aerial vehicle is provided, the device comprising:

[0010] An information determination module is used to obtain first image information of the first feature object when a first feature object of the drone hangar is detected, and determine the spatial coordinate information of the target drone relative to the first feature object according to the first image information; the first feature object is the outer contour of the drone hangar;

[0011] A heading determination module is used for, when a second characteristic object of the drone hangar is detected, obtaining characteristic image information of the second characteristic object, determining the heading angle of the target drone relative to the landing direction angle of the second characteristic object according to the characteristic image information, and determining the target heading deviation of the target drone according to the landing direction and a preset direction angle; the second characteristic object is located at a first preset position of the outer contour of the drone hangar;

[0012] The control module is used to control the target UAV to land in the UAV hangar according to the spatial coordinate information and the target heading deviation.

[0013] According to another aspect of the present invention, there is provided an electronic device, the electronic device comprising:

[0014] at least one processor; and

[0015] a memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the UAV vision-guided landing method described in any embodiment of the present invention.

[0017] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the UAV vision-guided landing method described in any embodiment of the present invention when executed.

[0018] The technical solution of the embodiment of the present invention is as follows: when a first characteristic object of a drone hangar is detected, first image information of the first characteristic object is obtained, and spatial coordinate information of the target drone relative to the first characteristic object is determined based on the first image information, so as to determine in real time the spatial coordinate information of the drone relative to the outer circle of the drone hangar during landing; when a second characteristic object of the drone hangar is detected, characteristic image information of the second characteristic object is obtained, and the heading angle of the target drone relative to the landing direction angle of the second characteristic object is determined based on the characteristic image information, and the target heading deviation of the target drone is determined based on the landing direction angle and the preset direction angle; the second characteristic object is located at a first preset position of the outer contour of the drone hangar, so as to accurately determine the landing direction angle, thereby accurately deriving whether the target drone has yawed; finally, the target drone is controlled to land in the drone hangar based on the spatial coordinate information and the target heading deviation, so as to achieve high-precision landing of the drone.

[0019] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 is a flow chart of a method for visually guided landing of a UAV provided according to an embodiment of the present invention;

[0022] Figure 2a is a schematic diagram of a drone hangar applicable to an embodiment of the present invention;

[0023] Figure 2b is a top view of a drone hangar applicable to an embodiment of the present invention;

[0024] Figure 2c is a side view of a drone hangar to which an embodiment of the present invention is applicable;

[0025] Figure 3 is a flow chart of another method for visually guided landing of a UAV provided according to an embodiment of the present invention;

[0026] Figure 4 is a flow chart of another method for visually guided landing of a UAV provided according to an embodiment of the present invention;

[0027] Figure 5 is a structural schematic diagram of a vision-guided landing device for an unmanned aerial vehicle provided according to an embodiment of the present invention;

[0028] Figure 6 1 is a schematic diagram of the structure of an electronic device for implementing the vision-guided landing method for a drone according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] Embodiment 1

[0032] Figure 1 This is a flowchart of a method for visually guiding a drone to land in an embodiment of the present invention. This embodiment is applicable to the case where a drone lands in a drone hangar. The method can be performed by a drone visually guiding a landing device. The drone visually guiding a landing device can be implemented in the form of hardware and / or software. The drone visually guiding a landing device can be configured in any electronic device with network communication function. Figure 1 As shown, the UAV vision-guided landing method of the present invention includes the following process:

[0033] S110. When a first characteristic object of the drone hangar is detected, first image information of the first characteristic object is obtained, and spatial coordinate information of the target drone relative to the first characteristic object is determined based on the first image information; the first characteristic object is the outer contour of the drone hangar.

[0034] The first image information is the image information including the first feature object taken by the shooting device. The spatial coordinate information can be understood as the spatial three-dimensional coordinates of the target drone relative to the first feature object, which can effectively reflect the relative position information of the target drone and the drone hangar.

[0035] like Figure 2a As shown in FIG. 1 , a schematic diagram of the structure of the drone hangar applicable to the present embodiment is shown. The target drone flies out of the drone hangar to perform a mission, and then returns to the drone hangar after the mission is completed. Figure 2a It can be seen that the drone hangar is hemispherical, with a ring-shaped outer ring structure at the top. Figure 2b As shown, it is a top view of the drone hangar, and the annular outer ring structure can be clearly seen, which is used as the first characteristic object of this embodiment, so that when the drone lands and returns to the drone hangar, it can be used as a reference to accurately determine the relative position of the target drone with respect to the drone hangar.

[0036] Specifically, in response to the landing command of the target UAV, the pod carried by the target UAV is controlled to adjust the shooting equipment to a positive downward angle to ensure that the landing area of ​​the UAV hangar can be clearly photographed. The target UAV further includes an identification module, which is used to identify the features associated with the UAV hangar, that is, the identification module will also receive the landing command of the target UAV, and search for the features associated with the UAV hangar in real time during the landing process to guide the UAV to land.

[0037] Furthermore, when the target UAV lands at a preset distance from the UAV hangar, the recognition module can clearly identify the first feature object and pass the recognition result to the control module of the target UAV, so that the control module controls the shooting device to shoot the first feature object and obtain image information. By analyzing the pixel information in the image information and the actual position information of the first feature object, the spatial coordinate information of the target UAV relative to the first feature object can be accurately obtained.

[0038] In this embodiment, the recognition module can be used to identify features using a recognition model. The recognition model consists of target detection and YOLO recognition algorithms. The construction of the recognition model includes steps A1-A3:

[0039] Step A1: Obtain a training data set.

[0040] The training data set includes image information of the first feature object, the second feature object, and the third feature object from different perspectives, ensuring that the data set can cover the target object feature performance under various lighting, angles, distances, etc. Figure 2b and 2c Taking the first feature object, the second feature object and the third feature object shown as an example, image information of the first feature object, the second feature object and the third feature object at different viewing angles is collected as a training data set.

[0041] Step A2: training the pre-trained recognition model based on the training data set to accurately obtain weight information for adjusting parameters in the pre-trained recognition model; the pre-trained recognition model includes target detection and YOLO recognition algorithms.

[0042] Step A3: When the weight information satisfies the preset weight condition, the pre-trained recognition model corresponding to the weight information satisfying the preset weight condition is used as the recognition model to facilitate accurate recognition of the feature object during the subsequent landing of the UAV.

[0043] The technical solution of this embodiment, by collecting image information of feature objects from different perspectives as a training data set, lays a data foundation for subsequent model training, so that the algorithm can effectively identify and locate feature objects in a real environment, and further trains the pre-trained recognition model based on the training data set to continuously adjust the parameters in the pre-trained recognition model, thereby obtaining a recognition model with accurate parameters, so as to improve the accuracy of subsequent feature object recognition using the recognition model.

[0044] S120. When a second characteristic object of the drone hangar is detected, characteristic image information of the second characteristic object is obtained, and the heading angle of the target drone relative to the landing direction angle of the second characteristic object is determined based on the characteristic image information, and the target heading deviation of the target drone is determined based on the landing direction angle and the preset direction angle; the second characteristic object is located at a first preset position of the outer contour of the drone hangar.

[0045] The second feature object is located at the first preset position of the outer contour of the drone hangar. The second feature object can be understood as a marker that is easy for the drone to quickly identify during flight. Therefore, the second feature object is set at the first preset position on the outer contour of the drone hangar to ensure that the drone can quickly identify the second feature object after taking off, so as to accurately determine the heading angle of the target drone relative to the landing direction angle of the second feature object. For example, Figure 2b As shown, the second feature is a triangular mark located at a first preset position on the outer contour of the drone hangar.

[0046] Specifically, the characteristic image information can be understood as image information taken by the shooting device and containing the second characteristic object. The characteristic image information is processed to obtain the reference image information of the second characteristic object. The heading angle of the target UAV relative to the landing direction angle of the second characteristic object is obtained by analyzing the pixel information in the reference image information. The difference between the landing direction angle of the target UAV and the preset direction angle is further used as the target heading deviation, thereby adjusting the heading of the UAV according to the target heading deviation.

[0047] Optionally, the preset direction angle can be the take-off direction angle of the target UAV, and the take-off direction angle is the direction angle of the heading angle of the target UAV relative to the second feature object when the target UAV takes off. Specifically, in response to the take-off command of the target UAV, the target UAV takes off and identifies the second feature object on the UAV hangar in real time, and uses the direction angle of the heading angle of the target UAV relative to the second feature object when the target UAV takes off as the take-off direction angle. Furthermore, during the landing process of the target UAV, the landing direction angle of the heading angle of the target UAV relative to the second feature object during the landing process of the target UAV is determined in real time, so as to accurately determine the target heading deviation of the target UAV according to the difference between the take-off direction angle and the landing direction angle.

[0048] S130: Control the target UAV to land in the UAV hangar according to the spatial coordinate information and the target heading deviation.

[0049] The flight parameters may include but are not limited to the flight trajectory and attitude of the target UAV.

[0050] Specifically, the spatial coordinate information and the target heading deviation are obtained, and the attitude that the drone needs to adjust is determined according to the positive and negative and size of the target heading deviation. The attitude may include adjusting the direction and adjusting the angle. The adjustment of the direction can be determined based on the comparison result between the target heading deviation and the preset deviation. For example, if the target heading deviation>preset deviation, it means that the heading of the drone is biased to the right (assuming that the left and right are judged according to the direction of the drone's advance), and the heading needs to be adjusted to the left; if the target heading deviation<preset deviation, it means that the heading of the drone is biased to the left, and the heading needs to be adjusted to the right. Further, the adjustment amount of the yaw angle is determined according to the size of the target heading deviation. A preset proportional coefficient can be set, and the target heading deviation is multiplied by the preset proportional coefficient to obtain the adjustment angle of the yaw angle. If the adjustment value = 0.1 and the target heading deviation = 30 degrees, then the adjustment angle = 0.1*30 = 3 degrees.

[0051] Furthermore, the flight trajectory of the target UAV is determined based on the spatial coordinate information and the actual location of the UAV hangar.

[0052] In this embodiment, optionally, after determining the spatial coordinate information of the target UAV relative to the first feature object, the method further includes: obtaining spatial coordinate information of a preset number of frames within a preset time range from the current moment as reference spatial coordinate information; taking the average of the reference spatial coordinate information to obtain updated spatial coordinate information, so as to adjust the flight parameters of the target UAV according to the updated spatial coordinate information and the target heading deviation.

[0053] Specifically, the preset number of frames can be set according to actual needs, and no specific limitation is made here. For example, the preset number of frames is 3. In addition, the spatial coordinate information of the preset number of frames within the preset time range from the current moment is obtained as the reference spatial coordinate information in order to obtain the spatial coordinate information of the target drone that has been recorded for the most recent preset number of frames. Because the interval time is too long, the difference between the spatial coordinate information of the two frames before and after may be too large, and it is meaningless to take the average of the reference spatial coordinate information to obtain the updated spatial coordinate information. Further taking the average of the reference spatial coordinate information to obtain the updated spatial coordinate information is to reduce the influence of environmental noise and transient errors, and obtain more stable current spatial coordinate information, so that the flight parameters of the target drone can be adjusted more accurately according to the spatial coordinate information and the target heading deviation, to ensure the accuracy of the drone landing.

[0054] The technical solution of the embodiment of the present invention is as follows: when a first characteristic object of a drone hangar is detected, the first image information of the first characteristic object is obtained, and the spatial coordinate information of the target drone relative to the first characteristic object is determined based on the first image information, so as to determine in real time the spatial coordinate information of the drone relative to the outer circle of the drone hangar during landing; when a second characteristic object of the drone hangar is detected, the characteristic image information of the second characteristic object is obtained, and the heading angle of the target drone relative to the landing direction angle of the second characteristic object is determined based on the characteristic image information, and the target heading deviation of the target drone is determined based on the landing direction angle and the preset direction angle; the second characteristic object is located at a first preset position of the outer contour of the drone hangar, so as to accurately determine the landing direction angle, thereby accurately deriving whether the target drone has yawed; finally, the target drone is controlled to land in the drone hangar based on the spatial coordinate information of the first characteristic object and the target heading deviation, so as to achieve high-precision landing of the drone.

[0055] Embodiment 2

[0056] Figure 3 This is a flow chart of another method for visually guiding the landing of a drone provided by an embodiment of the present invention. The technical solution of this embodiment further optimizes the process of determining the spatial coordinate information of the target drone relative to the first feature object in the above embodiment on the basis of the above embodiment. This embodiment can be combined with various optional solutions in one or more of the above embodiments. Figure 3 As shown, the UAV vision-guided landing method of the present invention includes the following process:

[0057] S210, obtaining first size information of the first feature object, and determining spatial coordinate information of the target drone relative to the first feature object according to the first image information and the first size information; the first size information is actual size information of the first feature object.

[0058] Specifically, the first image information records the pixel information of the first feature object, and the relationship between the pixel information of the first feature object and the first size information is established. After obtaining the first image information and the first size information, the spatial coordinate information of the target drone relative to the first feature object can be accurately determined based on the relationship between the pixel information of the first feature object and the first size information.

[0059] In this embodiment, optionally, determining the spatial coordinate information of the target drone relative to the first feature object according to the first image information and the first size information includes steps B1-B3:

[0060] Step B1, determining the second size information and the center point pixel coordinates of the first feature object according to the first image information; the second size information is the pixel size information of the first feature object in the first image information.

[0061] Specifically, the center pixel coordinates can be determined based on the intrinsic parameter matrix of the shooting device or the center pixel coordinates can be understood as the coordinate information of the center point on the first image information. The second size information can be understood as the second size information of the first feature object obtained by separating the background and foreground of the first image information.

[0062] Step B2: determining vertical axis coordinate information in the vertical axis direction according to the first size information and the second size information.

[0063] Specifically, the vertical axis coordinate information is expressed using the following formula:

[0064] ;

[0065] Among them, Z represents the vertical axis coordinate information, f y is the vertical focal length of the shooting device, h image is the first size information, h real It is the second size information.

[0066] Step B3: determine the horizontal axis coordinate information and the vertical axis coordinate information according to the vertical axis coordinate information and the center point pixel coordinates.

[0067] Specifically, the horizontal axis coordinate information and the vertical axis coordinate information are expressed by the following formula:

[0068] ;

[0069] ;

[0070] Among them, x is the horizontal axis coordinate information, y is the vertical axis coordinate information, (u, v) is the pixel coordinate of the center point, and f x The horizontal focal length of the shooting device.

[0071] S220. When a second characteristic object of the drone hangar is detected, characteristic image information of the second characteristic object is obtained, and the heading angle of the target drone relative to the landing direction angle of the second characteristic object is determined based on the characteristic image information, and the target heading deviation of the target drone is determined based on the landing direction angle and the preset direction angle; the second characteristic object is located at a first preset position of the outer contour of the drone hangar.

[0072] Specifically, the second feature object has orientation angle information of a preset orientation on the drone hangar, and determining the heading angle of the target drone relative to the landing direction angle of the second feature object based on the feature image information can include: determining a reference orientation angle of the second feature object based on the feature image information; the reference orientation angle is used to describe the angle information of the preset orientation indicated by the second feature object in the feature image information; and determining the landing direction angle of the target drone relative to the preset orientation of the second feature object based on the reference orientation angle and the orientation angle information.

[0073] Among them, the orientation angle information is used to describe the angle information of the preset orientation of the second feature object marked on the drone hangar, that is, it can reflect the actual orientation angle of the target drone in the drone hangar, so as to determine the reference orientation angle of the second feature object indicated in the characteristic image information captured during the landing of the target drone, and then compare the reference orientation angle with the orientation angle information to accurately obtain the landing direction angle of the target drone relative to the preset orientation of the second feature object, thereby ensuring the accurate determination of the landing direction angle.

[0074] Furthermore, the preset direction angle can be the take-off direction angle of the target UAV. Then, in response to the take-off command of the target UAV, the second feature object can be photographed by the photographing device on the target UAV to obtain reference feature image information, and the feature object orientation angle of the second feature object at take-off can be determined according to the reference feature image information; the feature object orientation angle is used to describe the angle information of the preset orientation indicated by the second feature object in the reference feature image information; according to the feature object orientation angle and the orientation angle information, the take-off direction angle of the target UAV relative to the preset orientation of the second feature object is determined. Then, the absolute value of the difference between the take-off direction angle and the landing direction angle is used as the target heading deviation of the target UAV.

[0075] S230: Control the target UAV to land in the UAV hangar according to the spatial coordinate information and the target heading deviation.

[0076] The technical solution of the embodiment of the present invention obtains the first size information of the first feature object, and determines the spatial coordinate information of the target drone relative to the first feature object based on the first image information and the first size information; the first size information is the actual size information of the first feature object, that is, the combination of the actual size information and pixel information of the first feature object can more accurately reflect the spatial coordinate information of the target drone relative to the first feature object. Further, when the second feature object of the drone hangar is detected, the characteristic image information of the second feature object is obtained, and the heading angle of the target drone relative to the landing direction angle of the second feature object is determined based on the characteristic image information, and the target heading deviation of the target drone is determined based on the landing direction angle and the preset direction angle; then, the target drone is controlled to land in the drone hangar based on the spatial coordinate information and the target heading deviation, so as to achieve high-precision landing of the drone.

[0077] Embodiment 3

[0078] Figure 4 This is a flow chart of another method for visually guided landing of a drone provided by an embodiment of the present invention. The technical solution of this embodiment further optimizes the determination of spatial coordinate information in the above embodiment on the basis of the above embodiment. This embodiment can be combined with various optional solutions in one or more of the above embodiments. Figure 4 As shown, the UAV vision-guided landing method of the present invention includes the following process:

[0079] S310. After obtaining the first image information of the first feature object, if the target drone is at a preset distance from the drone hangar and / or the first feature object in the first image information is incomplete, obtain the second image information of the third feature object taken by the shooting device; the third feature object is located at a second preset position inside the drone hangar.

[0080] Specifically, the third feature is located at the second preset position at the bottom of the drone hangar, for example, Figure 2b and 2c As shown, the third preset position can be the middle position of the bottom of the drone hangar. The third feature object is set because when the drone lands at a certain position, due to the field of view limitation of the shooting device, it may be impossible to shoot the first feature object. Therefore, the third feature object is set to facilitate the acquisition of the second image information of the third feature object shot by the shooting device. In the case of incomplete first feature object, the spatial coordinate information of the target drone relative to the third feature object can still be accurately determined based on the second image information and the third size information.

[0081] S320. Determine spatial coordinate information of the target UAV relative to the third feature object according to the second image information and the third size information; the third size information is actual size information of the third feature object.

[0082] Specifically, determining the spatial coordinate information of the target drone relative to the third feature object according to the second image information and the third size information may include steps C1-C3:

[0083] Step C1, determining fourth size information and center point pixel coordinates of the third feature object according to the second image information; the fourth size information is the pixel size information of the first feature object in the first image information.

[0084] Step C2: determining vertical axis coordinate information in the vertical axis direction according to the third size information and the fourth size information.

[0085] Specifically, the vertical axis coordinate information may be expressed by the following formula:

[0086] ;

[0087] Among them, Z represents the vertical axis coordinate information, f y is the vertical focal length of the shooting device, h real is the third size information, h image It is the fourth size information.

[0088] Step C3: determine the horizontal axis coordinate information and the vertical axis coordinate information according to the vertical axis coordinate information and the center point pixel coordinates.

[0089] Specifically, the horizontal axis coordinate information and the vertical axis coordinate information are expressed by the following formula:

[0090] ;

[0091] ;

[0092] Among them, x is the horizontal axis coordinate information, y is the vertical axis coordinate information, (u, v) is the pixel coordinate of the center point, and f x The horizontal focal length of the shooting device.

[0093] S330. When a second characteristic object of the drone hangar is detected, characteristic image information of the second characteristic object is obtained, and the heading angle of the target drone relative to the landing direction angle of the second characteristic object is determined based on the characteristic image information, and the target heading deviation of the target drone is determined based on the landing direction angle and the preset direction angle; the second characteristic object is located at a first preset position of the outer contour of the drone hangar.

[0094] S340: Control the target UAV to land in the UAV hangar according to the spatial coordinate information and the target heading deviation.

[0095] The technical solution of this embodiment, after obtaining the first image information of the first feature object, determines if the target drone is at a preset distance from the drone hangar and / or the first feature object in the first image information is incomplete, then obtains the second image information of the third feature object captured by the shooting device; that is, it realizes real-time judgment of the distance of the drone close to the drone hangar and whether the first feature object in the first image information is complete, avoiding the adjustment of the drone posture due to the limitation of the field of view. Further, the spatial coordinate information of the target drone relative to the third feature object is determined based on the second image information and the third size information, so as to control the target drone to land in the drone hangar based on the spatial coordinate information of the third feature object and the target heading deviation, and realizes the accurate determination of the spatial coordinate information.

[0096] Embodiment 4

[0097] Figure 5 This is a schematic diagram of the structure of a UAV visual guidance landing device provided by an embodiment of the present invention. This embodiment is applicable to the situation where a UAV lands in a UAV hangar. The UAV visual guidance landing device can be implemented in the form of hardware and / or software. The UAV visual guidance landing device can be configured in any electronic device with network communication function. Figure 5 As shown, the UAV vision-guided landing device of the present invention comprises:

[0098] The information determination module 310 is used to obtain first image information of the first feature object when a first feature object of the drone hangar is detected, and determine the spatial coordinate information of the target drone relative to the first feature object according to the first image information; the first feature object is the outer contour of the drone hangar;

[0099] The heading determination module 320 is used to, when a second feature object of the drone hangar is detected, obtain feature image information of the second feature object, determine the heading angle of the target drone relative to the landing direction angle of the second feature object according to the feature image information, and determine the target heading deviation of the target drone according to the landing direction and a preset direction angle; the second feature object is located at a first preset position of the outer contour of the drone hangar;

[0100] The control module 330 is used to control the target UAV to land in the UAV hangar according to the spatial coordinate information and the target heading deviation.

[0101] On the basis of the above embodiment, optionally, the information determination module includes a judgment unit and a reference coordinate information determination unit; the judgment unit is used to obtain the second image information of the third feature object captured by the shooting device if the target drone is at a preset distance from the drone hangar and / or the first feature object in the first image information is incomplete; the third feature object is located at a second preset position inside the drone hangar. The reference coordinate information determination unit is used to determine the spatial coordinate information of the target drone relative to the third feature object based on the second image information and the third size information, so as to control the target drone to land in the drone hangar based on the spatial coordinate information and the target heading deviation; the third size information is the actual size information of the third feature object.

[0102] Based on the above embodiment, optionally, the second feature object has orientation angle information of a preset orientation on the UAV hangar, and the heading determination module is used to determine a reference orientation angle of the second feature object based on the feature image information; the reference orientation angle is used to describe the angle information of the preset orientation indicated by the second feature object in the feature image information; based on the reference orientation angle and the orientation angle information, the landing direction angle of the target UAV relative to the preset orientation of the second feature object is determined.

[0103] Based on the above embodiment, optionally, the preset direction angle is a take-off direction angle of the target UAV, and the take-off direction angle is a direction angle of the heading angle of the target UAV relative to the second feature object when the target UAV takes off.

[0104] Based on the above embodiment, optionally, the information determination module further includes a response unit, which is used to control the pod carried by the target UAV to adjust the shooting device to a positive downward angle in response to a landing instruction of the target UAV.

[0105] Based on the above embodiment, optionally, the information determination module includes a coordinate information determination unit; the coordinate information determination unit is used to obtain first size information of the first feature object, and determine the spatial coordinate information of the target drone relative to the first feature object according to the first image information and the first size information; the first size information is the actual size information of the first feature object;

[0106] On the basis of the above embodiment, optionally, the coordinate information determining unit includes a pixel coordinate determining subunit, a first coordinate information determining subunit, and a second coordinate information determining subunit;

[0107] A pixel coordinate determination subunit is used to determine the second size information and center point pixel coordinates of the first feature object based on the first image information; the second size information is the pixel size information of the first feature object in the first image information; a first coordinate information determination subunit is used to determine the vertical axis coordinate information in the vertical axis direction based on the first size information and the second size information; a second coordinate information determination subunit is used to determine the horizontal axis coordinate information and the vertical axis coordinate information based on the vertical axis coordinate information and the center point pixel coordinates.

[0108] Based on the above embodiment, optionally, the vertical axis coordinate information is expressed by the following formula:

[0109] ;

[0110] Among them, Z represents the vertical axis coordinate information, f y is the vertical focal length of the shooting device, h image is the first size information, h real It is the second size information.

[0111] Based on the above embodiment, optionally, the following formula is used to represent the horizontal axis coordinate information and the vertical axis coordinate information:

[0112] ;

[0113] ;

[0114] Among them, x is the horizontal axis coordinate information, y is the vertical axis coordinate information, (u, v) is the pixel coordinate of the center point, and f x The horizontal focal length of the shooting device.

[0115] On the basis of the above embodiment, optionally, the image acquisition unit also includes a judgment subunit, which is used to obtain second image information of a third feature object captured by a shooting device if the target drone is at a preset distance from the drone hangar and / or the first feature object in the first image information is incomplete, so as to determine the spatial coordinate information of the target drone relative to the third feature object based on the second image information and the third size information; the third feature object is located at a second preset position at the bottom of the drone hangar, and the third size information is the actual size information of the third feature object.

[0116] Based on the above embodiment, optionally, the information determination module also includes an updating unit, which is used to obtain spatial coordinate information of a preset number of frames within a preset time range from the current moment as reference spatial coordinate information; taking the average of the reference spatial coordinate information to obtain updated spatial coordinate information, so as to adjust the flight parameters of the target UAV according to the updated spatial coordinate information and the target heading deviation.

[0117] The UAV vision-guided landing device provided in the embodiment of the present invention can execute the UAV vision-guided landing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0118] Embodiment 4

[0119] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0120] Figure 6 The structural schematic diagram of an electronic device that can be used to implement the UAV visual guidance landing method of an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0121] like Figure 6As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. The input / output (I / O) interface 15 is also connected to the bus 14.

[0122] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0123] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the drone vision-guided landing method.

[0124] In some embodiments, the drone vision-guided landing method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the drone vision-guided landing method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the drone vision-guided landing method in any other appropriate manner (e.g., by means of firmware).

[0125] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0126] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0127] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0128] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0129] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0130] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0131] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0132] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for visually guided landing of an unmanned aerial vehicle, characterized in that: The method comprises: When a first feature object of the drone hangar is detected, first image information of the first feature object is obtained, and spatial coordinate information of the target drone relative to the first feature object is determined according to the first image information; the first feature object is the outer contour of the drone hangar; When a second characteristic object of the drone hangar is detected, characteristic image information of the second characteristic object is obtained, a heading angle of the target drone relative to a landing direction angle of the second characteristic object is determined according to the characteristic image information, and a target heading deviation of the target drone is determined according to the landing direction angle and a preset direction angle; the second characteristic object is located at a first preset position of the outer contour of the drone hangar; Control the target UAV to land in the UAV hangar according to the spatial coordinate information and the target heading deviation; After obtaining the first image information of the first feature object, the method further includes: If the target drone is at a preset distance from the drone hangar and / or the first feature in the first image information is incomplete, obtaining second image information of a third feature captured by a photographing device; the third feature is located at a second preset position inside the drone hangar; The spatial coordinate information of the target UAV relative to the third characteristic object is determined according to the second image information and the third size information, so as to control the target UAV to land in the UAV hangar according to the spatial coordinate information and the target heading deviation; the third size information is the actual size information of the third characteristic object.

2. The method according to claim 1, characterized in that The second feature object has orientation angle information of a preset orientation on the drone hangar, and determining the landing direction angle of the heading angle of the target drone relative to the second feature object according to the feature image information includes: Determine a reference orientation angle of the second feature object according to the feature image information; the reference orientation angle is used to describe angle information of a preset orientation indicated by the second feature object in the feature image information; The landing direction angle of the target UAV relative to the preset direction of the second feature object is determined according to the reference direction angle and the direction angle information.

3. The method according to claim 1 or 2, characterized in that: The preset direction angle is a take-off direction angle of the target UAV, and the take-off direction angle is a direction angle of the heading angle of the target UAV relative to the second feature object when the target UAV takes off.

4. The method according to claim 1, characterized in that: Determining spatial coordinate information of the target drone relative to the first feature object according to the first image information includes: Acquire first size information of the first feature object; the first size information is actual size information of the first feature object; The spatial coordinate information of the target UAV relative to the first feature object is determined according to the first image information and the first size information.

5. The method according to claim 4, characterized in that Determining spatial coordinate information of the target drone relative to the first feature object according to the first image information and the first size information includes: Determine second size information and center point pixel coordinates of the first feature object according to the first image information; the second size information is pixel size information of the first feature object in the first image information; Determine vertical axis coordinate information in the vertical axis direction according to the first size information and the second size information; The horizontal axis coordinate information and the vertical axis coordinate information are determined according to the vertical axis coordinate information and the center point pixel coordinates.

6. The method according to claim 1, characterized in that After determining the spatial coordinate information of the target UAV relative to the first feature object according to the first image information, the method further includes: Obtaining spatial coordinate information of a preset number of frames within a preset time range from the current moment as reference spatial coordinate information; The reference space coordinate information is averaged to obtain updated space coordinate information, so as to control the target UAV to land in the UAV hangar according to the updated space coordinate information and the target heading deviation.

7. The method according to claim 1, characterized in that Before determining the spatial coordinate information of the target drone relative to the first feature object according to the first image information, the method further includes: In response to a landing instruction of the target UAV, the pod carried by the target UAV is controlled to adjust the shooting equipment to a positive downward angle.

8. A vision-guided landing device for unmanned aerial vehicles, characterized in that: The device comprises: An information determination module is used to obtain first image information of the first feature object when a first feature object of the drone hangar is detected, and determine the spatial coordinate information of the target drone relative to the first feature object according to the first image information; the first feature object is the outer contour of the drone hangar; A heading determination module is used for, when a second characteristic object of the drone hangar is detected, obtaining characteristic image information of the second characteristic object, determining the heading angle of the target drone relative to the landing direction angle of the second characteristic object according to the characteristic image information, and determining the target heading deviation of the target drone according to the landing direction and a preset direction angle; the second characteristic object is located at a first preset position of the outer contour of the drone hangar; A control module, used for controlling the target UAV to land in a UAV hangar according to the spatial coordinate information and the target heading deviation; Among them, the information determination module is also used to obtain the second image information of the third feature object taken by the shooting device if the target drone is at a preset distance from the drone hangar and / or the first feature object in the first image information is incomplete; the third feature object is located at a second preset position inside the drone hangar; the spatial coordinate information of the target drone relative to the third feature object is determined based on the second image information and the third size information, so as to control the target drone to land in the drone hangar based on the spatial coordinate information and the target heading deviation; the third size information is the actual size information of the third feature object.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the vision-guided landing method for an unmanned aerial vehicle according to any one of claims 1 to 7.

Images