An unmanned aerial vehicle obstacle avoidance method based on hand-eye coordination

Abstract

The application relates to the field of unmanned aerial vehicle applications, in particular to an unmanned aerial vehicle obstacle avoidance method based on hand-eye coordination. The method comprises the following steps: planting protection unmanned aerial vehicles and fixedly arranging positioning unmanned aerial vehicles in a working area; positioning target crops in the working area by the planting protection unmanned aerial vehicles, and confirming an initial flight route of the planting protection unmanned aerial vehicles moving to the target crops; collecting a plurality of feature points about obstacles in the initial route by the planting protection unmanned aerial vehicles and the positioning unmanned aerial vehicles respectively, and performing feature matching to obtain positioning coordinate information; determining a height parameter about the obstacles and an area parameter about the obstacles according to the positioning coordinate information; and determining an obstacle avoidance flight route of the planting protection unmanned aerial vehicles moving to the target crops according to the height parameter about the obstacles and the area parameter about the obstacles. Compared with the prior art, the application can effectively solve the problem of formulating an obstacle avoidance flight route in a small area range, and the operability is stronger, and the work efficiency is greatly improved.

Description

Technical Field

[0001] This application relates to the field of drone applications, and in particular to a drone obstacle avoidance method based on hand-eye coordination. Background Technology

[0002] In agricultural production, agricultural drones can be used to remotely and in real-time monitor target crops. When controlling an agricultural drone for operation, the drone first needs to plan its obstacle avoidance flight route using a GPS satellite navigation system, then use computer vision to enable the drone to perceive the farmland environment, and finally reach the vicinity of the target crop and collect relevant information about it.

[0003] The GPS satellite navigation system plans the flight path of a drone based on latitude and longitude coordinates, then performs the corresponding calculations to obtain the obstacle avoidance flight path.

[0004] However, in practical applications, when drones operate in a small area, the GPS satellite navigation system cannot achieve accurate positioning within that area. This results in inaccurate planning of obstacle avoidance flight routes for drones by the GPS satellite navigation system, which uses latitude and longitude as a coordinate system. Therefore, the GPS satellite navigation system is not suitable as an obstacle avoidance flight strategy for agricultural drones operating in a small area. Summary of the Invention

[0005] This application provides a hand-eye coordination-based obstacle avoidance method for agricultural drones, which can solve the problem that GPS satellite navigation systems cannot provide accurate obstacle avoidance flight strategies for agricultural drones operating in small areas.

[0006] The technical solution of this application is a drone obstacle avoidance method based on hand-eye coordination, including:

[0007] S1: The agricultural drone is set up and moved within the work area, and a positioning drone is fixedly set above the work area at a height higher than that of the agricultural drone.

[0008] S2: Receive crop disease information strategy through plant protection drone and locate the target crop in the operation area according to the crop disease information strategy, and confirm the initial flight path of plant protection drone to move to the target crop.

[0009] S3: Collect several feature points of obstacles in the initial path using agricultural drones and positioning drones respectively, to obtain a first initial image and a second initial image respectively, and perform feature matching on the feature points in the first initial image and the second initial image to obtain the positioning coordinate information of several feature points;

[0010] S4: Based on the positioning coordinates of several feature points, determine the height parameters and area parameters of the obstacle accordingly;

[0011] S5: Determine the obstacle avoidance flight path for the agricultural drone to move to the target crop based on the obstacle's height and area parameters.

[0012] Optionally, step S3 includes:

[0013] S31: Determine the relative positional relationship between the agricultural drone and the positioning drone;

[0014] S32: Collect several feature points of obstacles in the initial path using agricultural drones and positioning drones respectively, and obtain the first initial image and the second initial image respectively;

[0015] S33: Based on the relative positional relationship and epipolar constraint, feature matching is performed on feature points in the first and second initial images to obtain the positioning coordinate information of several feature points.

[0016] Optionally, the agricultural drone is equipped with a mobile vision acquisition system, which includes a mobile camera;

[0017] The positioning drone is equipped with a fixed visual acquisition system, which includes a fixed camera.

[0018] And, step S31 includes:

[0019] S311: Determine the intrinsic and extrinsic parameters of the moving and fixed cameras respectively;

[0020] S312: Determine the relative positional relationship between the agricultural drone and the positioning drone in the visual acquisition coordinate system based on the intrinsic and extrinsic parameters of the mobile and fixed cameras.

[0021] Optionally, step S33 includes:

[0022] S331: Either determine the feature points in the first initial image or determine the feature points in the second initial image;

[0023] S332: Based on the relative positional relationship and epipolar constraint conditions, or according to the position of the feature points in the first image, determine the position of the feature points in the second initial image accordingly to obtain the world coordinate information of several feature points, or according to the position of the feature points in the second initial image, determine the position of the feature points in the first initial image to obtain the world coordinate information of several feature points;

[0024] S333: Convert the world coordinate information of several feature points into the positioning coordinate information in the visual acquisition coordinate system to obtain the positioning coordinate information of several feature points.

[0025] Optionally, step S4 includes:

[0026] S41: Based on the positioning coordinates of several feature points, determine the height parameters of the obstacle using a mobile vision acquisition system;

[0027] S42: Based on the positioning coordinates of several feature points, determine the area parameters of the obstacle using a fixed vision acquisition system.

[0028] Optionally, step S5 includes:

[0029] S51: Based on the height and area parameters of the obstacle, and the relative positional relationship between the agricultural drone and the positioning drone in the visual acquisition coordinate system, determine the obstacle avoidance flight path of the agricultural drone to the target crop.

[0030] Beneficial effects:

[0031] This application replaces the GPS visual measurement system with a positioning drone, and combines it with agricultural drones to locate and judge obstacles in a small area, specifically obtaining information about the height and area of ​​the obstacles, and then deriving a reasonable obstacle avoidance flight route.

[0032] Compared with existing technologies, this application can effectively and specifically solve the problem of formulating obstacle avoidance flight routes within a small area. Furthermore, since the positioning drone is set within a small area, its sensitivity and operability are stronger, and the application location can be adjusted according to local conditions, which greatly improves the working efficiency of agricultural drones and fully guarantees the overall operational efficiency.

[0033] In summary, this application can solve the problem that GPS satellite navigation systems cannot provide accurate obstacle avoidance flight strategies for agricultural drones operating in small areas. Attached Figure Description

[0034] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1 This is a schematic diagram of the hand-eye coordination control strategy in the embodiments of this application;

[0036] Figure 2This is a flowchart illustrating a hand-eye coordination-based obstacle avoidance method for unmanned aerial vehicles (UAVs) in an embodiment of this application.

[0037] Figure 3 This is a schematic diagram showing the positions of the world coordinate system and the positioning coordinate system of the mobile camera and the fixed camera in the embodiments of this application;

[0038] Figure 4 This is an example diagram illustrating the calculation of obstacle area parameters using a positioning drone in this application embodiment;

[0039] Figure 5 Satellite images of the test site in the embodiments of this application;

[0040] Figure 6 The satellite images collected by the GPS visual measurement system in this embodiment cover an area of ​​800,000 square meters.

[0041] Figure 7 This is a data table of measurement results for check points on a plane in the GPS visual measurement system of this application embodiment. Detailed Implementation

[0042] The embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described below do not represent all embodiments consistent with this application. They are merely examples of systems and methods consistent with some aspects of this application as detailed in the claims.

[0043] The initial meaning of hand-eye coordination in the embodiments of this application refers to the coordination of fine hand movements with visual assistance. This control concept was later applied to the field of industrial robot control and is widely used in the visual positioning and tracking control strategy of robotic arms. Its core is to use computer vision feedback signals to plan the robot's motion curve, so that the robot can complete the target capture and tracking tasks through visual positioning.

[0044] In the embodiments of this application, such as Figure 1 As shown, Figure 1 This is a schematic diagram of the hand-eye coordination control strategy in the embodiments of this application. The application idea of ​​the hand-eye coordination control mode in the agricultural environment includes: labeling the plant protection drone as "hand", defining the vision system of the low-altitude positioning drone above the work area as "eye", and using the "eye" to observe the "movement of the hand" in real time and track and locate it, thus forming a closed loop of visual detection and drone flight control.

[0045] This application provides an embodiment of a drone obstacle avoidance method based on hand-eye coordination, such as... Figure 2 As shown, Figure 2This is a flowchart illustrating a hand-eye coordination-based obstacle avoidance method for unmanned aerial vehicles (UAVs) according to an embodiment of this application, including:

[0046] S1: The agricultural drone is set up in the work area, and a positioning drone is fixed above the work area at a height higher than that of the agricultural drone.

[0047] Specifically, when agricultural drones operate in small work areas, high accuracy is required for navigation and obstacle avoidance. Therefore, GPS positioning and navigation systems are insufficient. In this positioning system, in addition to the agricultural drone itself, a positioning drone with a flight altitude of 50 meters is also installed. A visual acquisition device equipped with a gimbal is installed below the positioning drone to perform real-time observation of the agricultural drone and the work area. Since the hovering altitude of the positioning drone remains constant, controlling the stability of its flight altitude is sufficient to ensure that the internal and external parameters of its visual acquisition system remain essentially unchanged.

[0048] Coordinate transformations using computer vision can provide ground base station computers with overall positional information and geometric parameters of agricultural drones and obstacles. This provides precise coordinates and obstacle edge size data for the implementation of drone obstacle avoidance strategies. Combined with the agricultural drone's eyes-in-hand visual information acquisition system, it provides strategies for controlling the agricultural drone to perform accurate and reasonable obstacle avoidance maneuvers.

[0049] S2: Receive crop disease information strategy through agricultural drones and locate the target crop in the operation area according to the crop disease information strategy, and confirm the initial flight path of the agricultural drone to the target crop.

[0050] Specifically, as a visual navigation and obstacle avoidance system for agricultural drones, after the agricultural drone receives crop disease information and strategies from the GPS hyperspectral imaging system, the system analyzes the strategies and guides the agricultural drone to locate the target crop. During the flight of the agricultural drone, electronic eyes are used to mark obstacles and plan flight routes and obstacle avoidance routes.

[0051] S3: Collect several feature points of obstacles in the initial path using agricultural drones and positioning drones respectively, to obtain a first initial image and a second initial image, and perform feature matching on the feature points in the first initial image and the second initial image to obtain the positioning coordinate information of several feature points.

[0052] Step S3 includes:

[0053] S31: Determine the relative positional relationship between the agricultural drone and the positioning drone.

[0054] Agricultural drones are equipped with a mobile vision acquisition system, which includes a mobile camera.

[0055] The positioning drone is equipped with a fixed visual acquisition system, which includes a fixed camera.

[0056] Specifically, the positioning drone uses a mobile vision acquisition system to locate the pixel coordinates of the acquired images, forming an "eyes on hand" vision control system that calibrates the positional relationship between the agricultural drone and obstacles on the working plane in real time.

[0057] The agricultural drone itself uses a fixed vision acquisition system to locate the pixel coordinates of the acquired images, forming an "eyes in hand" vision control system. This system identifies obstacles and can determine the distance and height relationship between the agricultural drone and the obstacle by measuring the proportion of the obstacle's edge features that occupy the entire image area.

[0058] By coupling the two sets of image feedback information, "eyes on hand" and "eyes in hand", a relatively accurate obstacle avoidance strategy can be provided for the drone's obstacle avoidance maneuvers.

[0059] Step S31 includes:

[0060] S311: Determine the intrinsic and extrinsic parameters of the moving and fixed cameras respectively.

[0061] Specifically, within the visual acquisition coordinate system of computer vision acquisition, the positional relationship of any target point p in the initial flight path can be represented by a 3×1 column vector:

[0062] C P = [P] x P y P z ] T (1-1)

[0063] The orientation of a spatial object W can be described by a 3×3 matrix consisting of the directions of three coordinate vectors of a world coordinate system {W} fixed to the object relative to the visual acquisition coordinate system {C}.

[0064] C x = W xcosθ- W ysinθ (1-2)

[0065] C y = W xsinθ+ W ycosθ (1-3)

[0066] C z = W z P (1-4)

[0067] The above formula can be simplified to:

[0068]

[0069] in:

[0070]

[0071] Therefore, coordinate system {C} can be obtained by rotating coordinate system {W} around a certain coordinate axis of {C}. The rotation matrix around the x, y, and z axes is as follows:

[0072]

[0073]

[0074]

[0075] The three equations (1-7) to (1-9) above can be written as follows:

[0076]

[0077] This matrix is ​​a rotation matrix and is necessarily orthogonal in three-dimensional space, that is:

[0078]

[0079] Rotation matrix It can be used as a coordinate transformation matrix to change the coordinates of P in coordinate system {W}. W P transforms to the coordinates of the midpoint. C P.

[0080] It can also be used as an operator to transform vectors or objects in {W} into {C}.

[0081] The coordinates of point P can be obtained by transforming the world coordinate system {W} to {C} using the formula shown below:

[0082]

[0083] in C P WO Let {W} be the position of the origin of coordinate system {W} within coordinate system {C}.

[0084] If the coordinate transformation is a translation, then:

[0085] C P = W P+ C P WO (1-13)

[0086] If the coordinate transformation is a rotation, then:

[0087]

[0088] Equation (1-14) can be written as follows:

[0089]

[0090] The position vectors of point P in {W} and {C} are respectively augmented as follows:

[0091]

[0092] Substituting the parameters, equation (1-14) becomes:

[0093]

[0094] in:

[0095] Let be the extrinsic parameter matrix of the camera.

[0096] From the pinhole model, we can see that:

[0097]

[0098] Where (u,v) are the pixel coordinates on the imaging plane, and the pixel coordinates of the intersection of the optical axis centerline and the imaging plane are denoted as (u0,v0), k x k y This is the magnification factor. in T is the intrinsic parameter matrix, where T is the intrinsic parameter matrix of the visual acquisition system. in T contains four parameters, hence it is called the four-parameter model of the visual system.

[0099] According to the Faugeras camera calibration method:

[0100]

[0101] Let matrix

[0102] but:

[0103]

[0104] Substituting (1-19) into (1-18) yields:

[0105]

[0106] Expanding gives:

[0107] c z·u = t 11 · w x + t 12 · w y + t 13 · w z + t 14 (1 - 21)

[0108] c z·v = t 21 · w x + t 22 · w y + t 23 · w z + t 24 (1 - 22)

[0109] c z = t 31 · w x + t 32 · w y + t 33 · w z + t 34 (1 - 23)

[0110] After arranging:

[0111] t 11 · w x + t 12 · w y + t 13 · w z + t 14 +(-u·t 34 )+(-u· w x·t 31 )+(-u· w y·t 32 )+(-u· w z·t 33 ) = 0 (1 - 24)

[0112] t 21 · w x + t 22 · w y + t 23 · w z + t 24 +(-v·t 34 )+(-v· w x·t 31 )+(-v· w y·t 32)+(-v· w z·t 33 )=0 (1-25)

[0113] For n spatial points with known coordinates in the {W} coordinate system, the following equation can be derived from (1-24) and (1-25):

[0114]

[0115] It can be rewritten as:

[0116] W9T9+W3T3=0 (1-26)

[0117] In the formula:

[0118] T9 = [t 11 t 12 t 13 t 14 t 21 t 22 t 23 t 24 t 34 ] T (1-27)

[0119] T3=[t 31 t 32 t 33 ] T

[0120]

[0121]

[0122] Agricultural drones need to be designed to operate in chaotic low-altitude environments. An index function needs to be constructed using equation (1-26) and ||T3||=1 as constraints.

[0123] W R =||W9T9+W3T3|| 2 +λ||T3|| 2 -1) (1-28)

[0124] Expanding it, we get:

[0125]

[0126] Right now:

[0127]

[0128] W R Take the partial derivatives with respect to T3 and r, and set them to 0, so that W RTo obtain the minimum value, we can get:

[0129]

[0130]

[0131]

[0132] This allows us to solve for the intrinsic and extrinsic parameters of the moving and fixed cameras, providing coordinates for locating the drone.

[0133] S312: Determine the relative positional relationship between the agricultural drone and the positioning drone in the visual acquisition coordinate system based on the intrinsic and extrinsic parameters of the mobile and fixed cameras.

[0134] S32: Collect several feature points of obstacles in the initial path using agricultural drones and positioning drones respectively, and obtain the first initial image and the second initial image respectively.

[0135] S33: Based on the relative positional relationship and epipolar constraint, feature matching is performed on feature points in the first and second initial images to obtain the positioning coordinate information of several feature points.

[0136] Step S33 includes:

[0137] S331: Either determine the feature points in the first initial image or determine the feature points in the second initial image.

[0138] S332: Based on the relative positional relationship and epipolar constraint conditions, or according to the position of the feature points in the first image, determine the position of the feature points in the second initial image accordingly to obtain the world coordinate information of several feature points, or according to the position of the feature points in the second initial image, determine the position of the feature points in the first initial image to obtain the world coordinate information of several feature points.

[0139] Specifically, in agricultural drone operation scenarios, the image data collected by different computer vision systems have constraints, and the images collected by the same computer vision acquisition system at different locations also have some constraints, which are classified into epipolar constraints, consistency constraints, uniqueness constraints, and continuity constraints according to different measurement environments. Since this vision acquisition system provides navigation and obstacle avoidance strategies for drones flying in farmland, the image acquisition sensor will be affected by external factors such as aperture and light intensity. After experimental verification, this system adopts epipolar constraint conditions.

[0140] When a drone performs measurements using a computer vision system, it is necessary to match the image data coordinates of the feature points of the target object in the flight environment in two or more images. This is called feature matching, which is the pixel coordinates of the same feature points in different image data.

[0141] The parameters in the diagram are defined as follows:

[0142] Scene P: A spatial point within the field of view from both viewpoints of the visual acquisition system;

[0143] c1, c2: The optical axis center points of two different viewpoints;

[0144] π1, π2: Imaging planes of the visual acquisition system at different viewpoints;

[0145] Plane Pc1c2: outer polar plane, that is, the plane formed by scene point P and the optical axis centers c1 and c2 of the two cameras;

[0146] e1, e2: External poles, the intersections of the line connecting the optical axis center points c1, c2 of the visual acquisition system and the imaging planes π1, π2.

[0147] c1c2: Baseline, the line connecting the centers of the optical axes of the vision acquisition system.

[0148] m1e1, m2e2: The intersection of the epipolar plane and the imaging plane is called the epipolar line.

[0149] As shown in the figure, the so-called feature matching of the measured object is the corresponding point of feature point P on the imaging plane of the two sets of vision acquisition systems. By changing the coordinate position of the corresponding point, the flight movement changes of the UAV in space can be found.

[0150] After the feature points of the target to be tested are marked on one image, the epipolar equation of the feature point on another image can be determined based on the relative positional relationship between the two sets of computer vision acquisition systems. By using the method of searching along the epipolar line, the search speed can be improved and the search area can be reduced.

[0151] External limit constraints typically employ a four-parameter model, i.e.:

[0152]

[0153] In the formula k x k is the magnification factor in the x-axis direction. y Let (u0, v0) be the magnification factor along the y-axis, (u0, v0) be the image coordinates of the optical axis center point, R be the rotation matrix in the external parameters of the vision acquisition system, and P be the position vector in the external target feature parameters. W x, W y, W z represents the coordinates of the target in the world coordinate system.

[0154]

[0155] n = [n x n y n z ]for W x in the visual acquisition coordinate system C o C x C y C The direction vector in z;

[0156] o = [o x o y o z ] T for W y in the visual acquisition coordinate system C o C x C y C The direction vector in z;

[0157] a = [a x a y a z ]for W z in the visual acquisition coordinate system C o C x C y C The direction vector in z.

[0158] S333: Convert the world coordinate information of several feature points into the positioning coordinate information in the visual acquisition coordinate system to obtain the positioning coordinate information of several feature points.

[0159] S4: Based on the positioning coordinates of several feature points, determine the height parameters and area parameters of the obstacle.

[0160] Step S4 includes:

[0161] S41: Based on the positioning coordinates of several feature points, determine the height parameters of the obstacle using a mobile vision acquisition system.

[0162] Specifically, during the visual acquisition process of agricultural drones, it is not necessary to accurately measure the three-dimensional information of the target for visual calibration. The purpose of visual navigation of agricultural drones is to avoid obstacles. The obstacles are treated as a whole, and it is sufficient to give the approximate spatial position between each obstacle and between the agricultural drone and the obstacle. Although the position and attitude of the target point obtained in this way are a fuzzy relative relationship, they can correctly reflect the spatial relative position between the agricultural drone and the target object.

[0163] Computer vision uses cameras to observe targets. First, the intrinsic and extrinsic parameters of a fixed camera can be obtained, and the variables of the T matrix can be defined:

[0164]

[0165] Let the world coordinates of the punctuation mark be:

[0166]

[0167] Set up two cameras to observe the target point. W P has homogeneous coordinates K1 = (u1, v1, 1) in the pixel coordinate system. T K2 = (u2, v2, 1) T The parameter matrices T of the two cameras are [T 31 t 41 ] and [T 32 t 42 From equation (1-21), we can obtain:

[0168]

[0169]

[0170] Lianlide:

[0171]

[0172] Simplifying, we get:

[0173]

[0174] Can be set as follows:

[0175]

[0176] And its antisymmetric matrix is ​​m x Due to the properties of antisymmetric arrays:

[0177] m x ·m=0

[0178] We can obtain:

[0179]

[0180] Expand it:

[0181]

[0182] Divide both sides of the equation by have to:

[0183]

[0184] As shown in formula (1-40), if the pixel coordinates of a spatial point in a fixed camera are known, the equation of a straight line can be obtained on the image of a moving camera using the camera's fundamental matrix. This allows us to determine the pixel coordinates of the spatial point on the moving camera and thus the position of the observed point. The converse is also true.

[0185] S42: Based on the positioning coordinates of several feature points, determine the area parameters of the obstacle using a fixed vision acquisition system.

[0186] Specifically, assuming the target lies in a plane perpendicular to the camera's optical axis centerline, and the target's area is known, the camera coordinates are established at the center of the visual acquisition system's optical axis, and the Z-axis of the environment coordinate system is parallel to the optical axis centerline. The direction from the optical axis centerline to the feature target is defined as the positive Z-axis, and the image coordinates along the horizontally increasing direction are defined as the positive X-axis. A world coordinate system is established at the target's centroid, with its coordinate axes parallel to the coordinate axes of the localization coordinate system. Figure 3 As shown, Figure 3 This is a schematic diagram showing the positions of the world coordinate system and the positioning coordinate system of the mobile camera and the fixed camera in the embodiments of this application.

[0187] The relationship between the positioning coordinate system and the pixel coordinate system shows that:

[0188]

[0189] Since obstacle recognition via computer vision requires the extraction of a large number of feature points, these feature points can be indexed for ease of description. Therefore, equation (1-41) can be transformed into:

[0190]

[0191] The correspondence between the local coordinate system and the world coordinate system shows that:

[0192]

[0193] Similarly, after subscripting the feature points, equation (1-43) can be transformed into:

[0194]

[0195] During the image information acquisition process, since the positioning drone does not need to consider the shape and height of obstacles in the image, its main functions, in addition to calibrating the coordinate positions of the agricultural drone and the obstacle points, also need to determine the area occupied by the obstacle, so as to provide control strategies for the agricultural drone to avoid obstacles.

[0196] like Figure 4 As shown, Figure 4This is an example diagram illustrating the calculation of obstacle area parameters using a positioning drone in this application embodiment. The target is positioned along the X... W The axis is divided into n parts, each of which can be approximated as a rectangular strip. Let P1 be the four vertices of the i-th rectangle. i , P1 i+1 ,

[0197] The area of ​​the target can then be expressed as:

[0198]

[0199] In the formula for Y in the world coordinate system W The coordinates of the axis, for X in the world coordinate system W The coordinates of the axis;

[0200] S represents the area of ​​the target.

[0201] Combining equations (1-42), (1-44), and (1-45), we get:

[0202]

[0203] In the formula, S1 is the area of ​​the target on the image.

[0204] P can be obtained through equation (1-46) z The calculation formula is as follows:

[0205]

[0206] In the world coordinate system, the distance L from feature point P to centroid OW can be expressed as:

[0207]

[0208] Therefore, by combining equations (1-47) and (1-48), the target distance can be obtained:

[0209]

[0210] By extracting the feature points and then analyzing them according to the above methods, we can obtain the distance between the obstacle and the agricultural drone, as well as the area of ​​the obstacle.

[0211] S5: Based on the height and area parameters of the obstacle, determine the location of the agricultural drone to move to the target crop.

[0212] Step S5 includes:

[0213] S51: Based on the height and area parameters of the obstacle, and the relative positional relationship between the agricultural drone and the positioning drone in the visual acquisition coordinate system, determine the obstacle avoidance flight path of the agricultural drone to the target crop.

[0214] The advantages of the embodiments of this application will now be illustrated by the process of calibrating obstacles using a GPS visual measurement system.

[0215] With the continuous development of satellite positioning technology, the navigation and positioning accuracy of transportation vehicles such as automobiles, airplanes, and ships is constantly improving. This system uses differential GPS to locate the UAV itself.

[0216] Differential GPS technology involves placing a GPS receiver on a reference station for observation. Based on the pre-known coordinates of the reference station, the distance between the reference station and the satellites is calculated, and this data is continuously transmitted from the reference station. Differential GPS tracks and observes GPS satellites from a reference station at known geographical coordinates, measuring corrections to the observed pseudorange or carrier phase values. Differential GPS positioning can be categorized into three types based on the method of information transmission from the differential GPS reference station: position difference, pseudorange difference, and phase difference.

[0217] The simplest differential method is position differential, which can be derived from any GPS receiver by integrating and converting it from scratch. After observing four satellites, the GPS receiver mounted on the reference station can perform three-dimensional spatial positioning and calculate the exact location of the reference station. Due to trajectory errors, clock deviations, SA policy factors, weather effects, multipath effects, and other influences, the calculated location differs from the initial location of the reference station, resulting in an error. The reference station uses its internal structure to correct the error before transmitting the position, which is then used by the user station to continue using it and to correct the calculated user station coordinates, improving positioning accuracy. All of this assumes that the reference station and the user station must be observing the same set of satellites. The position differential method is suitable for situations where the distance between the customer and the reference station is less than 100 kilometers.

[0218] The most widely used method currently is pseudorange differential. Almost all commercial differential GPS receivers utilize this technology. It requires the receiver at the reference station to predict the distance to the satellites and compare the original, potentially erroneous value with the calculated result. The length error between all satellites is then transmitted to the user, who uses this error to correct the observed pseudorange. Finally, the user uses the corrected pseudorange to determine their own coordinates, thus avoiding common errors and ensuring that subsequent positioning is essentially consistent with the original. Similar to position differential, pseudorange differential can eliminate errors shared by both reference stations, but as the distance between the user and the reference station increases, systematic errors emerge that cannot be eliminated by any method. The length between the user and the reference station plays a crucial three-dimensional role in accuracy.

[0219] The most widely used method currently is pseudorange differential. Almost all commercial differential GPS receivers utilize this technology. It requires the receiver at the reference station to predict the distance to the satellites and compare the original, potentially erroneous value with the calculated result. The length error between all satellites is then transmitted to the user, who uses this error to correct the observed pseudorange. Finally, the user uses the corrected pseudorange to determine their own coordinates, thus avoiding common errors and ensuring that subsequent positioning is essentially consistent with the original. Similar to position differential, pseudorange differential can eliminate errors shared by both reference stations, but as the distance between the user and the reference station increases, systematic errors emerge that cannot be eliminated by any method. The length between the user and the reference station plays a crucial three-dimensional role in accuracy.

[0220] With the advent of differential GPS, the position of objects can be provided in real time with meter-level accuracy, meeting the requirements of navigation, underwater operations, and other engineering applications. Position differential, pseudorange differential, and phase differential methods have been widely applied in various engineering operations. With further advancements, an even more precise measurement technique has emerged—carrier phase differential technology. Also known as RTK technology, it boasts centimeter-level accuracy and is based on simultaneously processing the carrier phase data from two observation stations.

[0221] The system first employs satellite vision as the "eyes-on-hand" visual system in hand-eye coordination. If there is a target to be measured within the satellite image, the mapping from the ground to the satellite camera's imaging plane is the satellite vision measurement. The two-dimensional position of the ground target is measured using computer vision technology.

[0222] Ignoring satellite lens distortion, the visual system parameter model shown in (2-16) is adopted. Since a 2D positioning system is used, the altitude parameter is negligible, i.e. w z = 0

[0223] Eliminate by formula (2-21) w The linear equation obtained from z is:

[0224]

[0225] in:

[0226] t = [t 11 t 12 t 14 t 21 t 22 t 24 t 31 t 32 ] T

[0227] Let t′ = t / t 34

[0228] From formula (2-1), we know that if we need to establish the transformation relationship between the world coordinate system and the pixel coordinate system of satellite images, we need to calculate t′. If we substitute the position coordinates of a point in the image, we can obtain two linear equations. There are a total of 8 unknown parameters in the formula, so we can calculate t′ by substituting 4 points. Simplifying formula (2-1), we get:

[0229] u= w x·t 11 + w y·t 12 +t 14 - w x·u·t 31 - w y·u·t 32 (2-2)

[0230] u= w x·t 21 + w y·t 22 +t 24 - w x·v·t 31 - w y·v·t 32

[0231] To verify the accuracy of the GPS visual measurement system, a test site was accessed via Google satellite map. The acquired pixel resolution was 548×826. Figure 5 As shown, Figure 5 This is a satellite image of the test site in an embodiment of this application.

[0232] The site measures 105 meters by 68 meters, covering approximately 10.71 acres. Four points on the edge are identified as A, B, C, and D. Their world coordinates are obtained from the latitude and longitude coordinates on a satellite map, as follows: A (123.4218, 4.9187), B (123.4226, 41.9186), C (123.4227, 41.9195), and D (123.4219, 41.9196). The pixel coordinates of points A, B, C, and D can be calculated using image coordinates and pixel coordinates. The measured lengths of the X and Y axes in the image are 145 and 218, respectively.

[0233] For example, if the image coordinates of point A are (42, 62), the pixel coordinates can be calculated using the relationship between the calibration coordinates and the image coordinates:

[0234] 42 / 145 = u / 548

[0235] 62 / 218 = v / 826

[0236] The pixel coordinates of point A are (159, 235). Following the same algorithm, the pixel coordinates of points B, C, and D are (351, 224), (378, 587), and (200, 610), respectively. Substituting the latitude, longitude, and pixel coordinates of these four points into equation (2-1) yields the following system of linear equations:

[0237] 123·t 11 +41·t 12 +t 14 -19557·t 31 -6519·t 32 =159

[0238] 123·t 21 +41·t 22 +t 24 -28905·t 31 -9635·t 32 =235

[0239] 123·t 11 +41·t 22 +t 14 -43173·t 31 -14391·t 32 =351

[0240] 123·t 21 +41·t 22 +t 24 -27552·t 31 -9184·t 32 =224

[0241] 123·t11 +41·t 12 +t 14 -46494·t 31 -15498·t 32 =378

[0242] 123·t 21 +41·t 22 +t 24 -72201·t 31 -24067·t 32 =587

[0243] 123·t 11 +41·t 12 +t 14 -24600·t 31 -8200·t 32 =200

[0244] 123·t 21 +41·t 22 +t 24 -75030·t 31 -25010·t 32 =610

[0245] The linear equations show that if calculations are performed solely using latitude and longitude, the coordinates of points A, B, C, and D are too close together, with only the fourth decimal place changing, thus failing to yield a precise result. Therefore, it can be concluded that defining the world coordinates of the UAV's hand-eye coordination system image using GPS satellite imagery is not suitable for accurately calibrating coordinates when the working area is small, making it unsuitable for agricultural drone flight strategies.

[0246] After multiple measurements and calculations, it was found that only when the image acquisition area is over 700,000 square meters can the coordinates extracted from GPS images have high accuracy. Therefore, it is necessary to re-extract Google satellite images, such as... Figure 6 As shown, Figure 6 The satellite imagery collected by the GPS visual measurement system in this embodiment covers an area of ​​800,000 square meters.

[0247] The image resolution is 669×317 pixels. The X and Y axis lengths in the image are measured to be 157 and 118 respectively. The image coordinates of points A, B, C, and D are measured to be A(52,51), B(79,44), C(83,90), and D(55,97), respectively. Their pixel coordinates are calculated to be (222,137), (337,118), (354,242), and (234,261). The satellite map... The latitude and longitude coordinates are: A(123.4154, 41.9173), B(123.4206, 41.9169), C(123.4213, 41.9208), D(123.4161, 41.9212). It can be seen that the latitude and longitude coordinates of these four points change starting from the second decimal place. By inputting any two points in the image into the image world coordinate calculation model, their latitude and longitude coordinates can be obtained. The calculation error is as follows: Figure 7 As shown, Figure 7 This is a data table of measurement results for check points on a plane in the GPS visual measurement system of this application embodiment.

[0248] As shown in the table above, when the test area is large enough, the accuracy of calculations using GPS latitude and longitude is at most 3.29%. However, to obtain measurement results, the intrinsic and extrinsic parameters of the GPS camera must be strictly fixed. Each time satellite images are retrieved and magnified to find measurement points, it is difficult to strictly calibrate the magnification factor, thus making it impossible to find the focal length at different magnification factors, leading to increased errors in image feature points. Therefore, using a GPS visual measurement system to provide eyes-on-hand world coordinates for UAV obstacle avoidance has significant limitations and uncertainties.

[0249] The embodiments of this application have been described in detail above, but the content is only a preferred embodiment of this application and should not be considered as limiting the scope of this application. All equivalent changes and improvements made within the scope of this application should still fall within the patent coverage of this application.

Claims

1. A method for obstacle avoidance in unmanned aerial vehicles (UAVs) based on hand-eye coordination, characterized in that, include: S1: The agricultural drone is set up and moved within the work area, and a positioning drone is fixedly set above the work area at a height higher than that of the agricultural drone. S2: Receive crop disease information strategy through plant protection drone and locate the target crop in the operation area according to the crop disease information strategy, and confirm the initial flight path of plant protection drone to move to the target crop. S3: Collect several feature points of obstacles in the initial flight path using agricultural drones and positioning drones respectively, to obtain a first initial image and a second initial image, and perform feature matching on the feature points in the first initial image and the second initial image to obtain the positioning coordinate information of several feature points; S4: Based on the positioning coordinates of several feature points, determine the height parameters and area parameters of the obstacle accordingly; S5: Determine the obstacle avoidance flight path for the agricultural drone to move to the target crop based on the obstacle height parameters and obstacle area parameters; Step S3 includes: S31: Determine the relative positional relationship between the agricultural drone and the positioning drone; S32: Collect several feature points of obstacles in the initial flight path using agricultural drones and positioning drones respectively, and obtain the first initial image and the second initial image respectively; S33: Based on the relative positional relationship and epipolar constraint, feature matching is performed on feature points in the first initial image and the second initial image to obtain the positioning coordinate information of several feature points; The agricultural drone is equipped with a mobile vision acquisition system, which includes a mobile camera. The positioning drone is equipped with a fixed visual acquisition system, which includes a fixed camera. And, step S31 includes: S311: Determine the intrinsic and extrinsic parameters of the moving and fixed cameras respectively; S312: Determine the relative positional relationship between the agricultural drone and the positioning drone in the visual acquisition coordinate system based on the intrinsic and extrinsic parameters of the mobile and fixed cameras.

2. The obstacle avoidance method for unmanned aerial vehicles based on hand-eye coordination according to claim 1, characterized in that, Step S33 includes: S331: Either determine the feature points in the first initial image or determine the feature points in the second initial image; S332: Based on the relative positional relationship and epipolar constraint conditions, or according to the position of the feature points in the first image, determine the position of the feature points in the second initial image accordingly to obtain the world coordinate information of several feature points, or according to the position of the feature points in the second initial image, determine the position of the feature points in the first initial image to obtain the world coordinate information of several feature points; S333: Convert the world coordinate information of several feature points into the positioning coordinate information in the visual acquisition coordinate system to obtain the positioning coordinate information of several feature points.

3. The obstacle avoidance method for unmanned aerial vehicles based on hand-eye coordination according to claim 2, characterized in that, Step S4 includes: S41: Based on the positioning coordinates of several feature points, determine the height parameters of the obstacle using a mobile vision acquisition system; S42: Based on the positioning coordinates of several feature points, determine the area parameters of the obstacle using a fixed vision acquisition system.

4. The obstacle avoidance method for unmanned aerial vehicles based on hand-eye coordination according to claim 1, characterized in that, Step S5 includes: S51: Based on the height and area parameters of the obstacle, and the relative positional relationship between the agricultural drone and the positioning drone in the visual acquisition coordinate system, determine the obstacle avoidance flight path of the agricultural drone to the target crop.

Images