Visual fusion precise landing method and unmanned aerial vehicle nest differential positioning system

Abstract

The invention belongs to the technical field of unmanned aerial vehicle positioning, and particularly discloses a visual fusion precise landing method and an unmanned aerial vehicle nest differential positioning system. Comprising the following steps: S1, establishing an unmanned aerial vehicle assisted landing model comprising a positioning module and an identification module; s2, setting a flight route of the unmanned aerial vehicle according to a predetermined task target; s3, using the position information provided by the positioning module to control the unmanned aerial vehicle to fly to the task target; and S4, using an identification module and a positioning module to control the unmanned aerial vehicle to identify the ground landing target and complete landing of the unmanned aerial vehicle. According to the method, the problem that the recognition rate is reduced along with the rising of the height of the unmanned aerial vehicle and the increase of a shielding part is effectively solved in actual application, and finally, the unmanned aerial vehicle can autonomously land on the unmanned aerial vehicle nest in the 6-level wind environment, and the landing precision is kept within the range of + / -20 cm.

Description

Technical Field

[0001] The invention belongs to the technical field of unmanned aerial vehicle positioning, and specifically discloses a vision fusion precise landing method and a unmanned aerial vehicle nest differential positioning system. Background Art

[0002] At present, the UAV landing technology mainly uses the navigation method that integrates the global navigation satellite system and the visual guidance positioning technology to achieve autonomous positioning and landing of the UAV. However, with the increasing complexity and refinement of the inspection tasks, the landing accuracy of the UAV directly affects the inspection effect and the safety of the equipment in the operation and inspection of power equipment. Therefore, how to ensure the accuracy and safety of the autonomous landing of the UAV has become one of the current challenges. At present, the positioning and landing of most UAVs are mainly based on the global navigation satellite system, supplemented by the visual guidance positioning technology. The UAV relies on satellite positioning technology to obtain geographic location information to ensure the accuracy of the flight path. When the UAV flies above the nest, it achieves accurate landing by visually identifying the ground landing mark pattern (ArUco mark) that guides the landing. However, traditional satellite positioning mainly uses single-point positioning technology. Because this technology is affected by errors such as satellite orbits, signal propagation paths, and satellite clocks, its positioning accuracy is low, generally only about 10 meters, resulting in a large deviation between the flight position of the UAV and the actual location of the nest, so that the ArUco mark cannot be captured by the airborne camera, resulting in the UAV being unable to land accurately in the nest. In addition, when the drone is far away from the ArUco mark on the nest or the ArUco mark is blocked by external objects, the drone cannot accurately identify the ArUco mark on the nest, which greatly increases the safety risk of the drone when landing.

[0003] Patent application number CN202311246696.1 is a method, medium and device for precise positioning of hybrid drones based on ArUco code and infrared, including: obtaining data captured by infrared sensors on drones, obtaining height data between drones and the ground through data analysis, obtaining images of the ground below captured by the shooting equipment on drones, using computer vision technology to detect ArUco codes, obtaining the coordinates of the four vertices of the ArUco code and the ID of the ArUco code, combining the height measured by the infrared sensor with the position information of the ArUco code, and calculating the distance and degree of deviation of the current drone through the four vertex positions; fine-tuning is performed according to the calculated distance and degree of deviation, and during the landing process, the detection of ArUco codes and the height measurement of infrared sensors are continuously updated to achieve precise landing. The above method realizes location information positioning for drones through ArUco codes. When the ArUco code is blocked by external objects, the drone cannot accurately identify the ArUco mark on the nest, which also increases the safety risk of the drone when landing. Therefore, it is urgent for those skilled in the art to solve the above technical problems. Summary of the invention

[0004] The technical problem to be solved by the present invention is that in the above-mentioned prior art, the UAV cannot accurately detect the blocked ArUco code, which may easily cause danger when the UAV lands.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0006] A vision fusion precision landing method and a UAV nest differential positioning system, comprising the following steps: step S1: establishing a UAV assisted landing model including a positioning module and an identification module; step S2: setting a UAV flight route according to a predetermined mission target; step S3: using the position information provided by the positioning module to control the UAV to fly to the mission target; step S4: using the identification module and the positioning module to control the UAV to identify a ground landing target and complete the landing of the UAV.

[0007] Further, in step S1, the positioning module is a PTK positioning module, the recognition module includes an ArUco-enhanced target and a HOG+SVM target detection module, the PTK positioning module provides location information to the drone, the ArUco-enhanced target provides a visual auxiliary target to the drone, and the HOG+SVM target detection module detects the ground landing point mark.

[0008] Furthermore, in step S2, the UAV flight route is set according to the mission sequence, the UAV takes off and traverses each mission point and finally returns to the initial take-off point.

[0009] Furthermore, in step S3, the UAV uses the position information provided by the differential positioning module to fly to the mission point. When the position information provided by the differential positioning module is near the mission point, the camera is turned on and ground image information is provided to the HOG+SVM target detection module at a certain frequency.

[0010] The further steps of PTK positioning technology are as follows: first, the reference station calculates based on the carrier phase observation value, then uses the known precise coordinates of the reference station to correct the positioning carrier phase observation value, then sends the carrier phase correction data to the mobile station user, and the mobile station user uses the carrier phase correction data to correct its single point positioning data, and finally solves the mobile station coordinate value;

[0011] The relationship between PTK positioning technology and satellite geometric distance can be expressed as:

[0012]

[0013] In order to reduce the influence of various errors in satellite observation, the carrier phase observation value can be expressed as:

[0014]

[0015] In the formula, is the satellite i carrier phase observation value, λ represents the actual wavelength of the carrier, is the true geometric distance from satellite i to the GNSS receiver, and T r (i) are the ionospheric and tropospheric delays, f is the carrier frequency, δt r is the receiver clock error, δt (i) is the satellite clock error, is the integer ambiguity, is the amount of carrier phase measurement noise;

[0016] If satellite i is tracked by both rover GNSS receiver m and reference station GNSS receiver r, the carrier phase observations of satellite i by m and r are With observed value It can be expressed as:

[0017]

[0018] Carrier phase observations of satellite i are made through m and r With observed value Difference processing, denoted as The single-difference carrier phase measurement value is It can be expressed as:

[0019]

[0020] In the formula,

[0021]

[0022]

[0023]

[0024] δt mr =δt m -δt r (6-8)

[0025]

[0026]

[0027] In the case of a short baseline, the clock differences of the two receivers m and r are the same, and the ionospheric delay equal Tropospheric delay Equal to So formula (6-4) can be simplified to:

[0028]

[0029] In the formula, represents the integer ambiguity after single difference of carrier phase, represents the measurement noise caused by the multipath effect of the reference station receiver r and the mobile station receiver m;

[0030] Reference station r and user mobile station m will observe multiple satellites. At this time, the other satellite tracked by reference station r and user mobile station m is denoted as j. From formula (6-11), it can be seen that the carrier phase observation value observed by user mobile station m for satellite j is The carrier phase observations of satellite j from reference station r Do the difference processing, at this time:

[0031]

[0032] The single-difference carrier phase observation With the single-difference carrier phase observation Do double difference processing and get:

[0033]

[0034] In the formula,

[0035]

[0036] From formula (6-13), we can see that the carrier phase observation value after double difference processing is The clock difference between the reference station receiver r and the mobile station receiver m is δt mr It is the same for different satellites, and the error is eliminated by double difference, and the double difference integer ambiguity Using the baseline vector r between the reference station r and the user mobile station m mr , solve the high-precision position coordinates of the user mobile station m;

[0037] For short baselines, the distances of GNSS satellites from the reference station and the rover are greater than the baseline length, so the observations of the same satellite at the user rover and the reference station are parallel to each other. The first difference of the distances of m and r to satellite i is Equal to the baseline vector r from m to r mr Observe the direction of satellite i at the reference station The opposite number of the projection length on

[0038]

[0039] Similarly, we can get:

[0040]

[0041] Substituting equation (6-17) and equation (6-18) into equation (6-14), we can obtain:

[0042]

[0043] Substituting equation (6-19) into (6-13) we can get:

[0044]

[0045] According to the double-difference model, if n different satellites are observed by two receivers at the same time, then the n single-difference carrier phase observations will generate n(n-1) double-difference carrier phase observations. The n-1 independent double-difference carrier phase observations are expressed as Each double-difference carrier phase observation has a double-difference observation equation similar to (6-20). These n-1 double-difference observation equations form the following matrix equation group:

[0046]

[0047] According to formula (6-21), each double difference integer ambiguity At this time, the baseline vector r mr , differential positioning data is obtained, and carrier phase differential positioning is realized.

[0048] Further, in step S4, the HOG+SVM module identifies the ArUco-enhanced mark. If the ArUco-enhanced mark is detected, the relative position information between the UAV and the ground marker is returned. According to this information, the UAV control signal is sent to adjust the horizontal position of the UAV in the air to align with the ground landing point, and then the UAV is controlled to land; if the landing point mark is not detected, the UAV flight altitude is adjusted or it flies nearby to further search for the target point position. If the landing point mark is not detected within a certain period of time, it is determined to stop continuing to execute the flight mission; the HOG+SVM target detection module returns the position information of the ArUco-enhance mark in the image, and captures a sub-image according to the position information and enters the ArUco module. The ArUco module uses Hough transform to capture the ArUco mark for identification, and returns the corresponding landing point description information.

[0049] Furthermore, the HOG+SVM module includes a support vector machine. First, the distance between the support vector and the separation hyperplane is maximized to obtain the separation hyperplane. According to the segmentation situation, the SVM can be divided into a linearly separable support vector machine, a soft-margin support vector machine, and a nonlinear support vector machine.

[0050] The HOG feature extraction process is as follows: Image preprocessing: First, grayscale the image, then perform Gamma correction to complete the normalization of the image, and calculate the image pixel gradient. First, use the gradient operator [0,1,0] and [1,0,1]T to perform convolution operations on the original image in the x-axis and y-axis directions respectively to obtain the gradient components gradscalx and gradscaly, and then calculate the gradient size and direction of the pixel point;

[0051] The horizontal and vertical gradients of the pixel are:

[0052] The gradient magnitude and direction at this pixel are:

[0053]

[0054] Finally, the cell pixel gradient is calculated, and 2 is added to the first and second dimensions of the gradient histogram respectively. The pixel point with coordinates (1, 1) in the gradient direction map has a gradient direction of 80, and corresponds to the fifth dimension in the gradient histogram. The corresponding value 2 in the gradient amplitude is directly placed in the fifth dimension. The gradient histogram block of the statistical block is a larger descriptor composed of cells. At the same time, L2-norm is used to normalize the contrast of each block. The HOG feature vectors of the statistical block are concatenated to obtain the HOG features of the gradient histogram window of the statistical window. The feature vectors of the non-overlapping windows are concatenated to obtain the gradient direction of the calculated image, and the feature vector used for the final classification is obtained.

[0055] Furthermore, an outer nested ring is added on the basis of the ArUco marker, where the ArUco marker is responsible for obtaining the landing point information, and the added outer nested ring is expected to improve the recognition accuracy when the landing point is blocked;

[0056] First, you need to get the center position of the circle in the image and the minimum radius of the circle r min , obtain the sub-image through the above information interception, and then obtain the relevant description information of the landing point by calling the ArUco recognition algorithm;

[0057] Perform edge detection on the ArUco-enhanced marker to obtain the foreground point;

[0058] Similar to the principle of detecting a straight line, the general form equation of a circle is rewritten as (xa) by coordinate transformation 2 +(yb) 2 =r 2 , then a point on the circle boundary in the XY coordinate system corresponds to a circle in the AB coordinate system;

[0059] By counting the number of circles at the local intersections and taking the maximum value, we can obtain the center position (a, b) of the circle in the original image. At the same time, once a circle is detected at a certain radius r, the value of r is also determined.

[0060] When obtaining the position information of multiple rings and the center position of the circle may be offset, first obtain the mass center of the circle in the image according to formula 6-23 And according to the minimum radius r min A square image B is captured from the picture. The captured image contains ArUco markers, and the landing point information is obtained through the traditional ArUco recognition algorithm;

[0061]

[0062] Further, in step S4, the UAV lands at the ground target point. If the position of the point is the initial take-off position, a flight mission is completed; otherwise, the UAV takes off again and jumps to step S3.

[0063] A drone nest differential positioning system controls the drone positioning and target identification and landing through the above-mentioned visual fusion precise landing method. The drone nest differential positioning system includes a positioning module and a target detection module. The positioning module is used to provide the drone with position information so as to control the drone's flight route. The target detection module is used to provide the drone with landing target information so as to control the drone's landing. The drone nest differential positioning system also includes an intelligent nest and auxiliary electronic equipment, which are mainly used to sense the drone's position status, analyze the equipment status, process the RTK positioning module and the target detection module data to determine whether the drone's landing conditions are met, and control the drone to land accurately.

[0064] The present invention has the following beneficial effects:

[0065] The present invention uses RTK (Real-time kinematic) differential technology to achieve centimeter-level positioning in a short time, avoiding the errors caused by the single-point positioning technology used in traditional satellite positioning, ensuring the accurate and safe landing of the UAV. The improved ArUco-enhanced marker effectively solves the problem that the ArUco marker cannot be correctly identified due to factors such as occlusion, and further improves the recognition rate when the marker is blocked by debris. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 It is a structural schematic diagram of the present invention.

[0067] Figure 2 It is a reference diagram for the landing of the UAV of the present invention.

[0068] Figure 3 It is a schematic diagram of PTK positioning of the present invention.

[0069] Figure 4 It is the existing ArUco code marking diagram.

[0070] Figure 5 It is the ArUco code marker image to be detected.

[0071] Figure 6 It is a schematic diagram of ArUco code mark detection and recognition results.

[0072] Figure 7 This is an example diagram of the ArUco-enhanced marking of the present invention.

[0073] Figure 8 It is a schematic diagram of the Hough transform detection ring of the present invention.

[0074] Fig. 9 This is an example of an ArUco marker capture.

[0075] Fig.10 It is image a taken by the drone of the present invention.

[0076] Fig.11 It is image b taken by the drone of the present invention.

[0077] Fig.12 It is the image c taken by the drone of the present invention.

[0078] Fig.13 It is the image d taken by the drone of the present invention.

[0079] Fig.14 It is the image e taken by the drone of the present invention.

[0080] Fig.15 It is the image f taken by the drone of the present invention.

[0081] Fig.16 It is the image g taken by the drone of the present invention.

[0082] Fig.17 This is an example of image annotation of the present invention.

[0083] Fig.18 This is an example diagram of the gradient histogram calculation of the present invention. DETAILED DESCRIPTION

[0084] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific preferred embodiments.

[0085] In the description of the present invention, it should be understood that the terms "left side", "right side", "upper part", "lower part" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. "First", "second" and the like do not indicate the importance of the components, and therefore cannot be understood as limiting the present invention. The specific dimensions used in this embodiment are only for illustrating the technical solution, and do not limit the scope of protection of the present invention.

[0086] Embodiment 1: At present, drones relying on satellite navigation systems have initially achieved the goal of autonomous landing. However, due to the large satellite positioning error and vulnerability to attack, the accuracy and safety of autonomous fixed-point landing of drones are directly affected. Figure 1 and Figure 2 It can be seen that a method for precise landing with visual fusion is provided. The present invention provides a UAV nest differential positioning system and a visually assisted precise landing technology, which mainly includes a UAV assisted landing model composed of a positioning module and a recognition module, wherein the positioning module includes an RTK positioning module, an ArUco-enhanced target, a ground target detection and visual recognition module, wherein:

[0087] The RTK positioning module mainly sends the carrier phase measurement value received by the base station to the UAV receiver, performs differential operation with the carrier phase measurement value of the UAV receiver itself, and finally solves the baseline vector and the initial integer ambiguity, thereby completing the high-precision positioning of the UAV.

[0088] ArUco-enhanced target, ArUco marker is responsible for acquiring landing point information, and the added outer nested ring (ArUco-enhanced) is intended to improve the recognition accuracy when the landing point is blocked. The ArUco-enhanced target is arranged adjacent to the machine nest equipment to provide specified visual auxiliary coordinates to the air;

[0089] The ground target detection and visual recognition module mainly detects the ground landing point mark through the HOG+SVM target detection algorithm, recognizes the ArUco-enhanced target, and extracts the position coordinates and rotation angle of the onboard camera relative to the identification code;

[0090] The drone is equipped with a camera and has a visual recognition mode. It cooperates with the RTK positioning module, ArUco-enhanced target, and ground target detection module to form a drone assisted landing model. When the drone takes off, it mainly includes the following steps:

[0091] Step 1: Set the UAV flight route according to the mission sequence. The UAV takes off and traverses each mission point and finally returns to the initial take-off point.

[0092] Step 2: The drone uses the location information provided by the differential positioning module to fly to the mission point. When the location information provided by the differential positioning module is near the mission point, the camera is turned on and ground image information is provided to the HOG+SVM module at a certain frequency.

[0093] Step 3: Target Recognition Module

[0094] (1) The HOG+SVM module recognizes the ArUco-enhanced marker. If it is detected, it returns the relative position information between the drone and the ground marker. Based on this information, it sends a drone control signal to adjust the drone's horizontal position in the air to align with the landing point on the ground, and then controls the drone to land. If the landing point marker is not detected, the drone adjusts its flight altitude or flies nearby to further search for the target point. If the landing point marker is not detected within a certain period of time, it is determined that the drone's navigation equipment may be interfered with by the signal and deviate from the flight route, and the drone stops executing the flight mission.

[0095] (2) The HOG+SVM module returns the location information of the ArUco-enhance mark in the image. Based on this information, a sub-image is intercepted and sent to the ArUco module. This module uses Hough transform to intercept the ArUco mark and uses traditional methods to identify it, and returns the corresponding landing point description information.

[0096] Step 4: The drone lands at the target point on the ground. If the point is the initial take-off position, the flight mission is completed. Otherwise, the drone takes off again and jumps to step 2.

[0097] Example 2: Reference Figure 3 It can be seen that the principle of differential positioning is mainly based on the satellite orbit errors of the mobile station and the reference station in the same area, the delay errors of the ionosphere and troposphere in the atmosphere, and the satellite clock errors, etc. have certain spatial correlations. By differentially processing the satellite observation values ​​of the mobile station and the reference station, most of the satellite observation errors can be reduced or eliminated. The differential positioning system is generally composed of more than two GNSS receivers. One receiver is fixedly placed on a measuring station with known precise coordinates as a reference station; the other receiver is placed on a moving device as a user mobile station. By performing real-time differential processing on the data observed by the reference station GNSS receiver and the data of the known coordinate points, the positioning measurement error information of the reference station to the satellite can be obtained, and then the error correction information is broadcast to the mobile station in real time through the data communication link. The mobile station uses the error correction data and the data observed by its own GNSS receiver to perform correction and solution, and high-precision positioning data can be obtained.

[0098] Existing differential positioning technology cannot be applied in high-precision positioning scenarios. Therefore, this embodiment mainly adopts carrier phase differential technology, namely PTK positioning method. First, the reference station calculates based on the carrier phase observation value, and then uses the known precise coordinates of the reference station to correct the positioning carrier phase observation value. Then, the carrier phase correction data is sent to the mobile station user. The mobile station user uses the carrier phase correction data to correct its single-point positioning data, and finally solves the mobile station coordinate value.

[0099] The relationship between carrier phase and satellite geometric distance can be expressed as:

[0100]

[0101] If various errors in satellite observations are taken into account, the carrier phase observation can be expressed as:

[0102]

[0103] In the formula, is the satellite i carrier phase observation value, λ represents the actual wavelength of the carrier, is the true geometric distance from satellite i to the GNSS receiver, and are the ionospheric and tropospheric delays, f is the carrier frequency, δt r is the receiver clock error, δt (i) is the satellite clock error, is the integer ambiguity, is the amount of carrier phase measurement noise. If satellite i is tracked simultaneously by mobile station GNSS receiver m and reference station GNSS receiver r, the carrier phase observations of satellite i by m and r are With observed value It can be expressed as:

[0104]

[0105] Carrier phase observations of satellite i are made through m and r With observed value Difference processing, denoted as The single-difference carrier phase measurement value is It can be expressed as:

[0106]

[0107] In the formula,

[0108]

[0109] It can be seen that the satellite clock δt (i)After single difference, it is eliminated. In the case of short baseline, the difference between the clocks of the two receivers m and r is the same, and the ionospheric delay It can be considered equal to Tropospheric delay It can be considered equal to So formula (6-4) can be simplified to:

[0110]

[0111] In the formula, represents the integer ambiguity after single difference of carrier phase, It represents the measurement noise caused by multipath effect of reference station receiver r and mobile station receiver m.

[0112] Since the reference station r and the user mobile station m can observe multiple satellites at a certain moment, if the other satellite tracked by the reference station r and the user mobile station m is denoted as j, it can be seen from formula (6-11) that the carrier phase observation value observed by the user mobile station m for satellite j is The carrier phase observations of satellite j from reference station r Doing the difference, we can get:

[0113]

[0114] The single-difference carrier phase observation With the single-difference carrier phase observation Doing double difference processing, we can get:

[0115]

[0116] In the formula,

[0117]

[0118] From formula (2-13), we can see that the carrier phase observation value after double difference processing is The clock difference between the reference station receiver r and the mobile station receiver m is δt mr It is the same for different satellites and can be eliminated by double difference. If the phase measurement noise is ignored, the double difference integer ambiguity is solved The baseline vector r between the reference station r and the user mobile station m can be used mr , solve the high-precision position coordinates of the user mobile station m.

[0119] For short baselines, the distances of GNSS satellites from the reference station and the rover will be greater than the baseline length, so the observations of the same satellite at the user rover and the reference station can be considered parallel to each other. Equal to the baseline vector r from m to r mr Observe the direction of satellite i at the reference station The opposite number of the projection length on

[0120]

[0121] Similarly, we can get:

[0122]

[0123] Substituting equation (6-17) and equation (6-18) into equation (6-14), we can obtain:

[0124]

[0125] Substituting equation (6-19) into (6-13) we can get:

[0126]

[0127] According to the double-difference model, if n different satellites are observed by two receivers at the same time, then the n single-difference carrier phase observations will generate n(n-1) double-difference carrier phase observations. If these n-1 mutually independent double-difference carrier phase observations are expressed as Each double-difference carrier phase observation has a double-difference observation equation similar to (6-20), so these n-1 double-difference observation equations can form the following matrix equation group:

[0128]

[0129] The double difference integer ambiguity can be obtained by formula (6-21): So we can solve the baseline vector r mr , differential positioning data is obtained, and carrier phase differential positioning is realized. In this process, most errors are eliminated except for the multipath effect phase measurement noise, so the carrier phase differential technology has a very high positioning accuracy and can usually achieve centimeter-level positioning.

[0130] Example 3: The ArUco marker was proposed by Rafael Munoz and Sergio Garrido. The marker consists of a black base and a white square. Due to its high robustness and fast calculation, it is widely used in computer vision related applications. Figure 4 , Figure 5 and Figure 6It can be seen that when using ArUco markers and the camera is far away from the marker, the traditional benchmark marker detection algorithm is very sensitive to the marker size. Therefore, large-sized markers are required during the landing process of the drone. However, the camera may not be able to fully capture the oversized marker, resulting in marker detection failure. Based on the above problems, the traditional benchmark algorithm, YOLO V3, YOLO V3-spp and YOLO V3-tiny were compared and the experimental results showed that the recognition rate of the traditional benchmark algorithm is better than that of the YOLO series algorithm in the absence of occlusion. However, when there is internal occlusion or edge occlusion, the recognition rate of the traditional benchmark algorithm is much lower than that of the YOLO series algorithm. When the ArUco marker is internally occluded, the recognition rate gradually decreases as the drone height increases and the occluded part increases. At this time, YOLO v3-spp performs better. In the edge occlusion experiment, the recognition rate of YOLO V3-tiny also decreases with the increase of height and occlusion area, but the recognition rates of YOLO V3 and YOLO V3-spp remain basically stable;

[0131] Reference Figure 7 , Figure 8 and Fig. 9 It can be seen that in order to solve the above problem, this embodiment adds an outer nested ring on the basis of the ArUco mark, where the ArUco mark is responsible for obtaining the landing point information, and the added outer nested ring is expected to improve the recognition accuracy when the landing point is blocked. Due to the addition of the ring mark, the traditional ArUco mark detection algorithm cannot be used normally. Therefore, in order to extract the ArUco mark information, it is first necessary to obtain the center position of the circle in the image. and the minimum radius of the circle r min , obtain a sub-image through the above information interception (the sub-image contains ArUco markers and reduces the proportion of occlusions outside the ArUco markers), and then obtain the relevant description information of the landing point by calling the ArUco recognition algorithm;

[0132] Perform edge detection on the ArUco-enhanced marker to obtain the foreground point. If there is a circle, the circular outline must belong to the foreground point.

[0133] Similar to the principle of detecting a straight line, the general form equation of a circle is rewritten as (xa) by coordinate transformation 2 +(yb) 2 =r 2 , then a point on the circle boundary in the XY coordinate system corresponds to a circle in the AB coordinate system;

[0134] By counting the number of circles at the local intersections and taking the maximum value, we can obtain the center position (a, b) of the circle in the original image. At the same time, once a circle is detected at a certain radius r, the value of r is also determined.

[0135] In actual application, multiple ring position information may be obtained, and the center position of the circle may be offset. To solve the above problem, first use formula 6-23 to obtain the mass center of the circle in the image. And according to the minimum radius r min A square image B is captured from the image. The captured image will contain ArUco markers, and the landing point information is obtained through the traditional ArUco recognition algorithm.

[0136]

[0137] Example 4: Although ArUco markers and ArUco-enhanced markers can be detected and identified by the above method, the problem of recognition failure may occur due to problems such as the landing marker being blocked by debris. At the same time, the influence of lighting factors on recognition is also great. Fig.10 , Fig.11 , Fig.12 , Fig.13 , Fig.14 , Fig.15 , Fig.16 and Fig.17 It can be seen that in order to solve the above problems, the present invention decided to introduce the traditional machine learning methods HOG+SVM and DPM+SVM as experimental comparison to determine the implementation method of the final target detection and recognition algorithm in the recognition and control module. The target recognition algorithm based on the classifier requires a certain number of pictures for training. Therefore, a total of 4 groups of videos were shot using the DJI Phantom 3 drone. The video grouping is shown in Table 1:

[0138]

[0139] Table 1 Dataset image grouping

[0140] Since ArUco markers cannot be detected successfully when they are occluded or when there are straight lines in the image background, the dataset is directly shot using ArUco-enhanced markers;

[0141] Dataset overview: Video A and Video D have different background environments, and the landing mark is unobstructed. In Video A, the background environment is rectangular floor tiles. When the drone flies above 3 meters, there are about 30% to 50% green plants. At this time, if the Hough transform is used alone to detect the mark position, the detection difficulty is further increased. The ground in Video D is gravel and there are no green plants. Because the marker has a certain reflectivity, the image quality is different in different directions.

[0142] The marker occlusion methods in Video B and Video C are different. In Video B, a black stone with a length and width of approximately 3×2 cm is placed in the middle of the ArUco marker, and in Video C, two leaves are used to cover the nested ring position of the landing point marker. The other information in the image is basically the same as in Video A.

[0143] Image screening: The dataset includes only two types of data, namely ArUco-enhanced marked images and background images. The proportion of landing marks in the image will decrease as the altitude of the drone increases. When the drone is flying at a high altitude, the captured image is easily blurred due to the influence of lighting. Such images are used as positive samples for training. If the landing mark only exists in less than 40% of the captured image, it is not used as a positive sample. Negative samples are plants or various floor tiles taken during image collection.

[0144] Image annotation: Image annotation is completed by the open source software labelImg. In this picture, the number of positive samples is 1, and the number of negative samples is 8, including the background images such as the bushes and the floor tiled ground in the image.

[0145] Example 5: After preliminary landing point detection and identification, it is also necessary to determine the location of the ring in advance. Fig.18 It can be seen that this embodiment adopts the HOG+SVM method, which specifically includes:

[0146] Support Vector Machines:

[0147] Support Vector Machine (SVM) was proposed by Vapnik and is generally used in classification tasks. The specific principle is to find the distance from the support vector to the separation hyperplane, and find the separation hyperplane under this condition. According to the segmentation situation, SVM can be divided into three categories: linear separable support vector machine, soft margin support vector machine, and nonlinear support vector machine.

[0148] HOG feature extraction process:

[0149] The process of HOG (Histogram of Oriented Gradient) feature extraction is:

[0150] Image preprocessing:

[0151] Because the color information in the HOG feature does not work, the image is first grayed out. In order to improve the robustness of the detector, Gamma correction is required to complete the normalization of the image. The purpose is to reduce the influence of factors such as local illumination, and also to reduce noise interference.

[0152] Calculate image pixel gradients:

[0153] First, use the gradient operators [0,1,0] and [1,0,1]T to perform convolution operations on the original image in the x-axis and y-axis directions respectively to obtain the gradient components gradscalx and gradscaly, and then calculate the gradient size and direction of the pixel point;

[0154] Suppose there is a pixel (x, y) in the image, then the horizontal and vertical gradients of the pixel are:

[0155]

[0156] The gradient magnitude and direction at this pixel are:

[0157]

[0158] Calculate the cell pixel gradient:

[0159] Fig.18 The gradient direction map (Gradient Direction) and gradient magnitude map (Gradient Magnitude), as well as the gradient histogram (Histogram of Gradients) are 8×8 respectively;

[0160] exist Fig.18 The pixel with coordinates (1, 4) in the gradient direction map has a gradient direction of 10. There is no value 10 in the gradient histogram, but it is between 0 and 20, so 10 = 0 × 0.5 + 20 × 0.5. The corresponding coordinate value in the gradient magnitude map is 4, so add 2 to the first and second dimensions of the gradient histogram.

[0161] In the gradient direction map, for the pixel point with coordinates (1, 1), its gradient direction is 80, and it corresponds to the fifth dimension in the gradient histogram. Then the corresponding value 2 in the gradient amplitude is directly placed in the fifth dimension.

[0162] Gradient histogram of statistical block:

[0163] A block is a larger descriptor composed of cells. If we assume that a cell is a 9-dimensional HOG feature and there are 4 cells in a block, we can know that the HOG feature of a block is 36-dimensional. At the same time, using L2-norm to normalize the contrast of each block can effectively reduce the influence of factors such as local illumination changes;

[0164] Gradient histogram of the statistical window:

[0165] The HOG feature of the window is obtained by concatenating the HOG feature vectors of the block;

[0166] Compute the gradient orientation histogram of an image:

[0167] The gradient histogram of the image is obtained by concatenating the feature vectors of all non-overlapping windows, which is also the feature vector used in the final classification.

[0168] The preferred embodiments of the present invention are described in detail above; however, the present invention is not limited to the specific details in the above embodiments. Within the technical concept of the present invention, various equivalent transformations can be made to the technical solutions of the present invention, and these equivalent transformations all belong to the protection scope of the present invention.

Claims

1. A visual fusion precision landing method, characterized in that: The method comprises the following steps: step S1: establishing a UAV assisted landing model including a positioning module and an identification module; step S2: setting a UAV flight route according to a predetermined mission target; step S3: using the position information provided by the positioning module to control the UAV to fly toward the mission target; step S4: using the identification module and the positioning module to control the UAV to identify the ground landing target and complete the landing of the UAV.

2. The visual fusion precision landing method according to claim 1, characterized in that: In step S1, the positioning module is a PTK positioning module, and the recognition module includes an ArUco-enhanced target and a HOG+SVM target detection module. The PTK positioning module provides location information to the drone, the ArUco-enhanced target provides a visual auxiliary target to the drone, and the HOG+SVM target detection module detects the ground landing point mark.

3. The visual fusion precision landing method according to claim 1, characterized in that: In step S2, the UAV flight route is set according to the mission sequence, the UAV takes off and traverses each mission point and finally returns to the initial take-off point.

4. The visual fusion precision landing method according to claim 1, characterized in that: In step S3, the UAV uses the position information provided by the differential positioning module to fly to the mission point. When the position information provided by the differential positioning module is near the mission point, the camera is turned on and ground image information is provided to the HOG+SVM target detection module at a certain frequency.

5. The visual fusion precision landing method according to claim 2, characterized in that: The steps of PTK positioning technology are as follows: first, the reference station calculates based on the carrier phase observation value, then uses the known precise coordinates of the reference station to correct the positioning carrier phase observation value, then sends the carrier phase correction data to the mobile station user, and the mobile station user uses the carrier phase correction data to correct its single point positioning data, and finally solves the mobile station coordinate value; The relationship between PTK positioning technology and satellite geometric distance can be expressed as: In order to reduce the influence of various errors in satellite observation, the carrier phase observation value can be expressed as: In the formula, is the satellite i carrier phase observation value, λ represents the actual wavelength of the carrier, is the true geometric distance from satellite i to the GNSS receiver, and are the ionospheric and tropospheric delays, f is the carrier frequency, δt r is the receiver clock error, δt (i) is the satellite clock error, is the integer ambiguity, is the amount of carrier phase measurement noise; If satellite i is tracked by both rover GNSS receiver m and reference station GNSS receiver r, the carrier phase observations of satellite i by m and r are With observed value It can be expressed as: Carrier phase observations of satellite i are made through m and r With observed value Difference processing, denoted as The single-difference carrier phase measurement value is It can be expressed as: In the formula, δt mr =δt m -δt r (6-8) In the case of a short baseline, the clock differences between the two receivers m and r are the same, and the ionospheric delay equal Tropospheric delay equal So formula (6-4) can be simplified to: In the formula, represents the integer ambiguity after single difference of carrier phase, represents the measurement noise caused by the multipath effect of the reference station receiver r and the mobile station receiver m; Reference station r and user mobile station m will observe multiple satellites. At this time, the other satellite tracked by reference station r and user mobile station m is denoted as j. From formula (6-11), it can be seen that the carrier phase observation value observed by user mobile station m for satellite j is The carrier phase observations of satellite j from reference station r Do the difference processing, at this time: The single-difference carrier phase observation With the single-difference carrier phase observation Do double difference processing and get: In the formula, From formula (6-13), we can see that the carrier phase observation value after double difference processing is The clock difference between the reference station receiver r and the mobile station receiver m is δt mr It is the same for different satellites, and the error is eliminated by double difference, and the double difference integer ambiguity Using the baseline vector r between the reference station r and the user mobile station m mr , solve the high-precision position coordinates of the user mobile station m; For short baselines, the distances of GNSS satellites from the reference station and the rover are greater than the baseline length, so the observations of the same satellite at the user rover and the reference station are parallel to each other. The first difference of the distances of m and r to satellite i is Equal to the baseline vector r from m to r mr Observe the direction of satellite i at the reference station The opposite number of the projection length on Similarly, we can get: Substituting equation (6-17) and equation (6-18) into equation (6-14), we can obtain: Substituting equation (6-19) into (6-13) we can get: According to the double-difference model, if n different satellites are observed by two receivers at the same time, then the n single-difference carrier phase observations will generate n(n-1) double-difference carrier phase observations. The n-1 mutually independent double-difference carrier phase observations are expressed as Each double-difference carrier phase observation has a double-difference observation equation similar to (6-20). These n-1 double-difference observation equations form the following matrix equation group: According to formula (6-21), each double difference integer ambiguity At this time, the baseline vector r mr , differential positioning data is obtained, and carrier phase differential positioning is realized.

6. The visual fusion precision landing method according to claim 1, characterized in that: In step S4, the HOG+SVM module identifies the ArUco-enhanced mark. If the ArUco-enhanced mark is detected, the relative position information between the drone and the ground marker is returned. According to this information, the drone control signal is sent to adjust the horizontal position of the drone in the air to align with the ground landing point, and then the drone is controlled to land. If the landing point mark is not detected, the drone flight altitude is adjusted or it flies nearby to further find the target point position. If the landing point mark is not detected within a certain period of time, it is determined to stop continuing to execute the flight mission. The HOG+SVM target detection module returns the position information of the ArUco-enhance mark in the image, and captures a sub-image according to the position information and enters the ArUco module. The ArUco module uses Hough transform to capture the ArUco mark for identification and returns the corresponding landing point description information.

7. The visual fusion precision landing method according to claim 2, characterized in that: The HOG+SVM module includes support vector machines. First, the distance between the support vector and the separation hyperplane is maximized to obtain the separation hyperplane. According to the segmentation situation, SVM can be divided into linear separable support vector machines, soft margin support vector machines and nonlinear support vector machines. The HOG feature extraction process is as follows: Image preprocessing: First, grayscale the image, then perform Gamma correction to complete the normalization of the image, and calculate the image pixel gradient. First, use the gradient operator [0,1,0] and [1,0,1]T to perform convolution operations on the original image in the x-axis and y-axis directions respectively to obtain the gradient components gradscalx and gradscaly, and then calculate the gradient size and direction of the pixel point; The horizontal and vertical gradients of the pixel are: The gradient magnitude and direction at this pixel are: Finally, the cell pixel gradient is calculated, and 2 is added to the first and second dimensions of the gradient histogram respectively. The pixel point with coordinates (1, 1) in the gradient direction map has a gradient direction of 80, and corresponds to the fifth dimension in the gradient histogram. The corresponding value 2 in the gradient amplitude is directly placed in the fifth dimension. The gradient histogram block of the statistical block is a larger descriptor composed of cells. At the same time, L2-norm is used to normalize the contrast of each block. The HOG feature vectors of the statistical block are concatenated to obtain the HOG features of the gradient histogram window of the statistical window. The feature vectors of the non-overlapping windows are concatenated to obtain the gradient direction of the calculated image, and the feature vector used for the final classification is obtained.

8. The visual fusion precision landing method according to claim 6, characterized in that: On the basis of ArUco markers, an outer nested ring is added. The ArUco marker is responsible for obtaining the landing point information. The added outer nested ring is expected to improve the recognition accuracy when the landing point is blocked. First, you need to get the center position of the circle in the image and the minimum radius of the circle r min , obtain the sub-image through the above information interception, and then obtain the relevant description information of the landing point by calling the ArUco recognition algorithm; Perform edge detection on the ArUco-enhanced marker to obtain the foreground point; Similar to the principle of detecting a straight line, the general form equation of a circle is rewritten as (xa) by coordinate transformation 2 +(yb) 2 =r 2 , then a point on the circle boundary in the XY coordinate system corresponds to a circle in the AB coordinate system; By counting the number of circles at the local intersections and taking the maximum value, we can obtain the center position (a, b) of the circle in the original image. At the same time, once a circle is detected at a certain radius r, the value of r is also determined. When obtaining the position information of multiple rings and the center position of the circle may be offset, first obtain the mass center of the circle in the image according to formula 6-23 And according to the minimum radius r min A square image B is captured from the picture. The captured image contains ArUco markers, and the landing point information is obtained through the traditional ArUco recognition algorithm; 9. The visual fusion precision landing method according to claim 1, characterized in that: In step S4, the drone lands at the target point on the ground. If the point is the initial take-off position, a flight mission is completed; otherwise, the drone takes off again and jumps to step S3.

10. A drone nest differential positioning system, comprising the visual fusion precision landing method according to any one of claims 1 to 8, characterized in that: The drone nest differential positioning system includes a positioning module and a target detection module. The positioning module is used to provide location information for the drone to control the drone's flight route, and the target detection module is used to provide landing target information for the drone to control the drone's landing. The drone nest differential positioning system also includes an intelligent nest and associated electronic equipment, which are mainly used to sense the drone's position status, analyze the equipment status, process the RTK positioning module and target detection module data to determine whether the drone's landing conditions are met, and control the drone to land accurately.

Images