A split flying car positioning and docking method based on visual depth information

Abstract

The present invention relates to the technical field of flying cars, and in particular to a split-type flying car positioning and docking method based on visual depth information. The steps are as follows: S1: Increase the positioning accuracy during docking through dual-code markers; S2: Construct a target detection scene library; S3: Introduce the depth information obtained by structured light into the pose matrix, and obtain accurate relative coordinates through multi-source information fusion correction. The present invention provides a split-type flying car positioning and docking method based on visual depth information. It has high accuracy, strong fault tolerance, and high flexibility through multi-level markers. It is suitable for various complex environments. Even when it is 50% blocked, it can still provide stable and reliable pose information, thereby improving the reliability and stability of the visual guidance system.

Description

Technical Field

[0001] The present invention relates to the technical field of flying cars, and in particular to a positioning and docking method for a split flying car based on visual depth information. Background Art

[0002] The split flying car is a "global, intelligent, and distributed" integrated equipment that has the ability to drive on land and fly in the air. The split flying car has the functions of vertical take-off and landing of drones and road driving of ground cars, and will play an important role in the future transportation field. In order to achieve accurate positioning and docking of drones and ground vehicles;

[0003] When the drone performs outdoor docking operations, it is easily disturbed by wind, causing shaking when hovering to search for targets, making it impossible for the camera to accurately detect the target platform. Different lighting, tilt, and occlusion will also change and weaken the characteristics of the target, further increasing the possibility of the onboard detection system losing the target;

[0004] Single visual positioning methods, whether monocular or binocular, cannot achieve high accuracy and have poor stability. Existing target detection algorithms cannot accurately locate moving targets.

[0005] Split flying cars have not yet become popular, resulting in the incomplete development of docking mechanisms for this application scenario.

[0006] To this end, we designed a split flying car positioning and docking method based on visual depth information to provide another technical solution to the above technical problems. Summary of the invention

[0007] Based on this, it is necessary to provide a split flying car positioning and docking method based on visual depth information to solve the technical problems raised in the above background technology.

[0008] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0009] A split flying car positioning and docking method based on visual depth information, the steps are as follows:

[0010] S1: Double-code markers are used to increase positioning accuracy during docking;

[0011] S2: Build a target detection scene library;

[0012] S3: The depth information obtained by structured light is introduced into the pose matrix, and the accurate relative coordinates are obtained through multi-source information fusion and correction.

[0013] As a preferred implementation of the method for positioning and docking a split flying car based on visual depth information provided by the present invention, in the step S1, the double-code identifier consists of an outer frame, an H graphic and a double Apriltag code;

[0014] The outer frame can be quickly found and listed as an alternative docking area when the drone searches for a target at a long distance;

[0015] The H-shaped pattern can provide distinct morphological features for the screening process through obvious corner features and symmetrical structure, thereby ensuring the accuracy of the screening process;

[0016] The dual Apriltag code is used to provide stable position information through 8 corner points for estimating spatial position and three-dimensional posture.

[0017] As a preferred implementation of the method for positioning and docking a split flying car based on visual depth information provided by the present invention, when the outer frame is clustered and calculated in pixel points, the convex quadrilateral shape can be more efficiently recognized.

[0018] As a preferred implementation of the split flying car positioning and docking method based on visual depth information provided by the present invention, the center position of the marker is preliminarily calculated through the pixel coordinates of each corner point in the obvious corner point features of the H figure, providing appropriate orientation guidance for the drone.

[0019] As a preferred implementation of the method for positioning and docking a split flying car based on visual depth information provided by the present invention, the target detection scene library is constructed in step S2, and the steps are as follows:

[0020] The model is constructed by collecting environmental information and target information. For different docking scenarios, the appropriate number of fully connected layers and neurons are set according to the appearance characteristics of the docking platform, and different neural network layers are adjusted to train models suitable for different platform characteristics.

[0021] As a preferred implementation of the method for positioning and docking a split flying car based on visual depth information provided by the present invention, in the step S3, the depth information obtained by the structured light is introduced into the pose matrix, and the accurate relative coordinates are obtained through multi-source information fusion correction, and the steps are as follows:

[0022] The position and direction information of the target point is obtained by performing adaptive thresholding, continuous boundary segmentation, quadrilateral fitting, homography transformation, and decoding on the markers. The depth frame and color frame are aligned and the depth information is fused by configuring the depth stream and color stream.

[0023] It can be seen without a doubt that the above-mentioned technical solution of the present application can definitely solve the technical problem to be solved by the present application.

[0024] At the same time, through the above technical solutions, the present invention has at least the following beneficial effects:

[0025] The present invention provides a split-type flying car positioning and docking method based on visual depth information. It has high accuracy, strong fault tolerance and high flexibility through multi-level markers, and is applicable to various complex environments. Even when it is 50% blocked, it can still provide stable and reliable posture information, thereby improving the reliability and stability of the visual guidance system.

[0026] During the docking process, the posture information from the visual and depth cameras is integrated, reducing the impact of single sensor errors, providing more comprehensive and accurate positioning and docking results, achieving more accurate scene understanding and position inference, and the docking accuracy reaches the millimeter level. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0028] Figure 1 It is a schematic diagram of the distance measurement data curve of the present invention;

[0029] Figure 2 It is a schematic diagram of experimental data of the present invention;

[0030] Figure 3 This is a schematic diagram of the angle measurement result comparison experiment of the present invention;

[0031] Figure 4 It is a schematic diagram of the docking process of the present invention;

[0032] Figure 5 A schematic diagram of the design of the marker of the present invention;

[0033] Figure 6 It is a schematic diagram of the stability experiment comparison between the double code and the single code of the present invention;

[0034] Figure 7 This is a schematic diagram of the center point recognition results under different environments of the present invention;

[0035] Figure 8 It is a schematic diagram of fusion depth information of the present invention;

[0036] Fig. 9 A schematic diagram of the identification logic of the present invention;

[0037] Fig.10 It is a schematic diagram of the structured light three-dimensional visual perspective model of the present invention;

[0038] Fig.11 It is a schematic diagram of the docking process of the present invention;

[0039] Fig.12 It is a schematic diagram of the recognition of the letter "H" in a complex environment of the present invention;

[0040] Fig.13 It is a schematic diagram of the falling point of the present invention. Implementation

[0041] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0042] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings.

[0043] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features and technical solutions in the embodiments may be combined with each other.

[0044] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0045] Reference Figure 1-Figure 13 , a split flying car positioning and docking method based on visual depth information, the steps are as follows:

[0046] By introducing depth information to the traditional visual system, and from the active end (coupling of visual information with other information) and the passive end (design of markers and docking device), including the guidance of the entire process from the drone flying over the car to completing the docking task, the docking stability is effectively improved, and the accuracy is increased from centimeters to millimeters. During the terminal docking process, this application integrates the depth information obtained by the D435i camera to further correct the posture information solved by the visual system, keeping the terminal accuracy within 6mm, and conducting terminal accuracy comparison experiments and airborne comparison experiments respectively.

[0047] When conducting the accuracy verification experiment on the ground, the depth camera is fixed with a slide rail and a camera bracket, and a marker is placed at one end. The slide rail is slowly moved to physically measure the actual value of the camera distance to the marker, and then the output depth value and monocular ranging value are read. The data obtained by the three methods are fitted using a Gaussian function to obtain the ranging data curve, such as Figure 2 As shown;

[0048] According to the experimental data scatter plot, after Gaussian curve fitting, it can be found that when the distance is within 500mm, the depth camera has an advantage over the monocular vision ranging. After the distance range of 500~600mm, the accuracy advantage of the depth camera no longer exists, and the error increases rapidly. After the distance exceeds 600mm, although the fitting curve of the monocular vision ranging error has a lower error value, the fluctuation range is large and the stability is reduced.

[0049] In order to compare the stability of the three methods in the actual UAV landing and docking process, the present invention uses the GNSS ranging module, monocular vision ranging, and depth information ranging to obtain experimental values. A total of 20 experiments were carried out, and the results of the third experiment are as follows: Figure 2 As shown;

[0050] The GNSS ranging module has poor sensitivity, low information feedback frequency, and a minimum change detection unit of 0.1m. This obviously cannot be the data source for the docking guidance system, but its advantage is good linearity and stable output of altitude information. Visual ranging can accurately output actual altitude information in actual flight, with accuracy controlled at the centimeter level, but due to the shaking of the drone, as the distance gradually decreases, there is a situation where the marker moves out of the field of view. There is no loss of depth information, and the accuracy is higher than the visual ranging system within the end (<600mm), and the stability is good;

[0051] When the UAV is docked, the estimation of the three-dimensional attitude angle is necessary data for the flight control system, and its accuracy will greatly affect the docking result. The estimation of the pitch angle, roll angle, and azimuth angle are all based on the same internal parameter matrix, so it is only necessary to measure the accuracy of one of the angles to deduce the estimation results of the other two angles. In the actual experiment, the marker is placed on the coordinate paper with the baseline, and the true value is calculated by measuring the angle between the visual image screen-the marker-the baseline, and then compared with the measurement value output by the visual system and the angle derived from the GPS module. A total of 20 experiments were carried out, and the data of the 14th experiment was measured to obtain the following Figure 3 ;

[0052] In the angle measurement result comparison experiment, it can be found that thanks to the advantage of the large distance between the center points of the dual codes, the visual guidance system can provide very accurate angle information. The maximum error of GPS during the entire experiment is about 12 degrees, while the visual guidance system can stably control the error value within 0.5 degrees. It has obvious advantages in angle detection and can provide better data support for the onboard computer to estimate the three-dimensional attitude angle. Figure 4 ;

[0053] Multi-level markers

[0054] It has good visual accuracy within different distance ranges. The marker is designed with dual codes. Accurate posture information is obtained through cross-fusion algorithm. The obvious corner point features can provide distinct morphological features for the screening process, and make the positioning base point have more redundant information to ensure that the drone still has good recognition accuracy during the shaking process. The spatial position and three-dimensional posture calculation are carried out through the 8 corner point information, creating good recognition conditions for target detection and achieving accurate terminal docking.

[0055] Spatial position and 3D posture calculation, through PnP (full name Perspective-n-Points), an algorithm for calculating the relative 3D posture of two planes. The more corner points involved, the higher the calculation accuracy. "n" refers to the number of corner points, and 8 corner points is the P8P algorithm. A minimum of 3 points are required for calculation, that is, P3P, to solve the problem of object motion positioning from 3D to 2D point pairs, that is, given the coordinates of the object in the world coordinate system and the pixel coordinates of the object in the camera's image plane, the camera's posture (six degrees of freedom, position coordinates and three direction angles) is solved.

[0056] The marker consists of an outer frame, an H graphic, and a double Apriltag code, which are visual recognition objects at different stages. The outer frame can be quickly found and listed as a candidate docking area when the drone searches for targets at a long distance. When performing pixel point clustering calculations, the convex quadrilateral shape can be recognized more efficiently. When all detected convex quadrilaterals are screened after the outer frame recognition is completed, the obvious corner point features and symmetrical structure of the H graphic can provide distinct morphological features for the screening process, ensuring the accuracy of the screening process, and preliminarily calculating the center position of the marker through the pixel coordinates of each corner point, providing appropriate orientation guidance for the drone.

[0057] The pixel clustering calculation specifically uses the k-means algorithm, a color-based unsupervised learning clustering algorithm that does not require samples. It "segments" the image according to certain rules and puts the same or similar target pixels together. In this way, an object can be extracted from the entire image. The steps are as follows:

[0058] Group the same or similar target pixels together, and assign the number of pixels to a specific class, setting the pixels to be divided into three categories;

[0059] S1: Select the initial centers of the three classes. The initial center points are selected completely randomly.

[0060] S2: Calculate the distances from the remaining points to these three centers;

[0061] S3: Classify the points with the shortest distance from the center point into one category;

[0062] S4: Divide all data points in sequence;

[0063] S5: recalculate the center;

[0064] S6: Repeat steps S2-S5 until the center point does not change.

[0065] According to this algorithm, the outer contour of the marker image, that is, the quadrilateral wheel, can be segmented in the image. Therefore, the outer contour of the marker in various environments can be segmented without learning samples, and the center coordinate value can be calculated based on the segmented contour information to guide the drone for initial positioning.

[0066] After the drone descends to the specified height based on the location information obtained in the previous step, the dual-code recognition of Apriltag is turned on. The eight corner points can provide stable location information for estimating the spatial position and three-dimensional posture, thereby achieving accurate terminal docking. As the drone gradually approaches the vehicle, the markers provide 2D and 3D information in a hierarchical manner to help the drone continuously adjust its position and posture. Figure 5 The outer contour and secondary outer contour of the marker can help the drone quickly and accurately find the docking target when it is far away. The clear corner point layout and unique contour make the marker have better resistance to environmental interference when it is initially identified. The inner layer mainly focuses on the accuracy and stability of near-end docking, and improves the recognition stability of the visual system under the condition of drone shaking through the redundant information of multiple corner points;

[0067] A target detection scene library was built based on machine learning;

[0068] For different docking scenarios, combined with the appearance characteristics of the docking platform, the appropriate number of fully connected layers and neurons are set, and different neural network layers are adjusted to train models suitable for different platform characteristics, so that they can still accurately and stably identify the target under the influence of tilt, lighting, and occlusion. Through training and collecting data, optimizing model parameters, and building a split flying car target detection scenario database model; the center point recognition results in different environments are as follows: Figure 7 .

[0069] When dealing with complex environments and docking platforms, it is particularly important to introduce machine learning. The neural network model is trained with a large amount of data, and neurons are continuously optimized to improve the recognition success rate. A neural network model is established for different environments and different docking platforms, and the impact of different angles, occlusions, and other situations on the success rate is studied. The model is constructed by collecting environmental and target information. For different docking scenarios, the appropriate number of fully connected layers and neurons are set, and different neural network layers are adjusted in combination with the appearance characteristics of the docking platform. Models suitable for different platform characteristics are trained so that they can still accurately and stably identify the target under the influence of tilt, lighting, and occlusion. The collected data set is trained, the model parameters are optimized, the accuracy and speed of target detection are improved, and a split flying car target detection scene database model is constructed.

[0070] The drone relies on GPS and inertial navigation systems to move to the landing point. When the vertical height is 3±0.2m and the horizontal distance is 0±0.2m, the visual guidance system is turned on. The visual guidance system further locates based on the acquired position information, and finally obtains the position and direction information of the target point by performing adaptive threshold, continuous boundary segmentation, quadrilateral fitting, homography transformation, decoding and matching with the Apriltag library. After that, the drone is guided to a height of 1±0.02m and a horizontal error of 0±0.02m. After entering the terminal positioning, the depth camera is turned on, and after configuring the depth stream and color stream, the depth frame and the color frame are aligned, the depth information is integrated, the accuracy is improved to within 6mm, and a precise landing is performed.

[0071] Multi-source fusion technology that integrates visual and depth information.

[0072] During the end docking process, the close-range visual error will be magnified. The depth information obtained by the structured light is introduced into the pose matrix, and the accurate relative coordinates are obtained through multi-source information fusion correction. The position and direction information of the target point are obtained by adaptive thresholding, continuous boundary segmentation, quadrilateral fitting, homography transformation, decoding and matching with the Apriltag library. After configuring the depth stream and color stream, the depth frame is aligned with the color frame, and the depth information is fused, such as Figure 8 .

[0073] Monocular vision ranging and binocular vision ranging are two common methods of using visual systems to obtain depth information. However, methods based entirely on image matching have great difficulties, and the accuracy and correctness of matching are difficult to guarantee. In addition, during the docking process of the drone, the image is easily disturbed by lighting, tilt angle, and mechanical structure. It is difficult to complete high-precision docking operations relying solely on visual guidance. The structured light technology is comprehensively applied to encode or characterize the projected light source to obtain an image modulated by the depth of the object surface. The structured light source has many feature points or codes, so it provides many matching corner points, which can easily match feature points. Depth cameras have higher accuracy, faster calculation speed, and are not limited by the field of view. The RealSense D435i depth camera is selected for ranging. The ranging principle diagram is shown as follows. Fig.10 shown.

[0074] Magnetic cone rod docking system

[0075] Based on the cone rod structure and magnetic suction principle, an adaptive docking platform is constructed, and a visual sensor feedback system is established based on the cone rod shape detection to achieve real-time feedback of visual position information, and automatically adjust the position and posture of the docking device according to the visual information. By real-time monitoring and analysis of the guidance system data during the docking process, collision sensors or force sensors are added to detect unexpected collisions or abnormal forces to ensure that the docking device can respond to changes in a timely manner, achieving more accurate docking. The cone rod magnetic docking device based on force feedback can itself be used as a machine vision target to provide relative yaw angle measurement information. The magnetic suction design improves the maneuverability of the drone and the ground vehicle body, and the docking device can respond to unexpected collisions or abnormal forces in a timely manner.

[0076] Docking experiments

[0077] By preprocessing the image, a binary image is obtained, and then the contour is screened to determine that the "H" that meets the preset contour relationship is the target marker. The recognition logic flow chart is as follows: Fig. 9 As shown. Binary images only have black and white colors (that is, only 0 and 1). Use the Suzuki contour tracking algorithm to find the contour. The steps are as follows:

[0078] First, determine the starting point of the boundary, that is, the starting pixel point of the contour tracing (usually the upper leftmost pixel on the contour);

[0079] Secondly, the sequence numbers of different boundaries are marked with corresponding tracking rules, and then the outer boundaries or holes are determined according to the sequence numbers, all contours are searched out, and finally a contour relationship tree is established.

[0080] After obtaining the boundary, the intersection of the two straight boundary lines is used to determine the inflection point. After counting the number and relative position of the inflection points, it can be determined whether it is the target "H". By judging whether the boundary relationship of "border-H-double code" satisfies the hierarchical relationship tree of the corner point quantity: The "H" characters that satisfy the relationship tree can be considered as target markers, which can filter out the interference in the environment and greatly reduce the misjudgment rate. In the experiment, 100% correct recognition can be achieved. The recognition of the "H" characters in complex environments is shown in the figure. Fig.12 ;

[0081] Comparison of drop point experiments using different methods:

[0082] No matter which guidance method is used, when applied to the actual docking and landing scene of the split flying car, it can only bring a certain degree of accuracy improvement. The final landing accuracy is also affected by many factors such as flight control, drone hardware, environmental wind, etc. However, under the same other conditions, the landing accuracy can still be maximized by changing the guidance method. Therefore, the experiment can intuitively show the help of various guidance methods in improving accuracy. After the drone takes off from a fixed location, each time it flies any distance in any direction, it relies on GPS to return autonomously and relies on the system to guide the landing to obtain the following landing points. The position and angle deviation between the landing point and the take-off origin can be visualized to visualize the accuracy advantage of the guidance system. A landing point experiment was carried out on a circular target. Two guidance methods were used to complete 20 take-offs and landings respectively. Invalid data with too large errors were removed, and the relative position of the landing point and the final landing direction of the drone were retained to obtain the landing point map. Fig.13 .

[0083] The preferred embodiments of the present invention disclosed above are only used to help explain the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the present invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A method for positioning and docking a split flying car based on visual depth information, characterized in that: Here are the steps: S1: Double-code markers are used to increase positioning accuracy during docking; S2: Build a target detection scene library; S3: The depth information obtained by structured light is introduced into the pose matrix, and the accurate relative coordinates are obtained through multi-source information fusion correction; In the step S1, the double-code marker is composed of an outer frame, an H graphic and a double Apriltag code; The outer frame can be quickly found and listed as an alternative docking area when the drone searches for a target at a long distance; The H-shaped pattern can provide distinct morphological features for the screening process through obvious corner features and symmetrical structure, thereby ensuring the accuracy of the screening process; The dual Apriltag code is used to provide stable position information through 8 corner points for estimating spatial position and three-dimensional posture; In the step S2, the target detection scene library is constructed, and the steps are as follows: The model is constructed by collecting environmental information and target information. For different docking scenarios, the appropriate number of fully connected layers and neurons are set according to the appearance characteristics of the docking platform, and different neural network layers are adjusted to train models suitable for different platform characteristics. In the step S3, the depth information obtained by the structured light is introduced into the pose matrix, and the accurate relative coordinates are obtained through multi-source information fusion correction. The steps are as follows: The position and direction information of the target point is obtained by performing adaptive thresholding, continuous boundary segmentation, quadrilateral fitting, homography transformation, and decoding on the markers. The depth frame and color frame are aligned and the depth information is fused by configuring the depth stream and color stream.

2. The method for positioning and docking a split flying car based on visual depth information according to claim 1, characterized in that: When the outer frame is calculated by pixel clustering, the convex quadrilateral shape can be recognized more efficiently.

3. The method for positioning and docking a split flying car based on visual depth information according to claim 1, characterized in that: The center position of the marker is preliminarily calculated through the pixel coordinates of each corner point in the obvious corner point features of the H figure, providing appropriate azimuth guidance for the drone.

Images