Apparatus and method for positioning unmanned aerial vehicle using water surface platform camera module

Abstract

The present application belongs to the technical field of optical positioning, and particularly relates to a device and method for positioning unmanned aerial vehicles (UAVs) using a water surface platform camera module. The device comprises a telephoto camera module and a close-up camera module equipped with an RGB camera and a depth camera, which are used to identify Aruco codes on different parts of the UAV body, obtain multiple frames of photos, and use a data processing unit to identify and convert data sources required for positioning according to a pre-set program, iteratively compensate for changes in adjacent frames of photos caused by water flow disturbance platforms, and calculate UAV point cloud coordinates. The present application solves the problems of positioning errors and poor real-time performance caused by light and ocean currents during the positioning of UAVs on existing water surface platforms, and is suitable for use in the working conditions of recovering UAVs on water surface platforms. Based on the same inventive concept, the present application also provides a readable recording medium and system storing the program of the method, which can be called by a processing circuit to execute the above method.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of optical positioning technology, and discloses a method, device, system and recording medium storing a program for executing the method for positioning an unmanned aerial vehicle by using a water surface platform camera module. BACKGROUND

[0002] An unmanned aerial vehicle (UAV) is an important component of air-sea cross-domain cooperative work due to its advantages of vertical take-off and landing, fixed-point hovering and flexible flight. In order to recycle and charge the UAV after work, the UAV needs to be accurately positioned to accurately fall onto the recycling platform. However, the positioning accuracy of the passive vision-based positioning method is affected by light and other reasons, and the disturbance of the uneven water surface to the platform will also cause large errors in the originally accurate positioning data, which will seriously affect the subsequent recycling work of the UAV. It is very difficult to achieve real-time and high-precision UAV positioning in a disturbed water environment.

[0003] The binocular vision technology uses two cameras to shoot the same two-dimensional image of a three-dimensional scene at different angles, and then extracts the depth information of the three-dimensional scene based on the principle of triangulation to realize positioning. Due to its advantages of simplicity, reliability and relatively low cost, it is widely used in various fields. However, this passive vision method needs to match the images according to the visual features in the depth solving process, and it is difficult to match in environments with poor light and scenes lacking visual features (such as white walls and sky), resulting in large distance errors.

[0004] Due to the waves and disturbances of the water surface, the platform shakes, causing the relative position between the platform center and the UAV to change. If the UAV is to be accurately positioned, the attitude error caused by the instability of the platform must be eliminated. Generally, an inertial measurement unit (IMU) is used for real-time motion compensation. Although its initial accuracy is high, it needs to calculate the attitude information by integrating the angular acceleration, and the error will accumulate over time, making it difficult to independently complete high-precision scene correlation. In contrast, point cloud registration involves feature extraction, matching and optimization, and can independently use the front and rear frames for motion estimation without drift problems, but its real-time performance is not as good as that of the IMU. SUMMARY

[0005] To solve the problems presented in the background art, the application provides a device for positioning a UAV by using a water surface platform camera module, comprising: a data processing unit, a telephoto camera module, a close-up camera module, a machine side Aruco code arranged on the side of the UAV, and a belly Aruco code arranged on the bottom of the UAV; the belly Aruco code and the machine side Aruco code are the same in all parameters except for the identity ID; the shooting distance of the telephoto camera module is 0.8-5 m, the telephoto camera module is arranged on one side edge of the water surface platform, the lens of the telephoto camera module faces the belly Aruco code, and the main shaft of the lens of the telephoto camera module is at an angle of 40-50 degrees with the horizontal plane; the shooting distance of the close-up camera module is 0.3-3 m, the close-up camera module is arranged on the adjacent side edge of the water surface platform, the lens of the close-up camera module faces the machine side Aruco code, and the main shaft of the lens of the close-up camera module is at an angle of 25-35 degrees with the horizontal plane; the telephoto camera module and the close-up camera module each comprise an RGB camera and a depth camera, and are in communication connection with the data processing unit; and the data processing unit is in communication connection with a UAV flight control unit.

[0006] Preferably, the data processing unit comprises a registration module for registering the RGB camera image and the depth camera image to obtain positioning information; a platform motion compensation module for compensating for the change in the positioning information caused by water disturbance by calculating the conversion matrix obtained from the front and rear point cloud positioning information; and a data source selection module for selecting the Aruco code whose two-dimensional shape is closest to a square as the data source for calculation when both Aruco codes are imaged; and selecting the UAV point cloud information whose Euclidean distance from the center of the Aruco code is the smallest as the data source for calculation within the range that can be imaged by the telephoto camera module and the close-up camera module.

[0007] Preferably, the main shaft of the lens of the telephoto camera module is at an angle of 45 degrees with the horizontal plane; and the main shaft of the lens of the close-up camera module is at an angle of 30 degrees with the horizontal plane.

[0008] Based on the same inventive concept, the application also provides a method for positioning a UAV by using a water surface platform camera module, which uses the device for positioning a UAV by using a water surface platform camera module, and comprises the following steps:

[0009] S1. When the UAV enters the imaging range of the water surface platform camera module, the UAV flight control unit adjusts the flight pose by using the point cloud information sent back by the data processing unit, so that the Aruco code beacon carried by the UAV is clearly recognized by the camera module;

[0010] S2. The intrinsic parameters and extrinsic parameters of each camera are obtained by chessboard calibration; after the two-dimensional information of each point is obtained by registering the RGB camera image and the depth camera image of the module, the corresponding depth information is obtained by the depth camera image, and the target point is converted into a three-dimensional coordinate, so as to obtain the UAV point cloud information;

[0011] S3. Detecting the Aruco code of the UAV, obtaining the corresponding three-dimensional coordinates of the four corners of the detected Aruco code by the pixel values of the four corners, calculating the center coordinates of the Aruco code as the positioning reference point, and if the module detects the side Aruco code of the UAV, converting the center coordinates of the side Aruco code into the center coordinates of the belly Aruco code through the conversion matrix;

[0012] S4. Compensating for the change of the positioning information caused by the water disturbance by calculating the conversion matrix obtained from the front and rear point cloud positioning information;

[0013] S5. When two Aruco codes are photographed at the same time, selecting the Aruco code whose two-dimensional shape is closest to a square as the data source for calculation; and in the range that can be photographed by the telephoto camera module and the close-up camera module, selecting the UAV point cloud information whose Euclidean distance from the center of the Aruco code to the center of the UAV point cloud is the smallest as the data source for calculation.

[0014] Preferably, the registration in S2 further includes the following steps:

[0015] The pixel coordinates are converted into spatial coordinates through the intrinsic matrix of each camera, the rotation vector and the translation vector are solved by using the extrinsic matrix of the camera, the coordinates of each point on the picture photographed by the RGB camera are mapped into the spatial coordinates of the depth camera through rotation and translation operations, and the UAV point cloud information is obtained.

[0016] Preferably, the compensation in S4 further includes the following steps:

[0017] The feature points are extracted, a preliminary correspondence between the point clouds of adjacent two frames is established by using a greedy search method, an initial value T0 of the conversion matrix is generated, coarse matching is realized through conversion, a plurality of rounds of iteration are performed through the voxelized generalized iterative closest point algorithm, the deviation between the coordinate measurement value and the true value after each iteration is calculated, the difference between the ratio of the deviation between the two times and 1 reaches a set threshold value, and the iteration is stopped, the final conversion matrix T is obtained, the rotation matrix R is calculated from T, the disturbance suffered by the platform is compensated, and the compensated positioning reference point coordinates of the UAV relative to the camera are obtained.

[0018] Preferably, the deviation is the mean absolute error.

[0019] Another scheme of the application provides a non-transitory readable recording medium for storing one or more programs containing a plurality of instructions, which when executed, will cause a processor to execute the above-mentioned method for positioning a UAV by using a water surface platform camera module.

[0020] The application also provides a UAV positioning system using a water surface platform camera module, comprising a processing circuit and a memory electrically coupled to the processing circuit, wherein the memory is configured to store at least one program, the program comprises a plurality of instructions, and the processing circuit executes the program to perform the UAV positioning method using the water surface platform camera module.

[0021] Compared with the prior art, the UAV positioning device and method using the water surface platform camera module have the following beneficial effects: (1) The application proposes a method of directly obtaining positioning information by registering RGB images and depth images to solve the problem of low adaptability of general passive vision to complex water surface environments. The depth camera uses infrared to solve depth and can obtain high-precision distance information in overcast environments. By registering two images with different resolutions through the internal and external parameters of the RGB camera and the depth camera, the depth corresponding to each pixel point can be directly obtained, and then the three-dimensional coordinates can be converted to quickly obtain point cloud information. Therefore, the application can not only improve positioning accuracy but also improve real-time positioning acquisition.

[0022] (2) In addition, the water surface environment is not stable, and the platform is prone to sway. Motion estimation through IMU will increase cumulative error with time due to angular velocity integration. By directly obtaining point cloud data of the front and rear frames as input, introducing mean absolute error, and calculating the average error ratio of the front and rear point clouds as an iteration condition, the VGICP algorithm is improved to obtain the rotation matrix in the conversion matrix, motion compensation is performed on the positioning information, the positioning accuracy is iteratively optimized, and the influence of water disturbance is reduced.

[0023] (3) In order to obtain the positioning of the UAV in the full range, a judgment function is designed for the Aruco code detected in different situations, the similarity to the square is introduced to judge which one to use for positioning when multiple Aruco codes are detected in a single module, the Euclidean distance between the center of the Aruco code and the center of the image is introduced, the distance is compared to determine which module data to use absolutely, and the corresponding point cloud-based motion estimation and motion compensation are performed to dynamically select the optimal data source and obtain more accurate preliminary positioning information. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 FIG. 1 is a structural schematic diagram of a UAV positioning device using a water surface platform camera module in an embodiment of the application;

[0025] Figure 2 FIG. 4 is a schematic diagram of the configuration of an Aruco code of a UAV on a body in an embodiment of the application;

[0026] Figure 3 FIG. 6 is a flowchart of RGB camera image and depth camera image registration in an embodiment of the application.

[0027] Figure 4 Flow chart for determining reference positioning fiducials in embodiments of the invention;

[0028] Figure 5 Flow chart for improved iterative VGICP in embodiments of the invention;

[0029] Figure 6 Flow chart for determining data sources for positioning calculations in embodiments of the invention. DETAILED DESCRIPTION

[0030] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application. The described embodiments are part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making innovative labor fall within the scope of protection of the present application.

[0031] The technical solutions and drawings will be described below to specifically explain the implementation process of the present application, including a device for positioning a UAV using a water surface platform camera module and a method for positioning a UAV using a water surface platform camera module. Figures 1-6 The technical solutions and drawings will be described below to specifically explain the implementation process of the present application, including a device for positioning a UAV using a water surface platform camera module and a method for positioning a UAV using a water surface platform camera module.

[0032] A device for positioning a UAV using a water surface platform camera module, comprising a data processing unit, a telephoto camera module, a close-up camera module, a machine side Aruco code placed on the side of the UAV, and a belly Aruco code placed on the bottom of the UAV. The belly Aruco code and the machine side Aruco code are the same in all parameters except for the identity ID. The telephoto camera module has a shooting distance of 2-5 m and is arranged on one side edge of the water surface platform with the lens pointing towards the belly Aruco code and the lens main shaft being at an angle of 45 degrees to the horizontal plane. The close-up camera module has a shooting distance of 0.3-3 m and is arranged on the adjacent side edge of the water surface platform with the lens pointing towards the machine side Aruco code and the lens main shaft being at an angle of 30 degrees to the horizontal plane. The telephoto camera module and the close-up camera module each comprise an RGB camera and a depth camera and are in communication connection with the data processing unit. The data processing unit is in communication connection with the UAV flight control unit.

[0033] The data processing unit comprises a registration module for registering the RGB camera image and the depth camera image to obtain positioning information; a platform motion compensation module for compensating for changes in the positioning information caused by water disturbance by calculating a conversion matrix obtained from the front and rear point cloud positioning information; a data source selection module for selecting the Aruco code whose imaged two-dimensional shape is closest to a square as the data source for calculation when two Aruco codes are simultaneously imaged; and selecting the unmanned aerial vehicle point cloud information whose Euclidean distance from the center of the Aruco code to the center of the unmanned aerial vehicle point cloud is smallest as the data source for calculation within the range that both the telephoto camera module and the close-up camera module can image.

[0034] The specific manner of using the above device for unmanned aerial vehicle positioning by using the water surface platform camera module is as follows:

[0035] Step 1: Corresponding settings are made to the water surface unmanned aerial vehicle recovery platform and the unmanned aerial vehicle, and the settings are as follows:

[0036] For the platform used for recovering the unmanned aerial vehicle, two camera modules (a telephoto camera module and a close-up camera module) are installed on the platform to observe and position the unmanned aerial vehicle in the air, and then the unmanned aerial vehicle is adjusted according to the positioning information. When the distance between the unmanned aerial vehicle and the platform reaches a certain threshold, the clamping arms on the platform are tightened so that the unmanned aerial vehicle reaches the water surface platform and is fixed at the center of the platform by the clamping arms for recovery and charging.

[0037] Settings of the unmanned aerial vehicle: Aruco codes with a side length of 10 cm are installed on the bottom and sides of the unmanned aerial vehicle. These two two-dimensional codes are identical except for the id information and serve as the source of positioning information. The four corner points of the Aruco code are detected by the camera, and the center point is calculated as the preliminary positioning information of the unmanned aerial vehicle after the three-dimensional coordinate information of the corner points is calculated. Two-dimensional codes are installed in different places to enable comprehensive detection by the two modules on the water surface platform. The ids are different to enable the selection of a more appropriate positioning data source by using the judgment framework and to obtain more accurate results.

[0038] Water surface platform setting: two modules are set on the adjacent two sides of the water surface platform, each module contains an RGB camera and an infrared depth camera, the data with subscript rgb in the subsequent data is the data of the RGB camera, and the data with subscript dep is the data of the infrared depth camera; one of the two modules is a R132 camera of Yin Niu (i.e. a close-up camera module), the observation distance is 0.3m-3m, and the other is a C158 camera of Yin Niu (i.e. a telephoto camera module), the observation distance is 2m-5m, and the observation distances of the two are different. The C158 camera is placed at an angle of 45 degrees with the plane of the platform and is used to observe the Aruco code at the bottom of the unmanned aerial vehicle; the R132 is placed at an angle of 30 degrees and is used to observe the Aruco code at the side of the unmanned aerial vehicle, and the installation angles and observation ranges of the two are different to ensure that the Aruco code of the unmanned aerial vehicle is observed in all directions. The image size obtained by the RGB camera is 800x600, and the image resolution obtained by the infrared depth camera is 544x360, so the two need to be registered before point cloud acquisition because their resolutions are different.

[0039] 1.4) Re-calibration of camera module: Since this system is used in a water environment, a water-tight shell needs to be added to the module, so the internal and external parameters of the camera will increase the error due to the influence of the added water-tight shell, and a chessboard calibration needs to be performed to re-acquire the internal parameters of the RGB image in this environment; and the infrared emitter of the infrared depth camera is blocked with black tape, and the internal parameters H rgb , dep obtained by Zhang Zhengyou calibration are acquired dep , dep and R rgb , rgb .

[0040] Second step: registration module action, register the RGB image and the depth image of the camera module, after acquiring the two-dimensional information of each point, the corresponding depth information is acquired through the depth image, and then the target point is converted into three-dimensional coordinates, so as to facilitate the acquisition of point cloud information. The registration of the two is actually to obtain the conversion matrix between the two-dimensional coordinates of the RGB image and the depth image, and the registration content is as follows:

[0041] 2.1) For a point Q in space, the spatial coordinates and pixel coordinates in the depth camera coordinate system are Q dep and q dep , and the spatial coordinates and pixel coordinates in the RGB camera coordinate system are Q rgb and q rgb

[0042] Q dep = H dep -1 q dep '

[0043] Q rgb = H rgb -1 q rgb (1)

[0044] In the formula, H rgb and H dep are the intrinsic matrices of the cameras, q rgb and q dep are the homogeneous coordinates. Since the resolutions of the two cameras are different, and their coordinate systems are also different, they can be related by a rotation and translation transformation, that is:

[0045] Q dep = RQ rgb + T (2)

[0046] where R is a rotation matrix and T is a translation vector. Finally, the coordinates of the point can be obtained by projecting Q dep using H dep :

[0047] q dep = H dep Q dep (3)

[0048] 2.2) Using the extrinsic parameters of the two cameras to solve the rotation vector and the translation vector, the extrinsic matrix also consists of a rotation matrix and a translation matrix. For a point Q in the global coordinate system, the transformation to the coordinate system of the two cameras is as shown in formula (4):

[0049] Q dep = R dep Q + T dep

[0050] q rgb = R rgb Q + T rgb (4)

[0051] In the formula, R dep , T dep are the rotation matrix and translation vector contained in the extrinsic matrix of the depth camera, and R rgb , T rgb are the rotation matrix and translation vector contained in the extrinsic matrix of the RGB camera.

[0052] 2.3) Express the point Q using Q rgb , R rgb and T rgb :

[0053] Q = R rgb -1 Q rgb -R rgb-1 T rgb (5)

[0054] Substitute the above formula into the first formula of formula (4) to obtain:

[0055] Q dep =R dep R rgb -1 Q rgb -R dep R rgb -1 T rgb +T dep

[0056] =(R dep R rgb -1 )Q rgb +T dep -(R dep R rgb -1 )T rgb (6)

[0057] Together with formula (2), we obtain:

[0058] R=R dep R rgb -1 ,

[0059] T=T dep -RT rgb (7)

[0060] Further, Q dep corresponding to the depth coordinate q dep , facilitating the acquisition and storage of three-dimensional coordinates and overall point cloud.

[0061] 2.4) In the process of obtaining three-dimensional points, taking the RGB camera coordinate system as the reference, the pixel in the RGB image is (u1, v1), the projection coordinates of the corresponding depth image are (u2, v2, z), and the overall projection coordinates are (u1, v1, z). Using the camera intrinsic parameters, we obtain:

[0062]

[0063] z c =z / 1000 (8)

[0065] where u0, v0 are the pixel coordinates of the optical center on the RGB image, f / dx and f / dy are the focal length of the camera in x and y direction, which can be directly obtained in the intrinsic parameters. The three-dimensional coordinates of each pixel in the RGB image are calculated, and the point cloud in the camera coordinate system is obtained.

[0066] Third step: detect the Aruco code and take the center value as the initial positioning value:

[0067] 3.1) Detect the Aruco code of the UAV, and the pixel values of the four corners of the detected Aruco code are (u1, v1), (u2, v2), (u3, v3), and (u4, v4). The three-dimensional coordinates in the camera are directly obtained from the second step, (x1, y1, z1), (x2, y2, z2), (x3, y3, z3), and (x4, y4, z4).

[0068] 3.2) Perform initial positioning of the UAV. The positioning of the UAV is based on the center of the Aruco code at the bottom of the UAV. If the module observes the Aruco code at the bottom of the UAV, the positioning coordinates (x0, y0, z0) of the UAV are:

[0069] x c =(x1+x2+x3+x4) / 4,

[0070] y o =(y1+y2+y3+y4) / 4,

[0071] z o =(z1+z2+z3+z4) / 4 (9)

[0073] If the module observes the Aruco code on the side of the UAV, and the center coordinates of the code are (x0', y0', z0'), then the positioning coordinates (x0, y0, z0) of the UAV are:

[0074] (x0, y0, z0) = T sc (x 0’ ,y 0’ ,z 0’ ) (10)

[0075] where T sc is the conversion matrix from the center of the side code to the center of the bottom code, and the three-dimensional coordinates (x0, y0, z0) of the object relative to the camera are obtained.

[0076] Fourth step: while the UAV is falling steadily, the recovery platform is shaking due to the disturbance of the water body. Since the obtained positioning information of the UAV is equivalent to the platform and camera coordinate system, only the attitude change of the platform (camera) relative to the previous frame is needed to start the platform motion compensation module to implement water disturbance compensation in the calculation of positioning.

[0077] 4.1) The three-dimensional point cloud information at this moment can be obtained from the second step, which is used as input together with the point cloud information of the previous frame. Since the attitude change relative to the previous frame is obtained, the point cloud of the previous frame is set as the target point cloud Q(q1, q2……q n ), and the point cloud of this frame is set as the source point cloud P(P1, P2……P m ). Where n and m are the number of point clouds. The conversion matrix initial value T0 is obtained by performing FPFH coarse registration on the two point clouds. First, for a point P s in the point cloud, find its neighbor P t in the neighborhood. Based on the vector between the two points and the normal vector obtained by the nearest neighbor search, define a local coordinate system. The three axes are calculated as follows:

[0078] X = n s

[0079] Y = (p ι -p s ) × X

[0080] W = X × Y (11)

[0082] Where n s is the normal vector of P s , n t is the normal vector of P t , and according to the coordinate system, the triplet feature between P s and P t is represented as:

[0083]

[0084] β = Y·n t ,

[0085] θ = arctan(Z·n t , X·n t ) (12)

[0086] When calculating the triplet of P s , first calculate the triplet of the source point and five neighbor points, then calculate the triplet of the neighbor points and their neighborhood, and assign a distance-based weight to them. Finally, assign the triplet combination to P s, the histogram is calculated by using formula (13) and formula (14) to obtain the statistical features:

[0087]

[0088] Wherein, the SPF (Surface Point Feature Histograms) function is based on the normal vector, the angle information between points to calculate the local geometric features of a point, the formula is as follows:

[0089]

[0090] Through the extracted feature points, and combined with the greedy search method, the preliminary correspondence between point clouds is established, so as to realize coarse matching, and generate a conversion matrix as the initial value T0.

[0091] 4.2) Assuming that each point in the two point clouds is subject to Gaussian distribution, the reason is that due to the error of measurement and other links, the measured value of the position of each point is actually deviated from the true value p k and q k . Under ideal conditions, the formula for converting from the P coordinate system to the Q coordinate system is:

[0092] q k = Rp k +T (15)

[0093] For the points in the source point cloud and the target point cloud, the coordinates are subject to Gaussian distribution, and there are:

[0094]

[0095] Wherein, and are the mean coordinates of the two groups of point clouds, and the covariance matrices of the point clouds Q and P are obtained and

[0096] 4.3) Voxelization of the target point cloud. Set the n*n*n point cloud domain for the target point cloud for voxelization:

[0097]

[0098] Wherein, N i is the number of points in the point cloud voxel domain;

[0099] Substitute the conversion matrix initial value T0 after coarse registration in 4.1). In order to solve the problem that VGICP may over-iterate, the mean absolute error (MAE) is introduced as a standard for measuring error convergence in the iteration process. The mean absolute error can be expressed as:

[0100]

[0101] where, where N is the number of point pairs, T i-1 is the updated transformation matrix in the last iteration. Before the first iteration, T i-1 is the transformation matrix T0obtained from the coarse registration, MAE is set to NUM_MAX which is a value of infinity, so that the following multiple iterations are performed.

[0102] 4.4) Update the transformation matrix T, and the mean absolute error MAE is used to judge

[0103] The distance between the p k and q k transformed points in the target point cloud after the i-th iteration is calculated as:

[0104]

[0105] The distribution satisfies the Gaussian distribution:

[0106]

[0107] where, is the mean of , and is the covariance of . The transformation T that maximizes the log-likelihood is estimated by the following way and is modified to facilitate efficient computation:

[0108]

[0109] That is:

[0110]

[0111] From the voxelization in 4.3), we have:

[0112]

[0113] Let this iteration be the i-th iteration, according to formula (18), the MAE i value of the i-th iteration is obtained, the ratio between the MAE i after the i-th iteration and the MAE i-1 of the i-1-th iteration is calculated to describe the similarity of the results of the two iterations:

[0114]

[0115] ​The ratio is usually between 0 and 1, if the ratio is closer to 1, the results of two iterations are more similar, the improvement brought by subsequent iterations may be smaller, when K≥0.97, the iteration is stopped, and the final conversion matrix T is obtained.

[0116] 4.4) Since the obtained positioning information of the UAV is equivalent to the platform and camera coordinate system, only the attitude change of the platform (camera) relative to the previous frame needs to be obtained. The initial positioning coordinates of the UAV are (x0, y0, z0), and the final conversion matrix is:

[0117]

[0118] The rotation matrix R is obtained from it, the UAV is motion compensated, and the final coordinates about the camera (x c ’,y c ’,z c ’) are obtained:

[0119] (x c′ , y c′ , z c′ ) = R(x o , y o , z o )(26)

[0120] Finally, according to the different modules, the coordinates in the platform center coordinate system are converted to obtain the positioning (x c , y c , z c ) about the platform center:

[0121] (x c , y c , z c ) = R 158 / 132 (x c′ , y c′ , z c′ )(27)

[0122] Step 5: Set the judgment framework, according to whether the RGB camera detects the Aruco code, whether it detects multiple Aruco codes, whether a single module or multiple modules detect the Aruco code, start the data source selection module for corresponding processing and data source selection, dynamically select the optimal data source, the specific content is as follows:

[0123] 5.1) For the case where multiple Aruco codes are detected by the RGB camera in a module, set the evaluation function, the formula is as follows:

[0124]

[0125] Among them, Area of the Aruco code in the image detected by the module, Circum Aruco Circumference of the Aruco code in the image detected by the module RGB camera, (Circum Aruco / 4) 2 Area of the square under this circumference, by evaluating the degree of similarity of the Aruco code detected by RGB in the RGB image with the square, the evaluation score score is obtained for comparison, the closer to 1, the more similar the degree of square, the smaller the degree of tangential distortion, to determine the Aruco code for positioning.

[0126] 5.2) For the case of multiple modules detecting Aruco code at the same time, different types are processed accordingly.

[0127] C158 module and R132 module because of the different observation distance and setting angle, their observation range is also different, when the UAV is more than 3m away, only C158 can observe the two-dimensional code at the bottom of the UAV, when the UAV is lowered to below 0.8m, the actual distance is generally within 2m, only R132 can observe the two-dimensional code at the side of the UAV, when the height of the UAV is between 0.8-3m, two camera modules can observe at most of the time, in this case, an evaluation function needs to be set for corresponding data source processing and motion estimation processing, first, the resolution of the RGB image is a x b, the evaluation function is as follows:

[0128]

[0129] Among them, (u0, v0) is the center of the image, by evaluating the Euclidean distance between the center of the Aruco code and the center of the image, the smaller the r value, the closer the detected Aruco code to the image, the smaller the possibility of radial distortion. Therefore, the module with smaller detected r value is used for subsequent processing.

[0130] 5.3) For the case where only one module detects in the previous frame and two modules detect in the next frame, let the result detected by C158 in the previous frame be P c158 , two modules detect in the next frame at the same time, save the point cloud (recorded as the previous frame) as Q c158 and Q c132 When formula (27) is obtained, the data of C158 is processed, the normal motion estimation and compensation are performed with Q c158 and P c158 When the data of Q c132 is processed, since there is no point cloud based on R132 in the previous frame, the point cloud Q c158 obtained by C158 can be directly used.c158 A motion estimation is performed to obtain a transformation matrix T c158 of C158 by the following equation:

[0131] T R132 = T CR T C158 (30)

[0132] A transformation matrix T R132 of camera R132 is obtained by the following equation: CR A rotation matrix R132 is extracted to compensate the preliminary positioning information obtained by T c158 and T R132 is a transformation relationship matrix between T

[0133] Those skilled in the art will appreciate that embodiments of the application can be readily used as a method, apparatus, or computer program product. Accordingly, the application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the application can take the form of a computer program product on one or more computer- readable storage media (e.g., magnetic disks, CD-ROMs, optical storage media, etc.) having computer usable program code embodied thereon.

[0134] The application is described in reference to the flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processing system or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart Figure 1 one or more functions specified in the flowchart block or blocks. Figure 1 one or more functions specified in the flowchart block or blocks.

[0135] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks. Figure 1 one or more functions specified in the flowchart block or blocks. Figure 1 one or more functions specified in the flowchart block or blocks.

[0136] These computer program instructions can also be loaded into a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable data processing device to generate a computer implemented process, and thus the instructions executed on the computer or other programmable data processing device provide a process for implementing the functions specified in the flowchart Figure 1 one flowchart or multiple flowcharts and / or blocks Figure 1 one block or multiple blocks.

[0137] Assembling the above method steps into a program and storing it in a hard disk or other non-transient storage medium constitutes an embodiment of the "non-transient readable recording medium" of the present application; and electrically connecting the storage medium with a computer processor, and through data processing, the unmanned aerial vehicle positioning using the water surface platform camera module can be completed, which constitutes an embodiment of the "unmanned aerial vehicle positioning system using the water surface platform camera module" of the present application.

[0138] Finally, it should be noted that: the above only for the preferred embodiments of the present application, and not for limiting the present application, although the foregoing detailed description of the present application is made with reference to the foregoing embodiments, for those skilled in the art, it still can be modified to the technical solutions recorded in the foregoing embodiments, or equivalent replacement of some technical features. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A device for positioning a drone using a surface platform camera module, characterized in that include: Data processing unit, telephoto camera module, close-up camera module, side Aruco code placed on the side of the drone, and belly Aruco code placed on the bottom of the drone; except for the identity ID, the belly Aruco code and the side Aruco code are the same in other parameters; the shooting distance of the telephoto camera module is 0.8-5m, and it is arranged on one edge of the surface platform, with the lens facing the belly Aruco code, and the lens main axis is 40-50 degrees to the horizontal plane; the shooting distance of the close-up camera module is 0.3-3m, and it is arranged on the surface platform The adjacent edge of the camera is facing the Aruco code on the side of the aircraft, and the main axis of the lens is 25-35 degrees to the horizontal plane; the telephoto camera module and the close-up camera module both include an RGB camera and a depth camera, which are respectively communicated with the data processing unit, and the data processing unit is communicated with the UAV flight control unit; the data processing unit includes a registration module for registering the RGB camera image and the depth camera image to obtain positioning information; the platform motion compensation module is used to compensate for changes in positioning information caused by water disturbances by calculating the conversion matrix obtained by the front and rear point cloud positioning information; The data source selection module is used to select the Aruco code whose two-dimensional imaged shape is closest to a square as the data source for calculation reference when two Aruco codes are captured simultaneously; within the range that can be captured by both the telephoto camera module and the macro camera module, the drone point cloud information with the smallest Euclidean distance between the center of the Aruco code and the center of the drone point cloud is selected as the data source for calculation reference.

2. The device for positioning a drone using a surface platform camera module according to claim 1, characterized in that: The main axis of the lens of the telephoto camera module is 45 degrees to the horizontal plane; the main axis of the lens of the close-up camera module is 30 degrees to the horizontal plane.

3. A method for positioning a drone using a surface platform camera module, comprising: using the apparatus for positioning a drone using a surface platform camera module according to any one of claims 1 to 2, and completing the following steps: S1. When the drone enters the camera module's range, the drone's flight control unit uses the point cloud information sent back by the data processing unit to adjust its flight posture so that its Aruco code beacon can be clearly identified by the camera module. S2. Obtain the intrinsic and extrinsic parameters of each camera through checkerboard calibration; The module's RGB camera image and depth camera image are registered to obtain the two-dimensional information of each point. The corresponding depth information is obtained through the depth camera image and then the target point is converted into three-dimensional coordinates to obtain the drone point cloud information. S3. Detect the drone's Aruco code. Obtain the corresponding three-dimensional coordinates using the pixel values ​​of the four corners of the detected Aruco code in step S2. The center coordinates of the ventral Aruco code are used as the positioning reference point. If the module detects the aircraft-side Aruco code, the center coordinates of the aircraft-side Aruco code are converted to the center coordinates of the ventral Aruco code using a conversion matrix. S4. Compensating for changes in positioning information caused by water disturbances by calculating the conversion matrix obtained from the previous and next point cloud positioning information; S5. When two Aruco codes are captured simultaneously, the Aruco code whose two-dimensional imaged shape is closest to a square is selected as the data source for the calculation reference. Within the range captured by both the telephoto and macro camera modules, the drone point cloud information with the smallest Euclidean distance between the center of the Aruco code and the center of the drone point cloud is selected as the data source for the calculation reference.

4. The method for positioning a UAV using a surface platform camera module according to claim 3, wherein: The registration in step S2 also includes the following steps: The pixel coordinates are converted into spatial coordinates through the intrinsic parameter matrix of each camera, and the rotation vector and translation vector are solved using the extrinsic parameter matrix of the camera. Through rotation and translation operations, the coordinates of each point on the image taken by the RGB camera are mapped to the spatial coordinates of the depth camera to obtain the drone point cloud information.

5. The method for positioning a UAV using a surface platform camera module according to claim 4, wherein: The compensation in step S4 also includes the following steps: By extracting feature points and combining them with a greedy search method, a preliminary correspondence between two adjacent frame point clouds is established, generating the initial value T0 of the transformation matrix. Rough matching is achieved through transformation, and multiple rounds of iterations are performed using the voxelized generalized iterative closest point algorithm. The deviation between the coordinate measurement value and the true value after each iteration is calculated. The iteration is stopped when the difference between the ratio of the current and the next two deviations and 1 reaches the set threshold. The final transformation matrix T is obtained, and the rotation matrix R is calculated from T to compensate for the disturbance of the platform, thereby obtaining the compensated positioning reference point coordinates of the drone with respect to the camera.

6. The method for positioning a UAV using a surface platform camera module according to claim 5, characterized in that: The deviation is the mean absolute error.

7. A non-transitory readable recording medium for storing one or more programs including a plurality of instructions, characterized in that: When the instruction is executed, the processor will be caused to execute the method for positioning a drone using a surface platform camera module as described in claim 6.

8. A drone positioning system using a surface platform camera module, characterized in that It includes a processing circuit and a memory electrically coupled thereto, characterized in that the memory is configured to store at least one program, the program includes multiple instructions, and the processing circuit runs the program to execute the method of using a surface platform camera module to locate a drone as described in claim 6.

Images