A robust UAV-unmanned ship collaborative relative visual positioning method and system

Abstract

The present invention discloses a robust UAV-unmanned ship collaborative relative visual positioning method and system. The present invention uses a corner detection algorithm based on image entropy and uses the nonlinear mapping result of image entropy as the threshold of corner detection of the Shi-Tomasi algorithm, thereby achieving the effect of adaptively adjusting the corner detection threshold. At the same time, the present invention uses a corner matching method based on geometric feature learning to support an asymmetric tolerant corner matching process. Finally, the present invention proposes a KF-GRU model for updating states and covariances without a Kalman filter model. In general, the present invention provides accurate pose estimation results for the UAV-unmanned ship collaborative system and improves the safety of unmanned system collaboration.

Description

Technical Field

[0001] The present invention belongs to the field of relative visual positioning (RVL) of unmanned aerial vehicle (UAV) and unmanned ship collaboration based on image processing, and in particular relates to a robust UAV-unmanned ship collaborative relative visual positioning method and system. Background Art

[0002] Collaborative unmanned surface vehicles have been widely used in civil and military scenarios. The drone's posture is an important parameter for determining the drone's position in the entire unmanned system. Accurate positioning is an important prerequisite for ensuring the relative position between the drone and the unmanned ship remains stable. It can avoid accidents such as the drone falling into the water due to inaccurate relative position during the coordinated landing process with the unmanned ship, thereby ensuring the safety of drone flight.

[0003] However, in the coordinated landing mission, the safe landing area is small and is always in six-degree-of-freedom motion. In addition, the sparse feature points in the water environment are also an inevitable problem for visual positioning. Therefore, the relative visual positioning method has become the key to solving the positioning problem of the surface unmanned collaborative system.

[0004] In recent years, machine vision has become a common approach to RVL problems due to the wide availability of visual sensors and the rapid development of image learning. Specifically, machine vision systems use cameras and other visual sensors to capture real-time images and data of the surrounding environment. Then, AI algorithms are used to process the visual information to determine the relative position and orientation of the drone relative to other objects or high-precision landmarks. Therefore, using appropriate algorithms to process visual information has become a key factor in improving the visual positioning accuracy of drones. Summary of the invention

[0005] In response to the problems that existing UAV pose estimation lacks robustness and is overly dependent on feature points, the present invention provides a robust UAV-unmanned ship collaborative relative visual positioning method, system, medium and device, which aims to perform online estimation of the UAV's pose based on image data collected by the UAV's visual sensor, provide accurate pose estimation results for the UAV-unmanned ship collaborative system, and improve the safety of unmanned system collaboration.

[0006] To achieve the above object, on the one hand, the present invention provides a robust UAV-unmanned ship collaborative relative visual positioning method, the method comprising the following steps:

[0007] Step 1: The current cooperative identification image is collected through the UAV airborne visual sensor, and the collected image is adaptively detected based on image entropy to obtain the corner points in the image.

[0008] Step 2: Use the corner point matching method based on geometric feature learning to match the corner points in the image obtained in step 1 with the template corner points in the image template library to obtain the matching corner point pair information, and thereby obtain the measurement value of the drone's posture.

[0009] Step 3: Based on the drone pose measurements obtained in step 2, use the KF-GRU model to get the estimated value of the drone pose.

[0010] On the other hand, the present invention provides a robust UAV-UAV collaborative relative visual positioning system, comprising:

[0011] The corner point calculation module is used to collect the current cooperative identification image through the UAV airborne visual sensor, and perform adaptive corner point detection based on image entropy on the collected image to obtain the corner points in the image;

[0012] The corner point pair information acquisition module uses a corner point matching method based on geometric feature learning to match the corner points in the image with the template corner points in the image template library to obtain matching corner point pair information, and thereby obtain the measured value of the drone's posture;

[0013] The pose estimation module is used to obtain the estimated value of the drone's pose based on the drone's pose measurement value using the KF-GRU model.

[0014] On the other hand, the present invention provides a computer-readable storage medium, comprising instructions, which, when executed on a computer, enable the computer to execute the drone-unmanned ship collaborative relative visual positioning method as described in any one of the above.

[0015] On the other hand, the present invention provides a drone-unmanned ship collaborative relative visual positioning device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the processor implements the contents of any of the above methods when executing the computer program.

[0016] Compared with the prior art, the present invention has the following beneficial effects:

[0017] 1. Adopt the corner detection algorithm based on image entropy, and use the nonlinear mapping result of image entropy as the threshold of Shi-Tomasi corner detection to achieve the effect of adaptively adjusting the corner detection threshold.

[0018] 2. Use the corner matching method based on geometric feature learning to support asymmetric tolerant corner matching process.

[0019] 3. A KF-GRU model is proposed to update the state and covariance without the Kalman filter model. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is an overview diagram of a robust UAV-unmanned ship collaborative relative visual positioning method in an embodiment of the present application.

[0021] Figure 2 This is a comparison result diagram of the corner point detection part of the embodiment of the present application.

[0022] Figure 3 This is a comparison result diagram of the corner point matching part of the embodiment of the present application.

[0023] Figure 4 This is a comparison result diagram of the pose estimation part of the embodiment of the present application. DETAILED DESCRIPTION

[0024] The present invention will be described in detail below with reference to the accompanying drawings and preferred examples, and the purpose and effects of the present invention will become more clear. It should be understood that the specific examples described herein are only used to explain the present invention and are not intended to limit the present invention.

[0025] The present application provides a robust UAV-UAV collaborative relative visual positioning method. The present application proposes an adaptive corner point detection algorithm based on image entropy, and uses a corner point matching method based on geometric feature learning to match the detected corner points with template corner points. Finally, the KF-GRU method is used to estimate the UAV posture. A highly robust collaborative relative visual positioning method can be established. The method specifically includes:

[0026] Step 1: The current cooperative identification image is collected through the UAV airborne visual sensor, and the collected image is adaptively detected based on image entropy to obtain the corner points in the image.

[0027] Step 2: Use the corner point matching method based on geometric feature learning to match the corner points in the image obtained in step 1 with the template corner points in the image template library to obtain the matching corner point pair information, and thereby obtain the measurement value of the drone's posture.

[0028] Step 3: Based on the drone pose measurements obtained in step 2, use the KF-GRU model to get the estimated value of the drone pose.

[0029] Further, the step 1 specifically includes:

[0030] Step 1_1: Calculate the intensity difference of all pixels in the current acquired image window and use it to calculate the R score in Shi-Tomasi.

[0031] Step 1_2: Calculate the image entropy of the current captured image and use it to calculate the corner threshold in the Shi-Tomasi algorithm.

[0032] Step 1_3: Use the Shi-Tomasi algorithm to detect corners of the image, and judge and filter the corners obtained by the Shi-Tomasi algorithm based on the corner threshold obtained in step 12.

[0033] Further, the step 2 specifically includes:

[0034] Step 2_1: Calculate the edge features in the image collected by the drone's onboard visual sensor in step 1.

[0035] Step 2_2: Pass the edge features obtained in step 2_1 into the graph neural network to generate enhanced corner feature vectors, calculate the cost matrix and adjust its dimension.

[0036] Step 2_3: Use the Hungarian algorithm to pair the corner points in the image collected by the drone's airborne visual sensor obtained in step 1_3 with the corner points in the image in the template library to obtain the corresponding corner point pair information.

[0037] Step 2_4: By matching the 2D-2D coordinates of the corner points, the mapping relationship between the 2D coordinates in the image and the 3D coordinates in the real world is obtained.

[0038] Step 2_5: Use the DH algorithm and combine the mapping relationship between 2D and 3D coordinates in step 2_4 to obtain the UAV pose measurement results.

[0039] Further, step 3 is specifically as follows:

[0040] Step 3_1: Establish the UAV visual measurement system equation and collect the true value of the UAV pose in advance, and input the true value of the UAV pose and the UAV pose measurement value into the GRU network.

[0041] By training the GRU network, the update function relationship between the state and the covariance is obtained, corresponding to GRU S and GRU P Two networks.

[0042] Step 3_2: Using the update function relationship between the state and covariance obtained in step 3_1, adjust the traditional Kalman filter to obtain the KF-GRU model.

[0043] Step 3_3: Use the KF-GRU model in step 3_2 to process the obtained drone pose measurement value to obtain the final drone pose estimate.

[0044] like Figure 1 As shown, in a preferred embodiment, the present application provides a specific process of a robust UAV-UAV ship collaborative relative visual positioning method, comprising the following steps:

[0045] S1: In the collaborative mission between the UAV and the unmanned ship, the current image is obtained through the UAV's onboard visual sensor.

[0046] S2: Let I(x,y) be the grayscale intensity of the pixel in the image, let ω(x,y) be the weight of the pixel in the window, which is a Gaussian distribution with the window center (x,y) as the origin, (u,v) represents the displacement in the X and Y directions in the pixel coordinates, and the eigenvalues ​​of the matrix M are calculated by formula 2.

[0047]

[0048]

[0049] Among them, E(u,v) is defined as the key indicator for evaluating corner detection, namely, intensity difference; R is the rotation matrix, and λ1 and λ2 are the eigenvalues ​​of M.

[0050] S3: Calculate the R score in the Shi-Tomasi algorithm using the eigenvalues ​​in S2:

[0051] R=min(λ1,λ2).

[0052] S4: Process the original image obtained in step 1 and count the proportion of the grayscale intensity of each pixel in the image, that is, the probability of different grayscale intensities appearing in the image, recorded as p i .

[0053] S5: replace p in S4 i Import formula 3 to calculate the image entropy Em.

[0054]

[0055] S6: Import the image entropy in S5 into Formula 4 to adaptively adjust the corner threshold θ in the Shi-Tomasi algorithm.

[0056]

[0057] where θ min =0.04,θ max =0.6.

[0058] S7: Using the adaptive threshold in S6, detect the corner points obtained by the Shi-Tomasi algorithm and use formula 5 to screen the corner points.

[0059]

[0060] Where R i is the i-th corner point currently detected.

[0061] S8: Assume that the image acquired by the UAV visual sensor in S1 is The images in the template are The S7 Substitute the corner coordinates of the graph into formula 6 to obtain the edge features and

[0062]

[0063] Where n is the total number of corner points in the image, The same logic can be applied. (x i ,y i ) is a graph The coordinates of the i-th corner point in the image, the coordinates of the two corner points closest to the i-th corner point are (x i,n1 ,y i,n1 ) and (x i,n2 ,y i,n2 ).

[0064] S9: Bring the edge features obtained in S8 into the graph neural network to obtain the enhanced corner feature vector nf. This feature vector is denoted as G, F is recorded as F. F and G are calculated by formula 7 T The inner product of is used to obtain the cost matrix C.

[0065] C=F⊙G T (7)

[0066] S10: Set κ to the maximum number of corner points detected by the Shi-Tomasi algorithm, and let G C and F C Figure and Figure The number of corner points in .

[0067] S11: Compare G described in S10 C and F C .

[0068] If G C =F C , then it represents the graph and Figure The number of corner points in G is the same, and the cost matrix C is a square matrix; if G C ≥κ and F C ≥κ, then it represents the graph and Figure The number of corner points in G is the same, which is the maximum number of corner points κ, and the cost matrix C is a square matrix; if G C <F C <κ, then it represents the graph The number of corner points is less than that of If the number of corner points is too large, a virtual zero column needs to be added to the cost matrix C; if F C <G C <κ, then it represents the graph The number of corner points is less than that of If the number of corner points is too large, a virtual zero row needs to be added to the cost matrix C.

[0069] S12: Using the Hungarian algorithm to parse the graph and Figure Pair the corner points in to get the corresponding corner point pairs.

[0070] S13: Using the corner point pairs obtained in S12, obtain the 2D-2D coordinate matching relationship between the corner points in the captured image and the template image, and then combine the 2D-3D coordinate matching relationship of the corner points in the template image to obtain the mapping relationship between the 2D coordinates and the 3D coordinates of the corner points in the captured image.

[0071] S14: The UAV posture measurement result is obtained by using the DH algorithm and combining the mapping relationship between the 2D and 3D coordinates of the corner points in S13.

[0072] S15: Establish the UAV vision measurement system equations as shown in Formula 8 and Formula 9.

[0073] The collected pose data true value and the pose measurement result obtained by S14 are input into the GRU network to obtain the update function relationship between the state and the covariance f(·) and h(·), which correspond to GRU S and GRU P Two networks.

[0074] y t =f(y t-1 )+w t (8)

[0075] z t =y t +v t (9)

[0076] Among them, w t ~N(0,Q),v t ~N(0,R), Q and R are process noise and measurement noise respectively.

[0077] S16: Using f(·) and h(·) obtained in S15, adjust the traditional Kalman filter KF to obtain Formula 10 and Formula 11.

[0078]

[0079]

[0080] The left side of the → symbol represents the traditional Kalman filter, and the right side represents the rewritten Kalman filter.

[0081] Specifically, is the estimated value of the state at time t, is the set of state estimates at the previous n moments, A is the state transfer matrix, is the estimated value of the covariance matrix at time t, is a collection of covariance matrix estimates for the previous n moments.

[0082] S17: Use the KF-GRU model to process the drone pose measurement value obtained in S14 to obtain the final drone pose estimation value.

[0083] Based on any of the above embodiments, the present application also provides a robust UAV-unmanned ship collaborative relative visual positioning system, the system comprising:

[0084] The corner point calculation module is used to collect the current cooperative identification image through the UAV airborne visual sensor, and perform adaptive corner point detection based on image entropy on the collected image to obtain the corner points in the image;

[0085] The corner point pair information acquisition module uses a corner point matching method based on geometric feature learning to match the corner points in the image with the template corner points in the image template library to obtain matching corner point pair information, and thereby obtain the measured value of the drone's posture;

[0086] The pose estimation module is used to obtain the estimated value of the drone's pose based on the drone's pose measurement value using the KF-GRU model.

[0087] Based on any of the above embodiments, the embodiment of the present application also provides a computer-readable storage medium, including instructions, which, when executed on a computer, enable the computer to execute the described drone-unmanned ship collaborative relative visual positioning method.

[0088] Based on any of the above embodiments, an embodiment of the present application also provides a drone-unmanned ship collaborative relative visual positioning device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, and when the processor executes the computer program, the drone-unmanned ship collaborative relative visual positioning method is implemented.

[0089] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0090] The following is based on the attached Figures 2 to 4 The comparison results between the embodiment of the present application and other algorithms are further described.

[0091] Figure 2 The result of corner detection on a real landing sign image taken by a drone.

[0092] Specifically, the "algorithm in this paper" in the figure is the detection image of the corner detection part in the embodiment of the present application, "Harris corner detection" is the detection image for detecting corners using the Harris algorithm, "Fast corner detection" is the detection image obtained using the Fast algorithm, "FOAGDD corner detection" is the corner detection image obtained using the FOAGDD algorithm, and "SOGGDD corner detection" is the corner detection image generated using the SOGGDD algorithm.

[0093] Through the Figure 2 By comparing the results in , it can be seen that the detection effect of the corner point detection part in the embodiment of the present application is better.

[0094] Figure 3 is the corner point matching result.

[0095] Specifically, the "algorithm in this paper" in the figure is the matching result of the corner point matching algorithm in the embodiment of the present application, "Brisk corner point matching" is the corner point matching result obtained using Brisk, "ORG corner point matching" is the matching result obtained using the ORG algorithm, and "SIFT corner point matching" is the corner point matching result diagram obtained by the SIFT algorithm.

[0096] Through the Figure 3 By comparing the results in , it can be seen that the corner point matching method in the embodiment of the present application is more accurate.

[0097] Figure 4 is the pose estimation error result.

[0098] The true value of the drone's posture is obtained by fusing the IMU module and RTK module in the flight control during the collaborative mission. Figures (a) to (f) are the error curves of the X (Cm), Y (cm), - (cm), roll (°), pitch (°) and yaw (°) parameters in the posture data.

[0099] Specifically, the "algorithm in this paper" is the error of the drone pose result obtained in the embodiment of this application. "LSTM-KF" is the error of processing the measured pose result using the LSTM-KF algorithm, and "traditional extended Kalman" is the error of processing the measured pose result using the traditional Kalman filter.

[0100] Through the Figure 4 By comparing the results in , it can be seen that the pose estimation method in the embodiment of the present application obtains a pose estimation result with smaller error.

[0101] Those skilled in the art can understand that the above are only preferred examples of the invention and are not intended to limit the invention. Although the invention is described in detail with reference to the above examples, those skilled in the art can still modify the technical solutions recorded in the above examples or replace some of the technical features therein with equivalents. Any modification, equivalent replacement, etc. made within the spirit and principle of the invention shall be included in the protection scope of the invention.

Claims

1. A robust UAV-UAV collaborative relative visual positioning method, characterized in that The method comprises the following steps: Step 1: The current cooperative identification image is collected by the UAV airborne visual sensor, and the collected image is subjected to adaptive corner point detection based on image entropy to obtain the corner points in the image; Step 2: Use the corner point matching method based on geometric feature learning to match the corner points in the image obtained in step 1 with the template corner points in the image template library to obtain the matching corner point pair information, and use this to obtain the measurement value of the drone's posture; Step 3: Based on the drone pose measurement value obtained in step 2, the KF-GRU model is used to obtain the estimated value of the drone pose, specifically: Step 3_1: Establish the UAV visual measurement system equation and collect the true value of the UAV posture in advance, and input the true value of the UAV posture and the UAV posture measurement value into the GRU network; By training the GRU network, the update function relationship between the state and the covariance is obtained; Step 3_2: Using the update function relationship between the state and covariance obtained in step 3_1, adjust the traditional Kalman filter to obtain the KF-GRU model; Step 3_3: Use the KF-GRU model in step 3_2 to process the obtained drone pose measurement value to obtain the final drone pose estimate.

2. A robust UAV-UAV collaborative relative visual positioning method according to claim 1, characterized in that: Step 1 is as follows: Step 1_1: Calculate the intensity difference of all pixels in the current acquired image window and use it to calculate the R score in the Shi-Tomasi algorithm; Step 1_2: Calculate the image entropy of the current captured image and use it to calculate the corner threshold in the Shi-Tomasi algorithm; Step 1_3: Use the Shi-Tomasi algorithm to detect corners of the image, and judge and filter the corners obtained by the Shi-Tomasi algorithm based on the corner threshold obtained in step 1_2.

3. A robust UAV-UAV collaborative relative visual positioning method according to claim 2, characterized in that: The corner point threshold θ is calculated using the following formula: Among them, θ min is the minimum threshold of the set corner point, θ max is the maximum threshold of the set corner points, and Em is the image entropy.

4. A robust UAV-UAV collaborative relative visual positioning method according to claim 2 or 3, characterized in that: Step 2 is as follows: Step 2_1: Calculate the edge features in the image collected by the UAV airborne visual sensor in step 1; Step 2_2: Pass the edge features obtained in step 2_1 into the graph neural network to generate enhanced corner feature vectors, calculate the cost matrix and adjust its dimension; Step 2_3: Use the Hungarian algorithm to pair the corner points in the image collected by the drone's airborne visual sensor obtained in step 1_3 with the corner points in the image in the template library to obtain corresponding corner point pair information; Step 2_4: Obtain the mapping relationship between the 2D coordinates in the image and the 3D coordinates in the real world through the matching relationship of the 2D-2D coordinates of the corner points; Step 2_5: Use the DH algorithm and combine the mapping relationship between 2D and 3D coordinates in step 2_4 to obtain the drone pose measurement results.

5. A robust UAV-UAV collaborative relative visual positioning method according to claim 4, characterized in that: The dimensions of the cost matrix are specifically: If the number of corner points of the image acquired by the UAV visual sensor is less than the number of corner points of the image in the template library, and the number of corner points of the image in the template library is less than the maximum number of corner points detected by the Shi-Tomasi algorithm, a virtual zero column is added to the cost matrix; If the number of corner points of the image in the template library is less than the number of corner points of the image acquired by the UAV vision sensor, and the number of corner points of the image in the template library is less than the maximum number of corner points detected by the Shi-Tomasi algorithm, a virtual zero row is added to the cost matrix.

6. The method for robust UAV-UAV collaborative relative visual positioning according to claim 1, characterized in that: The KF-GRU model is expressed as: in is the estimated value of the state at time t, is the set of state estimates at the previous n moments, f(·) is the state update function, is the estimated value of the covariance matrix at time t, is the set of covariance matrix estimates for the previous n moments, and h(·) is the update function of the covariance.

7. A robust UAV-UAV collaborative relative visual positioning system, characterized in that include: The corner point calculation module is used to collect the current cooperative identification image through the UAV airborne visual sensor, and perform adaptive corner point detection based on image entropy on the collected image to obtain the corner points in the image; The corner point pair information acquisition module uses a corner point matching method based on geometric feature learning to match the corner points in the image with the template corner points in the image template library to obtain matching corner point pair information, and thereby obtain the measured value of the drone's posture; The pose estimation module is used to obtain the estimated value of the drone's pose based on the drone's pose measurement value using the KF-GRU model; specifically: Establish the UAV visual measurement system equation and collect the true value of the UAV posture in advance, and input the true value of the UAV posture and the UAV posture measurement value into the GRU network; By training the GRU network, the update function relationship between the state and the covariance is obtained; The traditional Kalman filter is adjusted using the updated function relationship between the state and covariance obtained through training to obtain the KF-GRU model. The KF-GRU model is used to process the obtained UAV pose measurements to obtain the final UAV pose estimation value.

8. A computer-readable storage medium, comprising instructions, which, when executed on a computer, enables the computer to execute the drone-unmanned ship collaborative relative visual positioning method as described in any one of claims 1-6.

9. A UAV-UAV ship collaborative relative visual positioning device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.

Images