Unmanned aerial vehicle visual positioning method fusing deep learning video target positioning uncertainty
By extracting the uncertainty of the target image coordinates of the deep learning model and calculating the target object square coordinates with the drone video data, the accuracy reduction problem caused by the uncertainty of the target object square position extraction in the prior art is solved, and a high-precision target object square coordinate solution is achieved.
Patent Information
- Application Number
- CN202411857322.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-17
- Publication Date
- 2025-04-11
AI Technical Summary
In the prior art, due to factors such as the target morphology, observation platform and deep learning algorithm, there is uncertainty and error in the extraction of the image position of the target object in the video image, which in turn reduces the accuracy of the acquired ground coordinates of the target object.
By extracting the uncertainty of the target image square based on deep learning, and combining the image square coordinates of the drone video target, the object square coordinates of the target are calculated, and the rationality of the image square-object square mapping results are used to correlate the position on the drone video target map with the object square coordinates to achieve high-precision object square coordinates of target objects of different scales.
It has achieved the acquisition of high-precision target object square coordinates, which is widely applicable and is less affected by the observation scale and target object size.
Smart Images
Figure CN119810194A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a visual positioning method, in particular to a UAV visual positioning method integrating deep learning video target positioning uncertainty. Background Art
[0002] In recent years, deep learning has made breakthrough progress in the field of target detection. In particular, the emergence of algorithms such as YOLO, SSD, and FasterR-CNN has greatly promoted the development of target detection technology. These algorithms not only improve the speed and accuracy of detection, but also increase the ability to handle complex scenes. By acquiring video streams through cameras or other sensors, and based on the acquired video images, using deep learning and other methods, using target detection algorithms and deep neural networks to automatically extract and classify image features, it is possible to identify target objects in the video, extract the coordinates of the target objects on the image, and determine the image position coordinates of the target objects. It has a wide range of applications in intelligent monitoring, autonomous driving and other fields.
[0003] The object coordinates can be further calculated from the image coordinate information. The conversion between the image coordinate system and the object coordinate system is widely used in photogrammetry, such as map making, land planning, resource exploration and other fields. Through the precise image-object coordinate conversion technology, the feature points on the film can be matched with the actual ground points, thereby achieving accurate measurement and description of ground targets.
[0004] However, in the prior art, due to the influence of factors such as target morphology, observation platform, and deep learning algorithm, there is still uncertainty and error in the extraction of the image-side position of the target object in the video image, which leads to a decrease in the accuracy of the obtained ground coordinates of the target object. Summary of the invention
[0005] The purpose of the present invention is to provide a UAV visual positioning method integrating deep learning video target positioning uncertainty. The present invention has the characteristics of obtaining high-precision target object coordinates.
[0006] The technical solution of the present invention is a UAV visual positioning method integrating deep learning video target positioning uncertainty, comprising the following steps:
[0007] S1, obtain drone video data;
[0008] S2. Perform deep learning target detection on drone video data, extract the uncertainty of the network detection model's prediction of the target image coordinates and the coordinates of the four corners of the rectangular box of the target image position;
[0009] S3, according to the four corner coordinates of the rectangular frame of the target image position, solve the four corner coordinates of the target object;
[0010] S4. Compare the shape of the quadrilateral formed by the four corner coordinates of the target object with the rectangular frame of the target image. Combined with the uncertainty of model target extraction, determine whether the affine transformation relationship is satisfied. If so, the positioning coordinates of the target object are obtained. If not, corrections are made until the affine transformation relationship is satisfied.
[0011] In the aforementioned UAV visual positioning method integrating deep learning video target positioning uncertainty, in step S3, the calculation method of the collinearity equation inverse algorithm is:
[0012]
[0013] Where, X A , Y A , Z A is the coordinate of the target object; X S , Y S , Z S is the object coordinate of the photography center; a i 、b i 、c i (i=1, 2, 3) are the 9 direction cosine angle parameters composed of the 3 exterior azimuth elements of the image, which are known values; x, y, f are the target image coordinates.
[0014] In the aforementioned UAV visual positioning method integrating deep learning video target positioning uncertainty, in step S3, the calculation method of the forward intersection method includes the following steps:
[0015] 1. Get the rotation matrix R of the left and right images 1 ,R 2 ;
[0016] 2. Obtain the photographic baseline component B composed of the left and right images u ,B v ,B w ;
[0017]
[0018] Where, X S1 , X S2 , Y S1 , Y S2 , Z S1 , Z S2 are the coordinates of the centers of the exterior orientation elements of the left and right images on the ground, respectively;
[0019] 3. Calculate the image space auxiliary coordinates (u 1 ,v 1 ,w 1 ),(u 2 ,v 2 ,w2 ):
[0020] In the formula (x 1 ,y 1 ,-f)、(x 2 ,y 2 ,-f) represent the coordinates of each image point respectively;
[0021] 4. Calculate the projection coefficient N of each image point by point 1 、N 2 :
[0022]
[0023] 5. Calculate the X, Y, and Z coordinates of the target object point by point:
[0024] X=X S1 +N 1 u 1 =X S2 +N 2 u 2 ;
[0025] Y=Y S1 +N 1 v 1 =Y S2 +N 2 v 2 ;
[0026] Z=Z S1 +N 1 w 1 =Z S2 +N 2 w 2 .
[0027] In the aforementioned UAV visual positioning method integrating deep learning video target positioning uncertainty, in step S2, the uncertainty extraction method is:
[0028] S2.1. A priori set the prediction error to be Gaussian distributed, and obtain the following distribution of the measurement results: In the formula, σ * is the standard deviation of the prediction error distribution; f(x*, θ) represents the network detection model, x* represents the input data, θ represents the model parameters, and y* represents the target image coordinates;
[0029] S2.2. Use the maximum a posteriori estimate as the objective function L for training the network detection model:
[0030]
[0031] The network detection model is trained using the objective function L, and the σ* of each set of measurement results is predicted as the uncertainty;
[0032] S2.3. The average of the uncertainty σ* of the coordinates of the four corner points of the rectangular box of the target image position is used as the uncertainty of the network detection model in predicting the target image coordinates.
[0033] In the aforementioned UAV visual positioning method integrating deep learning video target positioning uncertainty, step S4 is specifically as follows:
[0034] Combined with the uncertainty of the network detection model, the quadrilateral composed of the four corners of the target object square coordinates is compared with the parallelogram formed by the affine transformation of the target image square rectangular frame, and the sum of the absolute values of the length differences of the two sets of quadrilaterals is set to s;
[0035] If s is greater than 1% of the perimeter of the parallelogram, it means that the deformation is too large. In this case, the target image rectangle is moved so that the four corner points of the target image rectangle traverse the eight surrounding image points.
[0036] Recalculate the four-corner coordinates and uncertainty of the target object space according to the traversed target image space coordinates, compare the quadrilateral formed by the four-corner coordinates of the target object space with the parallelogram formed after the affine transformation of the target image space rectangular frame, and let the sum of the absolute values of the length differences of the two sets of quadrilaterals be s;
[0037] If s is less than 1% of the perimeter and the uncertainty is less than 0.1, stop traversing and calculate the four corner coordinates of the target object space corresponding to the four corner coordinates of the target image rectangular frame; calculate the center coordinates of the target object space based on the four corner coordinates of the object space, thereby obtaining the positioning coordinates of the target object space;
[0038] If the conditions for obtaining the positioning coordinates of the target object still cannot be met after traversing 8 image points, move the target image rectangular frame so that the 4 corner points of the target image rectangular frame traverse the surrounding 16 image points, and repeat steps (3) and (4);
[0039] If the conditions cannot be met to obtain the positioning coordinates of the target object after traversing 16 image points, the four corner coordinates of the target object corresponding to the minimum value of the sum of s and uncertainty are selected, and the center coordinates of the four corners of the target object are calculated to obtain the positioning coordinates of the target object.
[0040] Compared with the prior art, the present invention has the following beneficial effects:
[0041] The present invention extracts the uncertainty of the target image space based on deep learning, calculates the object space coordinates of the target according to the image space coordinates of the drone video target, and associates the position on the drone video target map with the object space coordinates in combination with the rationality of the image space-object space mapping result, thereby achieving high-precision object space coordinate solution for targets of different scales, which is less affected by the observation scale and the size of the target and has a wide applicability. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION
[0043] The present invention will be further described below in conjunction with the embodiments, but they are not intended to limit the present invention.
[0044] Example:
[0045] like Figure 1 As shown, a UAV visual positioning method integrating deep learning video target positioning uncertainty includes the following steps:
[0046] S1, obtain drone video data;
[0047] S2. Perform deep learning target detection on drone video data, extract the uncertainty of the network detection model's prediction of the target image coordinates and the coordinates of the four corners of the rectangular box of the target image position;
[0048] The uncertainty extraction method is: first, model the error distribution of the prediction results a priori, and then use the maximum likelihood estimation method to design the objective function in the training phase. For the pixel coordinate prediction of the target positioning task, if the prediction error is set to be Gaussian distribution a priori, the distribution of the measurement results is obtained as follows:
[0049]
[0050] In the formula, σ * is the standard deviation of the forecast error distribution;
[0051] f(x*, θ) represents the network detection model, x* represents the input data, θ represents the model parameters, and y* represents the target image coordinates.
[0052] The maximum a posteriori estimation is used as the objective function for training the network detection model, which is expressed as:
[0053]
[0054] The network detection model is trained using the objective function L, and the σ* of each set of measurement results is predicted as the uncertainty to indicate the credibility of the prediction results.
[0055] The mean value of the uncertainty σ* of the coordinates of the four corner points of the rectangular box of the target image position is taken as the uncertainty of the network detection model in predicting the target image coordinates.
[0056] S3, according to the four corner coordinates of the rectangular frame of the target image position, solve the four corner coordinates of the target object;
[0057] According to the coordinates of the four corners of the rectangular frame of the target image position, the inverse algorithm of the collinear equation is used to obtain the coordinates of the four corners of the target object space.
[0058] The inverse formula for the collinear equation is:
[0059]
[0060] In the above formula, X A , Y A , Z A is the coordinate of the target object; X S , Y S , Z S is the object coordinate of the photography center; a i 、b i 、c i (i=1, 2, 3) are the 9 direction cosine angle parameters composed of the 3 exterior azimuth elements of the image, which are known values; x, y, f are the target image coordinates.
[0061] Alternatively, the four corner coordinates of the rectangular frame of the target image position can be used to solve the four corner coordinates of the target object position using the forward intersection method.
[0062] The calculation steps are:
[0063] 1. Get the rotation matrix R of the left and right images 1 ,R 2 ;
[0064] 2. Obtain the photographic baseline component B composed of the left and right images u ,B v ,B w ;
[0065]
[0066] In the formula, X S1 , X S2 , Y S1 , Y S2 , Z S1 , Z S2 They are the coordinates of the centers of the exterior orientation elements of the left and right images on the ground, respectively.
[0067] 3. Calculate the image space auxiliary coordinates (u 1 ,v 1 ,w1 ),(u 2 ,v 2 ,w 2 ):
[0068] In the formula (x 1 ,y 1 ,-f)、(x 2 ,y 2 ,-f) represent the coordinates of each image point respectively.
[0069] 4. Calculate the projection coefficient N of each image point by point 1 、N 2 :
[0070]
[0071] 5. Calculate the X, Y, and Z coordinates of the target object point by point:
[0072] X=X S1 +N 1 u 1 =X S2 +N 2 u 2 ;
[0073] Y=Y S1 +N 1 v 1 =Y S2 +N 2 v 2 ;
[0074] Z=Z S1 +N 1 w 1 =Z S2 +N 2 w 2 .
[0075] S4. Compare the shape of the quadrilateral formed by the four corner coordinates of the target object with the rectangular frame of the target image. Combined with the uncertainty of the network model, determine whether the affine transformation relationship is satisfied. If so, the positioning coordinates of the target object are obtained. If not, corrections are made until the affine transformation relationship is satisfied.
[0076] The affine transformation relationship is: the target image rectangular frame is affine transformed into a parallelogram, and the quadrilateral composed of the four corner coordinates of the target object is compared with the parallelogram. If the difference is too large, it means that the affine transformation relationship is not satisfied and needs to be corrected.
[0077] The judgment process is as follows:
[0078] (1) Combined with the uncertainty of the network detection model, the quadrilateral composed of the four corners of the target object square coordinates is compared with the parallelogram formed by the affine transformation of the target image square rectangular frame, and the sum of the absolute values of the length differences of the two sets of quadrilaterals is set to s;
[0079] (2) If s is greater than 1% of the perimeter of the parallelogram, it means that the deformation is too large, indicating that there is an obvious system error. In this case, move the target image rectangle so that the four corner points of the target image rectangle traverse the eight surrounding image points.
[0080] (3) Recalculate the four-corner coordinates and uncertainty of the target object space according to the traversed target image space coordinates, compare the quadrilateral formed by the four-corner coordinates of the target object space with the parallelogram formed after the affine transformation of the target image space rectangular frame, and let the sum of the absolute values of the length differences of the two sets of quadrilaterals be s;
[0081] (4) If s is less than 1% of the perimeter and the uncertainty is less than 0.1, stop traversing and calculate the four corner coordinates of the target object space corresponding to the four corner coordinates of the target image rectangular frame; calculate the center coordinates of the target object space based on the four corner coordinates of the object space, thereby obtaining the positioning coordinates of the target object space;
[0082] (5) If the conditions for obtaining the positioning coordinates of the target object still cannot be met after traversing 8 image points, move the target image rectangular frame so that the 4 corner points of the target image rectangular frame traverse the surrounding 16 image points, and repeat steps (3) and (4);
[0083] (6) If the conditions for obtaining the positioning coordinates of the target object still cannot be met after traversing 16 image points, the four corner coordinates of the target object corresponding to the minimum value of the sum of s and the uncertainty are selected, and the center coordinates of the four corners of the target object are calculated to obtain the positioning coordinates of the target object.
Claims
1. A UAV visual positioning method integrating deep learning video target positioning uncertainty, characterized by: The following steps are involved: S1, obtain drone video data; S2. Perform deep learning target detection on drone video data, extract the uncertainty of the network detection model's prediction of the target image coordinates and the coordinates of the four corners of the rectangular box of the target image position; S3, according to the four corner coordinates of the rectangular frame of the target image position, solve the four corner coordinates of the target object; S4. Compare the shape of the quadrilateral formed by the four corner coordinates of the target object with the rectangular frame of the target image. Combined with the uncertainty of model target extraction, determine whether the affine transformation relationship is satisfied. If so, the positioning coordinates of the target object are obtained. If not, corrections are made until the affine transformation relationship is satisfied.
2. The method for unmanned aerial vehicle visual positioning integrating deep learning video target positioning uncertainty according to claim 1, characterized in that: In step S3, the four corner coordinates of the target object are calculated by using the collinear equation inverse algorithm or by using the forward intersection method.
3. The method for unmanned aerial vehicle visual positioning integrating deep learning video target positioning uncertainty according to claim 2 is characterized in that: In step S3, the collinear equation is inverted The calculation method of the algorithm is: Where, X A , Y A , Z A is the coordinate of the target object; X S , Y S , Z S is the object coordinate of the photography center; a i 、b i 、c i (i=1, 2, 3) are the 9 direction cosine angle parameters composed of the 3 exterior azimuth elements of the image, which are known values; x, y, f are the target image coordinates.
4. The method for unmanned aerial vehicle visual positioning integrating deep learning video target positioning uncertainty according to claim 2, characterized in that: In step S3, the calculation method of the forward intersection method includes the following steps:
1. Get the rotation matrix R1, R2 of the left and right images; 2. Obtain the photographic baseline component B composed of the left and right images u ,B v ,B w ; Where, X S1 , X S2 , Y S1 , Y S2 , Z S1 , Z S2 are the coordinates of the centers of the exterior orientation elements of the left and right images on the ground, respectively; 3. Calculate the image space auxiliary coordinates (u1, v1, w1), (u2, v2, w2) of each image point point by point: In the formula, (x1, y1, -f) and (x2, y2, -f) represent the coordinates of each image point respectively; 4. Calculate the projection coefficients N1 and N2 of each image point by point:
5. Calculate the X, Y, and Z coordinates of the target object point by point: X=X S1 +N1u1=X S2 +N2u2; Y=Y S1 +N1v1=Y S2 +N2v2; Z=Z S1 +N1w1=Z S2 +N2w2。 5. The method for unmanned aerial vehicle visual positioning integrating deep learning video target positioning uncertainty according to claim 1, characterized in that: In step S2, the uncertainty extraction method is: S2.
1. A priori set the prediction error to be Gaussian distributed, and obtain the following distribution of the measurement results: In the formula, σ * is the standard deviation of the prediction error distribution; f(x*, θ) represents the network detection model, x* represents the input data, θ represents the model parameters, and y* represents the target image coordinates; S2.
2. Use the maximum a posteriori estimate as the objective function L for training the network detection model: The network detection model is trained using the objective function L, and the σ* of each set of predicted measurement results is used as the uncertainty; S2.
3. The average of the uncertainty σ* of the coordinates of the four corner points of the rectangular box of the target image position is used as the uncertainty of the network detection model in predicting the target image coordinates.
6. The method for unmanned aerial vehicle visual positioning integrating deep learning video target positioning uncertainty according to claim 1, characterized in that: Step S4 is specifically as follows: (1) Combined with the uncertainty of the network detection model, the quadrilateral composed of the four corners of the target object square coordinates is compared with the parallelogram formed by the affine transformation of the target image square rectangular frame, and the sum of the absolute values of the length differences of the two sets of quadrilaterals is set to s; (2) If s is greater than 1% of the perimeter of the parallelogram, it means that the deformation is too large. In this case, the target image rectangle is moved so that the four corner points of the target image rectangle traverse the eight surrounding image points. (3) Recalculate the four-corner coordinates and uncertainty of the target object space according to the traversed target image space coordinates, compare the quadrilateral formed by the four-corner coordinates of the target object space with the parallelogram formed after the affine transformation of the target image space rectangular frame, and let the sum of the absolute values of the length differences of the two sets of quadrilaterals be s; (4) If s is less than 1% of the perimeter and the uncertainty is less than 0.1, stop traversing and calculate the four corner coordinates of the target object space corresponding to the four corner coordinates of the target image rectangular frame; calculate the center coordinates of the target object space based on the four corner coordinates of the object space, thereby obtaining the positioning coordinates of the target object space; (5) If the conditions for obtaining the positioning coordinates of the target object still cannot be met after traversing 8 image points, move the target image rectangular frame so that the 4 corner points of the target image rectangular frame traverse the surrounding 16 image points, and repeat steps (3) and (4); (6) If the conditions for obtaining the positioning coordinates of the target object still cannot be met after traversing 16 image points, the four corner coordinates of the target object corresponding to the minimum value of the sum of s and the uncertainty are selected, and the center coordinates of the four corners of the target object are calculated to obtain the positioning coordinates of the target object.