Visual positioning method and system for unmanned aerial vehicle satellite image matching based on factor graph optimization

Abstract

The invention discloses a factor graph optimization-based visual positioning method and system for unmanned aerial vehicle satellite image matching. The method specifically comprises the following steps of: firstly, constructing an unmanned aerial vehicle satellite image matching-based visual positioning system AS-VPS; then based on an FGO algorithm, solving an optimal state by using a least square estimation algorithm to obtain state increment estimation; constructing an FGO-AS-VPS model, defining a front-end visual mileage factor, a rear-end measurement factor and a cost function thereof, and establishing optimal estimation of a global cost function; and finally, carrying out iterative solution on the FGO-AS-VPS model after the factor is introduced, and obtaining the optimal estimation. According to the method, the problem that the accuracy of rear-end image matching cannot meet the requirement of a front-end visual odometer is solved, effective integration of front-end information and rear-end information is ensured, and the navigation and positioning precision of the unmanned aerial vehicle is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of UAV vision positioning, in particular to a vision positioning method and system for UAV satellite image matching based on factor graph optimization. Background Art

[0002] With the rapid development of UAV technology, reliable positioning and navigation information is crucial for the intelligence of UAVs. Currently, UAVs mainly perform positioning and navigation through the Global Navigation Satellite System (GNSS). Since GNSS signals are easily blocked, resulting in poor or failed positioning, a vision positioning system (AS-VPS) based on remote sensing technology with sub-meter resolution through UAV satellite image matching has become an alternative solution in GNSS-denied environments.

[0003] In the vision positioning system (AS-VPS) for UAV satellite image matching, the front-end module uses frame-by-frame UAV images to establish a visual odometer for relative positioning, and the back-end module uses the matching of UAV aerial images and satellite images to achieve absolute positioning, and feeds the absolute positioning information back to the front-end visual odometer. However, due to factors such as seasonal changes in satellite images, the image matching in the back-end may fail. Even if the image matching is successful, the accuracy of the image matching cannot meet the requirements of the front-end visual odometer, and there is also a problem of discontinuous absolute positioning information in the back-end. Therefore, there is an urgent need to develop a flexible and robust technical solution to effectively integrate the front and back ends of the vision positioning system (AS-VPS) for UAV satellite image matching, so as to improve the positioning performance of the vision positioning system (AS-VPS) for UAV satellite image matching. Summary of the Invention

[0004] The purpose of the present invention is to provide a vision positioning method and system for UAV satellite image matching that can solve the position error problem in the back-end, ensure the effective integration of front and back-end information, and has high navigation and positioning accuracy.

[0005] The technical solution for achieving the purpose of the present invention is: a vision positioning method for UAV satellite image matching based on factor graph optimization, including the following steps:

[0006] Step 1, establish a front-end module and a back-end module. The front-end module matches frame-by-frame aerial images of the UAV to determine odometer information for relative positioning. The back-end module matches UAV aerial images and satellite images to achieve absolute positioning, and feeds the absolute positioning information back to the visual odometer of the front-end module to construct a vision positioning system AS-VPS based on UAV satellite image matching;

[0007] Step 2: Based on the FGO algorithm, construct the position as the state variable, construct the global cost function, obtain the optimal state estimate by searching for the maximum value of the global probability function, and use the least squares estimation algorithm to solve the optimal state to obtain the state increment estimate;

[0008] Step 3: Construct the FGO-AS-VPS model, introduce the front-end visual odometry factor, the back-end measurement factor and the cost function, integrate the front and back ends, and establish the optimal estimate of the global cost function;

[0009] Step 4: For the FGO-AS-VPS model after introducing the factors, set the initial values, update the Jacobian matrix, the residual vector, and the state increment, and perform iterative solution to obtain the optimal estimate.

[0010] A visual positioning system for UAV satellite image matching based on factor graph optimization, which is used to implement the visual positioning method for UAV satellite image matching based on factor graph optimization, includes a first unit to a fourth unit, and the functions of each unit are as follows:

[0011] The first unit: Establish a front-end module and a back-end module. The front-end module matches the frame-by-frame aerial images of the UAV to determine the odometry information for relative positioning. The back-end module matches the aerial images of the UAV with the satellite images to achieve absolute positioning, and feeds the absolute positioning information back to the visual odometry of the front-end module to construct a visual positioning system AS-VPS based on the matching of UAV satellite images;

[0012] The second unit: Based on the FGO algorithm, construct the position as the state variable, construct the global cost function, obtain the optimal state estimate by searching for the maximum value of the global probability function, and use the least squares estimation algorithm to solve the optimal state to obtain the state increment estimate;

[0013] The third unit: Construct the FGO-AS-VPS model, introduce the front-end visual odometry factor, the back-end measurement factor and its cost function, integrate the front and back ends, and establish the optimal estimate of the global cost function;

[0014] The fourth unit: For the FGO-AS-VPS model after introducing the factors, set the initial values, update the Jacobian matrix, the residual vector, and the state increment, and perform iterative solution to obtain the optimal estimate.

[0015] A mobile terminal includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the visual positioning method for UAV satellite image matching based on factor graph optimization.

[0016] A computer-readable storage medium stores a computer program thereon. When the program is executed by a processor, the steps in the visual positioning method for UAV satellite image matching optimized based on factor graph are implemented.

[0017] Compared with the prior art, the present invention has the following significant advantages:

[0018] (1) By using a factor graph to optimize the visual positioning of UAV satellite image matching, the problem that the inaccurate position in the backend cannot meet the requirements of the front-end visual odometer is solved, ensuring the effective integration of front-end and backend information.

[0019] (2) In the visual positioning system AS-VPS for UAV satellite image matching, considering the factor of discontinuous backend information, an FGO-AS-VPS model is constructed, improving the UAV navigation positioning accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a flowchart of the visual positioning method for UAV satellite image matching optimized based on factor graph according to the present invention.

[0021] Figure 2 It is a schematic structural diagram of the visual positioning system AS-VPS for UAV satellite image matching according to the present invention.

[0022] Figure 3 It is a schematic structural diagram of the FGO-AS-VPS model according to the present invention.

[0023] Figure 4 It is a schematic flowchart of the FGO iteration process according to the present invention.

[0024] Figure 5 It is a horizontal error comparison diagram of the front-end independent, backend independent, and FGO-AS-VPS schemes in the embodiments of the present invention, where (a) is the horizontal error comparison diagram of trajectory 1, (b) is the horizontal error comparison diagram of trajectory 2, and (c) is the horizontal error comparison diagram of trajectory 3. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0025] It is easy to understand that according to the technical solution of the present invention, without changing the essence of the present invention, those of ordinary skill in the art can envision various embodiments of the present invention. Therefore, the following detailed embodiments and drawings are only exemplary descriptions of the technical solution of the present invention, and should not be regarded as the whole of the present invention or as a limitation or restriction on the technical solution of the present invention.

[0026] Now, various exemplary embodiments of the present invention will be described in detail with reference to the drawings. It should be noted that: unless otherwise specifically stated, the relative arrangements of components and steps, numerical expressions, and numerical values set forth in these embodiments do not limit the scope of the present invention.

[0027] The following description of at least one exemplary embodiment is merely illustrative and is in no way a limitation on the present invention or its application or use.

[0028] Technologies, methods, and devices known to those of ordinary skill in the relevant art may not be discussed in detail, but where appropriate, such technologies, methods, and devices should be regarded as part of the specification.

[0029] In all the examples shown and discussed herein, any specific values should be construed as merely exemplary and not as a limitation. Thus, other examples of the exemplary embodiments may have different values.

[0030] As Figure 1 shown, a vision positioning method for UAV satellite image matching based on factor graph optimization according to the present invention includes the following steps:

[0031] Step 1: Establish a front-end module and a back-end module. The front-end module matches the frame-by-frame aerial images of the UAV to determine the odometer information for relative positioning, and the back-end module matches the aerial images of the UAV with satellite images to achieve absolute positioning, and feeds the absolute positioning information back to the visual odometer of the front-end module to construct a vision positioning system AS-VPS based on the matching of UAV satellite images;

[0032] Step 2: Based on the FGO algorithm, construct the position as the state variable, and construct the global cost function. Obtain the optimal state estimate by searching for the maximum value of the global probability function, and use the least squares estimation algorithm to solve the optimal state to obtain the state increment estimate;

[0033] Step 3: Construct the FGO-AS-VPS model, introduce the front-end visual odometry factor, the back-end measurement factor, and the cost function, integrate the front and back ends, and establish the optimal estimate of the global cost function;

[0034] Step 4: For the FGO-AS-VPS model after introducing the factors, set the initial values, update the Jacobian matrix, the residual vector, and the state increment, and perform iterative solution to obtain the optimal estimate.

[0035] As a specific example, in Step 1, a front-end module and a back-end module are established. The front-end module matches the frame-by-frame aerial images of the UAV to determine the odometer information for relative positioning, and the back-end module matches the aerial images of the UAV with satellite images to achieve absolute positioning, and feeds the absolute positioning information back to the visual odometer of the front-end module to construct a vision positioning system AS-VPS based on the matching of UAV satellite images, as Figure 2 shown, specifically as follows:

[0036] Step 1.1: Establish a front-end module. Build a visual odometer by matching consecutive downward-looking drone images. Based on the feature point set matched by LightGlue, determine the pixel coordinate transformation relationship between two frames, and transfer the pixel coordinates to the UTM coordinates to obtain the corresponding point coordinates of the pixel points in the UTM coordinate system:

[0037]

[0038] f W = β·(R Wu ·f u +t Wu )

[0039] In the formula, f k is the coordinate of the previous frame in the pixel coordinate system; f k+1 is the coordinate of the current frame in the pixel coordinate system, R Wu is the rotation matrix; t Wu is the displacement matrix; f u is the pixel coordinate; f W is the UTM coordinate system coordinate; β is the scaling factor; (dx dy dθ) k,k+1 represents the coordinate deviation from the previous frame (the kth frame) to the current frame (the k + 1th frame);

[0040] Step 1.2: Establish a back-end module. Use the preprocessed satellite tile map and the drone image for feature matching, calculate the perspective transformation matrix, and map the pixel coordinates of the drone image to the geographic coordinate system:

[0041]

[0042] In the formula, are the geographical locations of the upper left and lower right corners of the satellite image; C x , C y are the pixel positions of the feature points of the drone image; L a , L o are the geographical coordinates of the feature points of the drone image; W and H are the width and height of the satellite image;

[0043] Step 1.3: Feed the absolute positioning information back to the front-end visual odometer to construct a visual positioning system AS-VPS based on the matching of drone and satellite images.

[0044] As a specific example, in Step 2, based on the FGO algorithm, construct the position as the state variable, and construct the global cost function. Obtain the optimal state estimate by searching for the maximum value of the global probability function, and use the least squares estimation algorithm to solve the optimal state to obtain the state increment estimate, specifically as follows:

[0045] Step 2.1. Construct the position as a state variable and construct the global cost function Solve the state estimation by searching for the maximum value of the global probability function to obtain the optimal state as follows:

[0046]

[0047] where L is the position state, p(·) is the joint probability density, L = (L1…,L N ) is the set of AS-VPS position states, and G(L1, L2,…,L N ) is the global probability density function; j = 1, 2,…, m, m ≤ N;

[0048] Step 2.2. According to the requirement of the minimum residual value for the optimal state, adopt the least squares estimation algorithm to transform the solution of the optimal state into:

[0049]

[0050] where E is the residual vector; is characterized as

[0051] Step 2.3. Obtain the state increment estimation ΔL i as:

[0052]

[0053] where is the Jacobian matrix, is the residual.

[0054] As a specific example, in Step 3, construct the FGO-AS-VPS model, introduce the front-end visual odometry factor, the back-end measurement factor and their cost functions, integrate the front and back ends, and establish the optimal estimation of the global cost function, as Figure 3 shown, specifically as follows:

[0055] Step 3.1. In the front-end module, according to the relationship between the state estimated by the UAV image matching and the state increment, set the noise to follow a Gaussian distribution with zero mean, and define the front-end visual odometry factor as follows:

[0056]

[0057] where p k , p k+1 are the states at times k and k + 1; N(·) is the Gaussian distribution function; is the covariance matrix; is the state increment at times k and k + 1;

[0058] Furthermore, the cost function of the front-end visual odometry factor is obtained as follows:

[0059]

[0060] wherein, is the front-end cost function at time k + 1, and h Odo () is the front-end odometry function;

[0061] Step 3.2: In the backend module, according to the backend measurement position, set the noise to follow a Gaussian distribution with zero mean, and define the backend measurement factor as follows:

[0062]

[0063] wherein, is the backend measurement position; is the covariance matrix;

[0064] Furthermore, the cost function of the backend measurement factor is obtained as follows:

[0065]

[0066] wherein, is the backend cost function at time k + 1, and h Back () is the backend odometry function;

[0067] Step 3.3: Establish the optimal estimate (P * ) k+1 with the global cost function as follows:

[0068]

[0069] As a specific example, in Step 4, for the FGO-AS-VPS model after introducing the factor, set the initial values, update the Jacobian matrix, residual vector, and state increment, and perform iterative solution to obtain the optimal estimate, as shown in Figure 4 as follows:

[0070] Set the initial value to Update the Jacobian matrix Residual vector Use the least squares estimation algorithm to update the state increment, and the formula is:

[0071]

[0072] Furthermore, the updated state vector is obtained as follows:

[0073]

[0074] Iterate according to the above update method, update the state increment and state vector multiple times until the residual meets a predetermined threshold or the iteration count reaches a preset value, and obtain the optimal estimate.

[0075] The present invention also provides a visual positioning system for UAV satellite image matching based on factor graph optimization, which is used to implement the visual positioning method for UAV satellite image matching based on factor graph optimization, and includes a first unit to a fourth unit. The functions of each unit are as follows:

[0076] The first unit establishes a front-end module and a back-end module. The front-end module matches the frame-by-frame aerial images of the UAV to determine the odometer information for relative positioning, and the back-end module matches the aerial images of the UAV with satellite images to achieve absolute positioning, and feeds the absolute positioning information back to the visual odometer of the front-end module to construct a visual positioning system AS-VPS for UAV satellite image matching.

[0077] The second unit constructs the position as a state variable based on the FGO algorithm, constructs a global cost function, obtains the optimal state estimate by searching for the maximum value of the global probability function, and uses the least squares estimation algorithm to solve the optimal state to obtain the state increment estimate.

[0078] The third unit constructs an FGO-AS-VPS model, introduces front-end visual odometry factors, back-end measurement factors and their cost functions, integrates the front and back ends, and establishes the optimal estimate of the global cost function.

[0079] The fourth unit sets the initial value for the FGO-AS-VPS model after introducing factors, updates the Jacobian matrix, residual vector, and state increment, and performs iterative solution to obtain the optimal estimate.

[0080] The present invention also provides a mobile terminal, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the visual positioning method for UAV satellite image matching based on factor graph optimization.

[0081] The present invention also provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, it implements the steps in the visual positioning method for UAV satellite image matching based on factor graph optimization.

[0082] The following further describes the present invention in detail with reference to the accompanying drawings and specific embodiments.

[0083] Embodiment

[0084] In this embodiment, the front-end independent, back-end independent, and FGO-AS-VPS schemes are respectively used for visual positioning of UAV satellite image matching, and the experimental results are as Figure 5As shown in. From Figure 5 It can be seen from (a) to (c) in Figure 5 that compared with the front-end independence and the back-end independence, the horizontal positioning errors of trajectory 1 under the FOS-AS-VPS scheme are reduced by 44.18% and 31.9% respectively, and the maximum positioning errors are reduced by 31.72% and 51.60% respectively; the horizontal positioning errors of trajectory 2 are reduced by 29.5% and 22.7% respectively, and the maximum positioning errors are reduced by 41.52% and 56.48% respectively; the horizontal positioning errors of trajectory 3 are reduced by 27.2% and 27.0% respectively, and the maximum positioning errors are reduced by 31.30% and 46.93% respectively. The experimental results show that the visual positioning method based on factor graph optimization for UAV satellite image matching provided by the present invention effectively improves the positioning accuracy.

[0085] The above specific embodiments are only used to illustrate the technical method of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the examples, those of ordinary skill in the art should understand that the technical method of the present invention can be modified or equivalently replaced without departing from the spirit and scope of the technical method of the present invention, and all of them should be covered by the scope of the claims of the present invention.

Claims

1. A visual positioning method for UAV satellite image matching based on factor graph optimization, characterized in that It includes the following steps: Step 1: Establish a front-end module and a back-end module. The front-end module matches the frame-by-frame aerial images of the drone to determine the odometry information for relative positioning. The back-end module matches the aerial images of the drone with satellite images to achieve absolute positioning and feeds the absolute positioning information back to the visual odometry of the front-end module, constructing a visual positioning system AS-VPS based on the matching of drone satellite images; Step 2: Based on the FGO algorithm, construct the position as the state variable, and construct the global cost function. Obtain the optimal state estimate by searching for the maximum value of the global probability function, and use the least squares estimation algorithm to solve the optimal state to obtain the state increment estimate; Step 3: Construct the FGO-AS-VPS model, introduce the front-end visual odometry factor, the back-end measurement factor and the cost function, integrate the front and back ends, and establish the optimal estimate of the global cost function; Step 4: For the FGO-AS-VPS model after introducing the factors, set the initial values, update the Jacobian matrix, the residual vector, and the state increment, and perform iterative solution to obtain the optimal estimate.

2. The visual positioning method for UAV satellite image matching optimized based on factor graph according to claim 1, characterized in that, In Step 1, a front-end module and a back-end module are established. The front-end module matches the frame-by-frame aerial images of the drone to determine the odometry information for relative positioning. The back-end module matches the aerial images of the drone with satellite images to achieve absolute positioning and feeds the absolute positioning information back to the visual odometry of the front-end module, constructing a visual positioning system AS-VPS based on the matching of drone satellite images, specifically as follows: Step 1.1: Establish the front-end module. Establish the visual odometry by matching consecutive downward-looking drone images. Based on the feature point set matched by LightGlue, determine the pixel coordinate transformation relationship between two frames, and transfer the pixel coordinates to the UTM coordinates to obtain the corresponding point coordinates of the pixel points in the UTM coordinate system: f W = β·(R Wu ·f u + t Wu ) Where, f k is the coordinate of the previous frame in the pixel coordinate system; f k+1 is the coordinate of the current frame in the pixel coordinate system, R Wu is the rotation matrix; t Wu is the displacement matrix; f u is the pixel coordinate; f W is the coordinate in the UTM coordinate system; β is the scaling factor; (dx dy dθ) k,k+1 represents the coordinate deviation from the previous frame, i.e., the k-th frame, to the current frame, i.e., the k+1-th frame; Step 1.2: Establish the back-end module. Use the preprocessed satellite tile map to perform feature matching with the drone images, calculate the perspective transformation matrix, and map the pixel coordinates of the drone images to the geographic coordinate system: In the formula, are the geographical positions of the upper left corner and the lower right corner of the satellite image; C x , C y are the pixel positions of the feature points of the UAV image; L a , L o are the geographical coordinates of the feature points of the UAV image; W and H are the width and height of the satellite image; Step 1.3: Feed the absolute positioning information back to the front-end visual odometry, constructing a visual positioning system AS-VPS based on the matching of drone satellite images.

3. The visual positioning method for UAV satellite image matching optimized based on factor graph according to claim 2, characterized in that, In Step 2, based on the FGO algorithm, construct the position as the state variable, and construct the global cost function. Obtain the optimal state estimate by searching for the maximum value of the global probability function, and use the least squares estimation algorithm to solve the optimal state to obtain the state increment estimate, specifically as follows: Step 2.1: Construct the position as a state variable and construct the global cost function Solve the state estimation by searching for the maximum value of the global probability function to obtain the optimal state as follows: where L is the position state, p(·) is the joint probability density, L = (L1…, L N ) is the set of AS-VPS position states, G(L1, L2, …, L N ) is the global probability density function; j = 1, 2, …, m, m ≤ N; Step 2.2: According to the requirement of the minimum residual value for the optimal state, use the least squares estimation algorithm to transform the solution of the optimal state into: where E is the residual vector; characterized as Step 2.3: Obtain the state increment estimate ΔL i It is: In the formula is the Jacobian matrix, is the residual.

4. The visual positioning method for UAV satellite image matching optimized based on the factor graph according to claim 3, characterized in that, In Step 3, construct the FGO-AS-VPS model, introduce the front-end visual odometry factor, the back-end measurement factor and the cost function, integrate the front and back ends, and establish the optimal estimate of the global cost function, specifically as follows: Step 3.1: In the front-end module, according to the relationship between the state estimated by the drone image matching and the state increment, set the noise to follow a Gaussian distribution with zero mean, and define the front-end visual odometry factor as follows: where p k , p k+1 are the states at times k and k + 1; N(·) is the Gaussian distribution function; is the covariance matrix; is the state increment at times k and k + 1; Furthermore, the cost function of the front-end visual odometry factor is obtained as follows: In the formula, is the front-end cost function at the (k + 1)-th moment, and h Odo () is the front-end odometry function; Step 3.

2. In the backend module, according to the backend measurement positions, set the noise to follow a Gaussian distribution with zero mean, and define the backend measurement factor as follows: In the formula, is the rear measurement position; is the covariance matrix; Furthermore, obtain the cost function of the backend measurement factor as follows: wherein is the backend cost function at the (k + 1)-th moment, and h Back () is the backend odometer function; Step 3.3, establish the optimal estimate (P * ) k+1 as follows:

5. The visual positioning method for UAV satellite image matching optimized based on factor graph according to claim 4, characterized in that In step 4, for the FGO-AS-VPS model after introducing the factor, set the initial values, update the Jacobian matrix, residual vector, and state increment, and perform iterative solution to obtain the optimal estimate, specifically as follows: Set the initial value to Update the Jacobian matrix Residual vector Update the state increment using the least squares estimation algorithm, and the formula is: Furthermore, obtain the updated state vector as follows: Iteratively update the state increment and state vector multiple times until the residual meets a predetermined threshold or the iteration count reaches a preset value to obtain the optimal estimate.

6. A visual positioning system for UAV satellite image matching optimized based on factor graph, characterized in that, This system is used to implement the visual positioning method for UAV satellite image matching based on factor graph optimization described in any one of claims 1 to 5, and includes a first unit to a fourth unit. The functions of each unit are as follows: The first unit establishes a front-end module and a backend module. The front-end module matches the frame-by-frame aerial images of the UAV to determine the odometer information for relative positioning. The backend module matches the aerial images taken by the UAV with satellite images to achieve absolute positioning, and feeds back the absolute positioning information to the visual odometer of the front-end module to construct a visual positioning system AS-VPS based on the matching of UAV satellite images; The second unit constructs the position as a state variable based on the FGO algorithm, constructs a global cost function, obtains the optimal state estimate by searching for the maximum value of the global probability function, and uses the least squares estimation algorithm to solve for the optimal state to obtain the state increment estimate; The third unit constructs an FGO-AS-VPS model, introduces the front-end visual odometry factor, backend measurement factor, and their cost functions, integrates the front and backend, and establishes the optimal estimate of the global cost function; The fourth unit, for the FGO-AS-VPS model after introducing the factor, sets the initial values, updates the Jacobian matrix, residual vector, and state increment, and performs iterative solution to obtain the optimal estimate.

7. A mobile terminal, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the visual positioning method for UAV satellite image matching based on factor graph optimization described in any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the visual positioning method for UAV satellite image matching based on factor graph optimization described in any one of claims 1 to 5.

Images