A method for ground target positioning of UAV based on monocular camera

Abstract

The present invention discloses a method for positioning ground targets by a UAV based on a monocular camera, comprising: sequential image acquisition within a scene; solving the geometric relationship of sequential image pairs; neural network target detection; and multi-observation equation least squares target position solution. The method uses a monocular camera, combined with high-precision positioning information, attitude information and auxiliary prior information of the camera, and adopts a neural network detection model to detect the target, and solves the target position by multi-observation multi-equation least squares. The device used in the method is small, has a long battery life, fast target detection, high accuracy, low calculation amount, and can be calculated in real time. It can meet the requirements of lightweight, unmanned, and intelligent combat assistance, accurately locate the target during reconnaissance, and has high military and civilian value.

Description

Technical Field

[0001] The present invention relates to a method for positioning a ground target by an unmanned aerial vehicle (UAV), in particular to a method for positioning a ground target by an unmanned aerial vehicle (UAV) based on a monocular camera. Background Art

[0002] In the military, driven by emerging operational concepts such as "unmanned," "intelligent," and "convenient," drone-based targeting and strike methods are rapidly developing. Accurate, real-time target positioning has become a core technology for "unmanned" operations, playing a particularly important role in strategic and tactical reconnaissance, enhancing battlefield situational awareness, and improving weapon strike effectiveness. In the civilian sector, drone-based target positioning can assist in monitoring maritime and land transportation, natural disaster relief, and digital mapping.

[0003] Using drone platforms equipped with cameras to image the reconnaissance area, analyzing and processing sequential images to achieve autonomous navigation and target positioning, and obtaining the terrain and topography of the battlefield in real time is of great significance to combat.

[0004] However, traditional drone image positioning relies on stereo image pair observations using binocular or multi-camera cameras, combined with a certain number of ground control points or a high-precision digital elevation model (DEM) of the survey area. This solution offers high accuracy and produces a richer product. However, this method requires the deployment of a certain number of image control points, resulting in numerous unknown parameters, a large computational load, and time-consuming and labor-intensive solutions. Furthermore, the high cost and power consumption of binocular or multi-camera drones hinders long-range flight and beyond-visual-range target tracking and positioning. Summary of the Invention

[0005] Purpose of the invention: The technical problem to be solved by the present invention is to provide a method for positioning ground targets of UAVs based on a monocular camera in response to the shortcomings of the existing technology.

[0006] In order to solve the above technical problems, the present invention discloses a method for positioning a ground target by a UAV based on a monocular camera, comprising the following steps:

[0007] Step 1: Use a drone equipped with Real-Time Kinematic (RTK) technology and a high-precision Inertial Measurement Unit (IMU) system to acquire sequential images of the survey area.

[0008] Step 2: Determine the geometric relationship of the sequence images based on the obtained feature points of the same name and the UAV pose information, perform absolute orientation on the stereo image pairs in the sequence images, and obtain the geometric relationship of the sequence image pairs;

[0009] Step 3: Use the Yolo-v5 neural network to detect the target and obtain the target image points;

[0010] Step 4: Based on the geometric relationship of the sequence image pairs, the collinearity condition equation is used to solve the ground point coordinates corresponding to the target image point to complete the UAV ground target positioning based on the monocular camera.

[0011] When the drone is used to obtain sequential images of the survey area in step 1 of the present invention, the position information of the drone during the imaging process is obtained in real time by the real-time dynamic difference technology carried thereon.

[0012] When the drone is used to obtain sequential images of the survey area in step 1 of the present invention, the posture information of the drone during the imaging process is obtained in real time by the high-precision inertial navigation system carried thereon.

[0013] In step 1 of the present invention, the route planning and design of the UAV is performed so that the acquired sequence images of the survey area are sequence images with overlapping degrees.

[0014] The method for calculating the geometric relationship of the sequence image pairs in step 2 of the present invention includes:

[0015] Step 2-1: Combine the sequence images obtained in step 1 with the position and posture information during the imaging process to determine the position and posture information of each photo in the sequence images during the imaging process.

[0016] Step 2-2, based on the position and posture information of a single photo during the imaging process, the relative position and posture between the sequence images are restored, and the relationship between the stereo image pair and the ground point is determined by combining the real-time dynamic differential position information of the position.

[0017] The method for obtaining the target image point in step 3 of the present invention includes:

[0018] The Yolo-v5 model is trained according to the set goals, and the trained model is used to perform real-time target detection on the collected images to obtain target image points.

[0019] The collinearity condition equation described in step 4 of the present invention is:

[0020]

[0021]

[0022] Among them, X S ,Y S and Z S It represents the coordinate value of the projection center S in the space rectangular coordinate system selected on the ground when the photo is formed. The posture of the photo when it is formed is expressed by ω and κ represent, ω and κ are the rotation angles of the camera's principal optical axis around the Y, Z, and X axes, respectively. The six parameters mentioned above are the exterior orientation elements of the photo. The rotation matrix R represents the relationship with the exterior orientation elements. a1, b1, c1, a2, b2, c2, a3, b3, and c3 are used to represent the values ​​at each position of the rotation matrix calculation result. The rotation matrix is ​​expressed as:

[0023]

[0024] Among them, f is the camera principal distance, the observed quantity (x, y) is the coordinate of the target in the rectangular coordinate system of the image plane; the unknown quantity (X, Y, Z) is the position of the target in the rectangular coordinate system of the ground point.

[0025] The method of using the collinearity condition equation in step 4 of the present invention to solve the ground point coordinates corresponding to the target image point includes:

[0026] According to the collinearity condition equations, one photo constitutes two sets of equations, and a stereo pair consisting of a pair of photos constitutes four sets of equations. The same target appears simultaneously in different consecutive stereo pairs, and multiple equations can be composed of multiple observations. The precise position of the target (X, Y, Z) is solved by least squares, ultimately completing the UAV ground target positioning based on the monocular camera.

[0027] The stereo image pair described in step 2 of the present invention is two photos taken at different positions in the sequence of images with a certain degree of overlap, and adjacent image frames are used as the stereo image pair.

[0028] Beneficial effects:

[0029] 1. A method for ground target positioning using a monocular camera on a UAV is proposed. This method combines RTK (Real-Time Kinematic) and IMU (Inertial Measurement Unit) auxiliary information for real-time target resolution. UAVs using a monocular camera payload are compact, have extended flight range, and offer fast target detection with high accuracy, minimal computational effort, and real-time calculations, enabling rapid single- and multiple-target positioning. This method is highly valuable in providing unmanned, intelligent, and convenient military and civilian assistance.

[0030] 2. Introducing Yolo-V5 for rapid target detection and positioning. The latest Yolo-V5 neural network detection model can detect images at speeds up to 140 frames per second. This method enables real-time target detection and, combined with the geometric relationships between sequential image pairs, enables real-time target location calculation. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more apparent.

[0032] Figure 1 Flowchart of the present invention.

[0033] Figure 2 Schematic diagram of overlap during sequential image acquisition.

[0034] Figure 3 Schematic diagram of the position and posture of a single image in space.

[0035] Figure 4 Schematic diagram of the position and posture of the stereo image pair in the space and their relationship with the ground points in the sequence image. DETAILED DESCRIPTION

[0036] To facilitate understanding of the present invention, the present invention will be described more fully below with reference to the accompanying drawings. The accompanying drawings illustrate preferred embodiments of the present invention. However, the present invention may be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and comprehensive understanding of the present disclosure.

[0037] A method for ground target positioning of UAV based on monocular camera, such as Figure 1 As shown, the following steps are included:

[0038] Step 1: Use a drone equipped with real-time kinematic (RTK) technology and a high-precision inertial measurement unit (IMU) system to acquire sequential images of the survey area.

[0039] UAVs equipped with real-time dynamic differential RTK (Real-Time Kinematic) technology can obtain real-time position information during the UAV imaging process; UAVs equipped with high-precision inertial navigation IMU (Inertial Measurement Unit) systems can obtain real-time attitude information during the UAV imaging process. Figure 2 As shown in the figure, the overlap between the sequence images has a certain impact on the accuracy of target positioning solution. Therefore, it is necessary to ensure that the overlap between the sequence images is around 80%. In order to make the scale of the sequence images consistent, the UAV should maintain a constant altitude during flight.

[0040] Reasonable route planning and design should be carried out according to the actual situation of the survey area, maintaining a certain altitude, and obtaining a sequence of images with a certain degree of overlap in the survey area.

[0041] Step 2: Determine the geometric relationship of the sequence images based on the same-name feature points of the sequence images and the drone's pose information to complete the absolute orientation of the stereo image pair;

[0042] The parameters that determine the spatial position and posture of a photograph in the ground rectangular coordinate system at the moment of photographing are called the exterior orientation elements of the photograph. A photograph has six exterior orientation elements, three of which are the coordinate values ​​describing the spatial position of the photographic center S, called linear elements (X S ,Y S ,Z s The other three are the three angular elements that describe the spatial posture of the photographic beam ( ω,κ), the space is represented as Figure 3 shown.

[0043] exist Figure 3 In the figure, O-XYZ is the ground space rectangular coordinate system, that is, the position of the target in the space rectangular coordinate system is solved; S-XYZ is the image space auxiliary coordinate system, S is the photography center, (X S ,Y S ,Z S ) is the position of the photography center in the ground space rectangular coordinate system, o-xy is the image plane rectangular coordinate system, ( ω, κ) is a certain angle of rotation of the image plane around the axes of the S-XYZ coordinate system.

[0044] Combined with the sequence of images obtained in step 1, which includes the position information obtained by RTK (Real-Time Kinematic) and the imaging attitude obtained by IMU (Inertial Measurement Unit) during the imaging process, the position information of the sequence of images during the imaging process is obtained. The position coordinates of the imaging center S can be calculated based on the RTK (Real-Time Kinematic) observation value, and the three angular elements of the spatial attitude of the photographic beam can be obtained based on the results of IMU (Inertial Measurement Unit). This can determine the precise position and attitude of each photo during the imaging process, such as Figure 3 shown.

[0045] According to the pose information of a single image, the relative position and pose between the image pairs can be restored, and the relationship between the stereo image pairs and the ground points can be determined, such as Figure 4 shown.

[0046] Among them, O-XYZ is the ground space rectangular coordinate system, A is the ground point, and photos P1 and P2 are images covering the area taken by the drone. The spatial position and attitude relationship of the image pair are restored through the exterior orientation elements as follows: Figure 4As shown, S-xyz is the rectangular coordinate system of the image space, S is the photography center, o-xy is the rectangular coordinate system of the image plane, a1 and a2 are the positions of the ground point A on the photos P1 and P2 respectively.

[0047] Step 3: Use the Yolo-v5 neural network for target detection (reference: Zhang Hongqun, Ban Yongmiao, Guo Lingling, Jin Yunfei, Chen Ling. Remote Sensing Image Ship Detection Method Based on YOLOv5 [J]. Electronic Measurement Technology, 2021, 44(08): 87-92. DOI: 10.19651 / j.cnki.emt.2106199. And reference: Yang FA Real-Time Apple Targets Detection Method for Picking Robot Based on Improved YOLOv5 [J]. Remote Sensing, 2021, 13.);

[0048] In the early stage, the Yolo-v5 model is trained according to the target of interest, and the trained model is used to perform real-time target detection on the collected images.

[0049] Step 4: Based on the geometric relationship of the sequence image pairs, the collinearity condition equation is used to solve the ground point coordinates corresponding to the target image point.

[0050] Imaging theory shows that under ideal conditions, the image point, photographic center, and object point are located on the same straight line. The mathematical expression relationship between the image point, photographic center, and object point is called the collinearity equation. The collinearity equation can be expressed as:

[0051]

[0052]

[0053] Among them, (X S ,Y S ,Z S ) represents the coordinate value of the projection center S in the space rectangular coordinate system selected on the ground when the photo is imaged, and the posture when the photo is imaged is represented by ω and κ represent the six parameters of the photo's exterior orientation. In step 2, we learned that the exterior orientation was obtained using RTK (Real-Time Kinematic) and IMU (Inertial Measurement Unit). The conversion relationship between the exterior orientation and a1, b1, c1, a2, b2, c2, a3, b3, and c3 in the above formula is as follows:

[0054]

[0055]

[0056] f is the principal distance of the camera, and the observation quantity (x, y) is the coordinate of the target in the rectangular coordinate system of the image plane.

[0057] The undetermined value (X, Y, Z) is the position of the target point in the ground rectangular coordinate system. A single photograph can form two sets of equations, and a pair of sequential image pairs can form four sets of equations. If the same target appears simultaneously in consecutive stereo pairs, multiple observations can be used to form multiple equations, allowing the target point's undetermined value (X, Y, Z) to be calculated using least squares.

[0058] In a specific implementation, the present application provides a computer storage medium and a corresponding data processing unit, wherein the computer storage medium is capable of storing a computer program that, when executed by the data processing unit, can execute the invention content of the method for ground target positioning of an unmanned aerial vehicle based on a monocular camera provided by the present invention and some or all of the steps in each embodiment. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0059] Those skilled in the art can clearly understand that the technical solutions in the embodiments of the present invention can be implemented by means of computer programs and their corresponding general hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, in essence or in other words, the part that contributes to the prior art, can be embodied in the form of a computer program, i.e., a software product. The computer program software product can be stored in a storage medium and includes several instructions for enabling a device including a data processing unit (which can be a personal computer, a server, a single-chip microcomputer, a MUU, or a network device, etc.) to execute the methods described in various embodiments of the present invention or certain parts of the embodiments.

[0060] This invention provides a concept and method for ground target positioning using a monocular camera for a drone. While there are numerous methods and approaches for implementing this technical solution, the above-described preferred embodiments of the invention are merely exemplary. It should be noted that those skilled in the art may make improvements and modifications without departing from the principles of the invention, and such improvements and modifications are also within the scope of protection of this invention. Any components not specified in this embodiment may be implemented using existing technologies.

Claims

1. A method for positioning a ground target by an unmanned aerial vehicle based on a monocular camera, characterized in that: The following steps are involved: Step 1: Use a UAV equipped with real-time dynamic differential technology and a high-precision inertial navigation system to obtain sequential images of the survey area; Step 2: determine the geometric relationship of the sequence images based on the obtained feature points of the same name and the position and posture information of the drone, perform absolute orientation on the stereo image pairs in the sequence images, and obtain the geometric relationship of the sequence image pairs; Step 3, use the Yolo-v5 neural network to detect the target and obtain the target image points; Step 4: Based on the geometric relationship of the image pairs, the collinear condition equation theory is used to form an equation based on the observed quantities, and the coordinates of the ground points corresponding to the target image points are solved by the least square method to complete the positioning of the UAV to the ground target based on the monocular camera; The method for calculating the geometric relationship of the sequence image pair in step 2 includes: Step 2-1, combining the sequence images obtained in step 1 with the position and posture information during the imaging process to determine the position and posture information of each photo in the sequence images during the imaging process; Step 2-2, based on the position and posture information of a single photo during the imaging process, the relative position and posture between the sequence images are restored, and the relationship between the stereo image pair and the ground point is determined in combination with the real-time dynamic differential position information of the position.

2. The method for positioning a ground target by a UAV based on a monocular camera according to claim 1, characterized in that: When the drone is used to obtain the sequence images of the survey area in step 1, the position information of the drone during the imaging process is obtained in real time through the real-time dynamic difference technology carried by the drone.

3. The method for positioning a ground target by a UAV based on a monocular camera according to claim 2, characterized in that: When the drone is used to obtain the sequence images of the survey area in step 1, the attitude information of the drone during the imaging process is obtained in real time through the high-precision inertial navigation system carried by the drone.

4. The method for positioning a ground target by a UAV based on a monocular camera according to claim 3, characterized in that: In step 1, the flight path of the UAV is planned and designed so that the sequence images of the survey area obtained are sequence images with overlapping degrees.

5. The method for positioning a ground target by a UAV based on a monocular camera according to claim 4, characterized in that: The method for obtaining the target image point in step 3 includes: The Yolo-v5 model is trained according to the set goals, and the trained model is used to perform real-time target detection on the collected images to obtain target image points.

6. The method for positioning a ground target by a UAV based on a monocular camera according to claim 5, characterized in that: The collinearity condition equation described in step 4 is: Among them, X S ,Y S and Z S It represents the coordinate value of the projection center S in the spatial rectangular coordinate system selected on the ground when the photo is imaged, and the posture when the photo is imaged is expressed by ω and κ represent, ω and κ are the rotation angles of the camera's main optical axis around the Y, Z and X axes respectively; the above six parameters are the exterior orientation elements of the photo, and the rotation matrix R represents the relationship with the exterior orientation elements. a1, b1, c1, a2, b2, c2, a3, b3, c3 represent the values ​​at each position of the rotation matrix calculation result matrix. The rotation matrix is ​​expressed as: Among them, f is the camera principal distance, the observed quantity (x, y) is the coordinate of the target in the rectangular coordinate system of the image plane; the unknown quantity (X, Y, Z) is the position of the target in the rectangular coordinate system of the ground point.

7. The method for positioning a ground target by a UAV based on a monocular camera according to claim 6, characterized in that: The method of solving the ground point coordinates corresponding to the target image point using the collinear condition equation in step 4 includes: According to the collinearity condition equations, one photo constitutes two sets of equations, and a pair of photos constitutes four sets of equations. The same target appears simultaneously in different consecutive stereo pairs, and multiple equations can be composed of multiple observations. The precise position (X, Y, Z) of the target is solved by the least squares method, and finally the UAV ground target positioning based on the monocular camera is completed.

8. The method for positioning a ground target by a UAV based on a monocular camera according to claim 7, characterized in that: The stereo image pair described in step 2 is two photos taken at different positions in the sequence of images with overlapping degrees, and adjacent image frames are used as the stereo image pair.

Images