Unmanned aerial vehicle ground target positioning method and system based on monocular vision

Abstract

The invention discloses a monocular vision-based unmanned aerial vehicle ground target positioning method and system, and the method comprises the steps: building a north-east NED coordinate system, a body coordinate system, a camera coordinate system and a pixel coordinate system, setting an image plane auxiliary coordinate system and a ground auxiliary coordinate system, and obtaining a monocular vision-based unmanned aerial vehicle ground target positioning model; obtaining unmanned aerial vehicle pose and holder pose data, extracting pixel coordinates of a target in an aerial image of the unmanned aerial vehicle, and calculating an azimuth angle and an elevation angle of a target line-of-sight vector by combining internal parameters of a camera; calculating the three-dimensional coordinates of the target in the NED coordinate system according to the projection relationship among the coordinate systems; and converting the three-dimensional coordinate of the target in the NED coordinate system into a coordinate in a geodetic coordinate system to complete positioning. Meanwhile, the invention provides a non-flat ground positioning method. The method can solve the problem that traditional monocular positioning depends on more preconditions and is limited in operation, and can realize rapid and accurate positioning of a target.

Description

Technical Field

[0001] This invention belongs to the field of UAV positioning technology, specifically relating to a method and system for UAV ground target positioning based on monocular vision. Background Technology

[0002] In the context of modern information warfare, unmanned aerial vehicles (UAVs) have become crucial equipment for battlefield reconnaissance and precision strikes due to their advantages such as flexible deployment, strong stealth, and low cost. The precise location capabilities of UAVs for ground targets directly impact combat effectiveness and hold significant strategic importance in air-ground coordination, situational awareness, and fire guidance.

[0003] Currently, the mainstream UAV ground target visual positioning methods have the following significant drawbacks: (1) Positioning methods that integrate active ranging require active laser emission, which easily exposes the platform's position, reduces battlefield survivability, and the equipment size and power consumption are relatively large, limiting their application on small UAVs; (2) Positioning methods based on reference map matching rely on high-precision satellite reference maps or digital elevation models, which are limited in complex battlefields where reference maps are missing (such as remote battlefields) or where terrain and landforms have changed significantly (such as artillery-damaged areas); (3) Existing positioning methods based on monocular vision also have key bottlenecks: they require prior acquisition of the target's absolute size or strict limitation of the UAV's flight trajectory and gimbal angle, which cannot adapt to the needs of "unknown targets and flexible trajectories" on the battlefield; and they require dozens of observations and complex iterative calculations, which are time-consuming and difficult to cope with the rapidly changing battlefield environment; at the same time, they do not consider the altitude differences of non-flat ground, which leads to a significant increase in positioning errors in complex terrain areas, failing to meet the requirements of precision strikes. These technical bottlenecks restrict the positioning accuracy and combat effectiveness of UAVs in complex environments. Summary of the Invention

[0004] The purpose of this invention is to address the problems in the prior art by providing a method and system for UAV ground target localization based on monocular vision, which can achieve rapid and accurate target localization, is easy to implement, low in cost, and highly reliable.

[0005] To achieve the above objectives, the present invention provides the following technical solution: Firstly, a method for UAV ground target localization based on monocular vision is provided, including: Establish the NED coordinate system, body coordinate system, camera coordinate system and pixel coordinate system in the northeast, and set up the image plane auxiliary coordinate system and the ground auxiliary coordinate system to obtain the UAV localization model of ground target based on monocular vision; Acquire drone pose and gimbal attitude data, and extract the pixel coordinates of targets in the drone aerial images; Using a ground target localization model based on monocular vision UAV, the azimuth and elevation angles of the target's line of sight vector are calculated by combining camera internal parameters, UAV and gimbal data, and the pixel coordinates of the target in the image; and the three-dimensional coordinates of the target in the NED coordinate system are calculated based on the projection relationship between the coordinate systems. The target's three-dimensional coordinates in the NED coordinate system are converted to coordinates in the geodetic coordinate system to complete the positioning.

[0006] As a preferred embodiment, the NED coordinate system is established with the UAV's center of mass as the origin, the X-axis pointing due north, the Y-axis pointing due east, and the Z-axis perpendicular to the XOY plane pointing downwards towards the ground, denoted as O. v -X v Y v Z v ; The body coordinate system is established with the UAV's center of mass as its origin and is fixedly connected to the UAV. The X-axis is parallel to the body axis and points towards the nose; the Y-axis is perpendicular to the UAV's vertical plane of symmetry and points towards the right side of the fuselage; and the Z-axis is established in the UAV's vertical plane of symmetry, perpendicular to the X-axis and pointing downwards towards the fuselage, denoted as O. b -X b Y b Z b ; The camera coordinate system is established with the camera optical center as the origin, the Z-axis pointing forward along the camera optical axis, the X-axis parallel to the horizontal axis of the image physical coordinate system pointing to the right, and the Y-axis parallel to the horizontal axis of the image physical coordinate system pointing downward, denoted as O. c -X c Y c Z c ; The pixel coordinate system is established with the top left corner of the image as the origin, the X-axis pointing to the right along the top edge of the image, and the Y-axis pointing downwards along the left edge of the image, denoted as O. u -X u Y u .

[0007] As a preferred embodiment, the step of setting the image plane auxiliary coordinate system and the ground auxiliary coordinate system includes: For the auxiliary coordinate system of image plane M1, the origin O1 is the intersection of the optical axis and image plane M1. x u y The X-axis is parallel to the pixel coordinate system and points to the right, and the Y-axis is parallel to the pixel coordinate system and points upward, denoted as O1-X1Y1; For the ground-based auxiliary coordinate system M2, the origin O2 is the intersection of the optical axis and the ground M2. The X-axis and Y-axis are parallel to and opposite to the Y-axis and X-axis of the image plane M1, respectively, and are denoted as O2-X2Y2.

[0008] As a preferred embodiment, the step of using a monocular vision-based UAV ground target localization model, combined with camera internal parameters and the pixel coordinates of the target in the image from the UAV and gimbal data, to calculate the azimuth and elevation angles of the target's line-of-sight vector includes: The deflection angles of the target's line-of-sight vector relative to the optical axis in the horizontal and vertical directions are calculated using the following formula:

[0009]

[0010] In the formula, , These represent the positions P1 corresponding to target P in the auxiliary coordinate system O1-X1Y1 of the image plane; , These represent the pixel coordinates of the target location; The size is one pixel; This is the deflection angle in the vertical direction; This refers to the horizontal deflection angle. Focal length; set up , These represent the pitch and yaw angles of the gimbal, respectively. Since the positions of the gimbal's rotation center and the camera's optical center are relatively fixed, therefore... , This is also equal to the camera's pitch and yaw angles; The optical center O of the camera is calculated using the following formula. c Deflection angle at the origin:

[0011] In the formula, The angle between the target's line-of-sight vector and the horizontal plane; The angle between the projection of the target's line-of-sight vector onto the horizontal plane and the due north direction.

[0012] As a preferred embodiment, calculating the target's three-dimensional coordinates in the NED coordinate system based on the projection relationships between different coordinate systems includes the following steps: Assume the point where the drone perpendicularly intersects the ground is A, and the target's position is P. Considering the drone, camera, and target at point O... v The projection of the AP plane, in the NED coordinate system, is based on the UAV's centroid O. v and camera optical center O c The distance is L The distance between the optical center and the target on the horizontal plane is calculated as follows:

[0013]

[0014] In the formula, h The vertical distance between the UAV's center of mass and the camera's optical center; β The pitch angle of the UAV is equal to the angle between the X-axis of the aircraft's coordinate system and the horizontal plane, with upward pitch being positive. η Connect the center of mass of the UAV and the optical center of the camera to the horizontal plane X. v O v Y v The included angle is negative; H This refers to the relative altitude of the drone above the ground. Combining drones, cameras, and targets in X v O v Y v The projection of the plane onto the target, in the NED coordinate system, gives the following three-dimensional coordinates:

[0015] In the formula, γ The yaw angle of the UAV is equal to the angle between the X-axis of the aircraft's coordinate system and the due north direction.

[0016] As a preferred embodiment, the conversion of the target's three-dimensional coordinates in the NED coordinate system to coordinates in the geodetic coordinate system includes the following steps: According to the latitude B of the drone uav Longitude L uav and elevation H uav And the target's three-dimensional coordinates in the NED coordinate system ( x v , y v , z v The target's coordinates in the geodetic coordinate system are calculated using the following formula:

[0017] In the formula, R is the average radius of the Earth.

[0018] As a preferred embodiment, this also includes obtaining the vertical distance h from the target location to the horizontal plane of the UAV's takeoff point when the target's altitude and the target's altitude are different. p The drone's relative altitude to the horizontal plane is H. The target is photographed by changing the drone's position twice, with the drone's takeoff point being the same in both shots. p As a fixed value, the target's coordinates in the NED coordinate system are:

[0019] Converting the NED coordinate system to the geodetic coordinate system, the latitude and longitude of the target in the two photographs are equal, which meets the following requirements:

[0020] The vertical distance from the target location to the horizontal plane of the UAV takeoff point is obtained by calculating the average value. The target's position is obtained when the horizontal plane of the UAV takeoff point and the target's altitude are different, by taking information from any one of the photos.

[0021] As a preferred embodiment, the UAV and gimbal data include the UAV position, attitude angle, and gimbal attitude angle; The camera's internal parameters include focal length, field of view, sensor size, and lens distortion parameters.

[0022] Secondly, a monocular vision-based UAV ground target positioning system is provided, comprising: The monocular vision positioning model construction module is used to establish the NED coordinate system, body coordinate system, camera coordinate system and pixel coordinate system in the northeast, and set the image plane auxiliary coordinate system and the ground auxiliary coordinate system to obtain the UAV positioning model of ground target based on monocular vision. The data acquisition module is used to acquire the drone pose and gimbal attitude data, and extract the pixel coordinates of the target in the image from the drone aerial image; The NED coordinate calculation module is used to calculate the azimuth and elevation angles of the target's line-of-sight vector by using a ground target localization model based on monocular vision UAV, combined with camera internal parameters, UAV and gimbal data and the pixel coordinates of the target in the image; and, based on the projection relationship between the coordinate systems, to calculate the three-dimensional coordinates of the target in the NED coordinate system. The geodetic coordinate transformation module is used to convert the target's three-dimensional coordinates in the NED coordinate system to coordinates in the geodetic coordinate system, thus completing the positioning.

[0023] Thirdly, a computer-readable storage medium is provided, wherein at least one instruction is stored in the computer-readable storage medium, the at least one instruction being executed by a processor in an electronic device to implement the UAV ground target localization method based on monocular vision.

[0024] Compared with the prior art, the first aspect of the present invention has at least the following beneficial effects: This invention presents a monocular vision-based UAV ground target localization method that is easy to implement, requires no additional equipment, and can meet the needs of precise positioning in complex battlefield environments. By establishing a NED coordinate system (northeast-east), a body coordinate system, a camera coordinate system, and a pixel coordinate system, and setting up an image plane auxiliary coordinate system and a ground auxiliary coordinate system, a monocular vision-based UAV ground target localization model is obtained. Based on this model, UAV pose and gimbal attitude data, camera internal parameters, and the pixel coordinates of targets in the image are simultaneously acquired. Deep learning can be applied for target recognition and detection when acquiring pixel coordinates. Since the target may not be at the center of the camera's field of view, the azimuth and elevation angles of the target's line-of-sight vector are calculated. The three-dimensional coordinates of the target in the NED coordinate system are calculated based on the projection relationships between the coordinate systems. Finally, the three-dimensional coordinates of the target in the NED coordinate system are converted to coordinates in the geodetic coordinate system, completing the localization. This solves the problem of depth information loss during the conversion from the image coordinate system to the camera coordinate system for UAV ground target localization. The method of this invention does not require target size information, nor does it restrict the angle of the gimbal or the flight attitude and trajectory of the UAV. It solves the problem of traditional monocular positioning being "dependent on many prerequisites and limited in operation". It can achieve rapid and accurate target positioning in military combat environments and improve the UAV's low-cost and high-reliability precision strike support capability in information warfare.

[0025] Furthermore, if the terrain near the target location is complex, there may be a difference between the horizontal plane of the UAV's takeoff point and the altitude of the target. If the positioning theory for flat ground is applied, the result will be a large positioning error. This invention achieves positioning and fixes the UAV's takeoff point through "two-shot calibration". The UAV position is changed twice to shoot the target. The altitude difference between the target and the horizontal plane of the takeoff point is solved by using the condition that the two geodetic coordinate calculations are equal. This eliminates the positioning error caused by terrain undulations and achieves accurate positioning of ground targets by UAVs on non-flat ground. It can cover complex combat environments such as mountains and hills, and does not require dozens of observations and complex iterative calculations. It significantly shortens the time required to meet the needs of instantaneous battlefield response. The application scenarios can cover a variety of military missions such as battlefield reconnaissance, fire guidance, and air-ground coordination. Attached Figure Description

[0026] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the following drawings are only some of the embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a schematic diagram illustrating the principle of the UAV ground target localization method based on monocular vision according to an embodiment of the present invention; Figure 2(a) O between the UAV, camera and target in the NED coordinate system according to an embodiment of the present invention v AP plane projection relationship diagram; Figure 2(b) shows the X-axis distance between the UAV, camera, and target in the NED coordinate system according to an embodiment of the present invention. v O v Y v Plane projection relationship diagram; Figure 3 This is a schematic diagram illustrating the positioning principle on non-flat ground according to an embodiment of the present invention; Figure 4 This is an image containing a target taken by a drone in an embodiment of the present invention. Detailed Implementation

[0028] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0029] It should be noted that, in the description of the embodiments of the present invention, unless otherwise stated, "a plurality of" means two or more; the terms "upper", "lower", "left", "right", "front", "back", "horizontal", "vertical", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present invention.

[0030] Furthermore, in the description of this invention, unless otherwise explicitly specified and limited, the terms "connected" and "linked" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0031] Please see Figure 1 This invention proposes a method for UAV ground target localization based on monocular vision, which can improve the positioning accuracy and combat effectiveness of UAVs in complex battlefield environments. The method mainly includes the following steps: S1. Establish the NED coordinate system, body coordinate system, camera coordinate system and pixel coordinate system in the northeast, and set the image plane auxiliary coordinate system and the ground auxiliary coordinate system to obtain the UAV localization model of ground target based on monocular vision. S2. Obtain UAV pose and gimbal attitude data, and extract the pixel coordinates of the target in the image from the UAV aerial image; S3. Using a ground target localization model based on monocular vision UAV, combined with camera internal parameters and the pixel coordinates of the target in the UAV and gimbal data and images, calculate the azimuth and elevation angles of the target's line-of-sight vector; and calculate the three-dimensional coordinates of the target in the NED coordinate system based on the projection relationship between the coordinate systems. S4. Convert the target's three-dimensional coordinates in the NED coordinate system to coordinates in the geodetic coordinate system to complete the positioning.

[0032] In one possible implementation, when establishing the NED coordinate system, body coordinate system, camera coordinate system, and pixel coordinate system, the NED coordinate system is established with the UAV's centroid as the origin, the X-axis pointing due north, the Y-axis pointing due east, and the Z-axis perpendicular to the XOY plane pointing downwards towards the ground, denoted as O. v -X v Y v Z v ; The body coordinate system is established with the UAV's center of mass as its origin and is fixedly connected to the UAV. The X-axis is parallel to the body axis and points towards the nose; the Y-axis is perpendicular to the UAV's vertical plane of symmetry and points towards the right side of the fuselage; and the Z-axis is established in the UAV's vertical plane of symmetry, perpendicular to the X-axis and pointing downwards towards the fuselage, denoted as O. b -X b Y b Z b ; The camera coordinate system is established with the camera optical center as the origin, the Z-axis pointing forward along the camera optical axis, the X-axis parallel to the horizontal axis of the image physical coordinate system pointing to the right, and the Y-axis parallel to the horizontal axis of the image physical coordinate system pointing downward, denoted as O. c -X c Y c Z c ; The pixel coordinate system is established with the top left corner of the image as the origin, the X-axis pointing to the right along the top edge of the image, and the Y-axis pointing downwards along the left edge of the image, denoted as O. u -X u Y u .

[0033] In one possible implementation, when setting the image plane auxiliary coordinate system and the ground auxiliary coordinate system in step S1 of this embodiment, for the image plane M1 auxiliary coordinate system, the intersection of the optical axis and the image plane M1 is taken as the origin O1 (u x u y The X-axis is parallel to the pixel coordinate system and points to the right, and the Y-axis is parallel to the pixel coordinate system and points upward, denoted as O1-X1Y1; For the ground-based auxiliary coordinate system M2, the origin O2 is the intersection of the optical axis and the ground M2. The X-axis and Y-axis are parallel to and opposite to the Y-axis and X-axis of the image plane M1, respectively, and are denoted as O2-X2Y2.

[0034] In one possible implementation, the UAV and gimbal data mentioned in step S2 of this embodiment include the UAV position, attitude angle and gimbal attitude angle, and the camera internal parameters specifically include focal length, field of view, sensor size and lens distortion parameters, etc.; during the UAV's ground target positioning process, the UAV attitude and gimbal data output by the airborne sensor are directly collected, and the camera internal parameters are obtained through pre-calibration of the camera or the official data manual.

[0035] In one possible implementation, step S2 of this embodiment obtains the pixel coordinates of the target in the image. Deep learning can be applied to perform target recognition and detection to output the pixel coordinates. Here, the coordinates are obtained manually to avoid errors in the pixel coordinates.

[0036] In one possible implementation, since the target is not necessarily at the center of the camera's field of view, step S3 of this embodiment utilizes a ground target localization model based on monocular vision, combining camera internal parameters with the pixel coordinates of the target in the image and the data from the UAV and gimbal, to calculate the azimuth and elevation angles of the target's line-of-sight vector. The deflection angles of the target's line-of-sight vector relative to the optical axis in the horizontal and vertical directions are calculated using the following formula:

[0037]

[0038] In the formula, , These represent the positions P1 corresponding to target P in the auxiliary coordinate system O1-X1Y1 of the image plane; , These represent the pixel coordinates of the target location; The size is one pixel; This is the deflection angle in the vertical direction; This refers to the horizontal deflection angle. Focal length; set up , These represent the pitch and yaw angles of the gimbal, respectively. Since the positions of the gimbal's rotation center and the camera's optical center are relatively fixed, therefore... , This is also equal to the camera's pitch and yaw angles; The optical center O of the camera is calculated using the following formula. c Deflection angle at the origin:

[0039] In the formula, The angle between the target's line-of-sight vector and the horizontal plane; The angle between the projection of the target's line-of-sight vector onto the horizontal plane and the due north direction.

[0040] In one possible implementation, step S3 of this embodiment calculates the three-dimensional coordinates of the target in the NED coordinate system based on the projection relationship between the coordinate systems as follows: In the NED coordinate system, the drone, camera, and target are in O v The projection of the AP plane is shown in Figure 2(a). Assuming the point where the UAV perpendicularly intersects the ground is A, and the target's position is P, the UAV, camera, and target are projected onto plane O. v The projection of the AP plane, in the NED coordinate system, is based on the UAV's centroid O. v and camera optical center O c The distance is L The distance between the optical center and the target on the horizontal plane is calculated as follows:

[0041]

[0042] In the formula, h The vertical distance between the UAV's center of mass and the camera's optical center; β The pitch angle of the UAV is equal to the angle between the X-axis of the aircraft's coordinate system and the horizontal plane, with upward pitch being positive. η Connect the center of mass of the UAV and the optical center of the camera to the horizontal plane X. v O v Y v The included angle is negative; H This refers to the relative altitude of the drone above the ground. Drones, cameras, and targets in X v O v Y v The projection of the plane is shown in Figure 2(b), combining the UAV, camera, and target in the X... v O v Y v The projection of the plane onto the target, in the NED coordinate system, gives the following three-dimensional coordinates:

[0043] In the formula, γ The yaw angle of the UAV is equal to the angle between the X-axis of the aircraft's coordinate system and the due north direction.

[0044] In one possible implementation, step S4 of this embodiment introduces the Earth's average radius parameter, combined with the UAV's geodetic coordinates (B... uav Luav H uav The process of converting the target's three-dimensional coordinates in the NED coordinate system to coordinates in the geodetic coordinate system includes the following steps: According to the latitude B of the drone uav Longitude L uav and elevation H uav And the target's three-dimensional coordinates in the NED coordinate system ( x v , y v , z v The target's coordinates in the geodetic coordinate system are calculated using the following formula:

[0045] In the formula, R is the average radius of the Earth.

[0046] In one possible implementation, if the terrain near the target location is complex, and the altitude of the UAV takeoff point differs from that of the target, continuing to use positioning methods for flat ground will result in significant positioning errors. Therefore, based on the aforementioned positioning theory, this invention proposes a positioning method for non-flat terrain, specifically including: Please see Figure 3 Obtain the vertical distance h from the target location to the horizontal plane of the UAV takeoff point. p The drone's relative altitude to the horizontal plane is H. The drone takes the first shot, including the target. Then, the drone's position is arbitrarily changed, and the attitude angles of the drone and gimbal are adjusted to take a second shot. By changing the drone's position twice to shoot the target, the drone's takeoff point is the same in both shots, therefore h... p As a fixed value, the target's coordinates in the NED coordinate system are:

[0047] Converting the NED coordinate system to the geodetic coordinate system, the latitude and longitude of the target in the two photographs are equal, which meets the following requirements:

[0048] In the formulas, both the latitude and longitude equations contain only one unknown, h. p The vertical distance from the target position to the horizontal plane of the UAV takeoff point is obtained by calculating the average value of each value. The position of the target is obtained when the horizontal plane of the UAV takeoff point and the target altitude are different by taking information from any one of the photos.

[0049] Please see Figure 4 The following examples will further illustrate and introduce the UAV ground target localization method based on monocular vision according to the embodiments of the present invention.

[0050] The DJI Phantom 4 RTK was selected as the drone platform, with the image center point coordinates as u. x =2726, u y =1845; focal length f=8.8; pixel size l=2.41228. Taking a book as the target object, the drone takes a picture of the target location and analyzes its latitude and longitude as the target's real location, with coordinates (34.23465164, 108.95246510, 379.14).

[0051] Acquire the latitude and longitude of the drone, its altitude, the drone's attitude angle, and the gimbal's attitude angle during the shooting process.

[0052] Using Photoshop, we can obtain its coordinates in the pixel coordinate system as (2876, 1852).

[0053] Calculate the azimuth and elevation angles of the target's line-of-sight vector. The elevation angle is -63.91° and the azimuth angle is 79.45°.

[0054] Based on the projection relationship between the coordinate systems, the three-dimensional coordinates of the target in the NED coordinate system are calculated as (-2.16, -11.50, 23.76).

[0055] The target's coordinates in the NED coordinate system were converted to the geodetic coordinate system. The converted latitude, longitude and altitude are (34.23464957, 108.95246735, 379.07).

[0056] The final calculation showed that the error between the calculated target point and the actual location was 0.3094m.

[0057] Therefore, it can be seen that the UAV ground target localization method based on monocular vision in the embodiments of the present invention has high accuracy in localizing ground targets for a single UAV, and combined with the kill range of artillery shells, it is suitable for application in modern warfare environments.

[0058] Another embodiment of the present invention also proposes a UAV ground target localization system based on monocular vision, comprising: The monocular vision positioning model construction module is used to establish the NED coordinate system, body coordinate system, camera coordinate system and pixel coordinate system in the northeast, and set the image plane auxiliary coordinate system and the ground auxiliary coordinate system to obtain the UAV positioning model of ground target based on monocular vision. The data acquisition module is used to acquire the drone pose and gimbal attitude data, and extract the pixel coordinates of the target in the image from the drone aerial image; The NED coordinate calculation module is used to calculate the azimuth and elevation angles of the target's line-of-sight vector by using a ground target localization model based on monocular vision UAV, combined with camera internal parameters, UAV and gimbal data and the pixel coordinates of the target in the image; and, based on the projection relationship between the coordinate systems, to calculate the three-dimensional coordinates of the target in the NED coordinate system. The geodetic coordinate transformation module is used to convert the target's three-dimensional coordinates in the NED coordinate system to coordinates in the geodetic coordinate system, thus completing the positioning.

[0059] Another embodiment of the present invention provides a computer-readable storage medium storing at least one instruction, which is executed by a processor in an electronic device to implement the monocular vision-based UAV ground target localization method.

[0060] For example, the instructions stored in the memory can be divided into one or more modules / units. These modules / units are stored in a computer-readable storage medium and executed by the processor to complete the UAV ground target localization method based on monocular vision according to embodiments of the present invention. The one or more modules / units can be a series of computer-readable instruction segments capable of performing specific functions, which describe the execution process of the computer program on the server.

[0061] The electronic device may be a smartphone, laptop, PDA, or cloud server, among other computing devices. It may include, but is not limited to, a processor and memory. Those skilled in the art will understand that the electronic device may also include more or fewer components, or combinations of certain components, or different components; for example, it may also include input / output devices, network access devices, buses, etc.

[0062] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0063] The memory can be an internal storage unit of the server, such as the server's hard drive or memory. The memory can also be an external storage device of the server, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or FlashCard.

[0064] Furthermore, the memory may include both internal storage units of the server and external storage devices. The memory is used to store the computer-readable instructions and other programs and data required by the server. The memory can also be used to temporarily store data that has been output or will be output.

[0065] It should be noted that the information interaction and execution process between the above-mentioned module units are based on the same concept as the method embodiment. For details on their specific functions and technical effects, please refer to the method embodiment section. They will not be repeated here.

[0066] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0067] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to a photographing device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks.

[0068] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0069] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for UAV ground target localization based on monocular vision, characterized in that, include: Establish the NED coordinate system, body coordinate system, camera coordinate system and pixel coordinate system in the northeast, and set up the image plane auxiliary coordinate system and the ground auxiliary coordinate system to obtain the UAV localization model of ground target based on monocular vision; Acquire drone pose and gimbal attitude data, and extract the pixel coordinates of targets in the drone aerial images; Using a ground target localization model based on monocular vision UAV, combined with camera internal parameters and UAV and gimbal data and the pixel coordinates of the target in the image, the azimuth and elevation angles of the target's line of sight vector are calculated. Furthermore, based on the projection relationships between the various coordinate systems, the three-dimensional coordinates of the target in the NED coordinate system are calculated; The target's three-dimensional coordinates in the NED coordinate system are converted to coordinates in the geodetic coordinate system to complete the positioning.

2. The UAV ground target localization method based on monocular vision according to claim 1, characterized in that: The NED coordinate system is established with the UAV's center of mass as the origin, the X-axis pointing due north, the Y-axis pointing due east, and the Z-axis perpendicular to the XOY plane pointing downwards towards the ground, denoted as O. v -X v Y v Z v ; The body coordinate system is established with the UAV's center of mass as its origin and is fixedly connected to the UAV. The X-axis is parallel to the body axis and points towards the nose; the Y-axis is perpendicular to the UAV's vertical plane of symmetry and points towards the right side of the fuselage; and the Z-axis is established in the UAV's vertical plane of symmetry, perpendicular to the X-axis and pointing downwards towards the fuselage, denoted as O. b -X b Y b Z b ; The camera coordinate system is established with the camera optical center as the origin, the Z-axis pointing forward along the camera optical axis, the X-axis parallel to the horizontal axis of the image physical coordinate system pointing to the right, and the Y-axis parallel to the horizontal axis of the image physical coordinate system pointing downward, denoted as O. c -X c Y c Z c ; The pixel coordinate system is established with the top left corner of the image as the origin, the X-axis pointing to the right along the top edge of the image, and the Y-axis pointing downwards along the left edge of the image, denoted as O. u -X u Y u .

3. The UAV ground target localization method based on monocular vision according to claim 2, characterized in that, The steps for setting the image plane auxiliary coordinate system and the ground auxiliary coordinate system include: For the auxiliary coordinate system of image plane M1, the origin O1 is the intersection of the optical axis and image plane M1. x u y The X-axis is parallel to the pixel coordinate system and points to the right, and the Y-axis is parallel to the pixel coordinate system and points upward, denoted as O1-X1Y1; For the ground-based auxiliary coordinate system M2, the origin O2 is the intersection of the optical axis and the ground M2. The X-axis and Y-axis are parallel to and opposite to the Y-axis and X-axis of the image plane M1, respectively, and are denoted as O2-X2Y2.

4. The UAV ground target localization method based on monocular vision according to claim 3, characterized in that, The steps of using a ground target localization model based on monocular vision UAV, combined with camera internal parameters and the pixel coordinates of the target in the UAV and gimbal data and image, to calculate the azimuth and elevation angles of the target's line-of-sight vector include: The deflection angles of the target's line-of-sight vector relative to the optical axis in the horizontal and vertical directions are calculated using the following formula: In the formula, , These represent the positions P1 corresponding to target P in the auxiliary coordinate system O1-X1Y1 of the image plane; , These represent the pixel coordinates of the target location; The size is one pixel; This is the deflection angle in the vertical direction; This refers to the horizontal deflection angle. Focal length; set up , These represent the pitch and yaw angles of the gimbal, respectively. Since the positions of the gimbal's rotation center and the camera's optical center are relatively fixed, therefore... , This is also equal to the camera's pitch and yaw angles; The optical center O of the camera is calculated using the following formula. c Deflection angle at the origin: In the formula, The angle between the target's line-of-sight vector and the horizontal plane; The angle between the projection of the target's line-of-sight vector onto the horizontal plane and the due north direction.

5. The UAV ground target localization method based on monocular vision according to claim 4, characterized in that, The calculation of the target's three-dimensional coordinates in the NED coordinate system based on the projection relationship between the coordinate systems includes the following steps: Assume the point where the drone perpendicularly intersects the ground is A, and the target's position is P. Considering the drone, camera, and target at point O... v The projection of the AP plane, in the NED coordinate system, is based on the UAV's centroid O. v and camera optical center O c The distance is L The distance between the optical center and the target on the horizontal plane is calculated as follows: In the formula, h The vertical distance between the UAV's center of mass and the camera's optical center; β The pitch angle of the UAV is equal to the angle between the X-axis of the aircraft's coordinate system and the horizontal plane, with upward pitch being positive. η Connect the center of mass of the UAV and the optical center of the camera to the horizontal plane X. v O v Y v The included angle is negative; H This refers to the relative altitude of the drone above the ground. Combining drones, cameras, and targets in X v O v Y v The projection of the plane onto the target, in the NED coordinate system, gives the following three-dimensional coordinates: In the formula, γ The yaw angle of the UAV is equal to the angle between the X-axis of the aircraft's coordinate system and the due north direction.

6. The UAV ground target localization method based on monocular vision according to claim 5, characterized in that, The process of converting the target's three-dimensional coordinates in the NED coordinate system to coordinates in the geodetic coordinate system includes the following steps: According to the latitude B of the drone uav Longitude L uav and elevation H uav And the target's three-dimensional coordinates in the NED coordinate system ( x v , y v , z v The target's coordinates in the geodetic coordinate system are calculated using the following formula: In the formula, R is the average radius of the Earth.

7. The UAV ground target localization method based on monocular vision according to claim 6, characterized in that, This also includes obtaining the vertical distance h from the target location to the drone's takeoff point (horizontal plane) when the target's altitude differs from the drone's takeoff point's altitude. p The drone's relative altitude to the horizontal plane is H. The target is photographed by changing the drone's position twice, with the drone's takeoff point being the same in both shots. p As a fixed value, the target's coordinates in the NED coordinate system are: Converting the NED coordinate system to the geodetic coordinate system, the latitude and longitude of the target in the two photographs are equal, which meets the following requirements: The vertical distance from the target location to the horizontal plane of the UAV takeoff point is obtained by calculating the average value. The target's position is obtained when the horizontal plane of the UAV takeoff point and the target's altitude are different, by taking information from any one of the photos.

8. The UAV ground target localization method based on monocular vision according to claim 1, characterized in that, The drone and gimbal data include the drone's position, attitude angle, and gimbal attitude angle. The camera's internal parameters include focal length, field of view, sensor size, and lens distortion parameters.

9. A ground target localization system for unmanned aerial vehicles (UAVs) based on monocular vision, characterized in that, include: The monocular vision positioning model construction module is used to establish the NED coordinate system, body coordinate system, camera coordinate system and pixel coordinate system in the northeast, and set the image plane auxiliary coordinate system and the ground auxiliary coordinate system to obtain the UAV positioning model of ground target based on monocular vision. The data acquisition module is used to acquire the drone pose and gimbal attitude data, and extract the pixel coordinates of the target in the image from the drone aerial image; The NED coordinate calculation module is used to calculate the azimuth and elevation angles of the target's line-of-sight vector by using a ground target localization model based on monocular vision UAV, combined with camera internal parameters, UAV and gimbal data and the pixel coordinates of the target in the image; and, based on the projection relationship between the coordinate systems, to calculate the three-dimensional coordinates of the target in the NED coordinate system. The geodetic coordinate transformation module is used to convert the target's three-dimensional coordinates in the NED coordinate system to coordinates in the geodetic coordinate system, thus completing the positioning.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, which is executed by a processor in an electronic device to implement the monocular vision-based UAV ground target localization method as described in any one of claims 1 to 8.

Images