A tilt photography method for real-time target recognition and route generation of a UAV

Abstract

The application discloses a tilt photography method for real-time target identification and route generation of a UAV, relates to the technical field of identification and tracking, and comprises the following steps: high-altitude cruising of the UAV, target identification and tracking of the UAV, target positioning of the UAV, self-adaptive generation of a flight range, and automatic cruising of the UAV at a low altitude. The application enables the UAV to cruise, identify and track a target in a large range and a long distance, and effectively reduces the time and effort required for the UAV to cruise and search for a moving target in a large range and to move for a long time, thereby saving time and effort and being used for identifying and tracking a moving target.

Description

Technical Field

[0001] This invention relates to the field of identification and tracking technology, and in particular to an oblique photography method for real-time target identification and flight path generation of unmanned aerial vehicles (UAVs). Background Technology

[0002] The use of drones is becoming increasingly widespread. When using drones for inspections to observe and collect data on moving targets, the problem is that these targets are not fixed in one place for long periods and are constantly moving. Therefore, it is impossible for drones to directly reach the precise location of a fixed target. Currently, the common method is for staff to manually operate the drone or set a flight path for multiple patrols to first locate the target, and then observe and collect data from the specific location. However, when the distance between the drone's takeoff point and the moving target is too great, resulting in an excessively large search area, locating the moving target initially requires a significant amount of time and effort. Furthermore, multiple trips by the drone can easily cause the moving target to leave its original position. Summary of the Invention

[0003] To address the problems of current drones not being able to cruise, identify, and track targets in large-scale, long-distance scenarios, being time-consuming, and easily losing targets, and being unable to automatically generate flight paths and perform oblique photography around moving targets, this invention provides an oblique photography method for real-time target identification and flight path generation for drones.

[0004] To address the aforementioned technical problems, this invention employs the following technical solution: a method for real-time target recognition and flight path generation using oblique photography for unmanned aerial vehicles (UAVs), comprising the following steps: S1 The UAV performs high-altitude automatic cruise based on a defined flight path at a set flight altitude and completes adaptive oblique photography. S2 The UAV uses its onboard target recognition and tracking model to identify and track targets based on images obtained from oblique photography. S3 After identifying a target, the UAV obtains the pixel coordinates of the corresponding image and converts them into latitude and longitude coordinates according to the world coordinate system. S4 Based on the obtained target coordinate point set, the UAV's cruise range is obtained using a constraint model, convex hull algorithm, and minimum envelope rectangle algorithm. S5 Waypoints are adaptively generated based on the UAV's cruise range and position information, and waypoint tasks are designed for low-altitude automatic cruise of the UAV, during which the target is photographed.

[0005] Furthermore, in S1, the drone is equipped with embedded airborne equipment, optical sensors, a gimbal, a camera, and an onboard computer. The onboard computer is connected to the embedded airborne equipment, optical sensors, gimbal, and camera for control. The onboard computer has an SDK module and a widget remote control component. The SDK module is started to transmit information, and the widget remote control component is used to start the control service. The cruise altitude is set between 50m and 70m to define the route for the drone to cruise automatically.

[0006] Furthermore, in S2, data preprocessing, training of the target recognition model, and construction of the target tracking method are performed to obtain the target recognition and tracking model, and the target recognition and tracking model is deployed to the embedded airborne device.

[0007] Furthermore, the data preprocessing involves capturing multi-angle target images at a cruising altitude of 50m using oblique photography, labeling the targets to construct a dataset, and dividing the dataset into training, validation, and test sets for the recognition model in an 8:1:1 ratio. A YOLOv10 model is used, with MobileNet V3 and GSConv modules added for model training to obtain pt weight files. These weight files are then converted to an ONNX structure, and the TensorRT 8.5.0 application library is deployed on an embedded airborne device. A lightweight ONNX structure using int8 is used as the engine structure, and the recognition model is deployed on the embedded airborne device. During cruising, the model continuously performs model recognition on the camera stream. The target tracking method employs a sorting approach, obtaining detectors by associating target detection results. For each tracked target, a Kalman filter is used to predict its position, thus completing state prediction. The Hungarian algorithm is used to match the predicted results with the detection results using IOU for data association. Successful matches update the Kalman filter state, creating new trackers and deleting lost trackers to complete the tracking of detected targets.

[0008] Furthermore, in S3, the target recognition and tracking model runs continuously during the drone's cruise, and the mounted optical sensor is always perpendicular to the ground. Only the drone's motion state needs to be considered. When the drone recognizes a target through the camera, it obtains the pixel coordinates of the corresponding image. At the same time, it subscribes to drone information and performs coordinate transformation based on the obtained drone position information, transforming the pixel coordinate system to the drone's WGS84 world coordinate system, thus converting the pixel coordinates into latitude and longitude coordinates.

[0009] Furthermore, where the pixel coordinates are the center coordinates of the target bounding box identified during the UAV's cruise, and the center coordinates of the bounding box are taken as the target's pixel coordinates P(u,v), combined with the origin O(u0,v0) of the image coordinate system, we have:

[0010] x=(u-u0)*d x;

[0011] y = (v - v0) * d y ;

[0012] In the formula, d x The pixel size corresponding to the x-axis is obtained from the camera's intrinsic parameters; d y The pixel size corresponding to the y-axis is obtained from the camera's intrinsic parameters;

[0013] The image coordinates P(x,y) are calculated using the origin O(u0,v0) of the image coordinate system. The target camera coordinates can be calculated from the image coordinates P(x,y). Then, the UAV world coordinates are obtained through the rotation matrix R. Finally, the target position is calculated based on the deviation between the latitude and longitude lines and the UAV position coordinates.

[0014] The formula for converting image coordinates to target camera coordinates is:

[0015]

[0016] In the formula, f a H is the camera's focal length; H is the drone's altitude above the ground; O is the origin of the image coordinate system; O C O is the center of the camera coordinate system; W P1 represents the center of the UAV after the sensor is projected onto the ground; P1 represents the position of the target on the imaging plane; X P The coordinates of the target's position on the ground projection; Y is the x-coordinate of the target on the imaging plane; p The vertical coordinate of the target on the imaging plane;

[0017] The formula for the rotation matrix R is:

[0018]

[0019]

[0020] In the formula, x E The x-coordinate of the rotated coordinate system; y E Z is the ordinate of the rotated coordinate system. E Here are the height coordinates after rotation; ∝ is the yaw angle; θ is the pitch angle. x is the roll angle; p y is the x-coordinate of the coordinate system before rotation; p z is the ordinate of the coordinate system before rotation; p These are the height coordinates before rotation;

[0021]

[0022] In the formula, N is the radius of curvature of the Earth's geoid (or reference ellipsoid) at a certain point; a is the Earth's semi-major axis; and f is the Earth's oblateness. It is the radian representation of a given latitude;

[0023]

[0024] In the formula, m perdelat This represents the ground distance corresponding to a unit change in latitude.

[0025]

[0026] In the formula, m perdelon This represents the ground distance corresponding to a unit change in longitude.

[0027] object_lat=drone_lat+Δlat;

[0028] object_lon=drone_lon+Δlon;

[0029] In the formula, object_lat is the latitude of the target; object_lon is the longitude of the target; drone_lat is the latitude of the drone; drone_lon is the longitude of the drone; Δlat is the latitude position correction of the target relative to the drone; and Δlon is the longitude position correction of the target relative to the drone.

[0030] Furthermore, in S4, the target coordinate point set is obtained, a constraint model is constructed to reduce redundant points, and then the convex hull algorithm is used to obtain the minimum area polygon for the filtered points. Finally, the minimum envelope rectangle method is used to obtain the drone's cruising range with maximum efficiency.

[0031] The constraint model mainly includes cluster analysis to merge redundant points, setting target recognition confidence threshold, spatial clustering constraints, calculating the distance between each point after positioning, merging points into one point if the distance between points is less than 1m, storing the recognition confidence, filtering out points with confidence below 0.6, traversing all points, using the Haversine formula to calculate the distance d between each point, and storing the coordinates of other points within a radius of 10m.

[0032] The Haversine formula is:

[0033]

[0034] In the formula, λ1 represents the latitude of a known point A on the Earth's surface; λ2 represents the longitude of a known point A on the Earth's surface. λ2 is the latitude of a known point B on the Earth's surface; R is the Earth's radius, and the Earth's average radius is R = 6371 km.

[0035] Furthermore, the convex hull algorithm selects the point with the smallest latitude in the target point set as the starting point, sorts all other points according to their polar angle with the pole, and adds the points to the convex hull using a stack. For each new point added, it checks whether the current point causes the polygon to "turn right". If a "turn" occurs, the redundant point is removed until all points are processed. After completion, all vertices of the convex hull are arranged in counterclockwise order.

[0036] Furthermore, the minimum envelope rectangle method is as follows: traverse all edges of the convex hull, calculate the extreme values ​​of the edge and its normal and each point in its direction, calculate the area, compare the areas of all rectangles, record the vertices of the rectangle with the smallest area, and use the space enclosed by the vertices as the flight range of the route.

[0037] Furthermore, in S5, after the drone returns to the starting point after the cruise ends, the drone is set with low-altitude flight altitude, heading, and lateral overlap rate. Then, the ground length and width corresponding to the actual photo are calculated based on the set flight altitude, and the waypoint positions are calculated based on the overlap rate. After obtaining the waypoint positions, an action is set for each waypoint to control the gimbal rotation and drive the camera to take pictures. After the drone uploads the waypoint task, it starts automatic cruise to take pictures of the target.

[0038] The beneficial effects of this invention are as follows: This invention can identify and track moving targets using a YOLOv10 model deployed on a UAV, and save the corresponding pixels of the identified moving targets. It uses the target tracking algorithm sort to avoid repeatedly recording the pixel coordinates of the moving targets, and calculates the latitude and longitude coordinates of the moving targets based on information such as the UAV's position, flight attitude, and altitude above the ground. Then, it uses the convex hull algorithm to obtain the minimum envelope rectangle, i.e., the rectangular flight path range, designs waypoint actions, and automatically generates waypoints using the vertex coordinates of the rectangular range according to the set overlap rate. After the high-altitude cruise is completed, a low-altitude cruise is performed. During the cruise, waypoint actions are performed at the waypoints to adaptively complete the oblique photography of the target.

[0039] This invention enables drones to cruise, identify, and track targets over large areas and long distances without requiring on-site inspections by personnel. Target identification can be performed through oblique photography during the cruise, significantly reducing the need for drones to search for moving targets over large areas and for long-term tracking of moving targets. This saves time and effort and is used for identifying and tracking moving targets. Attached Figure Description

[0040] Figure 1The diagram shown is a flowchart of one embodiment of the present invention.

[0041] Figure 2 The diagram shows the structure of the improved YOLOv10 model.

[0042] Figure 3 The diagram shows the transformation relationship between the pixel coordinate system and the image coordinate system.

[0043] Figure 4 The diagram shows the transformation relationship between the camera coordinate system and the UAV world coordinate system.

[0044] Figure 5 The diagram shows the mapping relationship between the image coordinate system and the UAV coordinate system.

[0045] Figure 6 The diagram shows the implementation principle of the convex hull algorithm.

[0046] Figure 7 The image shows a waypoint map within a rectangular area. Detailed Implementation

[0047] An oblique photography method for real-time target recognition and flight path generation for unmanned aerial vehicles (UAVs) includes the following steps: S1 The UAV performs automatic high-altitude cruise based on a defined flight path at a set flight altitude and completes adaptive oblique photography. S2 The UAV uses its onboard target recognition and tracking model to identify and track targets based on the images obtained from the oblique photography. S3 After identifying a target, the UAV obtains the pixel coordinates of the corresponding image and converts them to latitude and longitude coordinates according to the world coordinate system. S4 Based on the obtained target coordinate point set, the UAV's cruise range is obtained through a constraint model, convex hull algorithm, and minimum envelope rectangle algorithm. S5 Waypoints are adaptively generated based on the UAV's cruise range and position information. Waypoint tasks are designed for automatic low-altitude cruise of the UAV, and target photography is performed.

[0048] Furthermore, in S1, the drone is equipped with embedded airborne equipment, optical sensors, a gimbal, a camera, and an onboard computer. The onboard computer is connected to the embedded airborne equipment, optical sensors, gimbal, and camera for control. The onboard computer has an SDK module and a widget remote control component. The SDK module is started to transmit information, and the widget remote control component is used to start the control service. The cruise altitude is set between 50m and 70m to define the route for the drone to cruise automatically.

[0049] Furthermore, in S2, data preprocessing, training of the target recognition model, and construction of the target tracking method are performed to obtain the target recognition and tracking model, and the target recognition and tracking model is deployed to the embedded airborne device.

[0050] Furthermore, the data preprocessing involves taking multi-angle target images at a cruising altitude of 50m using oblique photography, labeling the targets to construct a dataset, and dividing the dataset into training, validation, and test sets for the recognition model in an 8:1:1 ratio. A YOLOv10 model is used, with MobileNet V3 and GSConv modules added for model training to obtain pt weight files. These weight files are then converted to an ONNX structure. The TensorRT 8.5.0 application library is deployed on an embedded airborne device, using a lightweight int8 ONNX structure as the engine. The recognition model is deployed on the embedded airborne device, and model recognition is continuously performed on the camera stream during cruising. The target tracking method employs the sort method, obtaining detectors by associating target detection results. For each tracked target, a Kalman filter is used to predict its position, thus completing state prediction. The Hungarian algorithm is used to match the predicted and detected results using IOU for data association. Successful matches update the Kalman filter state, creating new trackers and deleting lost trackers to complete the tracking of detected targets.

[0051] Furthermore, in S3, the target recognition and tracking model is always running during the drone's cruise, and the mounted optical sensor is always perpendicular to the ground. When the drone recognizes a target through the camera, it obtains the pixel coordinates of the corresponding image and subscribes to the drone's position information. Based on the obtained drone position information, coordinate transformation is performed to convert the pixel coordinate system to the drone's WGS84 world coordinate system, thus converting the pixel coordinates into latitude and longitude coordinates.

[0052] Furthermore, where the pixel coordinates are the center coordinates of the target bounding box identified during the UAV's cruise, and the center coordinates of the bounding box are taken as the target's pixel coordinates P(u,v), combined with the origin O(u0,v0) of the image coordinate system, we have:

[0053] x=(u-u0)*d x ;

[0054] y = (v - v0) * d y ;

[0055] In the formula, d x The pixel size corresponding to the x-axis is obtained from the camera's intrinsic parameters; d y The pixel size corresponding to the y-axis is obtained from the camera's intrinsic parameters;

[0056] The image coordinates P(x,y) are calculated using the origin O(u0,v0) of the image coordinate system. The target camera coordinates can be calculated from the image coordinates P(x,y). Then, the UAV world coordinates are obtained through the rotation matrix R. Finally, the target position is calculated based on the deviation between the latitude and longitude lines and the UAV position coordinates.

[0057] The formula for converting image coordinates to target camera coordinates is:

[0058]

[0059] In the formula, f a H is the camera's focal length; H is the drone's altitude above the ground; O is the origin of the image coordinate system; O C O is the center of the camera coordinate system; W P1 represents the center of the UAV after the sensor is projected onto the ground; P1 represents the position of the target on the imaging plane; X P The coordinates of the target's position on the ground projection; Y is the x-coordinate of the target on the imaging plane; P The vertical coordinate of the target on the imaging plane;

[0060] The formula for the rotation matrix R is:

[0061]

[0062] In the formula, x E The x-coordinate of the rotated coordinate system; y E Z is the ordinate of the rotated coordinate system. E Here are the height coordinates after rotation; ∝ is the yaw angle; θ is the pitch angle. x is the roll angle; p y is the x-coordinate of the coordinate system before rotation; p z is the ordinate of the coordinate system before rotation; p These are the height coordinates before rotation;

[0063]

[0064] In the formula, N is the radius of curvature of the geoid (or reference ellipsoid) at a certain point; a is the semi-major axis of the Earth; and f is the flattening of the Earth. It is the radian representation of a given latitude;

[0065]

[0066] In the formula, m perdelat This represents the ground distance corresponding to a unit change in latitude.

[0067]

[0068] In the formula, m perdelon This represents the ground distance corresponding to a unit change in longitude.

[0069] object_lat=drone_lat+Δlat;

[0070] object_lon=drone_lon+Δlon;

[0071] In the formula, object_lat is the latitude of the target; object_lon is the longitude of the target; drone_lat is the latitude of the drone; drone_lon is the longitude of the drone; Δlat is the latitude position correction of the target relative to the drone; Δlon is the longitude position correction of the target relative to the drone.

[0072] Furthermore, in S4, the target coordinate point set is obtained, a constraint model is constructed to reduce redundant points, and then the convex hull algorithm is used to obtain the minimum area polygon for the filtered points. Finally, the minimum envelope rectangle method is used to obtain the drone's cruising range with maximum efficiency.

[0073] The constraint model mainly includes cluster analysis to merge redundant points, setting target recognition confidence threshold, spatial clustering constraints, calculating the distance between each point after positioning, merging points into one point if the distance between points is less than 1m, storing the recognition confidence, filtering out points with confidence below 0.6, traversing all points, using the Haversine formula to calculate the distance d between each point, and storing the coordinates of other points within a radius of 10m.

[0074] The Haversine formula is:

[0075]

[0076] In the formula, λ1 represents the latitude of a known point A on the Earth's surface; λ2 represents the longitude of a known point A on the Earth's surface. λ2 is the latitude of a known point B on the Earth's surface; R is the Earth's radius, and the Earth's average radius is R = 6371 km.

[0077] Furthermore, the convex hull algorithm selects the point with the smallest latitude in the target point set as the starting point, sorts all other points according to their polar angle with the pole, and adds the points to the convex hull using a stack. For each new point added, it checks whether the current point causes the polygon to "turn right". If a "turn" occurs, the redundant point is removed until all points are processed. After completion, all vertices of the convex hull are arranged in counterclockwise order.

[0078] Furthermore, the minimum envelope rectangle method is as follows: traverse all edges of the convex hull, calculate the extreme values ​​of the edge and its normal and each point in its direction, calculate the area, compare the areas of all rectangles, record the vertices of the rectangle with the smallest area, and use the space enclosed by the vertices as the flight range of the route.

[0079] Furthermore, in S5, after the drone returns to the starting point after the cruise ends, the drone is set with low-altitude flight altitude, heading, and lateral overlap rate. Then, the ground length and width corresponding to the actual photo are calculated based on the set flight altitude, and the waypoint positions are calculated based on the overlap rate. After obtaining the waypoint positions, an action is set for each waypoint to control the gimbal rotation and drive the camera to take pictures. After the drone uploads the waypoint task, it starts automatic cruise to take pictures of the target.

[0080] This invention discloses an oblique photography method for real-time target recognition and flight path generation of unmanned aerial vehicles (UAVs). An embodiment of this invention is described in detail below with reference to the accompanying drawings.

[0081] like Figure 1 As shown, the first step is high-altitude drone cruise. The drone is equipped with embedded airborne equipment, optical sensors, a gimbal, a camera, and an onboard computer. The onboard computer is connected to the embedded airborne equipment, optical sensors, gimbal, and camera for control. The embedded airborne equipment connects to the drone via USB-TTL and USB-USBC for information exchange. After the drone is powered on, the onboard computer starts up. The onboard computer connects to the drone via the SDK and controls the drone through the widget desktop component on the remote controller. The widget component on the remote controller is developed using the SDK and can realize functions such as target recognition and tracking, and drone flight control by calling the defined flight control code. The drone can use manual remote control or pre-planned route automatic cruise, among other cruise methods. The route can be planned based on historical data and human experience. The drone's cruise altitude range is between 50m and 70m, and it is adapted to build a target recognition model with a recognition altitude range of 50m to 70m.

[0082] The second step is drone target recognition. Model preparation: Data preprocessing involves setting up multi-angle target images taken by oblique photography at a cruising altitude between 50m and 70m, labeling the targets, and constructing a dataset. This dataset is divided into training, validation, and test sets for the recognition model in an 8:1:1 ratio. For example... Figure 2As shown in the diagram, Input represents the input layer, Backbone represents the backbone network, Neck represents the neck, Head represents the detection head, Conv represents the convolutional layer, SCDown represents spatial compression downsampling, UpSample represents upsampling, Concat represents concatenation for feature fusion, C2fCIB represents the improved convolutional module, SPPF represents the pooling layer, PSA represents the improved convolutional module, and C2f represents the improved convolutional module. The YOLOv10 model is used for target recognition, and MobileNet V3 and GSConv modules are added to achieve smooth deployment of the UAV, improving information extraction capabilities and model accuracy while reducing computational costs. After model training, a pt-type weight file is obtained. To achieve real-time recognition, the TensorRT architecture is used for deployment, and the weight file format is converted. Model structure conversion: First, after training, the file structure is converted to ONNX structure. Then, the TensorRT 8.5.0 application library is deployed on the embedded airborne device, using the official library's trtexec application and the int8 lightweight ONNX to the engine structure. Target Tracking: The `sort` method is used to track targets. Detectors are obtained by correlating target detection results, and bounding boxes are derived from these detectors. Kalman filtering is used to predict the positions of existing trackers, and the Hungarian algorithm is used to perform IOU matching between predicted and detected bounding boxes. Successfully matched detection results update the Kalman filter state; unmatched results create new trackers, while unmatched trackers are deleted, thus achieving real-time target tracking. During tracker construction, a newly created tracker represents a new target, storing its pixel position and UAV information. TensorRT is deployed in conjunction with the model engine structure. The application after deployment is as follows: the `.engine` model file is loaded, the UAV's real-time camera stream is acquired, and the acquired stream data is formatted into shapes and types suitable for inference. TensorRT is used to perform inference and obtain the inference results. The inferred bounding boxes and confidence scores are processed using the `sort` method. Trackers are set up, matching and unmatched trackers are built, tracker predictions are continuously performed, and the two are matched. If multiple frames are unmatched, the corresponding trackers are deleted, releasing occupied resources. After the UAV takes off, the recognition module is activated via a widget on the remote controller to complete real-time target recognition during the UAV's cruise process.

[0083] The third step is drone target localization. During the drone's cruise, the target recognition module operates continuously, and the onboard optical sensor always captures images perpendicularly. When the lens identifies a target, it acquires the pixel coordinates of the corresponding image. Simultaneously, it subscribes to drone information, performing coordinate transformation based on the acquired drone position. This transformation converts the pixel coordinates to the drone's WGS84 world coordinate system using the correspondence between coordinate systems, converting the pixel coordinates to latitude and longitude coordinates. The subscribed drone information includes the drone's geographical coordinates (e.g., longitude and latitude), flight attitude (e.g., yaw, pitch, roll), ground altitude, elevation, gimbal orientation, and drone attitude. Combined with... Figure 3 , Figure 4 as well as Figure 5 As shown, the acquisition of the UAV's WGS84 world coordinate system involves processes such as converting the pixel coordinate system to the image coordinate system, converting the image coordinate system to the camera coordinate system, and converting the camera coordinate system to the UAV's WGS84 world coordinate system.

[0084] The pixel coordinate system is transformed into the image coordinate system. During drone navigation, a target bounding box is generated upon target detection. The center coordinates of the bounding box are taken as the target's pixel coordinates P(u,v). Combined with the image coordinate system origin O(u0,v0), the following holds:

[0085] x=(u-u0)*d x ;

[0086] y = (v - v0) * d y ;

[0087] In the formula, d x The pixel size corresponding to the x-axis is obtained from the camera's intrinsic parameters; d y The pixel size corresponding to the y-axis is obtained from the camera's intrinsic parameters;

[0088] The image coordinates P(x,y) are calculated using the origin O(u0,v0) of the image coordinate system. The target camera coordinates can be calculated from the image coordinates P(x,y). Then, the UAV world coordinates are obtained through the rotation matrix R. Finally, the target position is calculated based on the deviation between the latitude and longitude lines and the UAV position coordinates.

[0089] The formula for converting image coordinates to target camera coordinates is:

[0090]

[0091] In the formula, f a H is the camera's focal length; H is the drone's altitude above the ground; O is the origin of the image coordinate system; O C O is the center of the camera coordinate system; W P1 represents the center of the UAV after the sensor is projected onto the ground; P1 represents the position of the target on the imaging plane; X PThe coordinates of the target's position on the ground projection; Y is the x-coordinate of the target on the imaging plane; P Let be the vertical coordinate of the target on the imaging plane.

[0092] After obtaining the target camera coordinates, the camera is in an orthogonal orientation, so its positive Z-axis is aligned with the body coordinates. To obtain the target world coordinates, only the yaw angle ∝ needs to be considered. The actual distance camera coordinates after offset can be obtained using the rotation matrix R. The formula for the rotation matrix R is:

[0093]

[0094] In the formula, x E The x-coordinate of the rotated coordinate system; y E Z is the ordinate of the rotated coordinate system. E Here are the height coordinates after rotation; ∝ is the yaw angle; θ is the pitch angle. x is the roll angle; p y is the x-coordinate of the coordinate system before rotation; p z is the ordinate of the coordinate system before rotation; p These are the height coordinates of the coordinates before rotation;

[0095] After obtaining the corrected longitude and latitude offset distances, they are converted into longitude and latitude differences in the two directions, and the longitude and latitude of the target can be calculated.

[0096]

[0097] In the formula, N is the radius of curvature of the geoid (or reference ellipsoid) at a certain point; a is the semi-major axis of the Earth; and f is the flattening of the Earth. It is the radian representation of a given latitude;

[0098]

[0099] In the formula, m perdelat This represents the ground distance corresponding to a unit change in latitude.

[0100]

[0101] In the formula, m perdelon This represents the ground distance corresponding to a unit change in longitude.

[0102] object_lat=drone_lat+Δlat;

[0103] object_lon=drone_lon+Δlon;

[0104] In the formula, object_lat is the latitude of the target; object_lon is the longitude of the target; drone_lat is the latitude of the drone; drone_lon is the longitude of the drone; Δlat is the latitude position correction of the target relative to the drone; Δlon is the longitude position correction of the target relative to the drone.

[0105] Store the converted latitude and longitude coordinates and the confidence value of the recognition box.

[0106] The fourth step is to generate the UAV flight path range. After the UAV completes its high-altitude cruise, target identification and localization are terminated via a widget component. Since the coordinates are continuously acquired during the identification process, a constraint model is designed to reduce redundant and false detection points. After obtaining valid points, a convex hull algorithm is used to obtain the minimum area polygon, and then a minimum area envelope rectangle algorithm is used to obtain the minimum area rectangle as the flight path range. The constraint model is mainly constrained in three aspects: cluster analysis to merge redundant points, setting a target identification confidence threshold, and spatial clustering constraints. Cluster analysis merges monitoring points within a 5m radius to optimize the number of points obtained. Confidence constraints set the minimum confidence threshold for model identification to 70%. Clustering constraints ensure that other targets exist within a 15m radius of the target.

[0107] After target identification and localization, the distance between each point is calculated. If the distance between each point is less than 1m, they are merged into one point, the confidence score of the identification is stored, and points with a confidence score lower than 0.6 are filtered out. At the same time, all points are traversed and only the coordinates of other points within a radius of 10m are stored.

[0108] After obtaining the coordinates of all points, the distance d between each point is calculated using the Haversine formula, which is:

[0109]

[0110] In the formula, λ1 represents the latitude of a known point A on the Earth's surface; λ2 represents the longitude of a known point A on the Earth's surface. λ2 is the latitude of a known point B on the Earth's surface; R is the Earth's radius, and the Earth's average radius is R = 6371 km.

[0111] Based on the results, determine whether there is an intersection within a distance of 10m and save the set of points that meets the requirements. Use the convex hull algorithm on the obtained set of points to find the polygon with the minimum area.

[0112] The convex hull algorithm selects the point with the smallest latitude in the target point set, i.e., the bottom left corner point P0, as the starting point. It sorts all other points in ascending order according to their polar angles from the pole. If the polar angles are the same, it selects the distance to P0 and sorts them in ascending order. It traverses the sorted point set from P0 and uses a stack to add points to the convex hull. For each new point added, it checks whether the current point causes the polygon to "turn right". If a "turn" occurs, the redundant point is removed. This continues until all points are processed. After completion, all vertices of the convex hull are arranged in counterclockwise order to obtain the convex hull point set.

[0113] Among them, such as Figure 6 As shown in the figure, the black rectangle S1 represents the smallest bounding rectangle with P0 and P3 as its bases, which is also the rectangle with the smallest area of ​​the polygon; the red rectangle S2 represents the smallest bounding rectangle with P0 and P5 as its bases; and the green rectangle S3 represents the smallest bounding rectangle with P5 and P3 as its bases. 11 The smallest bounding rectangle for the bottom edge; the purple rectangle S4 represents P9 and P. 11 The smallest bounding rectangle for the bottom edge is defined by the blue rectangle S5, which indicates that P9 and P3 are the smallest bounding rectangles for the bottom edge. For the obtained set of convex hull points, traverse all edges of the convex hull, calculate the extreme values ​​of the edge and its normal along each point in its direction, calculate the area, compare the areas of all rectangles, and record the vertex of the rectangle with the smallest area, which is the flight path range.

[0114] Step 5: Low-altitude drone auto-cruise. After obtaining the drone's flight range, low-altitude auto-cruise is performed based on the set overlap ratio and subscribed drone information. The drone's cruising altitude is set to 50m, maximum flight speed to 10m / s, automatic flight speed to 2m / s, lateral overlap ratio to 70%, and forward overlap ratio to 60%. The drone then performs a photo-taking task at the waypoint. Specifically, the drone information subscription obtains the current GPS / RTK position, and the actual distance to the ground corresponding to the drone's captured image is obtained based on the pinhole imaging principle. For example... Figure 7 As shown, the UAV automatically generates waypoints. Based on the acquired UAV flight range, the distances to the ground corresponding to the rectangle's length and width are calculated. The waypoint positions are then determined by combining the ground distances from the UAV imagery with the overlap ratio. The waypoint task is set to point-to-point shooting, with five gimbal rotation angles set for point-to-point shooting: pitch, yaw, and roll. The drone rotates in the following order: downwards at 45 degrees (-45,0,0), downwards at 45 degrees to the right (-45,90,0), downwards at 45 degrees to the rear (-45,180,0), downwards at 45 degrees to the left (-45,270,0), and downwards at (-90,0,0). The drone executes the flight path, flying and taking pictures according to the designed waypoints. After completion, a 3D scene of the target environment can be generated through 3D modeling.

[0115] Example: When using drones for patrol inspections to observe and collect data on spotted seals, the precise location of their habitat is difficult to determine because spotted seals are social animals with variable habitats. Currently, common methods for observing spotted seals involve staff manually controlling the drone or setting multiple flight paths to locate the habitat before conducting observations and data collection at that specific location. When the drone's takeoff point is too far from the habitat, resulting in an excessively large search area, locating the habitat initially often requires significant time and effort. Furthermore, the repeated drone trips can lead to spotted seals leaving their original habitat. The specific process is as follows:

[0116] The drone is equipped with an embedded device, Jetson Xavier NX. The ground station remotely connects to Jetson Xavier NX via a 4G network, starts the spotted seal recognition and flight path planning system of the device, runs the spotted seal recognition module, and achieves wide-range cruise by manual flight or setting a flight path. During the cruise, the drone automatically identifies targets, creates trackers for the recognition results of each frame in the drone image through the sort tracking algorithm, and uses Kalman filtering to predict the position. If the predicted box matches the tracking box in the subsequent frame image, it is the same target. If the tracker does not match for a long time, the target is no longer in the tracking range and the tracker is deleted to free up space. If a new tracking box is found, it indicates that a new target individual has been identified. The location of the identified target and the drone information are saved.

[0117] The target location is saved as pixel coordinates. Drone information includes the current relative altitude and drone coordinates. When drawing a detection box during drone recognition, the pixel coordinates of the two endpoints of the rectangular detection box (top left Q1, bottom right Q2) are recorded. The center of the rectangle (centerX, centerY) is calculated as the pixel center for recognizing the spotted seal.

[0118] centerX = (Q2(x) + Q1(x)) / 2;

[0119] centerY = (Q1(y) + Q2(y)) / 2;

[0120] The drone uses the Zenmuse P1 sensor, with a resolution of 8192×5460 pixels, a focal length of 35mm, and a sensor size of 35.9×24mm. During the transmission of images from the camera stream to the Jetson Xavier NX, the image is compressed to a resolution of 1920×1280 on the embedded device. Therefore, the image scaling factor is 4.26. To reconstruct the original image of the spotted seal's pixel coordinates, the scaling factor needs to be multiplied. When the spotted seal's detected pixel coordinates are (300, 300), the corresponding sensor pixel coordinates are (1278, 1278). Based on the pinhole imaging principle and the sensor's intrinsic parameters, a coordinate system transformation is performed. First, the sensor camera's pixel coordinate system is converted to the image coordinate system:

[0121]

[0122] In the formula, d x Where u is the sensor width / image width resolution, u is the x-coordinate of the pixel used to identify the spotted seal, and u0 is the x-coordinate of the pixel at the center of the image.

[0123]

[0124] In the formula, d y v is the vertical coordinate of the pixel used to identify the spotted seal, and v0 is the vertical coordinate of the pixel at the center of the image.

[0125] Therefore, the image coordinates of the spotted seal in the image coordinate system can be calculated as P(x,y)=p(-12.4mm,-6.4mm). The predefined drone flight altitude is 50m. The target camera coordinates can be calculated from the image coordinates P(x,y) and the flight altitude. During the drone's cruise, the sensor is vertically downward, so the gimbal angle is not considered; only the drone's angle change is taken into account. The drone's world coordinates are obtained through the rotation matrix R.

[0126]

[0127] In the formula, f a H is the camera's focal length; H is the UAV's altitude above the ground; P1 is the target's position on the imaging plane; X P The coordinates of the target's position on the ground projection; Y is the x-coordinate of the target on the imaging plane; P Let be the ordinate of the target on the imaging plane. Let XP be the vertical coordinate of the target on the imaging plane; calculated P(XP,YP) = P(-17.7m,-9.1m).

[0128] After obtaining the coordinates in the camera coordinate system, transform them to the UAV world coordinate system. Based on the UAV's attitude angles, including pitch, yaw, and roll angles, the coordinates in the UAV world coordinate system can be calculated. The yaw angle is 0 when the UAV's nose direction is relative to geographic north, and its range is (-180°, 180°). When the UAV's latitude and longitude are (40.052235, 116.243683), and the pitch, roll, and yaw angles are (0, 0, -90), the coordinates of the spotted seal can be calculated.

[0129] Restore coordinate system to world coordinate system:

[0130]

[0131] Calculate the position correction for each unit of latitude and longitude:

[0132]

[0133] The latitude and longitude corrections are calculated based on the corrected distance between the spotted seal and the center of the drone:

[0134]

[0135] Calculate and identify the coordinates of the spotted seal:

[0136] object_lat=drone_lat+Δlat=40.0520756716;

[0137] object_lon=drone_lon+Δlon=116.2437898542;

[0138] When a spotted seal is detected during a drone patrol, it is automatically stored as a data structure containing the pixel coordinates of the detection point, the drone's GPS position, and the flight altitude. When the operation workstation performs route planning, the airborne equipment will locate all the spotted seal data and calculate its world coordinates as described above, thus constructing a set of spotted seal coordinate points in the WGS84 coordinate system.

[0139] After the point set calculation is completed, the constraint model optimizes the spotted seal point set by spatial constraints. Isolated points are considered low-confidence points, and points are deleted if no other spotted seals are within a 50m radius. High-confidence filtering is performed because the spotted seals are distributed in groups, so the confidence threshold is increased to 0.7 to filter out falsely detected spotted seal points. Point set clustering merges multiple spotted seal points within a 20m radius into a central point as a single point group. During the constraint process, the distance between each point is calculated using the Haversine formula. Through these processes, multiple spotted seal aggregation points are finally obtained, and flight routes are planned based on these points.

[0140] After constraint, the Graham scan algorithm is used to calculate the convex hull of the given point set, which is the smallest convex polygon containing all points. Specifically, the point with the smallest latitude of the spotted seal is selected as the starting point of the bottom left corner P0. All other points are sorted in ascending order according to their polar angles from the pole. If the polar angles are the same, the points are sorted in ascending order according to their distance from P0. The sorted point set is traversed from P0 and points are added to the convex hull using a stack. For each new point added, it is checked whether the current point causes the polygon to "turn right". If a "turn" occurs, the redundant point is removed. This process continues until all points are processed. After completion, all vertices of the convex hull are arranged in counterclockwise order to obtain the convex hull point set.

[0141] like Figure 7 As shown, after obtaining the rectangular flight path range, oblique photography waypoints are generated. First, the endpoints of the rectangle are sorted by longitude, resulting in sorted point sets P1, P2, P3, and P4, where the longitude gradually increases from P1 to P4. The latitudes of P1 and P2 are compared; the point with the smaller latitude is the lower left point, and the point with the larger latitude is the upper left point. Similarly, the points on the upper and lower right sides are obtained by comparing P3 and P4, resulting in point sets Q1, Q2, Q3, and Q4 for the lower left, lower right, upper right, and upper left sides. Since the rectangular boundary may not be parallel to the geographic coordinate system, the included angle θ is calculated.

[0142] The formula for calculating the included angle θ is:

[0143]

[0144] The center of the rectangle is:

[0145]

[0146] dx = Q(X) - X;

[0147] dy = Q(y) - Y;

[0148]

[0149] In the formula, X and Y are the center of the rectangle, X1 and Y1 are the coordinates of the four endpoints of the rectangle after rotation, the bottom and side are parallel to the horizontal and vertical lines after Mercator projection, respectively, and dx and dy are the changes of each point relative to the center of the matrix.

[0150]

[0151] Where imageHeight and imageWidth are the actual ground width and height corresponding to the image captured by the sensor, respectively, W sensor H sensor Here are the sensor's corresponding width and height, H is the drone's height, W is the drone's width, and f is the height. a The focal length of the sensor;

[0152] The step size of the projected horizontal and vertical lines is:

[0153] stepX=imageWidth×(1-overlapW)=51.28×(1-0.6)=20.512m;

[0154] stepY=imageHeight×(1-overlapH)=34.28×(1-0.7)=10.284m;

[0155] In the formula, stepX is the step size of the horizontal line, stepY is the step size of the vertical line, overlapW is the flight path overlap rate, and overlapH is the lateral overlap rate.

[0156] The waypoints are generated by gridding, and the x and y coordinates of the four rectangle endpoints in the planar coordinate system are compared to obtain their extreme values ​​X. min X max Generate flight path based on lateral spacing stepX;

[0157] Number of lateral routes:

[0158]

[0159] Add waypoints to each route, with a stepY interval between them, to obtain the set of planar coordinates D of all waypoints within a rectangular area. Perform an inverse matrix transformation on the coordinates to restore the initial coordinate system position (X2,Y2).

[0160]

[0161] Transform the plane coordinate system to the WGS84 coordinate system, obtain the latitude and longitude coordinates of each point, and then convert them to radians;

[0162]

[0163] In the formula, Red represents radians, Deg represents latitude and longitude, and π is 3.14159265358979323846;

[0164] Upload the radian waypoints to the drone and add waypoint actions, including gimbal rotation and camera shooting. The gimbal rotation is set to shoot at five angles. The pitch angle, yaw angle, and roll angle are set to 45 degrees downward in front (-45,0,0), 45 degrees downward to the right (-45,90,0), 45 degrees downward to the rear (-45,180,0), 45 degrees downward to the left (-45,270,0), and directly downward (-90,0,0), respectively. After the gimbal rotates to the specified angle, the shooting is performed. The waypoint trigger is set to trigger upon arrival, and the execution mode is associated execution. That is, after arriving at the waypoint, the gimbal rotation at the five angles is performed in sequence and the shooting is performed after rotation, thus completing the acquisition of oblique photography data.

[0165] Of course, the above description is not intended to limit the present invention, and the present invention is not limited to the examples given above. Any changes, modifications, additions or substitutions made by those skilled in the art within the scope of the present invention should also fall within the protection scope of the present invention.

Claims

1. An oblique photography method for real-time target recognition and flight path generation of unmanned aerial vehicles (UAVs), characterized in that, Includes the following steps: The S1 drone performs automatic high-altitude cruise based on the set flight altitude and defined flight path, and completes adaptive oblique photography. The S2 drone uses an onboard target recognition and tracking model to identify and track targets based on images obtained from oblique photography. After the S3 UAV identifies the target, it obtains the pixel coordinates of the corresponding image and converts the pixel coordinates into latitude and longitude coordinates according to the world coordinate system. S4 obtains the drone's cruising range based on the acquired target coordinate point set using a constraint model, convex hull algorithm, and minimum envelope rectangle algorithm. The S5 adaptively generates waypoints based on the drone's cruising range and location information, designs waypoint tasks to conduct low-altitude automatic cruising of the drone, and photographs the target. Wherein, the pixel coordinates are the center coordinates of the target bounding box identified during the drone's cruise, and the center coordinates of the bounding box are taken as the target's pixel coordinates. Combined with the origin of the image coordinate system ,exist: ； ； In the formula, for The pixel size corresponding to the axis is obtained from the camera's intrinsic parameters; for The pixel size corresponding to the axis is obtained from the camera's intrinsic parameters; With the origin of the image coordinate system Solve image coordinates From image coordinates The target camera coordinates can be calculated, and then the rotation matrix can be used to determine the coordinates. Obtain the drone's world coordinate system coordinates, and finally calculate the target position based on the deviation between the latitude and longitude lines and the drone's position coordinates; The formula for converting image coordinates to target camera coordinates is: ； In the formula, The focal length of the camera; This refers to the drone's altitude above the ground. The origin of the image coordinate system; Centered on the camera coordinate system; The center of the sensor's projection onto the ground; To identify the position of the target on the imaging plane; The coordinates of the target's position on the ground projection; Let x be the x-coordinate of the target on the imaging plane; The coordinates of the target's position on the ground projection; The vertical coordinate of the target on the imaging plane; Rotation matrix The formula is: ； ； In the formula, The x-coordinate of the rotated coordinate system; The ordinate is the y-coordinate after rotation; These are the height coordinates after rotation. This refers to the yaw angle. The pitch angle; This refers to the roll angle. The x-coordinate is the coordinate before rotation; The ordinate is the coordinate before rotation; These are the height coordinates of the coordinates before rotation; ； In the formula, The radius of curvature of the Earth's geoid (or reference ellipsoid) at a given point; This is the Earth's semi-major axis; The oblateness of the Earth; It is the radian representation of a given latitude; ； In the formula, This represents the ground distance corresponding to a unit change in latitude. ； In the formula, This represents the ground distance corresponding to a unit change in longitude. ； ； ； ； In the formula, The latitude corresponding to the target; The longitude corresponding to the target; The latitude of the drone; The longitude of the drone; This is the correction amount for the target's latitudinal position relative to the UAV; This is the correction amount for the target's longitude position relative to the UAV.

2. The oblique photography method for real-time target recognition and flight path generation of a UAV according to claim 1, characterized in that, In S1, the drone is equipped with embedded airborne equipment, optical sensors, gimbal, camera, and onboard computer. The onboard computer is connected to the embedded airborne equipment, optical sensors, gimbal, and camera control. The onboard computer has an SDK module and a widget remote control component. The SDK module is started to transmit information, and the widget remote control component is used to start the control service. The cruise altitude is set between 50m and 70m to define the route for the drone to cruise automatically.

3. The oblique photography method for real-time target recognition and flight path generation of a UAV according to claim 2, characterized in that, In S2, data preprocessing, target recognition model training, and target tracking method construction are performed to obtain target recognition and tracking model, and the target recognition and tracking model is deployed to embedded airborne equipment.

4. The oblique photography method for real-time target recognition and flight path generation of a UAV according to claim 3, characterized in that, in, Data preprocessing involved capturing multi-angle target images at a cruising altitude of 50m using oblique photography, labeling the targets, and constructing a dataset. The dataset was then divided into training, validation, and test sets for the recognition model in an 8:1:1 ratio. A YOLOv10 model was used, with MobileNet V3 and GSConv modules added for model training to obtain pt weight files. These weight files were then converted to ONNX structures. The TensorRT 8.5.0 application library was deployed on an embedded airborne device, using an int8 lightweight ONNX as the engine structure. The recognition model was deployed on the embedded airborne device, and model recognition was continuously performed on the camera stream during cruising. The target tracking method employs the sort method, which obtains detectors by associating target detection results. For each tracked target, a Kalman filter is used to predict its position, thereby completing the state prediction. The Hungarian algorithm is used to match the prediction results with the detection results through IOU to associate the data. Successful matches update the Kalman filter state, and new trackers are created and lost trackers are deleted, thus completing the tracking of the detected targets.

5. The oblique photography method for real-time target recognition and flight path generation of a UAV according to claim 4, characterized in that, In S3, the target recognition and tracking model runs continuously during the drone's cruise, and the mounted optical sensor is always perpendicular to the ground. When the drone recognizes a target through the camera, it acquires the pixel coordinates of the corresponding image and subscribes to the drone's position information. Based on the acquired drone position information, it performs coordinate transformation, converting the pixel coordinate system to the drone's WGS84 world coordinate system, thus converting the pixel coordinates into latitude and longitude coordinates.

6. The oblique photography method for real-time target recognition and flight path generation of a UAV according to claim 5, characterized in that, In S4, the target coordinate point set is obtained, a constraint model is constructed to reduce redundant points, and then the convex hull algorithm is used to obtain the minimum area polygon for the filtered points. Finally, the minimum envelope rectangle method is used to obtain the drone's cruising range with maximum efficiency. The constraint model mainly includes cluster analysis to merge redundant points, setting a target recognition confidence threshold, applying spatial clustering constraints, calculating the distance between points after positioning, merging points if the distance is less than 1m, storing the recognition confidence, filtering out points with a confidence level below 0.6, and simultaneously traversing all points and using the Haversine formula to calculate the distance between each point. Access the coordinates of other points within a radius of 10m; The Haversine formula is: ； In the formula, Known points on the Earth's surface Latitude; Known points on the Earth's surface Longitude; Known points on the Earth's surface Latitude; Known points on the Earth's surface Latitude; Let be the Earth's radius. The Earth's average radius is: km.

7. The oblique photography method for real-time target recognition and flight path generation of a UAV according to claim 6, characterized in that, in, The convex hull algorithm selects the point with the smallest latitude in the target point set as the starting point, and sorts all other points according to their polar angle with the pole. It uses a stack to add points to the convex hull. For each new point added, it checks whether the current point causes the polygon to "turn right". If a "turn" occurs, the redundant point is removed until all points are processed. After completion, all vertices of the convex hull are arranged in counterclockwise order.

8. The oblique photography method for real-time target recognition and flight path generation of a UAV according to claim 7, characterized in that, in, The minimum envelope rectangle method is as follows: traverse all edges of the convex hull, calculate the extreme values ​​of the edge and its normal and each point in its direction, calculate the area, compare the areas of all rectangles, record the vertices of the rectangle with the smallest area, and use the space enclosed by all vertices as the flight range of the flight path.

9. The oblique photography method for real-time target recognition and flight path generation of a UAV according to claim 8, characterized in that, In S5, after the drone returns to its starting point after the cruise ends, the drone is set with low-altitude flight altitude, heading, and lateral overlap rate. Then, based on the set flight altitude, the ground length and width corresponding to the actual photo are calculated. The waypoint positions are calculated based on the overlap rate. After obtaining the waypoint positions, an action is set for each waypoint to control the gimbal rotation and drive the camera to take pictures. After the drone uploads the waypoint task, it begins automatic cruise to take pictures of the target.

Images