Oblique photography method for real-time target identification and route generation of unmanned aerial vehicle

Abstract

The invention discloses an oblique photography method for real-time target identification and route generation of an unmanned aerial vehicle, and relates to the technical field of identification and tracking, and the method comprises the following steps: unmanned aerial vehicle high-altitude cruise, unmanned aerial vehicle target identification and tracking, unmanned aerial vehicle target positioning, route flight range adaptive generation, and low-altitude unmanned aerial vehicle automatic cruise. According to the invention, the unmanned aerial vehicle can have the capabilities of cruising, identifying and tracking the target in a large-range and long-distance scene, on-site investigation of a worker is not needed, the situation that the unmanned aerial vehicle searches the moving target in a large-range cruising manner and moves the moving target for a long time is effectively reduced, time and labor are saved, and the unmanned aerial vehicle is used for identifying and tracking the moving target.

Description

Technical Field

[0001] The present invention relates to the technical field of identification and tracking, and particularly to an oblique photography method for real-time target identification and flight path generation of an unmanned aerial vehicle (UAV). Background Art

[0002] Currently, UAVs are used more and more widely. When using a UAV for inspection to complete the observation of moving targets and data collection, since the moving targets do not stay fixed at a certain position for a long time and will move at any time, it is impossible for the UAV to directly reach the accurate position of the fixed target. The common method now is that the staff operates the UAV manually or sets a flight path to conduct multiple cruises to first find the target, and then conduct observations and data collection for specific positions. However, when the distance between the UAV take-off point and the moving target is too far, resulting in a too large search range for the moving target, it takes a long time and effort to first locate the moving target, and it is also easy for the moving target to leave its original position during multiple round trips of the UAV. Summary of the Invention

[0003] Aiming at the problems that the current UAVs do not have the capabilities of cruising, identifying, and tracking targets in large-scale and long-distance scenarios, which are time-consuming and prone to losing targets, and at the same time cannot automatically generate a flight path around the moving target and conduct oblique photography, the present invention provides an oblique photography method for real-time target identification and flight path generation of a UAV.

[0004] To solve the above technical problems, the present invention adopts the following technical solutions: An oblique photography method for real-time target identification and flight path generation of a UAV, comprising the following steps: S1 The UAV defines a flight path according to the set flight altitude to conduct high-altitude automatic cruise of the UAV and complete adaptive oblique photography. S2 The UAV uses the carried target identification and tracking model and identifies and tracks the target according to the images obtained by oblique photography. S3 After the UAV identifies the target, it obtains the pixel coordinates of the image corresponding to the target and converts the pixel coordinates into longitude and latitude coordinates according to the world coordinate system. S4 According to the obtained set of target coordinate points, the UAV cruise range is obtained through a constraint model, a convex hull algorithm, and a minimum bounding rectangle algorithm. S5 According to the UAV cruise range and the UAV position information, waypoints are adaptively generated, a waypoint task is designed to conduct low-altitude automatic cruise of the UAV, and the target is photographed.

[0005] Further, in S1, the drone is equipped with an embedded on-board device, an optical sensor, a gimbal, a camera, and an on-board computer. The on-board computer is respectively connected to the embedded on-board device, the optical sensor, the gimbal, and the camera for control. The on-board computer has an SDK module and a widget remote control component. The SDK module is started for information transmission and the widget remote control component is used to start the control service. A cruising altitude ranging from 50m to 70m is set to define a flight path for the automatic cruising of the drone.

[0006] Further, in S2, data preprocessing, training a target recognition model, and constructing a target tracking method are carried out to obtain a target recognition and tracking model, and the target recognition and tracking model is deployed to the embedded on-board device.

[0007] Further, in data preprocessing, oblique photography is used to capture multi-angle target images at a cruising altitude of 50m, the targets are labeled to construct a dataset, and the dataset is divided into a training set, a validation set, and a test set for the recognition model in a ratio of 8:1:1. The yolov10 model is used, and the Mobilenet V3 and GSConv modules are added. Model training is carried out to obtain a pt weight file, the weight file is converted into an onnx structure, and the application library of TensorRT 8.5.0 is deployed on the embedded on-board device. The int8 lightweight onnx is used as the engine structure, and the recognition model is deployed in the embedded on-board device. During the cruising process, the model recognition is continuously carried out on the camera bitstream. The target tracking method adopts the sort method. A detector is obtained by associating the target detection results. The Kalman filter is used to predict the position of each tracking target to complete the state prediction. The Hungarian algorithm is used to match the prediction results with the detection results through IOU for data association. The results of successful matching update the Kalman filter state, and new trackers are created and lost trackers are deleted to complete the tracking of the detected targets.

[0008] Further, in S3, the target recognition and tracking model is always running during the drone's cruising process, and the mounted optical sensor is always perpendicular to the ground. Only the motion state of the drone needs to be considered. When the drone recognizes a target through the camera, the pixel coordinates of the corresponding image are obtained. At the same time, the drone information is subscribed, and coordinate conversion is carried out according to the obtained position information of the drone, converting the pixel coordinate system to the drone's WGS84 world coordinate system, so that the pixel coordinates are converted into longitude and latitude coordinates.

[0009] Further, the pixel coordinates are the center coordinates of the target bounding box recognized during the drone's cruising process. The center coordinates of the bounding box are taken as the pixel coordinates P(u, v) of the target. Combining with the origin O(u0, v0) of the image coordinate system, there is:

[0010] x = (u - u0) * d x;

[0011] y = (v - v0) * d y ;

[0012] In the formula, d x is the pixel size corresponding to the x-axis and is obtained from the camera internal parameters; d y is the pixel size corresponding to the y-axis and is obtained from the camera internal parameters;

[0013] Solve the image coordinates P(x, y) with the origin O(u0, v0) of the image coordinate system. The target camera coordinate system coordinates can be calculated from the image coordinates P(x, y), and then the UAV world coordinate system coordinates can be obtained through the rotation matrix R. Finally, calculate the target position based on the deviation between the longitude and latitude lines and the UAV position coordinates;

[0014] The formula for converting image coordinates to target camera coordinate system coordinates is:

[0015]

[0016] In the formula, f a is the focal length of the camera; H is the height of the UAV from the ground; O is the origin of the image coordinate system; O C is the center of the camera coordinate system; O W is the center of the UAV corresponding to the sensor projected onto the ground; P1 is the position of the recognized target on the imaging plane; X P is the position coordinate of the target on the ground projection; is the abscissa of the target on the imaging plane; Y p is the ordinate of the target on the imaging plane;

[0017] The formula for the rotation matrix R is:

[0018]

[0019]

[0020] In the formula, x E is the abscissa of the rotated coordinate; y E is the ordinate of the rotated coordinate; Z E is the height coordinate of the rotated coordinate, ∝ is the yaw angle; θ is the pitch angle; is the roll angle; x p is the abscissa of the coordinate before rotation; y p is the ordinate of the coordinate before rotation; z p is the height coordinate of the coordinate before rotation;

[0021]

[0022] where N is the radius of curvature of the Earth's geoid (or reference ellipsoid) at a certain point; a is the semi-major axis of the Earth; f is the flattening of the Earth; is the radian representation of a given latitude;

[0023]

[0024] where m perdelat is the ground distance corresponding to a unit change in latitude;

[0025]

[0026] where m perdelon is the ground distance corresponding to a unit change in longitude;

[0027] object_lat = drone_lat + Δlat;

[0028] object_lon = drone_lon + Δlon;

[0029] where object_lat is the latitude corresponding to the target; object_lon is the longitude corresponding to the target; drone_lat is the latitude of the drone; drone_lon is the longitude of the drone; Δlat is the latitude position correction amount of the target relative to the drone; Δlon is the longitude position correction amount of the target relative to the drone.

[0030] Furthermore, in S4, obtain the target coordinate point set, construct a constraint model to reduce redundant points, then use the convex hull algorithm for the filtered points to obtain the minimum area polygon, and then obtain the most efficient drone cruise range through the minimum bounding rectangle method;

[0031] Among them, the constraint model mainly includes clustering analysis to merge redundant points, setting the target recognition confidence threshold, performing spatial clustering constraints, calculating the distances between the located points. If the distances between the points are less than 1m, they are merged into one point, storing the recognition confidence, screening out points with a confidence lower than 0.6, and at the same time traversing all points, using the Haversine formula to calculate the distances d between the points, and storing the coordinates of other points within a radius of 10m;

[0032] The Haversine formula is:

[0033]

[0034] where is the latitude of the known point A on the Earth's surface; λ1 is the longitude of the known point A on the Earth's surface; φ is the latitude of the known point B on the earth's surface; λ2 is the latitude of the known point B on the earth's surface; R is the radius of the earth, and the average radius of the earth is: R = 6371 km.

[0035] Further, in the convex hull algorithm, the point with the smallest latitude in the target point set is selected as the starting point, and all other points are sorted by the polar angle with respect to the pole. A stack is used to add points to the convex hull. For each newly added point, it is checked whether the current point makes the polygon "turn right". If "turn right" occurs, the redundant points are removed until all points are processed. After completion, all vertices of the convex hull are arranged in counterclockwise order.

[0036] Further, the minimum bounding rectangle method is as follows: traverse all sides of the convex hull, calculate the extreme values of the side and its normal in the direction of each point, calculate the area, compare the areas of all rectangles, record the vertices of the rectangle with the smallest area, and use the space enclosed by the vertices as the flight range of the flight path.

[0037] Further, in S5, after the UAV returns to the starting point after the cruise ends, set the low-altitude flight altitude, heading, and side overlap rate for the UAV. Then calculate the ground length and width corresponding to the actual photos according to the set flight altitude, calculate the waypoint positions according to the overlap rate, calculate the waypoints based on the obtained flight range and the overlap degree. After obtaining the waypoint positions, set an action for each waypoint to control the pan-tilt head to rotate and drive the camera to take pictures. After the UAV uploads the waypoint task, it starts automatic cruising to take pictures of the target.

[0038] The beneficial effects of the present invention are as follows: The present invention can identify and track moving targets through the yolov10 model deployed on the UAV, save the corresponding pixel points of the identified moving targets, use the target tracking algorithm sort to avoid repeated recording of the pixel coordinates of moving targets, calculate the longitude and latitude position coordinates of the moving targets according to information such as the UAV position, flight attitude, and ground height, then use the convex hull algorithm to obtain the minimum bounding rectangle, that is, the rectangular flight path range, design waypoint actions, automatically generate waypoints using the vertex coordinates of the rectangular range for the set overlap rate, perform low-altitude cruising after the high-altitude cruising ends, and execute waypoint actions at the waypoints during the cruising process to adaptively complete the oblique photography of the target.

[0039] The present invention enables the UAV to have the capabilities of cruising, identifying, and tracking targets in a large range and long-distance scenarios, without the need for on-site investigation by staff, and can identify targets through the images taken by oblique photography during the cruising process, significantly reducing the situation of the UAV cruising in a large range to find moving targets and the moving targets moving for a long time, saving time and effort, and being used for identifying and tracking moving targets. Description of the Drawings

[0040] Figure 1The figure shows a flowchart of an embodiment of the present invention.

[0041] Figure 2 The figure shows a schematic diagram of the improved yolov10 model structure.

[0042] Figure 3 The figure shows a schematic diagram of the conversion relationship between the pixel coordinate system and the image coordinate system.

[0043] Figure 4 The figure shows a schematic diagram of the conversion relationship between the camera coordinate system and the UAV world coordinate system.

[0044] Figure 5 The figure shows a schematic diagram of the mapping relationship between the image coordinate system and the UAV coordinate system.

[0045] Figure 6 The figure shows a schematic diagram of the implementation principle of the convex hull algorithm.

[0046] Figure 7 The figure shows a waypoint map of a rectangular range. Specific implementation manner

[0047] An oblique photography method for real-time target recognition and route generation of an unmanned aerial vehicle (UAV) includes the following steps: S1. The UAV defines a route according to the set flight altitude for high-altitude automatic cruise of the UAV and completes adaptive oblique photography. S2. The UAV uses the target recognition and tracking model carried on it and recognizes and tracks the target according to the images obtained by oblique photography. S3. After the UAV recognizes the target, it obtains the pixel coordinates of the image corresponding to the target and converts the pixel coordinates into longitude and latitude coordinates according to the world coordinate system. S4. According to the obtained set of target coordinate points, the cruising range of the UAV is obtained through a constraint model, a convex hull algorithm, and a minimum bounding rectangle algorithm. S5. According to the cruising range of the UAV and the UAV position information, waypoints are adaptively generated, a waypoint task is designed for low-altitude automatic cruise of the UAV, and the target is photographed.

[0048] Further, in S1, the UAV is equipped with an embedded airborne device, an optical sensor, a gimbal, a camera, and an airborne computer. The airborne computer is respectively connected to the embedded airborne device, the optical sensor, the gimbal, and the camera in a control manner. The airborne computer has an SDK module and a widget remote control component. The SDK module is started for information transmission and the widget remote control component is used to start the control service. The cruising altitude is set between 50m and 70m to define the route for automatic cruise of the UAV.

[0049] Further, in S2, data preprocessing, training a target recognition model, and constructing a target tracking method are carried out to obtain a target recognition and tracking model, and the target recognition and tracking model is deployed to the embedded airborne device.

[0050] Further, in data preprocessing, oblique photography is used to capture multi-angle target images at a cruising altitude of 50m, and the targets are labeled to construct a dataset. The dataset is divided into a training set, a validation set, and a test set for the recognition model in a ratio of 8:1:1. The yolov10 model is used, and the Mobilenet V3 and GSConv modules are added for model training to obtain a pt weight file. The weight file is converted into an onnx structure, and the application library of TensorRT 8.5.0 is deployed on the embedded airborne device. The int8 lightweight onnx is used as the engine structure for recognition model deployment on the embedded airborne device, and the model recognition is continuously performed on the camera stream during the cruising process. Among them, the SORT method is adopted for target tracking. The detector is obtained by associating the target detection results. The Kalman filter is used to predict the position of each tracking target to complete the state prediction. The Hungarian algorithm is used to match the prediction results with the detection results through IOU for data association. The results of successful matching update the Kalman filter state, and new trackers are created and lost trackers are deleted to complete the tracking of the detected targets.

[0051] Further, in S3, the target recognition and tracking model is always running during the UAV cruising process, and the mounted optical sensor is always perpendicular to the ground. When the UAV recognizes a target through the camera, the pixel coordinates of the corresponding image are obtained, and at the same time, the position information of the UAV is subscribed. Coordinate transformation is performed according to the obtained position information of the UAV to convert the pixel coordinate system to the UAV WGS84 world coordinate system, so that the pixel coordinates are converted into longitude and latitude coordinates.

[0052] Further, among them, the pixel coordinates are the center coordinates of the target bounding box recognized during the UAV cruising process. The center coordinates of the bounding box are taken as the pixel coordinates P(u, v) of the target. Combining with the origin O(u0, v0) of the image coordinate system, there is:

[0053] x = (u - u0) * d x ;

[0054] y = (v - v0) * d y ;

[0055] In the formula, d x is the pixel size corresponding to the x-axis, obtained from the camera internal parameters; d y is the pixel size corresponding to the y-axis, obtained from the camera internal parameters;

[0056] The image coordinates P(x, y) are solved with the origin O(u0, v0) of the image coordinate system. The target camera coordinate system coordinates can be solved from the image coordinates P(x, y), and then the UAV world coordinate system coordinates are obtained through the rotation matrix R. Finally, the target position is calculated according to the deviation between the longitude and latitude lines and the UAV position coordinates;

[0057] The formula for converting image coordinates to the coordinates in the target camera coordinate system is as follows:

[0058]

[0059] In the formula, f a is the focal length of the camera; H is the height of the UAV from the ground; O is the origin of the image coordinate system; O C is the center of the camera coordinate system; O W is the center of the UAV corresponding to the projection of the sensor on the ground; P1 is the position of the recognized target on the imaging plane; X P is the position coordinate of the target on the ground projection; is the abscissa of the target on the imaging plane; Y P is the ordinate of the target on the imaging plane;

[0060] The formula for the rotation matrix R is:

[0061]

[0062] In the formula, x E is the abscissa of the rotated coordinate; y E is the ordinate of the rotated coordinate; Z E is the height coordinate of the rotated coordinate, ∝ is the yaw angle; θ is the pitch angle; is the roll angle; x p is the abscissa of the coordinate before rotation; y p is the ordinate of the coordinate before rotation; z p is the height coordinate of the coordinate before rotation;

[0063]

[0064] In the formula, N is the radius of curvature of the Earth's geoid (or reference ellipsoid) at a certain point; a is the semi-major axis of the Earth; f is the flattening of the Earth; is the radian representation of the given latitude;

[0065]

[0066] In the formula, m perdelat is the ground distance corresponding to the change in unit latitude;

[0067]

[0068] In the formula, m perdelon is the ground distance corresponding to the change in unit longitude;

[0069] object_lat = drone_lat + Δlat;

[0070] object_lon = drone_lon + Δlon;

[0071] In the formula, object_lat is the latitude corresponding to the target; object_lon is the longitude corresponding to the target; drone_lat is the latitude of the drone; drone_lon is the longitude of the drone; Δlat is the latitude position correction amount of the target relative to the drone; Δlon is the longitude position correction amount of the target relative to the drone.

[0072] Further, in S4, obtain the set of target coordinate points, construct a constraint model to reduce redundant points, then use the convex hull algorithm for the filtered points to obtain the minimum area polygon, and then use the minimum bounding rectangle method to obtain the most efficient drone cruise range;

[0073] Among them, the constraint model mainly includes clustering analysis to merge redundant points, setting the target recognition confidence threshold, performing spatial clustering constraints, calculating the distances between the located points. If the distance between points is less than 1m, they are merged into one point, storing the recognition confidence, screening out points with a confidence lower than 0.6, and at the same time traversing all points, using the Haversine formula to calculate the distance d between points, and storing the coordinates of other points within a radius of 10m.

[0074] The Haversine formula is:

[0075]

[0076] In the formula, is the latitude of the known point A on the earth's surface; λ1 is the longitude of the known point A on the earth's surface; is the latitude of the known point B on the earth's surface; λ2 is the latitude of the known point B on the earth's surface; R is the radius of the earth, and the average radius of the earth is: R = 6371 km.

[0077] Further, among them, the convex hull algorithm selects the point with the smallest latitude in the set of target points as the starting point, and sorts all other points according to the polar angle with the pole, uses a stack to add points to the convex hull. For each new point added, check whether the current point makes the polygon "turn right". If "turn right" occurs, remove the redundant points until all points are processed. After completion, all vertices of the convex hull are arranged in counterclockwise order.

[0078] Further, among them, the minimum bounding rectangle method is: traverse all sides of the convex hull, calculate the extreme values of the side and its normal line in the direction of each point, calculate the area, compare the areas of all rectangles, record the vertices of the rectangle with the smallest area, and use the space enclosed by the vertices as the flight range of the flight path.

[0079] Further, in S5, after the UAV returns to the starting point at the end of the cruise, set the low-altitude flight altitude, heading, and side overlap rate for the UAV. Then, calculate the ground length and width corresponding to the actual photos based on the set flight altitude, calculate the waypoint positions according to the overlap rate, calculate the waypoints based on the obtained flight range and the overlap degree, and after obtaining the waypoint positions, set an action for each waypoint to control the pan-tilt head to rotate and drive the camera to take pictures. After the UAV uploads the waypoint task, it starts automatic cruising to take pictures of the target.

[0080] The present invention discloses an oblique photography method for real-time target recognition and route generation of a UAV. The following describes a specific embodiment of the present invention in conjunction with the accompanying drawings.

[0081] As Figure 1 shown, in the first step, the UAV conducts high-altitude cruising. The UAV is equipped with an embedded airborne device, an optical sensor, a pan-tilt head, a camera, and an airborne computer. The airborne computer is respectively connected to the embedded airborne device, the optical sensor, the pan-tilt head, and the camera for control. The embedded airborne device performs information intertransmission with the UAV through usb-ttl and usb-usbc. After the UAV is powered on, the airborne computer is started. The airborne computer is connected to the UAV through the SDK and controls the UAV through the widget desktop component on the remote controller. The widget component on the remote controller is developed through the SDK and can implement functions such as target recognition and tracking, and UAV flight control by calling the defined flight control code. The UAV can adopt cruising methods such as manual remote control or pre-planned route automatic cruising. Among them, the route can be preferably planned according to historical data, human experience, etc. The cruising altitude range of the UAV is between 50m and 70m, and a target recognition model with an identification altitude range between 50m and 70m is adaptively constructed.

[0082] In the second step, UAV target recognition. Model preparation: Preprocess the data and set the multi-angle target images taken by oblique photography with a cruising altitude in the range of 50m to 70m for the identification height range, label the targets to construct a data set, and divide it into a training set, a validation set, and a test set of the recognition model according to a ratio of 8:1:1. As Figure 2As shown in the figure, Input in the figure represents the input layer, Backbone represents the backbone network, Neck represents the neck, Head represents the detection head, Conv represents the convolutional layer, SCDown represents spatial compression downsampling, UpSample represents upsampling, Concat represents concatenation for feature fusion, C2fCIB represents an improved convolutional module, SPPF represents the pooling layer, PSA represents an improved convolutional module, and C2f represents an improved convolutional module. The yolov10 model is used for object recognition, and the Mobilenet V3 and GSConv modules are added to achieve the smoothness of drone deployment, which can improve the information extraction ability while reducing the computational cost and enhance the model accuracy. After the model training is completed, a weight file of the pt type is obtained. To achieve real-time recognition, the TensorRT architecture is used for deployment, and the format of the weight file is converted. Model structure conversion: First, convert the file structure to the onnx structure after training is completed, and then deploy the application library of TensorRT 8.5.0 on the embedded airborne device. Use the trtexec application provided by the official library to convert the int8 lightweight onnx to the engine structure. Object tracking: Use the sort method to track the object. Obtain the detector by associating the object detection results and get the detection box through the detector. Use the Kalman filter to predict the position of the existing tracker, and use the Hungarian algorithm to perform IOU matching between the predicted box and the detection box. The detection results that match successfully update the Kalman filter state, and those that do not match successfully create a new tracker. At the same time, the unmatched trackers are deleted to complete the real-time tracking of the object. During the process of building the tracker, if a new tracker is created, it represents a new recognition target, and the pixel position of this point and the drone information are stored, and the TensorRT deployment is combined with the model engine structure. The deployed application is as follows: Load the.engine model file, obtain the real-time camera stream of the drone, format the obtained stream data into a shape and type suitable for inference, use TensorRT to perform inference and obtain the inference results, process the information such as the recognized box and confidence through the sort method, set the tracker, build the matching and unmatched trackers, continuously perform the prediction of the tracker, and match the two. If they do not match for multiple frames, delete the corresponding tracker and release the occupied resources; After the drone takes off, start the recognition module through the widget component on the remote control to complete the real-time recognition of the object during the drone cruise process.

[0083] Step 3: UAV target positioning. During the UAV cruise, the target recognition module is always running, and the mounted optical sensor is always vertically photographed. When the target is recognized by the lens, the pixel coordinates of the corresponding image are obtained. At the same time, the UAV information is subscribed, and coordinate conversion is performed according to the obtained UAV position. Through the corresponding relationship between coordinate systems, the pixel coordinate system is converted to the UAV WGS84 world coordinate system, and the pixel coordinates are converted to longitude and latitude coordinates. Among them, the subscribed UAV information includes the UAV geographical coordinates such as longitude and latitude, the UAV flight attitude such as yaw angle, pitch angle, roll angle, ground height, altitude, pan direction, UAV attitude, etc. Combining Figure 3 , Figure 4 and Figure 5 As shown, the acquisition of the UAV WGS84 world coordinate system has gone through processes such as pixel coordinate system to image coordinate system, image coordinate system to camera coordinate system, and camera coordinate system to UAV WGS84 world coordinate system.

[0084] When converting the pixel coordinate system to the image coordinate system, a target bounding box will be generated when the target is recognized during the UAV cruise. Take the center coordinates of the bounding box as the pixel coordinates P(u, v) of the target. Combining the origin O(u0, v0) of the image coordinate system, there is:

[0085] x = (u - u0) * d x ;

[0086] y = (v - v0) * d y ;

[0087] In the formula, d x is the pixel size corresponding to the x-axis, obtained from the camera internal parameters; d y is the pixel size corresponding to the y-axis, obtained from the camera internal parameters;

[0088] Solve the image coordinates P(x, y) with the origin O(u0, v0) of the image coordinate system. The target camera coordinate system coordinates can be solved from the image coordinates P(x, y), and then the UAV world coordinate system coordinates can be obtained through the rotation matrix R. Finally, calculate the target position according to the deviation between the longitude and latitude lines and the UAV position coordinates;

[0089] The formula for converting image coordinates to target camera coordinate system coordinates is:

[0090]

[0091] In the formula, f a is the focal length of the camera; H is the ground height of the UAV; O is the origin of the image coordinate system; O C is the center of the camera coordinate system; O W is the corresponding UAV center after the sensor is projected onto the ground; P1 is the position of the recognized target on the imaging plane; X Pis the position coordinate of the target on the ground projection; is the abscissa of the target on the imaging plane; Y P is the ordinate of the target on the imaging plane.

[0092] After obtaining the target camera coordinate system coordinates, the camera is in the orthographic state. Therefore, the positive direction of its Z-axis is consistent with the body coordinate system. To obtain the target world coordinates, only the yaw angle ∝ needs to be considered. The actual distance camera coordinate system coordinates after offset can be obtained through the rotation matrix R. The formula for the rotation matrix R is:

[0093]

[0094] In the formula, x E is the abscissa of the rotated coordinate; y E is the ordinate of the rotated coordinate; Z E is the height coordinate of the rotated coordinate, ∝ is the yaw angle; θ is the pitch angle; is the roll angle; x p is the abscissa of the coordinate before rotation; y p is the ordinate of the coordinate before rotation; z p is the height coordinate of the coordinate before rotation;

[0095] After obtaining the corrected offset distances of the longitude and latitude, convert them into the longitude and latitude differences in two directions, and the longitude and latitude of the target can be calculated;

[0096]

[0097] In the formula, N is the radius of curvature of the Earth's geoid (or reference ellipsoid) at a certain point; a is the semi-major axis of the Earth; f is the flattening of the Earth; is the radian representation of the given latitude;

[0098]

[0099] In the formula, m perdelat is the ground distance corresponding to the change in unit latitude;

[0100]

[0101] In the formula, m perdelon is the ground distance corresponding to the change in unit longitude;

[0102] object_lat = drone_lat + Δlat;

[0103] object_lon = drone_lon + Δlon;

[0104] Where object_lat is the latitude corresponding to the target; object_lon is the longitude corresponding to the target; drone_lat is the latitude of the drone; drone_lon is the longitude of the drone; Δlat is the latitude position correction amount of the target relative to the drone; Δlon is the longitude position correction amount of the target relative to the drone.

[0105] Store the converted longitude and latitude coordinates and the confidence value of the recognition frame.

[0106] Step 4, generate the flight range of the drone route. After the drone's high-altitude cruise ends, the target recognition and positioning are ended through the widget component. Since the coordinates are always obtained during the recognition process, a constraint model is designed to reduce redundant points and misdetected points. After obtaining the valid points, the convex hull algorithm is used to obtain the polygon with the smallest area, and then the minimum area bounding rectangle algorithm is used to obtain the minimum area rectangle as the flight range of the route. Among them, the constraint model mainly conducts constraints from three aspects: clustering analysis to merge redundant points, setting the confidence threshold for target recognition, and conducting spatial clustering constraints. Clustering analysis merges the monitoring points within 5m of each other to optimize the number of obtained points. Confidence constraint, set the minimum confidence threshold for model recognition to 70%. Clustering constraint, there are other targets within 15m of the target.

[0107] After target recognition and positioning, calculate the distances between points. If the distances between points are less than 1m, they are merged into one point, store the recognized confidence, filter out the points with a confidence lower than 0.6, and at the same time traverse all points and only store the coordinates of the points where there are other points within a radius of 10m;

[0108] After obtaining all the point coordinates, use the Haversine formula to calculate the distance d between points. The Haversine formula is:

[0109]

[0110] In the formula, is the latitude of the known point A on the earth's surface; λ1 is the longitude of the known point A on the earth's surface; is the latitude of the known point B on the earth's surface; λ2 is the latitude of the known point B on the earth's surface; R is the radius of the earth, and the average radius of the earth is: R = 6371km.

[0111] Judge whether there is an intersection within a 10m distance according to the result, save the point set that meets the requirements, and use the convex hull algorithm to obtain the polygon with the smallest area for the obtained point set.

[0112] The convex hull algorithm selects the point with the minimum latitude in the target point set, that is, the point P0 at the bottom leftmost as the starting point, sorts all other points in ascending order of the polar angle with respect to the pole. When the polar angles are the same, the distances to P0 are sorted in ascending order. Traverse the sorted point set from P0 and use a stack to add points to the convex hull: for each newly added point, check whether the current point makes the polygon "turn right". If "turn right" occurs, remove the redundant points and continue until all points are processed. After completion, all vertices of the convex hull are arranged in counterclockwise order to obtain the convex hull point set.

[0113] Among them, as Figure 6 shown, the black rectangular frame S1 in the figure represents the minimum circumscribed rectangle with P0 and P3 as the base, and it is also the minimum area rectangle of the polygon. The red rectangular frame S2 represents the minimum circumscribed rectangle with P0 and P5 as the base, and the green rectangular frame S3 represents the minimum circumscribed rectangle with P5 and P 11 as the base, and the purple rectangular frame S4 represents the minimum circumscribed rectangle with P9 and P 11 as the base, and the blue rectangular frame S5 represents the minimum circumscribed rectangle with P9 and P3 as the base. For the obtained convex hull point set, traverse all the edges of the convex hull, calculate the extreme values of the edge and its normal in the direction of each point, calculate the area, compare the areas of all rectangles, and record the vertices of the rectangle with the minimum area, that is, the flight range of the flight path.

[0114] Step 5, low-altitude UAV automatic cruise. After obtaining the UAV flight range, perform low-altitude automatic cruise according to the set overlap rate in combination with the subscribed UAV information. Set the UAV cruise altitude to 50m, the maximum flight speed to 10m / s, the automatic flight speed to 2m / s, the lateral overlap rate to 70%, and the heading overlap rate to 60%. When arriving at the waypoint, execute the UAV photography task. Among them, the UAV information subscription obtains the current GPS\RTK position, and obtains the actual distance on the ground corresponding to the UAV-taken image according to the principle of pinhole imaging. As Figure 7 shown, the UAV automatically generates waypoints, calculates the ground distances corresponding to the length and width of the rectangle according to the obtained UAV flight range, and solves the waypoint positions through the overlap rate in combination with the ground distance corresponding to the UAV image. Among them, the waypoint task is set to take pictures at the point, and the rotation angles of five pan-tilts to the point are set, including the pitch angle, yaw angle, and roll angle rotate in the order of 45 degrees downward in the forward direction (-45, 0, 0), 45 degrees downward in the due right direction (-45, 90, 0), 45 degrees downward in the due rear direction (-45, 180, 0), 45 degrees downward in the due left direction (-45, 270, 0), and directly downward (-90, 0, 0); the UAV executes the flight path, flies in sequence according to the designed waypoints and performs shooting. After completion, a three-dimensional scene of the target environment can be generated through three-dimensional modeling.

[0115] Example: When using a drone for inspection to observe and collect data on spotted seals, since spotted seals are social animals and their habitats are not fixed, it is impossible to accurately know the exact location of the spotted seal habitats. Currently, the common method for observing spotted seals is often that staff operate the drone manually or set a flight path for multiple cruises to first find the spotted seal habitats, and then conduct observations and data collection at specific locations. When the distance between the drone's takeoff point and the spotted seal habitats is too far, resulting in a large target search range, it often takes a lot of time and effort to first locate the position of the spotted seal habitats. Moreover, the problem that the spotted seals leave their original habitats easily occurs during the multiple round trips of the drone. The specific process is as follows:

[0116] The drone is equipped with an embedded device jetson Xavier NX. The ground station remotely connects to jetson Xavier NX through a 4G network, starts the spotted seal recognition and flight path planning system of the device, runs the spotted seal recognition module, and manually flies or sets a flight path to achieve a large-scale cruise of the drone. The drone automatically identifies targets during the cruise. A tracker is created for the recognition result of each frame in the drone's image through the tracking algorithm sort, and Kalman filtering is used for position prediction. If the predicted box matches the tracking box in the subsequent frame image, it is the same target. If the tracker is not matched for a long time, the target is no longer within the tracking range and the tracker is deleted to release space. If there is a new tracking box, it indicates a newly recognized target individual, and the recognized target position and drone information are saved;

[0117] The saved target position is in pixel coordinates. The drone information includes the current relative height and the drone coordinates. When the drone performs recognition and draws a detection box, the pixel coordinates of the two endpoints, the upper left Q1 and the lower right Q2, of the rectangular detection box are recorded, and the center of the rectangle (centerX, centerY) is calculated as the pixel center for identifying the spotted seal:

[0118] centerX = (Q2(x) + Q1(x)) / 2;

[0119] centerY = (Q1(y) + Q2(y)) / 2;

[0120] The drone sensor used is the Zenmuse P1, with a resolution of 8192×5460 pixels, a focal length of 35mm, a sensor size of 35.9×24mm. When the sensor transmits images to the jetson Xavier NX through the camera bitstream, the images will be compressed to a resolution of 1920×1280 for the embedded device. Therefore, the image scaling factor is 4.26. To restore the pixel coordinates of the original image of the spotted seal, the scaling factor needs to be multiplied. When the pixel coordinates of the spotted seal recognized are (300, 300), the corresponding sensor pixel coordinates are (1278, 1278). The coordinate system conversion is carried out based on the pinhole imaging principle and the internal parameters of the sensor. First, convert the pixel coordinate system of the sensor camera to the image coordinate system:

[0121]

[0122] In the formula, d x is the sensor width / image width resolution, u is the horizontal coordinate of the pixel of the recognized spotted seal, and u0 is the horizontal coordinate of the pixel at the center of the image;

[0123]

[0124] In the formula, d y is the sensor height / image height resolution, v is the vertical coordinate of the pixel of the recognized spotted seal, and v0 is the vertical coordinate of the pixel at the center of the image;

[0125] Thus, the coordinates of the spotted seal in the image coordinate system can be calculated as P(x, y) = p(-12.4mm, -6.4mm). The predefined flight height of the drone is 50m. From the image coordinates P(x, y) and the flight height, the target camera coordinate system coordinates can be solved. During the drone's cruise, the sensor is vertically downward, so the gimbal angle is not considered, and only the change in the drone's angle is considered. The world coordinate system coordinates of the drone are obtained through the rotation matrix R:

[0126]

[0127] In the formula, f a is the focal length of the camera; H is the height of the drone from the ground; P1 is the position of the recognized target on the imaging plane; X P is the position coordinate of the target on the ground projection; is the horizontal coordinate of the target on the imaging plane; Y P is the vertical coordinate of the target on the imaging plane, is the vertical coordinate of the target on the imaging plane; After calculation, P(XP, YP) = P(-17.7m, -9.1m).

[0128] After obtaining the coordinates in the camera coordinate system, convert the coordinates to the UAV world coordinate system. According to the UAV attitude angles, including pitch angle, yaw angle, and roll angle, the coordinates in the UAV world coordinate system can be calculated. Among them, the yaw angle is 0 when the nose direction is relative to the geographic north direction, and the range is (-180°, 180°). When the UAV latitude and longitude are (40.052235, 116.243683), and the pitch, roll, and yaw angles are (0, 0, -90), the coordinates of the spotted seal can be calculated:

[0129] Restore the coordinate system to the world coordinate system:

[0130]

[0131] Calculate the position correction amount corresponding to the unit longitude and latitude:

[0132]

[0133] Calculate the longitude and latitude correction amounts according to the correction distance of the spotted seal from the UAV center:

[0134]

[0135] Calculate the identified coordinates of the spotted seal:

[0136] object_lat = drone_lat + Δlat = 40.0520756716;

[0137] object_lon = drone_lon + Δlon = 116.2437898542;

[0138] During the UAV cruise, when a spotted seal is identified, it will be automatically stored as a data structure of the identified point pixel coordinates, UAV GPS position, and flight altitude. When the operation workstation conducts route planning, the on-board equipment will locate all the spotted seal data, calculate its world coordinates through the above calculation process, and construct a set of spotted seal coordinate points in the WGS84 coordinate system;

[0139] After the point set calculation is completed, the constraint model optimizes the spotted seal point set, performs spatial constraints. A single isolated point is regarded as a low-confidence point, and if there are no other spotted seals within 50m, the point will be deleted; high-confidence filtering, because spotted seals are clustered, so the confidence threshold is increased to 0.7 during identification to filter out misdetected spotted seal points; point clustering, multiple spotted seal points within 20m are merged into a central point as a single point group. During the constraint process, the distance between each point position is calculated through the Haversine formula. Through these processes, the aggregation points of multiple spotted seals are finally obtained, and the route is planned based on these points;

[0140] After the spotted seal point set is constrained, the Graham scan algorithm is used to calculate the convex hull of the given point set, that is, the smallest convex polygon containing all points. Specifically, the point with the smallest latitude of the spotted seal is selected as the starting point P0 at the bottom leftmost corner, and all other points are sorted in ascending order of the polar angle with respect to the pole. When the polar angles are the same, the distance to P0 is sorted in ascending order. Traverse the sorted point set from P0 and use a stack to add points to the convex hull: for each newly added point, check whether the current point makes the polygon "turn right". If "turn right" occurs, remove the redundant points and continue until all points are processed. After completion, all vertices of the convex hull are arranged in counterclockwise order to obtain the convex hull point set;

[0141] As Figure 7 shown, after obtaining the rectangular flight path range, oblique photography flight points are generated. First, sort the rectangular endpoints. Perform longitude sorting, and the sorted point set is P1, P2, P3, P4, where the longitude from P1 to P4 gradually increases. Compare the latitudes of P1 and P2, and the one with the smaller latitude is the lower left point position, and the one with the larger latitude is the upper left point position. Similarly, compare P3 and P4 to obtain the right side upper and lower point positions, and obtain the point sets Q1, Q2, Q3, Q4 of the lower left, lower right, upper right, and upper left. Since the rectangular boundary may not be parallel to the geographic coordinate system, calculate its included angle θ:

[0142] The formula for calculating the included angle θ is:

[0143]

[0144] The center of the rectangle is:

[0145]

[0146] dx = Q(X) - X;

[0147] dy = Q(y) - Y;

[0148]

[0149] In the formula, X and Y are the center of the rectangle, X1 and Y1 are the coordinates of the four rotated rectangle endpoints. After rotation, the bottom side and the side are parallel to the horizontal line and the vertical line after Mercator projection respectively. dx and dy are the change amounts of each point relative to the matrix center;

[0150]

[0151] Among them, imageHeight and imageWidth are the actual ground widths and heights corresponding to the images captured by the sensor, W sensor , H sensor are the corresponding width and height of the sensor, H is the height of the UAV, W is the width of the UAV, and f a is the focal length of the sensor;

[0152] The step lengths of the horizontal and vertical lines after projection are as follows:

[0153] stepX = imageWidth × (1 - overlapW) = 51.28 × (1 - 0.6) = 20.512 m;

[0154] stepY = imageHeight × (1 - overlapH) = 34.28 × (1 - 0.7) = 10.284 m;

[0155] In the formula, stepX is the step length of the horizontal line, stepY is the step length of the vertical line, overlapW is the flight line overlap rate, and overlapH is the side overlap rate;

[0156] Generate grid waypoints, compare the abscissa and ordinate of the four rectangular endpoints in the plane coordinate system, and obtain their extreme values X min 、X max , and generate flight lines according to the side spacing stepX;

[0157] Number of side flight lines:

[0158]

[0159] Add waypoints to each flight line, add waypoints at an interval of stepY, obtain the set D of plane coordinate points of all waypoints within the rectangle, and perform inverse matrix transformation on the coordinates to restore the positions (X2, Y2) in the initial coordinate system.

[0160]

[0161] Convert the plane coordinate system to the WGS84 coordinate system, convert the longitude and latitude coordinates of each point to radians after obtaining them;

[0162]

[0163] In the formula, Red is the radian, Deg is the longitude and latitude, and π is 3.14159265358979323846;

[0164] Upload the radian waypoints to the drone and add waypoint actions to the waypoints. The waypoint actions include gimbal rotation and camera shooting. Five angles are set for gimbal rotation for shooting. For each angle, the pitch angle, yaw angle, and roll angle are set as 45 degrees downward in the forward direction (-45, 0, 0), 45 degrees downward in the right direction (-45, 90, 0), 45 degrees downward in the backward direction (-45, 180, 0), 45 degrees downward in the left direction (-45, 270, 0), and directly below (-90, 0, 0). After the gimbal rotates to the specified angle, shooting is performed. The waypoint trigger is set to trigger on arrival, and the execution mode is associated execution, that is, after reaching the waypoint, the gimbal rotations at the five angles are performed in sequence and shooting is performed after rotation, completing the acquisition of oblique photography data.

[0165] Of course, the above description is not a limitation of the present invention, and the present invention is not limited to the above examples. Changes, modifications, additions, or substitutions made by those skilled in the art within the scope of the essence of the present invention should also fall within the protection scope of the present invention.

Claims

1. An oblique photography method for real-time target recognition and flight path generation of an unmanned aerial vehicle, characterized in that, It includes the following steps: S1: The drone conducts automatic high-altitude cruise according to the defined route based on the set flight altitude and completes adaptive oblique photography; S2: The drone uses the target recognition and tracking model carried on it and identifies and tracks the target based on the images obtained from the oblique photography; S3: After the drone identifies the target, it obtains the pixel coordinates of the corresponding image of the target and converts the pixel coordinates into longitude and latitude coordinates according to the world coordinate system; S4: According to the obtained set of target coordinate points, the cruise range of the drone is obtained through the constraint model, convex hull algorithm, and minimum bounding rectangle algorithm; S5: According to the cruise range of the drone and the drone position information, waypoints are adaptively generated, a waypoint task is designed to conduct automatic low-altitude cruise of the drone, and the target is photographed.

2. The oblique photography method for real-time target recognition and flight path generation of an unmanned aerial vehicle according to claim 1, characterized in that In S1, the drone is equipped with an embedded airborne device, an optical sensor, a gimbal, a camera, and an airborne computer. The airborne computer is respectively connected to the embedded airborne device, the optical sensor, the gimbal, and the camera in a control manner. The airborne computer has an SDK module and a widget remote control component. The SDK module is started for information transmission and the widget remote control component is used to start the control service. The cruise altitude is set between 50m and 70m to define the route for the automatic cruise of the drone.

3. A tilt photography method for real-time target recognition and flight path generation of an unmanned aerial vehicle according to claim 2, characterized in that, In S2, data preprocessing, training the target recognition model, and constructing the target tracking method are carried out to obtain the target recognition and tracking model, and the target recognition and tracking model is deployed to the embedded airborne device.

4. A tilt photography method for real-time target recognition and flight path generation of an unmanned aerial vehicle according to claim 3, characterized in that Among them, In data preprocessing, target images at multiple angles are taken by oblique photography at a cruise altitude of 50m, the targets are labeled to construct a data set, the data set is divided into a training set, a validation set, and a test set for the recognition model according to the ratio of 8:1:

1. The yolov10 model is used, and the Mobilenet V3 and GSConv modules are added. Model training is carried out to obtain the pt weight file, the weight file is converted into the onnx structure, the application library of TensorRT 8.5.0 is deployed on the embedded airborne device, the int8 lightweight onnx is used as the engine structure, and the recognition model is deployed in the embedded airborne device. During the cruise process, the model recognition is continuously carried out on the camera bitstream; Among them, the target tracking method adopts the sort method. The detector is obtained by associating the target detection results. The position of each tracking target is predicted using the Kalman filter to complete the state prediction. The Hungarian algorithm is used to match the prediction results with the detection results through the IOU for data association. The results of successful matching update the Kalman filter state, and new trackers are created and lost trackers are deleted to complete the tracking of the detected targets.

5. The oblique photography method for real-time target recognition and flight path generation of an unmanned aerial vehicle according to claim 4, characterized in that, In S3, the target recognition and tracking model is always running during the drone cruise, and the mounted optical sensor is always perpendicular to the ground. When the drone identifies the target through the camera, the pixel coordinates of the corresponding image are obtained. At the same time, the position information of the drone is subscribed, and coordinate conversion is carried out according to the obtained position information of the drone, converting the pixel coordinate system to the drone WGS84 world coordinate system to convert the pixel coordinates into longitude and latitude coordinates.

6. A tilt photography method for real-time target recognition and flight path generation of an unmanned aerial vehicle according to claim 5, characterized in that Among them, The pixel coordinates are the center coordinates of the target bounding box recognized during the UAV cruise. The center coordinates of the bounding box are taken as the pixel coordinates P(u, v) of the target. Combining with the origin O(u0, v0) of the image coordinate system, there is: x = (u - u0) * d x ; y = (v - v0) * d y ; where d x is the pixel size corresponding to the x-axis and is obtained from the camera internal parameters; d y is the pixel size corresponding to the y-axis and is obtained from the camera internal parameters; Solve the image coordinates P(x, y) with the origin O(u0, v0) of the image coordinate system. The target camera coordinate system coordinates can be solved from the image coordinates P(x, y), and then the UAV world coordinate system coordinates are obtained through the rotation matrix R. Finally, the target position is calculated based on the deviation between the longitude and latitude lines and the UAV position coordinates. The formula for converting image coordinates to target camera coordinate system coordinates is: Where f a is the focal length of the camera; H is the height of the UAV from the ground; O is the origin of the image coordinate system; O C is the center of the camera coordinate system; O W is the center corresponding to the projection of the sensor onto the ground; P1 is the position of the recognized target on the imaging plane; X P is the position coordinate of the target on the ground projection; is the abscissa of the target on the imaging plane; Y P is the ordinate of the target on the imaging plane; The formula for the rotation matrix R is: Where x E is the abscissa of the rotated coordinate; y E is the ordinate of the rotated coordinate; Z E is the height coordinate of the rotated coordinate, ∝ is the yaw angle; θ is the pitch angle; is the roll angle; x p is the abscissa of the coordinate before rotation; y p is the ordinate of the coordinate before rotation; z p is the height coordinate of the coordinate before rotation; where N is the radius of curvature of the Earth's geoid (or reference ellipsoid) at a certain point; a is the semi-major axis of the Earth; f is the flattening of the Earth; is the radian representation of the given latitude; where m perdelat is the ground distance corresponding to the change in unit latitude; where m perdelon is the ground distance corresponding to the change in unit longitude; object_lat = drone_lat + Δlat; object_lon = drone_lon + Δlon; In the formula, object_lat is the latitude corresponding to the target; object_lon is the longitude corresponding to the target; drone_lat is the latitude of the UAV; drone_lon is the longitude of the UAV; Δlat is the latitude position correction amount of the target relative to the UAV; Δlon is the longitude position correction amount of the target relative to the UAV.

7. A tilt photography method for real-time target recognition and flight path generation of an unmanned aerial vehicle according to claim 6, characterized in that, In S4, obtain the target coordinate point set, construct a constraint model to reduce redundant points, then use the convex hull algorithm for the filtered points to obtain the minimum area polygon, and then obtain the most efficient UAV cruise range through the minimum bounding rectangle method. Among them, the constraint model mainly includes clustering analysis to merge redundant points, setting the target recognition confidence threshold, performing spatial aggregation constraints, calculating the distances between the located points. If the distances between the points are less than 1m, they are merged into one point, store the recognition confidence, filter out the points with a confidence lower than 0.6, and at the same time traverse all points, use the Haversine formula to calculate the distances d between the points, and store the coordinates of other points within a radius of 10m. The Haversine formula is: In the formula, is the latitude of the known point A on the earth's surface; λ1 is the longitude of the known point A on the earth's surface; is the latitude of the known point B on the earth's surface; λ2 is the latitude of the known point B on the earth's surface; R is the radius of the earth, and the average radius of the earth is: R = 6371 km.

8. A tilt photography method for real-time target recognition and flight path generation of an unmanned aerial vehicle according to claim 7, characterized in that, Among them, The convex hull algorithm selects the point with the minimum latitude in the target point set as the starting point, and sorts all other points according to the polar angle with the pole. Use a stack to add points to the convex hull. For each newly added point, check whether the current point makes the polygon "turn right". If "turn right" occurs, remove the redundant points until all points are processed. After completion, all vertices of the convex hull are arranged in counterclockwise order.

9. The oblique photography method for real-time target recognition and flight path generation of an unmanned aerial vehicle according to claim 8, characterized in that, Among them, The minimum bounding rectangle method is: traverse all sides of the convex hull, calculate the extreme values of the side and its normal line in the direction of each point, calculate the area, compare the areas of all rectangles, record the vertices of the rectangle with the minimum area, and use the space enclosed by the vertices as the flight range of the flight path.

10. The oblique photography method for real-time target recognition and flight path generation of an unmanned aerial vehicle according to claim 9, characterized in that, In S5, after the UAV returns to the starting point after the cruise, set the low-altitude flight height, heading, and side overlap rate for the UAV. Then calculate the ground length and width corresponding to the actual photo according to the set flight height, calculate the waypoint positions according to the overlap rate, calculate the waypoints based on the obtained flight range and the overlap degree, and after obtaining the waypoint positions, set an action for each waypoint to control the pan-tilt rotation and drive the camera to take pictures. After the UAV uploads the waypoint task, it starts automatic cruising to take pictures of the target.

Images