A target positioning method based on UAV oblique photography

Abstract

The present application discloses a target positioning method based on unmanned aerial vehicle oblique photography, which relates to the field of unmanned aerial vehicle photography, including: constructing a target detection model based on a neural network, outputting the pixel coordinates of the target in the photo, and the pixel coordinates of the center point of the photo; obtaining the state parameters of the unmanned aerial vehicle and the camera; converting the pixel coordinates of the target and the pixel coordinates of the center point of the photo into the target image plane coordinates (x, y) and the image plane coordinates of the center point of the photo respectively; using UTM projection or Gauss-Krüger projection to convert the longitude and latitude of the unmanned aerial vehicle into projection coordinates; according to the pixel coordinates of the center point of the photo, calculating the projection coordinates of the center point of the photo on the ground; calculating the coordinates of the target in the ground projection coordinate system; using UTM back projection or Gauss-Krüger back projection to convert the coordinates of the target in the ground projection coordinate system into the longitude and latitude coordinates of the target on the ground; in view of the low accuracy of unmanned aerial vehicle photography measurement in the prior art, the present application improves the accuracy.

Description

Technical Field

[0001] The present application relates to the field of drone photography, and more specifically, to a target positioning method based on drone oblique photography; Background Art

[0002] With the rapid development of UAV technology, UAVs have been widely used in various fields, such as post-disaster search and rescue, emergency command, surveying and exploration, etc. In these application scenarios, using UAVs to accurately locate ground targets is a key technology. For example, in the process of post-disaster search and rescue, rescuers not only need to grasp the real-time position of the UAV, but also need to quickly obtain the geographic coordinates of the objects to be rescued photographed by the UAV in order to carry out rescue operations in a timely manner. Therefore, it is of great practical significance to study efficient and accurate UAV target positioning methods.

[0003] The basic principle of target positioning by UAV is to obtain the geographic coordinates of the target through a series of coordinate transformations and calculations based on the relative position of the target in the captured image, combined with the attitude information and camera parameters of the UAV. At present, the target positioning methods of UAV are mainly divided into two categories: traditional positioning methods based on sensors and intelligent positioning methods based on computer vision. Sensor-based positioning methods usually rely on high-performance GPS, IMU and other sensors to achieve target positioning by fusing multi-source sensor data, but such methods have high requirements for hardware equipment and the positioning effect is easily interfered by noise. Computer vision-based positioning methods use image processing and pattern recognition technology to achieve target positioning by analyzing the image data taken by the UAV. Such methods reduce the requirements for sensor performance and have stronger robustness and adaptability. However, most traditional computer vision positioning methods are limited to obtaining the relative position of the target in the image (such as pixel coordinates), which cannot be directly converted into the real geographic coordinates of the target, which restricts its effectiveness in practical applications. Therefore, it is urgent to deeply integrate advanced computer vision technology with the actual working scenes of UAVs, and further infer the geographic coordinates of the target on the basis of obtaining the relative position of the target to meet the needs of practical applications.

[0004] In the process of drone aerial photography, according to different shooting angles, it is usually divided into two modes: vertical photography and oblique photography. In vertical photography, since the shooting direction is perpendicular to the ground, the actual distance of the target from the drone can be inferred by the principle of similar triangles through the photo scale and the relative position of the target in the image, and then the geographic coordinates of the target can be calculated in combination with the position information of the drone. However, in the oblique photography mode, there is an angle between the shooting direction and the ground, which makes the mapping relationship between the position of the target in the image and its actual geographic location more complicated, and the target coordinates cannot be directly obtained by simple similarity calculation. Therefore, in the case of drone oblique photography, how to make full use of image information and drone attitude data, build an accurate mapping model between pixel coordinates, image plane coordinates and geographic coordinates, and realize the conversion of target positioning results from relative coordinates in the field of computer vision to practical geographic coordinates has become a key problem to be solved at present. Summary of the invention

[0005] 1. Technical issues to be solved

[0006] In view of the low accuracy of drone photogrammetry in the prior art, the present application provides a target positioning method based on drone oblique photography, which constructs a target detection model based on the YOLOv7 model, introduces the Swin-Transformer block module to enhance feature extraction and fusion capabilities, and combines drone state parameters, camera internal parameters, and UTM projection / Gauss-Krüger projection information to establish a mapping relationship between pixel coordinates, image plane coordinates, and ground coordinates, so as to achieve accurate conversion and positioning of the target from pixel coordinates to ground longitude and latitude coordinates;

[0007] 2. Technical solution

[0008] The purpose of this application is achieved through the following technical solutions:

[0009] The embodiment of this specification provides a method for positioning a target based on drone oblique photography, including: S1, constructing a target detection model based on a neural network to pre-process the photos taken by the drone, and outputting the pixel coordinates of the target in the photo , and the pixel coordinates of the center point of the photo ; S2, obtain the state parameters of the drone and the camera; according to the conversion relationship between pixel coordinates and image plane coordinates, the pixel coordinates of the target are converted to and the pixel coordinates of the center point of the photo , respectively converted into the target image plane coordinates and the photo center point image plane coordinates ; S3, use UTM projection or Gauss-Krüger projection to convert the longitude and latitude of the drone Convert to projected coordinates ; According to the pixel coordinates of the center point of the photo , calculate the projection coordinates of the center point of the photo on the ground ; S4, calculate the coordinates of the target in the ground projection coordinate system according to the flight altitude H of the drone, the tilt angle of the gimbal, the focal length f of the lens, the field of view angle and the pixel spacing d ; S5, using UTM back projection or Gauss-Krüger back projection, the coordinates of the target in the ground projection coordinate system Converted to the latitude and longitude coordinates of the target on the ground .

[0010] Furthermore, S1 outputs the pixel coordinates (u, v) of the target in the photo and the pixel coordinates of the center point of the photo. , including: constructing a target detection model based on the YOLOv7 model, the YOLOv7 model includes a backbone network and a detection head; preprocessing the input photos taken by the drone to obtain RGB images; extracting features from the obtained RGB images through the backbone network and outputting a feature matrix; using the detection head to train the output feature matrix, and combining different feature matrices through upsampling to output the type and border of the target; replacing the ELAN module in the YOLOv7 backbone network with a Swin-Transformer block module; the Swin-Transformer block module includes a window multi-head self-attention mechanism W-MSA module and a sliding window multi-head self-attention mechanism SW-MSA module; the W-MSA module divides the feature matrix into multiple non-overlapping windows, and the multi-head self-attention mechanism MSA calculates the attention score of the features in each independent window; the SW-MSA module fuses the feature matrices of multiple windows by shifting the shift window cyclically, and uses the Mask mechanism to fuse contextual information of different scales to interact features between different windows; setting the coordinates of the target to the position of the center point of the detection border, and according to the output target detection result, calculating the pixel coordinates of the target in the photo by the following formula : , , in, Indicates the pixel coordinates of the upper left corner of the detection border. Indicates the pixel coordinates of the lower right corner of the detection frame; according to the length w and width h of the photo, the pixel coordinates of the center point of the photo are calculated by the following formula : , .

[0011] Furthermore, the three feature matrices outputted are trained by the detection head, and different feature matrices are combined by upsampling to output the type and border of the target, including: inputting multiple feature matrices of different sizes outputted by the backbone network into the corresponding detection head respectively, and each detection head includes multiple convolution layers and upsampling layers; in each detection head, convolution operation is performed on the input feature matrix to extract target features at different scales to obtain a preliminary target feature matrix; in each detection head, the preliminary target feature matrix is ​​input into a multi-scale feature fusion module; the multi-scale feature fusion module includes an upsampling unit and a connection unit; the preliminary target feature matrix is ​​scale-normalized by the upsampling unit to unify the resolution of the preliminary target feature matrix to the same scale as the input image to obtain a scale-normalized target feature matrix; the scale-normalized target feature matrix is ​​spliced ​​by the connection unit to obtain a fused feature matrix; in each detection head, the fused feature matrix is ​​convoluted to output the type and position information of the target through a fully connected layer; according to the position information of the target, the non-maximum suppression (NMS) algorithm is used to remove overlapping detection frames to obtain the final target border coordinates. ,in, Indicates the coordinates of the upper left corner of the border. Indicates the coordinates of the lower right corner of the border;

[0012] Furthermore, the W-MSA module divides the feature matrix into multiple non-overlapping windows, and the multi-head self-attention mechanism MSA calculates the attention score of the features in each independent window, including: evenly dividing the feature matrix into multiple window areas of equal size and non-overlapping, each window area contains a sub-block of the feature matrix; inputting the feature sub-blocks in each window area into the multi-head self-attention mechanism MSA, calculating the attention score matrix of the features in the corresponding window area, and the attention score matrix reflects the correlation between the features at different positions in the corresponding window area; the multi-head self-attention mechanism MSA contains multiple parallel self-attention heads, each of which is independent. Calculate the attention score matrix, concatenate the attention score matrices calculated by different self-attention heads, and obtain the attention feature matrix of the corresponding window area through linear transformation; in each self-attention head, input the feature sub-blocks of the window area into the query vector matrix Q, the key vector matrix K and the value vector matrix V respectively, and calculate the attention score matrix by the dot product of Q and K. According to the attention score matrix and V, obtain the output feature matrix of the corresponding self-attention head; concatenate the attention feature matrices of all window areas to obtain the feature matrix output by the W-MSA module. The features of each position in the feature matrix are integrated with the contextual information in the window area;

[0013] Furthermore, the SW-MSA module fuses the feature matrices of multiple windows by shifting the Window cyclically, and uses the Mask mechanism to fuse contextual information of different scales to interact features between different windows, including: performing a cyclic shift operation on the feature matrix output by the W-MSA module, spatially shifting the feature sub-blocks of each window area so that the shifted window area partially overlaps with the original window area to form a shifted feature matrix; for the shifted feature matrix, using the multi-head attention mechanism MSA to calculate the attention score matrix of the features in the shifted window area to obtain the attention feature matrix of the shifted window area; using the Mask mechanism to generate a Mask matrix according to the overlapping relationship between the shifted window area and the original window area. The elements in the matrix indicate whether the features at the corresponding positions participate in the fusion of information of different scales; perform element-by-element multiplication of the Mask matrix and the shifted attention feature matrix, set the elements of the attention feature matrix at the corresponding positions of the elements in the Mask matrix to zero, and obtain the attention feature matrix that fuses information of different scales; perform a reverse shift operation on the attention feature matrix that fuses information of different scales, restore the shifted window area to the original position, and obtain the restored attention feature matrix to align the features between different windows; perform element-by-element addition of the restored attention feature matrix and the feature matrix output by the W-MSA module, and add the fused features between different windows to the original features element-by-element to obtain the output feature matrix of the SW-MSA module;

[0014] Further, S2, the pixel coordinates of the target and the pixel coordinates of the center point of the photo , respectively converted to the coordinates (x, y) and , including: obtaining the camera's yaw angle by parsing the EXIF ​​data of photos taken by the drone , and the obtained yaw angle Make corrections, the correction formula is as follows: ,in, Indicates the corrected yaw angle; Indicates the camera's yaw angle; obtains the drone's flight altitude H and the drone's flight latitude and longitude and the tilt angle of the gimbal; obtain the focal length f, field of view angle and pixel spacing d of the camera lens; use the following formula to calculate the pixel coordinates of the target and the pixel coordinates of the center point of the photo To perform coordinate transformation: , ,in, represents the target pixel coordinates, Indicates the pixel coordinates of the center point of the photo; represents the target image plane coordinates, The image plane coordinates representing the center point of the photo; and Indicates the physical size of a pixel in the x- and y-axis directions.

[0015] Further, S3, calculates the projection coordinates of the center point of the photo on the ground , including: the longitude and latitude of the drone flight obtained using UTM projection or Gauss-Krüger projection Convert to projection coordinates in ground coordinate system ;According to the obtained flight altitude H and gimbal tilt angle of the drone , the horizontal distance between the camera and the target is calculated by the following formula : ,in, It represents the distance between the optical center of the camera and the center point of the photo on the horizontal ground. H represents the flight altitude of the drone. Indicates the tilt angle of the gimbal; according to the horizontal distance and the corrected camera yaw angle , calculate the projection coordinates of the center point of the photo on the ground using the following formula : .

[0016] Further, S5, the coordinates of the target in the ground projection coordinate system Converted to the latitude and longitude coordinates of the target on the ground , including: according to the flight height H of the drone, the tilt angle of the gimbal , the focal length f of the lens, the field of view angle and the pixel spacing d, calculate the spatial resolution ratio of the photo; when the horizontal distance When it is less than the threshold, it is the perigee, and the spatial resolution ratio of the perigee is: , when the horizontal distance When it is within the preset range, it is the center point, and the spatial resolution ratio of the center point is: , when the horizontal distance When it is greater than the threshold, it is the apogee. The spatial resolution ratio of the apogee is: , , where k represents the distance between the camera's perigee and the center point on the photo. is half of the field of view; ratio represents the spatial resolution; f represents the focal length of the camera; according to the spatial resolution ratio of the photo, the distance between the target and the center point of the photo on the ground is calculated by the following formula : .

[0017] in, Indicates the ground distance of the target from the center point of the photo. Indicates the distance between the target and the center point of the photo. Ratio refers to the spatial resolution of the photo. In oblique aerial photography, the center point usually refers to the center of the oblique image, that is, the area near the intersection of the camera's main optical axis and the ground. The perigee refers to the ground area below the center point in the oblique image that is closer to the camera. The apogee refers to the ground area above the center point in the oblique image that is farther from the camera. Among them, k represents the distance between the camera's perigee and the center point in the photo. is half of the field of view; ratio represents the spatial resolution; f represents the focal length of the camera; according to the spatial resolution ratio of the photo, the distance between the target and the center point of the photo on the ground is calculated by the following formula : ,in, Indicates the ground distance of the target from the center point of the photo. It indicates the distance of the target from the center of the photo, and ratio refers to the spatial resolution of the photo;

[0018] Furthermore, the angle between the line connecting the target and the center point on the photo and the true north direction of the photo is calculated by the following formula: :

[0019]

[0020] in, Indicates the angle between the target and the center point in the photo; the angle calculated based on and the corrected photo yaw angle , the actual deflection angle is calculated by the following formula , that is, the line between the target and the center of the photo and the north direction: .

[0021] Furthermore, according to the projection coordinates of the center point of the photo on the ground , photo spatial resolution ratio, the actual deflection angle of the target relative to the true north direction , solve the projection coordinates of the ground target , and the projection coordinates Converted to the latitude and longitude coordinates of ground targets through UTM back-projection or Gauss-Krüger back-projection :

[0022] .

[0023] 3. Beneficial effects

[0024] Compared with the prior art, the advantages of this application are:

[0025] The target detection model based on the YOLOv7 model is introduced. The backbone network extracts multi-scale features, and the detection head is used for feature fusion and target detection. The pixel coordinates of the target in the photo can be accurately obtained, providing reliable initial information for the subsequent coordinate transformation. At the same time, the ELAN module in the YOLOv7 backbone network is replaced with the Swin-Transformerblock module, and the window multi-head self-attention mechanism W-MSA and the sliding window multi-head self-attention mechanism SW-MSA are introduced to enhance the feature extraction and fusion capabilities of the model, and improve the accuracy and robustness of target detection.

[0026] By acquiring the state parameters of the drone and the internal parameters of the camera, the mapping relationship between pixel coordinates, image plane coordinates and ground coordinates is established; the target pixel coordinates and the pixel coordinates of the center point of the photo are converted into image plane coordinates, eliminating the influence of camera lens distortion; the longitude and latitude of the drone are converted into projection coordinates using UTM projection or Gauss-Krüger projection, and the projection coordinates of the center point of the photo on the ground are calculated by combining the pixel coordinates of the center point of the photo, the internal parameters of the camera and the state parameters of the drone, and the association between the image plane coordinates and the ground coordinates is established; this coordinate conversion method fully considers the influence of the drone posture and camera parameters, and improves the accuracy of coordinate conversion;

[0027] According to the state parameters of the UAV and the internal parameters of the camera, the coordinates of the target in the ground projection coordinate system are calculated, and the UTM back projection or Gauss-Krüger back projection is used to convert them into ground longitude and latitude coordinates to achieve target positioning. In the coordinate conversion process, the flight altitude of the UAV, the tilt angle of the gimbal, the focal length of the camera, the field of view and other parameters are comprehensively considered to establish the geometric relationship between the target in the photo, the image plane and the ground coordinate system, ensuring the accuracy and reliability of the coordinate conversion. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 A schematic diagram of a target positioning method based on oblique photography of a drone in this application;

[0029] Figure 2 A diagram of the drone space modeling for this application;

[0030] Figure 3 The state parameter diagram of the drone for this application;

[0031] Figure 4 The latitude and longitude coordinates of the ground target for this application; DETAILED DESCRIPTION

[0032] The present application is described in detail below in conjunction with the accompanying drawings and specific embodiments;

[0033] Figure 1This is a flow chart of a target positioning method based on drone oblique photography of the present application, comprising: S1, constructing a target detection model based on a neural network to pre-process the photos taken by the drone, and outputting the pixel coordinates (u, v) of the target in the photo, as well as the pixel coordinates of the center point of the photo ; S2, obtain the state parameters of the drone and camera, the state parameters include the flight altitude H of the drone, the latitude and longitude of the drone flight , gimbal tilt angle, camera internal parameters, lens focal length f, field of view angle and pixel spacing d; according to the conversion relationship between pixel coordinates and image plane coordinates, the pixel coordinates of the target (u, v) and the pixel coordinates of the center point of the photo are converted into , respectively converted into the target image plane coordinates (x, y) and the image plane coordinates (m, n) of the center point of the photo; the image plane coordinate system is a coordinate system established with the center of the photo as the origin; image plane coordinates: expressed in physical units (such as millimeters), the origin is the intersection of the camera optical axis and the image plane, the coordinate axis is parallel to the edge of the image or the frame mark line, and is used to describe the actual physical position of the point in the image; pixel coordinates: expressed in pixels, the origin is located in the upper left corner of the image, the coordinate axis represents the number of columns and rows of pixels, and is used to directly describe the position of pixels in the image; S3, the longitude and latitude of the drone are projected using UTM projection or Gauss-Krüger projection Convert to projected coordinates ; According to the pixel coordinates of the center point of the photo , the focal length f of the lens, the pixel size d, the flight altitude H of the drone and the tilt angle of the gimbal, calculate the projection coordinates of the center point of the photo on the ground ; S4, calculate the coordinates of the target in the ground projection coordinate system according to the flight altitude H of the drone, the tilt angle of the gimbal, the focal length f of the lens, the field of view angle and the pixel spacing d ; S5, using UTM back projection or Gauss-Krüger back projection, the coordinates of the target in the ground projection coordinate system Converted to the latitude and longitude coordinates of the target on the ground ;

[0034] A target detection model based on neural network is constructed, and YOLOv7 is selected as the basic architecture of the target detection model. The YOLOv7 model is mainly composed of two parts: the backbone network and the detection head. The backbone network adopts classic convolutional neural network structures such as Deep Residual Network (ResNet) or Aggregated Residual Transformations (ResNeXt) to extract the features of the input image. The detection head adopts the Feature Pyramid Network (FPN) structure. Through upsampling and feature fusion, the different scale features extracted by the backbone network are fused to achieve the detection of targets of different sizes. The original image taken by the drone is read as a picture in BGR format. The BGR picture is converted to RGB picture, that is, the color channel order of the picture is converted from BGR to RGB. The RGB picture is resized and scaled to the fixed size required for the model input (such as 640x640). The scaled RGB picture is normalized and the pixel value is mapped from [0, 255] to [0, 1] range to suit the input requirements of the model;

[0035] The preprocessed RGB image is input into the backbone network of the YOLOv7 model; the backbone network extracts features from the input image through a series of convolutional layers, pooling layers and activation functions; at different stages of the backbone network, three feature matrices of different sizes are extracted, corresponding to different receptive fields and semantic information; for example, the outputs of the 3rd, 4th and 5th stages of the backbone network can be used as feature matrices of three different scales, denoted as C3, C4 and C5; the three feature matrices C3, C4 and C5 output by the backbone network are respectively input into different branches of the detection head; in each branch, a series of convolution operations and upsampling operations are performed on the feature matrix to extract target features at different scales; the low-resolution feature matrix (such as C5) is upsampled to the same size as the high-resolution feature matrix (such as C3) through upsampling operations; the upsampled feature matrix is ​​element-by-element added to the feature matrix of the corresponding scale to achieve the fusion of features of different scales; the fused feature matrix is ​​convoluted, and the type and border information of the target are output through the fully connected layer;

[0036] The detection head consists of multiple parallel detection branches, each branch corresponds to a feature matrix of a specific scale; the feature matrices C3, C4, and C5 output by the backbone network are respectively input into the corresponding detection branches; each detection branch contains multiple convolutional layers and upsampling layers for extracting and fusing feature information of different scales; the convolutional layer uses 3x3 or 1x1 convolution kernels to extract local features and adjust the number of channels; the upsampling layer uses an interpolation method (such as bilinear interpolation) to upsample the feature matrix to increase its resolution to the same as that of feature matrices of other scales; in each detection branch, a series of convolution operations are performed on the input feature matrix; the convolution operation uses convolution kernels of different sizes and numbers to extract target features of different scales and semantic levels; for example, multiple 3x3 convolutional layers can be used to extract local features, and 1x1 convolutional layers can be used to adjust the number of channels; the output of the convolution operation is a preliminary target feature matrix, which contains target information at different scales; in each detection branch, the preliminary target feature matrix is ​​input into the multi-scale feature fusion module;

[0037] The multi-scale feature fusion module consists of an upsampling unit and a connection unit; the upsampling unit performs an upsampling operation on the preliminary target feature matrix to increase its resolution to the same as the feature matrices of other scales; the connection unit concatenates or element-wise adds the upsampled feature matrix with the feature matrices of other scales to achieve the fusion of features of different scales; the upsampling unit uses an interpolation method (such as bilinear interpolation) to upsample the preliminary target feature matrix; the purpose of upsampling is to unify the resolution of feature matrices of different scales to the same scale as the input image; for example, if the size of the input image is 640x640, The preliminary target feature matrix is ​​upsampled to a size of 640x640; after upsampling, the scale-normalized target feature matrix is ​​obtained, and its resolution is the same as that of the input image; the connection unit concatenates or adds the scale-normalized target feature matrix element-wise; the concatenation operation concatenates feature matrices of different scales in the channel dimension to obtain a fused feature matrix containing multi-scale feature information; the element-wise addition operation adds the elements of corresponding positions of feature matrices of different scales to obtain a fused feature matrix; the fused feature matrix contains target information at different scales, which helps to improve detection performance;

[0038] A series of convolution operations are performed on the fused feature matrix to extract high-level semantic features. The convolution operation uses convolution kernels of different sizes and numbers, such as 3x3 or 1x1. The output of the convolution operation is classified and regressed through the fully connected layer. The fully connected layer outputs the type and location information of the target, including the category probability and bounding box coordinates of the target. For the target location information output by each detection head, the non-maximum suppression (NMS) algorithm is used for post-processing. The purpose of the NMS algorithm is to remove overlapping detection frames and retain the best detection results. First, the detection frames are sorted according to the category probability of the target, and the detection frame with the highest probability is selected as the reference frame. Then, the intersection over union (IoU) of the reference frame and other detection frames is calculated. Union (IoU), IoU represents the overlap of two boxes; if IoU is greater than a preset threshold (such as 0.5), the two boxes are considered to overlap, and the detection box with a lower probability is removed; repeat the above process until all detection boxes are processed; the final detection box is the optimal frame coordinates of the target (x1, y1, x2, y2), where (x1, y1) represents the coordinates of the upper left corner of the frame, and (x2, y2) represents the coordinates of the lower right corner of the frame;

[0039] The ELAN module in the YOLOv7 backbone network is replaced with the Swin-Transformer block module. The Swin-Transformer block module consists of two main components: Window-based Multi-head Self-Attention (W-MSA) and Shifted Window-based Multi-head Self-Attention (SW-MSA). The W-MSA module calculates self-attention within a local window to capture the relationship between local features. The SW-MSA module establishes connections between different windows by sliding windows to capture the relationship between global features. The ELAN module in the YOLOv7 backbone network is replaced with the Swin-Transformer block module to introduce the self-attention mechanism and improve the ability of feature representation.

[0040] The MSA module divides the feature matrix into multiple non-overlapping windows, and the multi-head self-attention mechanism MSA calculates the attention score of the features in each independent window; for the input feature matrix, it is divided into multiple window areas of the same size and non-overlapping; each window area contains a sub-block of the feature matrix, and the size of the sub-block is determined by the window size, usually set to 7x7 or 14x14; for each window area, the multi-head self-attention mechanism (Multi-head Self-Attention (MSA) calculates the attention score of the features in the window; the MSA mechanism generates an attention score matrix by calculating the similarity between features, which represents the correlation between features at different positions in the window; the input feature matrix is ​​evenly divided in the spatial dimension according to the preset window size; for example, if the size of the feature matrix is ​​56x56 and the window size is set to 7x7, the feature matrix can be divided into 8x8 window areas, and the size of each window area is 7x7; each window area contains a sub-block of the feature matrix, and the elements in the sub-block have adjacent spatial positions in the original feature matrix; the window areas do not overlap, that is, each feature belongs to only one window area; for each window area, the feature sub-block inside it is input into the multi-head self-attention mechanism MSA; the MSA mechanism generates an attention score matrix by calculating the similarity between features; the size of the attention score matrix is ​​the same as the size of the window area, and each element in the matrix represents the correlation between the features at the corresponding position and the features at other positions; the attention score matrix reflects the correlation between the features at different positions in the window area, and the features in the window can be weighted and aggregated through the attention score matrix;

[0041] The MSA mechanism consists of multiple parallel self-attention heads, and each self-attention head independently calculates the attention score matrix; for each self-attention head, the feature sub-blocks in the window area are input into the query matrix (Query), key matrix (Key) and value matrix (Value) respectively; the attention score matrix is ​​obtained by calculating the dot product of the query matrix and the key matrix, which represents the correlation between the features at different positions in the window; according to the attention score matrix and the value matrix, the output feature matrix of the self-attention head is obtained by weighted summation; the output feature matrices calculated by different self-attention heads are spliced ​​in the channel dimension; the spliced ​​feature matrix is ​​linearly transformed to obtain the attention feature matrix of the corresponding window area; for each self-attention head, the feature sub-blocks in the window area are input into three linear transformation layers respectively to obtain the query matrix Q, key matrix K and value matrix V; the size of the query matrix Q, key matrix K and value matrix V is (window_size, window_size, dim_head), where window_size represents the window size and dim_head represents the dimension of the self-attention head; the attention score matrix is ​​obtained by performing dot product calculation on the transpose of the query matrix Q and the key matrix K, which represents the correlation between the features at different positions in the window;

[0042] The attention score matrix is ​​scaled and softmax normalized to obtain the normalized attention weight matrix; the normalized attention weight matrix is ​​matrix multiplied with the value matrix V to obtain the weighted aggregated output feature matrix; the size of the output feature matrix is ​​(window_size, window_size, dim_head), which represents the weighted aggregation result of the self-attention head on the features in the window; for each window area, the corresponding attention feature matrix is ​​calculated by the multi-head self-attention mechanism MSA; the attention feature matrices of all window areas are spliced ​​in the spatial dimension to obtain the feature matrix output by the W-MSA module; the size of the output feature matrix is ​​the same as that of the input feature matrix, but the features at each position are integrated with the contextual information in the window area; through the W-MSA module, the features at each position in the feature matrix capture the contextual information in the local window, enhancing the ability of feature representation;

[0043] The SW-MSA module introduces a shifted window and mask mechanism based on the W-MSA module. Through the shift operation, the feature matrix output by the W-MSA module is shifted in the spatial dimension so that the shifted window area partially overlaps with the original window area. For the shifted feature matrix, the multi-head self-attention mechanism MSA is used to calculate the attention score matrix of the features in the shifted window area. The Mask mechanism is used to generate a Mask matrix based on the overlapping relationship between the shifted window area and the original window area to control the fusion of information of different scales. Through the element-by-element multiplication of the Mask matrix and the shifted attention feature matrix, the context information of different scales is fused. The fused attention feature matrix is ​​reversely shifted to restore the shifted window area to its original position to align the features between different windows. The restored attention feature matrix is ​​added element-by-element to the feature matrix output by the W-MSA module to obtain the output feature matrix of the SW-MSA module.

[0044] The feature matrix output by the W-MSA module is subjected to a cyclic shift operation to spatially shift the feature sub-blocks of each window area. The step length of the shift is usually set to half the window size. For example, if the window size is 7x7, the step length of the shift is 3. Through the cyclic shift operation, the shifted window area is partially overlapped with the original window area to form a shifted feature matrix. In the shifted feature matrix, the features of each position come from the features of different window areas in the original feature matrix, realizing feature interaction between different windows. The shifted feature matrix is ​​divided into multiple shifted window areas. For each shifted window area, the multi-head self-attention mechanism MSA is used to calculate the attention score matrix of the features within the window. The MSA mechanism generates an attention score matrix by calculating the similarity between the features within the shifted window area, which represents the correlation between the features at different positions within the shifted window.

[0045] According to the attention score matrix and the features in the shifted window area, the attention feature matrix of the shifted window area is obtained by weighted aggregation; according to the overlapping relationship between the shifted window area and the original window area, a Mask matrix is ​​generated; the size of the Mask matrix is ​​the same as the shifted feature matrix, and each element in the matrix takes the value of 0 or 1; for the position in the shifted window area that overlaps with the original window area, the element at the corresponding position in the Mask matrix is ​​set to 1, indicating that the feature at this position participates in the fusion of information of different scales; for the position in the shifted window area that does not overlap with the original window area, the element at the corresponding position in the Mask matrix is ​​set to 0, indicating that the feature at this position does not participate in the fusion of information of different scales; the Mask matrix is ​​multiplied element by element with the shifted attention feature matrix; the attention feature matrix elements at the corresponding positions of the elements of 1 in the Mask matrix remain unchanged, indicating that the features at this position participate in the fusion of information of different scales; the attention feature matrix elements at the corresponding positions of the elements of 0 in the Mask matrix are set to zero, indicating that the features at this position do not participate in the fusion of information of different scales; through the control of the Mask matrix, the selective fusion of information of different scales is realized, and the interference of irrelevant information is avoided;

[0046] The attention feature matrix that integrates information of different scales is subjected to a reverse shift operation to restore the displaced window area to its original position. The reverse shift operation is opposite to the previous circular shift operation, and the displaced window area is shifted in the opposite direction to restore it to its original position. For example, if the previous circular shift operation shifts the window area to the lower right by 3 pixels, the reverse shift operation shifts the window area to the upper left by 3 pixels to return it to its original position. Through the reverse shift operation, a restored attention feature matrix is ​​obtained, in which the features at each position correspond to the features at the same position in the original feature matrix. The restored attention feature matrix has the same size and position correspondence as the original feature matrix, and the alignment of features between different windows is achieved. The restored attention feature matrix is ​​element-by-element added to the feature matrix output by the W-MSA module. The feature matrix output by the W-MSA module represents the feature representation within the local window, and the restored attention feature matrix represents the feature representation that integrates information from different windows.

[0047] By element-by-element addition, the eigenvalues ​​of corresponding positions in the two feature matrices are added to obtain a feature representation that integrates local and global information; let the feature matrix output by the W-MSA module be F_w, and the restored attention feature matrix be F_sw, then the output feature matrix F_out of the SW-MSA module can be expressed as: F_out = F_w + F_sw; where the size of F_out is the same as that of F_w and F_sw, and the eigenvalue of each position is the sum of the eigenvalues ​​of F_w and F_sw at the corresponding position; by element-by-element addition, the output feature matrix of the SW-MSA module integrates the features within the local window and the features obtained by interaction between different windows, enhancing the ability of feature representation and capturing richer contextual information;

[0048] Set the coordinates of the target to the position of the center point of the detection frame. According to the output target detection results, calculate the pixel coordinates (u, v) of the target in the photo using the following formula: , , in, Indicates the pixel coordinates of the upper left corner of the detection border. Indicates the pixel coordinates of the lower right corner of the detection frame; according to the length w and width h of the photo, the pixel coordinates of the center point of the photo are calculated by the following formula : ; .

[0049] Define the coordinates of the target as the location of the center point of the detection bounding box. According to the output of the above target detection model, calculate the pixel coordinates (u, v) of the target on the photo: , ,in, Indicates the pixel coordinates of the upper left corner of the detection border. Indicates the pixel coordinates of the lower right corner of the detection frame; in addition, the pixel coordinates of the center point of the photo : , ;

[0050] Collect the status parameters of the drone and camera; at the same time, convert the pixel coordinates of the center point and the target of the photo into image plane coordinates; obtain the relevant parameters of the drone and camera; first, since the JPG photos taken by the drone have EXIF ​​information, that is, it contains the status information of the camera when shooting; therefore, by parsing the EXIF ​​data of the photo, the status information of the camera is obtained, and the yaw angle of the obtained photo is calculated. (angle relative to true north) to correct for ,in, Indicates the corrected yaw angle; Indicates the camera's yaw angle; then, the drone's flight altitude H (height relative to the ground), the drone's longitude and latitude (lat, lon), and the gimbal tilt angle are obtained through the status information released by the drone's communication module mavros; finally, refer to the factory manual of the drone's camera to supplement the camera's basic parameters, including the camera's internal parameters, the lens's focal length f, the field of view angle, and the pixel pitch d.

[0051] Coordinate conversion: Considering that the output of the YOLOv7 model is the coordinates of the target in the pixel coordinate system, however, the pixel coordinate system only represents the row and column number of the target in the digital image, and cannot accurately reflect the physical position of the pixel; therefore, in order to calculate the physical distance between the target and the center point in the photo, the pixel coordinates of the two are converted into and Convert to the coordinates (x, y) and (m, n) in the image plane coordinate system respectively: , ,in, represents the target pixel coordinates, represents the pixel coordinates of the center point of the photo; (x, y) represents the coordinates of the target image plane, Represents the image plane coordinates of the center point of the photo; dx and dy represent the physical sizes of the pixel in the x-axis and y-axis directions.

[0052] Figure 2 For the drone spatial modeling diagram of this application, the projection coordinates of the center point of the photo corresponding to the ground position are calculated according to the drone flight altitude, the drone latitude and longitude coordinates, the gimbal tilt angle and the photo yaw angle; the UTM projection or Gauss-Krüger projection is used to convert the obtained drone flight latitude and longitude (lat, lon) into the projection coordinates in the ground coordinate system ;According to the obtained flight altitude H and gimbal tilt angle of the drone , the horizontal distance between the camera and the target is calculated by the following formula : ,in, It represents the distance between the optical center of the camera and the center point of the photo on the horizontal ground. H represents the flight altitude of the drone. Indicates the tilt angle of the gimbal; according to the horizontal distance and the corrected camera yaw angle , calculate the projection coordinates of the center point of the photo on the ground using the following formula : ;

[0053] The latitude and longitude coordinates of the target are inferred by combining the image plane coordinates of the target, the image plane coordinates of the center point of the photo, the flight altitude of the drone, the tilt angle of the gimbal, the yaw angle of the photo, and the projection coordinates of the ground position corresponding to the center point of the photo;

[0054] Figure 3 The state parameter diagram of the drone in this application, G, w, They are the edge point, center point and any point on the photo, O is the center point of the camera, H is the height from the camera to the ground, and T is the height on the photo. The center point is the ground position corresponding to w on the photo, and the perigee is the position corresponding to G on the photo. The three angles are the angle between the camera center point and the ground center point and the vertical direction, the angle between the camera center point and the ground center point and the ground perigee, and the angle between the camera center point and the ground apogee and perigee. According to the flight altitude H of the drone and the tilt angle of the gimbal , the focal length f of the lens, the field of view angle and the pixel spacing d, the spatial resolution ratio of the photo is calculated by the following formula: When it is less than the threshold, it is the perigee. The perigee refers to the ground area that is closer to the camera and below the center point in the oblique image. To determine whether a target point is the perigee, you can compare the horizontal distance between the target point and the camera with the preset threshold. The spatial resolution ratio of the perigee is: , when the horizontal distance If it is within the preset range, it is the center point. The center point refers to the area near the intersection of the camera's main optical axis and the ground in the oblique image. In practical applications, you can determine whether the point is the center point based on whether the horizontal distance between the target point and the camera is within the preset range. The spatial resolution ratio of the center point is: , when the horizontal distance When it is greater than the threshold, it is the apogee. The apogee refers to the ground area that is far away from the camera and above the center point in the oblique image. To determine whether a target point is the apogee, you can compare the horizontal distance between the target point and the camera with the preset threshold. The spatial resolution ratio of the apogee is: , , where k represents the distance between the camera's perigee and the center point on the photo. is half of the field of view; ratio represents the spatial resolution; f represents the focal length of the camera. Setting the near-point threshold Thres_near and the far-point threshold Thres_far can effectively determine the type of target point in the oblique image, and then use different spatial resolution calculation formulas to improve the accuracy of target positioning. In practical applications, the judgment criteria can be dynamically adjusted according to the flight altitude, gimbal tilt angle and other parameters of the drone to adapt to different shooting conditions and application requirements.

[0055] Figure 4The latitude and longitude coordinates of the ground target in this application are calculated according to the spatial resolution ratio of the photo by the following formula: : ,in, It indicates the ground distance between the target and the center point of the photo, l indicates the photo distance between the target and the center point of the photo, and ratio refers to the spatial resolution of the photo;

[0056] The angle between the line connecting the target and the center point on the photo and the true north direction of the image is calculated using the following formula: :

[0057]

[0058] in, Indicates the angle between the target and the center point in the photo;

[0059] According to the calculated angle and the corrected photo yaw angle , the actual deflection angle is calculated by the following formula , that is, the line between the target and the center of the photo and the north direction: .

[0060] According to the projection coordinates of the center point of the photo on the ground , photo spatial resolution ratio, the actual deflection angle of the target relative to the true north direction , solve the projection coordinates (P, Q) of the ground target, and convert the projection coordinates (P, Q) into the latitude and longitude coordinates (Lat, Lon) of the ground target through UTM back projection or Gauss-Krüger back projection:

[0061] , .

Claims

1. A target positioning method based on UAV oblique photography, characterized in that: S1, build a neural network-based target detection model to pre-process the photos taken by the drone and output the pixel coordinates of the target in the photo , and the pixel coordinates of the center point of the photo ; S2, obtain the state parameters of the drone and the camera; according to the conversion relationship between pixel coordinates and image plane coordinates, convert the pixel coordinates of the target in the photo and the pixel coordinates of the center point of the photo , respectively converted into the target image plane coordinates and the photo center point image plane coordinates ; S3, use UTM projection or Gauss-Krüger projection to convert the longitude and latitude of the drone Convert to projected coordinates ; According to the pixel coordinates of the center point of the photo , calculate the projection coordinates of the center point of the photo on the ground ; S4, calculates the coordinates of the target in the ground projection coordinate system according to the flight altitude H of the drone, the tilt angle of the gimbal, the focal length f of the lens, the field of view angle and the pixel spacing d ; S5, using UTM back projection or Gauss-Krüger back projection, the coordinates of the target in the ground projection coordinate system Converted to the latitude and longitude coordinates of the target on the ground ; In S1, a neural network-based target detection model is built to pre-process the photos taken by the drone and output the pixel coordinates of the target in the photo. , and the pixel coordinates of the center point of the photo ,include: Build an object detection model based on the YOLOv7 model, which includes a backbone network and a detection head; Preprocess the input photos taken by the drone to obtain RGB images; The backbone network is used to extract features from the obtained RGB image and output the feature matrix; The detection head is used to train the output feature matrix, and different feature matrices are combined through upsampling to output the type and border of the target; Replace the ELAN module in the YOLOv7 backbone network with the Swin-Transformer block module; the Swin-Transformer block module includes the window multi-head self-attention mechanism W-MSA module and the sliding window multi-head self-attention mechanism SW-MSA module; The W-MSA module divides the feature matrix into multiple non-overlapping windows, and the multi-head self-attention mechanism MSA calculates the attention score of the features in each independent window; The SW-MSA module fuses the feature matrices of multiple windows by shifting the Window cyclically, and uses the Mask mechanism to fuse context information of different scales to interact features between different windows. Set the target coordinates to the position of the center point of the detection frame. According to the output target detection results, calculate the pixel coordinates of the target in the photo using the following formula : in, Indicates the pixel coordinates of the upper left corner of the detection border. Indicates the pixel coordinates of the lower right corner of the detection border; According to the length w and width h of the photo, the pixel coordinates of the center point of the photo are calculated by the following formula : ； The detection head is used to train the three feature matrices outputted, and different feature matrices are combined through upsampling to output the type and border of the target, including: Multiple feature matrices of different sizes output by the backbone network are input into corresponding detection heads respectively. Each detection head contains multiple convolutional layers and upsampling layers. In each detection head, a convolution operation is performed on the input feature matrix to extract target features at different scales and obtain a preliminary target feature matrix; In each detection head, the preliminary target feature matrix is ​​input into a multi-scale feature fusion module; the multi-scale feature fusion module includes an upsampling unit and a connection unit; An upsampling unit is used to normalize the scale of the preliminary target feature matrix, and the resolution of the preliminary target feature matrix is ​​unified to the same scale as the input image, so as to obtain a scale-normalized target feature matrix; The connection unit is used to concatenate the scale-normalized target feature matrix to obtain the fused feature matrix; In each detection head, the fused feature matrix is ​​convolved and the type and location information of the target are output through the fully connected layer; According to the location information of the target, the non-maximum suppression (NMS) algorithm is used to remove overlapping detection frames to obtain the final target frame coordinates. ,in, Indicates the coordinates of the upper left corner of the target frame. Indicates the coordinates of the lower right corner of the target bounding box; The W-MSA module divides the feature matrix into multiple non-overlapping windows, and the multi-head self-attention mechanism MSA calculates the attention score of the features in each independent window, including: The feature matrix is ​​evenly divided into multiple window areas of equal size and non-overlapping, each window area contains a sub-block of the feature matrix; The feature sub-blocks in each window area are input into the multi-head self-attention mechanism MSA, and the attention score matrix of the features in the corresponding window area is calculated. The attention score matrix reflects the correlation between the features at different positions in the corresponding window area; The multi-head self-attention mechanism MSA contains multiple parallel self-attention heads. Each self-attention head independently calculates the attention score matrix. The attention score matrices calculated by different self-attention heads are concatenated and linearly transformed to obtain the attention feature matrix of the corresponding window area. In each self-attention head, the feature sub-blocks of the window area are input into the query vector matrix Q, the key vector matrix K and the value vector matrix V respectively. The attention score matrix is ​​obtained by the dot product calculation of Q and K. According to the attention score matrix and V, the output feature matrix of the corresponding self-attention head is obtained. The attention feature matrices of all window areas are concatenated to obtain the feature matrix output by the W-MSA module. The features at each position in the feature matrix incorporate the contextual information in the window area. The SW-MSA module fuses the feature matrices of multiple windows by shifting the Window cycle, and uses the Mask mechanism to fuse context information of different scales to interact features between different windows, including: The feature matrix output by the W-MSA module is subjected to a cyclic shift operation, and the feature sub-blocks of each window area are shifted in space, so that the shifted window area partially overlaps with the original window area, forming a shifted feature matrix; For the feature matrix after displacement, the multi-head attention mechanism MSA is used to calculate the attention score matrix of the features in the window area after displacement, and the attention feature matrix of the window area after displacement is obtained; The Mask mechanism is used to generate a Mask matrix based on the overlapping relationship between the displaced window area and the original window area. The elements in the Mask matrix indicate whether the features at the corresponding position participate in the fusion of information at different scales. Perform element-by-element multiplication of the Mask matrix and the shifted attention feature matrix, set the attention feature matrix elements corresponding to the elements of the Mask matrix that are 0 to zero, and obtain the attention feature matrix that integrates information of different scales; Perform a reverse shift operation on the attention feature matrix that integrates information of different scales, restore the shifted window area to its original position, and obtain the restored attention feature matrix to align the features between different windows; The restored attention feature matrix is ​​added element by element to the feature matrix output by the W-MSA module, and the fused features between different windows are added element by element to the original features to obtain the output feature matrix of the SW-MSA module.

2. The target positioning method based on drone oblique photography according to claim 1, characterized in that: In S2, the state parameters of the drone and the camera are obtained; according to the conversion relationship between pixel coordinates and image plane coordinates, the pixel coordinates of the target are converted to and the pixel coordinates of the center point of the photo , respectively converted into the target image plane coordinates and the photo center point image plane coordinates ,include: Obtain the camera's yaw angle by parsing the EXIF ​​data of photos taken by the drone , and the obtained yaw angle Make corrections, the correction formula is as follows: in, Indicates the corrected yaw angle; Indicates the yaw angle of the camera; Get the flight altitude H of the drone and the latitude and longitude of the drone and the pan tilt angle; Get the focal length f, field of view angle and pixel pitch d of the camera lens; The pixel coordinates of the target are calculated by the following formula and the pixel coordinates of the center point of the photo To perform coordinate transformation: in, represents the pixel coordinates of the center point of the photo; (x, y) represents the coordinates of the target image plane, The image plane coordinates representing the center point of the photo; and Indicates the physical size of a pixel in the x- and y-axis directions.

3. The target positioning method based on drone oblique photography according to claim 2, characterized in that: S3 uses UTM projection or Gauss-Kruger projection to convert the longitude and latitude of the drone into Convert to projected coordinates ; According to the pixel coordinates of the center point of the photo , calculate the projection coordinates of the center point of the photo on the ground ,include: The longitude and latitude of the drone flight are obtained using the UTM projection or the Gauss-Krüger projection Convert to projection coordinates in ground coordinate system ; According to the obtained flight height H and gimbal tilt angle of the drone , the horizontal distance between the camera and the target is calculated by the following formula : According to the horizontal distance and the corrected camera yaw angle , calculate the projection coordinates of the center point of the photo on the ground using the following formula : 。 4. The target positioning method based on drone oblique photography according to claim 3 is characterized in that: S5 uses UTM back projection or Gauss-Krüger back projection to convert the coordinates of the target in the ground projection coordinate system Converted to the latitude and longitude coordinates of the target on the ground ,include: According to the flight height H of the drone, the tilt angle of the gimbal , the focal length f of the lens, the field of view angle and the pixel spacing d, calculate the spatial resolution ratio of the photo; When the horizontal distance When it is less than the threshold, it is the perigee, and the spatial resolution ratio of the perigee is: When the horizontal distance When it is within the preset range, it is the center point, and the spatial resolution ratio of the center point is: When the horizontal distance When it is greater than the threshold, it is the apogee. The spatial resolution ratio of the apogee is: Among them, k represents the distance between the camera's perigee and the center point on the photo. is half of the field of view; According to the spatial resolution ratio of the photo, the distance between the target and the center point of the photo on the ground is calculated by the following formula : in, Indicates the distance from the target to the center of the photo.

5. The target positioning method based on drone oblique photography according to claim 4, characterized in that: The angle between the line connecting the target and the center point on the photo and the true north direction of the image is calculated using the following formula: : in, Indicates the angle between the target and the center point in the photo; According to the calculated angle and the corrected photo yaw angle , the actual deflection angle is calculated by the following formula , that is, the angle between the line connecting the target and the center point of the photo and the true north direction: 。 6. The target positioning method based on drone oblique photography according to claim 5, characterized in that: According to the projection coordinates of the center point of the photo on the ground , photo spatial resolution ratio, the actual deflection angle of the target relative to the true north direction , solve the projection coordinates of the ground target , and the projection coordinates Converted to the latitude and longitude coordinates of ground targets through UTM back-projection or Gauss-Krüger back-projection : 。

Images