A monocular ranging method and system for unmanned aerial vehicles and personnel targets based on deep learning

Abstract

This invention discloses a monocular ranging method and system for UAVs and personnel targets based on deep learning. The method includes: detecting and identifying personnel targets using a deep learning target detection network; considering changes in the UAV's flight attitude, measuring the three-dimensional distance between the UAV and the personnel target using ranging theory; and calibrating the personnel target's position and distance measurement in the form of a bounding box, thus completing the identification, detection, and three-dimensional ranging of the personnel target from the UAV's perspective. This invention can achieve the identification, detection, and three-dimensional ranging of personnel targets from the UAV's perspective, and calibrate the target's position and distance measurement in the form of a bounding box.

Description

Technical Field

[0001] This invention relates to the field of target detection and ranging, and in particular to a monocular ranging method and system for UAVs and personnel targets based on deep learning. Background Technology

[0002] Target detection and ranging are key technologies in environmental perception, providing the algorithmic foundation for UAV emergency obstacle avoidance and path decision-making. Target ranging presupposes the identification and localization of objects in an image, i.e., target detection. Due to the complex and ever-changing environment during UAV flight, influenced not only by external environmental factors such as lighting, weather, noise, and shadows, but also by the image quality of the onboard camera itself, the robustness and generalization ability of algorithms are required to be higher, significantly increasing the difficulty of target detection. The distance between the UAV and the target is crucial information for understanding the surrounding environment and making control decisions. However, traditional target recognition algorithms suffer from poor accuracy and robustness in UAV flight scenarios, and struggle to accurately identify smaller targets, easily leading to false positives and false negatives. Furthermore, traditional monocular ranging methods do not consider changes in UAV flight attitude, resulting in significant measurement errors and poor performance in UAV scenarios. Therefore, proposing an improved method for personnel detection and monocular ranging on UAV platforms to achieve accurate identification and ranging of personnel targets is of great significance.

[0003] To address the problems of traditional monocular ranging technology, this invention utilizes deep learning to obtain the target's position in an image. By improving the YOLOv5 algorithm and employing a multi-scale feature fusion module and various attention mechanisms, it achieves accurate identification of personnel targets. Furthermore, by designing a monocular ranging model that considers attitude angles, it acquires 3D distance data between the UAV and the target, enabling precise identification and localization of personnel targets from the UAV's perspective, effectively overcoming the bottlenecks encountered by traditional monocular ranging technology.

[0004] This invention proposes a deep learning-based method for personnel identification and monocular ranging on unmanned aerial vehicle (UAV) platforms. By improving the deep learning network and optimizing the ranging theory, it achieves accurate identification and ranging of personnel targets from the UAV's perspective. This method can provide reliable target information support for UAVs flying in complex environments and has significant application prospects and value in improving target ranging and positioning accuracy. Summary of the Invention

[0005] This invention provides a monocular ranging method and system for UAVs and personnel targets based on deep learning. It can realize the identification, detection and three-dimensional ranging of personnel targets from the perspective of UAVs, and calibrate the target position and distance measurement in the form of a box.

[0006] The technical solution to achieve the purpose of this invention is as follows:

[0007] A monocular ranging method for UAVs and human targets based on deep learning includes:

[0008] Use a deep learning-based object detection network to detect and identify human targets;

[0009] Considering the changes in the flight attitude of the UAV, the three-dimensional distance between the UAV and the human target is measured using ranging theory.

[0010] The location and distance of personnel targets are calibrated in the form of a box, and the identification, detection and three-dimensional ranging of personnel targets from the perspective of the UAV are completed.

[0011] Furthermore, the method specifically includes the following steps:

[0012] (1) Calibrate the intrinsic parameters of the UAV's onboard camera and obtain the camera's focal length f and optical center coordinates (u0, v0);

[0013] (2) Obtain images frame by frame from the video stream captured by the UAV's onboard camera;

[0014] (3) Use deep learning target detection to identify the target person from frame-by-frame images and obtain the target's ranging feature points p(u, v); use the improved network Yolov5s-TinyVision based on the Yolov5s detection algorithm to perform target person detection and feature extraction, output the target person location information, and convert it into the target person's ranging feature points p(u, v);

[0015] (4) Use the fuselage sensors to read the UAV altitude H, UAV attitude angle data pitch angle β, and yaw angle δ;

[0016] (5) Input the camera focal length f, optical center coordinates (u0, v0) obtained in step (1), the personnel target ranging feature point p(u, v) obtained in step (3), and the aircraft attitude data obtained in step (4) into the optimized ranging model to obtain the three-dimensional projection distance L, lateral distance K, longitudinal distance D, and line distance O between the UAV and the personnel target. C P;

[0017] (6) Output the results of personnel target recognition, detection and ranging, and calibrate the target position and distance measurement in the form of a box, so as to complete the recognition, detection and three-dimensional ranging of personnel targets from the perspective of the UAV.

[0018] Furthermore, in step (1), when calibrating the intrinsic parameters of the UAV's onboard camera, the Zhang Zhengyou calibration method is adopted, and the calibration board is selected as a black and white checkerboard calibration board; the calibrated intrinsic parameter matrix is:

[0019]

[0020] Among them, f x =f / dx, f y = f / dy, where f is the camera focal length, and dx and dy are inherent camera parameters, representing the physical length of a unit pixel in the x and y directions, respectively.

[0021] Furthermore, in step (2), the airborne camera can be any type of camera.

[0022] Furthermore, in step (3), when using deep learning target detection to identify the target person from frame-by-frame images and obtain the target's ranging feature points p(u, v), the convolutional neural network model used in the deep learning target detection is a Yolov5s-TinyVision network based on the improved Yolov5s detection algorithm. It mainly includes the steps of using the deep learning neural network Yolov5s-TinyVision for feature extraction and using feature detection positioning information to obtain the ranging feature points p(u, v) of the target person. The deep learning neural network Yolov5s-TinyVision improves the network by using a small target feature processing module, a feature communication enhancement module, a multi-scale feature fusion module, and various attention mechanisms.

[0023] Furthermore, step (3) specifically includes the following steps:

[0024] (3-1) Using a drone scene dataset, the optimal anchor box size for personnel targets is obtained based on K-Means clustering and genetic algorithms; the dataset can be an actual shooting dataset or a network dataset for drone shooting scenarios, such as VisDrone2019, VisDrone2021, etc.

[0025] (3-2) Input image, use feature extraction network to initially extract target features X; where the feature extraction network can be any feature extraction network, such as CSPDarknet53, ResNet, etc.

[0026] (3-3) Use the small target feature processing module to process the initially extracted target features X to obtain features X that are more relevant to small targets. tiny The small target feature processing module is used to replace the cross-layer convolution and pooling layers in the feature extraction network. It consists of a space-to-depth (SPD) layer and a no-stride convolution layer.

[0027] (3-4) The target feature Xtiny is further processed using the feature communication enhancement module; the personnel target information is enhanced by feature communication using the CBAM attention mechanism to obtain the enhanced personnel target feature X.tiny-f ;

[0028] (3-5) Using a feature pyramid network, the enhanced personnel target features X tiny-f Processing is performed to obtain feature X. tiny-r The feature pyramid structure used can be any feature pyramid structure, such as FPN, PAN, BiFPN, etc.

[0029] (3-6) Use the detection head to target feature X tiny-r The process is performed to obtain relevant information on the target location of personnel.

[0030] (3-7) Output the personnel target location information, and obtain the coordinate information (x, y, y) of the personnel target bounding box in the image coordinate system. left y top ), (x right y bottom The coordinates of the ranging feature point p(u, v) of the personnel target are calculated as follows:

[0031]

[0032] Furthermore, the specific process of feature extraction for feature X using the small target feature processing module is as follows:

[0033]

[0034] In this context, Split(X, Scale) represents downsampling and slicing the feature map X by a scaling factor scale; Concat(·) represents concatenating the sliced ​​features along the channel direction; CBR(·) represents performing convolution, normalization, and activation function operations on the feature map in sequence; Sigmoid(·) represents performing Sigmoid function activation and normalization operations on the feature map; and CB(·) represents performing convolution and normalization operations on the feature map in sequence.

[0035] Furthermore, the specific calculation process of the feature communication enhancement module is as follows:

[0036] (3-4-1) Channel attention mechanism is used to enhance the target information of people in the channel; the formula for calculating the weight of each channel for channel attention is:

[0037] M C (X tiny =Sigmoid(MLP(AvgPool(X)) tiny ))+MLP(MaxPool(X tiny ))) (3)

[0038] X C =M C (Xtiny )·X tiny (4)

[0039] Where AvgPool(·) represents average pooling operation, MaxPool(·) represents max pooling operation, MLP(·) represents multilayer perceptron model, (·) represents element-based dot product, and (+) represents matrix addition.

[0040] (3-4-2) Obtain the feature X after channel attention enhancement C Then, a spatial attention mechanism is used to enhance the spatial target information of people, resulting in enhanced target features X. tiny-f The steps are as follows:

[0041] M S (X C )=Sigmoid(Conv(Concat(AvgPool(X C MaxPool(X) C (5)

[0042] X tiny-f =M S (X C )·X C (6)

[0043] Where Conv(·) represents the convolution operation;

[0044] Furthermore, steps (3-6) specifically include:

[0045] The three detection head structures of the Yolov5 Head are increased to four, and a small target detection layer is added; the detection head structure is a Transformer detection head structure; using the Transformer detection head structure, the features X obtained in steps (3-5) are processed. tiny-r The formula for utilizing the multiple attention mechanism of Transformer is as follows:

[0046] X trans-r =Conv(Concat(TB(Conv(X)) tiny-r )), Conv(X tiny-r ))) (7)

[0047] Wherein, TB(·) refers to the Transformer Block module based on the Transformer Encoder architecture, and the specific process is as follows:

[0048]

[0049] The Flatten(·) layer flattens a two-dimensional feature vector into a one-dimensional vector, LN(·) represents the normalization operation, and MultiAttention(·) represents the multi-head attention mechanism.

[0050] Furthermore, in step (4), the UAV altitude H is the relative altitude between the UAV and the plane where the target is located, and the UAV attitude angle data pitch angle β and yaw angle δ are the fuselage IMU data.

[0051] Furthermore, in step (5), the projected distance L between the UAV and the personnel target, the lateral distance K, the longitudinal distance D, and the line distance O between the UAV and the target are obtained. C The specific process of P is as follows:

[0052] (5-1) Input the camera's focal length f, optical center coordinates (u0, v0) in step (1), the personnel target ranging feature point p (u, v) in step (3), the UAV altitude H read by the fuselage sensor in step (4), and the UAV attitude angle data such as pitch angle β and yaw angle δ.

[0053] (5-2) Design a UAV ranging model, define a coordinate system and corresponding coordinate points, where P is the position of the target to be detected, p is the corresponding position of the target in the imaging plane, O is the center of the camera target surface, and O C For the camera optical center, O P uv is the pixel coordinate system, O W X W Y W For the world coordinate system, O W X' C Y' C Let be the projected coordinate system of the camera coordinate system onto the ground; let ε be the angle between the target P and the projection of the optical axis, and let ∠cO be the angle between the target P and the optical axis projection. C O is γ; target P is on axis O. W Y' C Let A be the projection point on the image, w be the lateral distance from point p to the center point of the image, h be the longitudinal distance from point p to the center point of the image, and O be the longitudinal distance between the target and the UAV. W B is D, the lateral distance BP is K, and the projected distance O is... W If P is L, then:

[0054]

[0055] (5-3) Using geometric relationships, we obtain the projected distance L between the UAV and the personnel target, the lateral distance K, the longitudinal distance D, and the distance O between the UAV and the target. C The calculation process for P is as follows:

[0056] Calculate |O based on geometric relations W A|:

[0057]

[0058] From ΔcO c p~ΔAO c P and Then the distance |AP| is:

[0059]

[0060] The calculated projection distance L is:

[0061]

[0062] The calculated lateral distance K is:

[0063]

[0064] The calculated longitudinal distance D is:

[0065]

[0066] The distance O between the UAV and the target was calculated. C P is:

[0067]

[0068] Furthermore, in step (6), the three-dimensional ranging output of the personnel target from the perspective of the UAV is the projected distance L, lateral distance K, longitudinal distance D, and the distance O between the target and the target calculated in step (5). C P.

[0069] A monocular ranging system for UAVs and personnel targets based on deep learning, comprising:

[0070] The personnel target detection unit uses a deep learning target detection network to detect and identify personnel targets.

[0071] The distance measurement unit takes into account the changes in the flight attitude of the UAV and uses ranging theory to measure the three-dimensional distance between the UAV and the personnel target.

[0072] The calibration unit is used to calibrate the personnel target position and measurement distance in the form of a box.

[0073] Compared with the prior art, the present invention has the following advantages:

[0074] 1. The improved Yolov5s-TinyVision deep learning network proposed in this invention improves the network by using a small target feature processing module, a feature communication enhancement module, a multi-scale feature fusion module design, and multiple attention mechanisms, making the network more sensitive to small objects and achieving accurate identification of people targets;

[0075] 2. The monocular ranging model considering attitude angle proposed in this invention does not require a fixed reference length, thus overcoming the deficiency of traditional monocular ranging theory that can only measure vertical distance. By considering the changes in the flight attitude of the UAV, the ranging theory is modified to realize the three-dimensional distance measurement between the UAV and the target, which has high accuracy.

[0076] 3. This invention, through an improved deep learning network and monocular ranging model, enables accurate identification and localization of personnel targets from the perspective of an unmanned aerial vehicle (UAV). This method exhibits high generalization and robustness, small measurement error, and has broad application scenarios and significant potential for wider application. Attached Figure Description

[0077] Figure 1 This is a schematic diagram of the overall process of an embodiment of the present invention.

[0078] Figure 2 This is a schematic diagram illustrating the improvement of the deep learning neural network Yolov5s-TinyVision in the implementation process of this invention.

[0079] Figure 3 This is a schematic diagram of the feature communication enhancement module in the implementation process of the present invention.

[0080] Figure 4 This is a schematic diagram of the detection head structure in the implementation process of the present invention.

[0081] Figure 5 This is a diagram showing the detection results of the deep learning neural network in the implementation process of this invention.

[0082] Figure 6 This is a schematic diagram of the monocular ranging model in the implementation process of the present invention.

[0083] Figure 7 This is a schematic diagram illustrating the identification, detection, and ranging results of personnel targets from the perspective of an unmanned aerial vehicle (UAV) in the implementation process of this invention. Detailed Implementation

[0084] The present invention will be further described below with reference to the accompanying drawings and specific implementation examples.

[0085] This invention provides a monocular ranging method for UAVs and personnel targets based on deep learning. It can identify, detect, and measure the distance to personnel targets from the UAV's perspective in three dimensions, and calibrate the target position and distance measurement using a bounding box, achieving high accuracy. This embodiment establishes a deep learning-based UAV platform personnel detection and monocular ranging example based on the Ubuntu 18.04 operating system, using the Python 3.8 programming language to build the PyTorch deep learning framework, and employing the CUDA 11.4 and cuDNN 8.3 neural network libraries to improve model training speed. The computer hardware configuration is as follows: CPU: Intel(R) Core(TM) i5-6400 CPU @ 2.70GHz; GPU: NVIDIA GeForce RTX 2080Ti.

[0086] like Figure 1 As shown, a monocular ranging method for UAVs and human targets based on deep learning specifically includes the following steps:

[0087] (1) The intrinsic parameters of the UAV's onboard camera were calibrated to obtain the camera's focal length f and optical center coordinates (u0, v0). The Zhang Zhengyou calibration method was used for the calibration of the UAV's onboard camera, and a black and white checkerboard calibration board was selected. The calibrated intrinsic parameter matrix is ​​as follows:

[0088]

[0089] Among them, f x =f / dx, f y = f / dy, where f is the camera focal length, and dx and dy are inherent camera parameters, representing the physical length of a unit pixel along the x and y axes, respectively. For the camera used in this example, the calibration result is: f x =2816, f y =2816, u0=2016, v0=1520.

[0090] (2) Obtain images frame by frame from the video stream captured by the UAV's onboard camera. For the UAV's onboard camera in this example, the acquired images are RGB three-channel images with a resolution of 4032*3024.

[0091] (3) Figure 2 As shown, an improved network based on the Yolov5s detection algorithm, Yolov5s-TinyVision, is used for person target detection and feature extraction to obtain person target location information and convert it into ranging feature points p(u, v) of the person target. The specific process is as follows:

[0092] (3-1) The VisDrone2019 drone scene dataset was used. This dataset mainly targets drone-captured scenes, has a wide coverage, and includes annotations for various common targets. The image detection task in this dataset includes 6471 training images, 548 validation images, and 3190 test images. This dataset contains a large number of human target scenes from the drone's perspective, with diverse environments in which people are located, which can effectively test the model's performance and is well-suited for the target detection task of this network. Images and annotation files of the category "person" were extracted, and pre-trained weights based on Yolov5s.pt were used for 100 epochs of training.

[0093] During network training, adaptive anchor boxes, image scaling, and data augmentation techniques were used to augment the input images. A genetic algorithm was used to mutate the results of YOLO's K-Means clustering to obtain the optimal anchor box size. Specifically, the optimal anchor box sizes for the four detection heads obtained based on the VisDrone dataset were: [2, 4, 3, 6, 4, 5, 3, 9], [5, 8, 5, 12, 7, 9, 6, 15], [8, 13, 11, 10, 8, 19, 13, 17], [12, 25, 19, 24, 18, 38, 39, 53]. It can be seen that for the small person target detection task, most people are standing, therefore the target anchor boxes are mostly small, narrow, and elongated rectangles. By designing targeted anchor box sizes, the accuracy of person target detection can be improved.

[0094] (3-2) Input a 4032*3024 resolution RGB three-channel image acquired by the UAV's onboard camera. Use a feature extraction network to initially extract the target's features X. The feature extraction network can be any network, such as CSPDarknet53 or ResNet. In this example, CSPDarknet53 is used as the feature extraction network, and its output feature channels are 1024.

[0095] (3-3) Use the small target feature processing module to process the initially extracted target features X to obtain features X that are more relevant to small targets. tiny The specific process of the small target feature processing module is as follows:

[0096]

[0097] In this example, Split(X, Scale) represents downsampling and slicing the feature map X by a scaling factor scale, and Concat(·) represents concatenating the sliced ​​features along the channel direction. In this example, CBR(·) performs a 1×1 convolution, standardization, and ReLU activation on the feature map sequentially; Sigmoid(·) performs Sigmoid activation and normalization on the feature map; CB(·) performs a 1×1 convolution and standardization on the feature map sequentially; and (·) represents element-wise element-wise multiplication.

[0098] In this example, scale = 2, so by applying the small target feature processing module to X, four sub-feature maps f can be obtained. 0，0 f 1，0 f 0，1 f 1，1 The shape is X, downsampled by a factor of 2. After obtaining the sub-feature maps, they are concatenated along the channel dimension and subjected to a 1×1 convolution filter with stride=1 to achieve non-stretch convolution, thus obtaining the feature X. tiny .

[0099] (3-4) such as Figure 3 As shown, the feature communication enhancement module is used to enhance the target feature X. tiny Further processing is performed. An attention mechanism is incorporated to enhance the feature exchange of personnel target information, resulting in the enhanced personnel target feature X. tiny-f The specific calculation process of the feature communication enhancement module is as follows:

[0100] Channel attention is used to enhance the target information of people in each channel. The formula for calculating the weight of each channel in channel attention is as follows:

[0101] M C (X tiny =Sigmoid(MLP(AvgPool(X)) tiny ))+MLP(MaxPool(X tiny ))) (3)

[0102] X C =M C (X tiny )·X tiny (4)

[0103] Where AvgPool(·) represents average pooling, MaxPool(·) represents max pooling, MLP(·) represents a multilayer perceptron model, and (+) represents matrix addition. In this example, we first process the feature X... tinyAverage pooling and max pooling operations are performed, and the resulting feature maps are fed into a two-layer neural network (MLP). Then, the features output by the MLP are summed element-wise, followed by sigmoid activation to obtain the MLP processed by channel attention. C (X tiny Finally, M C (X tiny ) and input feature map X tiny Perform element-wise dot product operations to generate the input features required by the spatial attention mechanism.

[0104] Obtain the feature X after channel attention enhancement C Then, a spatial attention mechanism is used to enhance the spatial target information of people, resulting in enhanced target features X. tiny-f The steps are as follows:

[0105] M S (X C )=Sigmoid(Conv(Concat(AvgPool(X C MaxPool(X) C (5)

[0106] X tiny-f =M S (X C )·X C (6)

[0107] Here, Conv(·) represents the convolution operation. In this example, we first process the feature X... C Average pooling and max pooling operations are performed, and then the two feature maps are concatenated along the channel direction. After a Conv operation, with a 7×7 kernel selected to reduce the channel dimension to 1, a Sigmoid activation operation is performed to obtain M processed by the channel attention mechanism. S (X C Finally, M S (X C ) and input feature map X C Perform a multiplication operation to obtain the final generated feature X. tiny-f .

[0108] (3-5) Using a feature pyramid network, the enhanced personnel target features X tiny-f Processing is performed to obtain feature X. tiny-r The feature pyramid structure used can be any structure, such as FPN, PAN, BiFPN, etc. This implementation example uses the FPN feature pyramid structure.

[0109] (3-6) such as Figure 4 As shown, the detection head is used to detect feature X. tiny-r The process involves processing to obtain information related to the target location of personnel. The specific process is as follows:

[0110] The three detection head structures of the YOLOv5 Head are increased to four, and a small target detection layer is added. The detection head structure is a Transformer detection head structure. The Transformer module is integrated into the C3 module of the YOLOv5 architecture to form the C3TR module, which processes the features X obtained in steps (3-5). tiny-r The formula for utilizing the multiple attention mechanism of Transformer is as follows:

[0111] X trans-r =Conv(Concat(TB(Conv(X)) tiny-r )), Conv(X tiny-r ))) (7)

[0112] That is, X is processed using 1×1 convolution and TB(·) operation respectively. tiny-r The processed features are then concatenated with the features obtained after 1×1 convolution along the channel direction, and then processed again using 1×1 convolution. Here, TB(·) refers to the Transformer Block module based on the Transformer Encoder architecture. The specific process of the TB(·) operation is as follows:

[0113]

[0114] The Flatten(·) layer flattens a two-dimensional feature vector into a one-dimensional vector, LN(·) represents the normalization operation, and MultiAttention(·) represents the multi-head attention mechanism. This example uses 8 attention heads, each randomly initialized. The formula for each single-head attention is:

[0115]

[0116] Where Q, K, V∈R N×C Q represents the query matrix, key matrix, and value matrix, respectively. i K i V i This indicates that Q, K, V are multiplied by the weight matrix of the self-attention mechanism. `softmax(·)` represents the softmax activation function. The Attention function is obtained after feature mapping for each single-head attention mechanism. i The final output of the multi-head attention mechanism is formed through a multi-scale feature fusion module:

[0117] Attention=Concat(Attention1, Attention2,...,Attention8) (10)

[0118] After processing, the output image of the personnel target detected and identified by the deep neural network is obtained, such as... Figure 5 As shown.

[0119] (3-7) From the bounding box location information of the personnel target, obtain the coordinate information (x, y, y) of the personnel target bounding box in the image coordinate system. left y top ), (x right y bottom For the image in this example, the coordinates of the target box are: (2207, 2314), (2256, 2433).

[0120] The coordinates of the ranging feature point p(u, v) of the personnel target are calculated as follows:

[0121]

[0122] In this example, the distance measurement feature point is calculated to be (2231.5, 2433).

[0123] (4) Use the fuselage sensors to read the UAV's altitude H, pitch angle β, and yaw angle δ. In this example, the fuselage sensors read the UAV's altitude as 3.8m, the UAV's pitch angle as 6°, and the yaw angle as 5°.

[0124] (5) Input the camera focal length f, optical center coordinates (u0, v0) obtained in step (1), the personnel target ranging feature point p(u, v) obtained in step (3), and the aircraft attitude data obtained in step (4) into the optimized ranging model to obtain the projected distance L, lateral distance K, longitudinal distance D, and line distance O between the UAV and the personnel target. C P. The specific process is as follows:

[0125] (5-1) such as Figure 6 The UAV ranging model shown defines a coordinate system and corresponding coordinate points. Here, P represents the position of the target to be detected, and p is the target's corresponding position in the imaging plane. O is the center of the camera target surface. C For the camera optical center, O P uv is the pixel coordinate system, O W X W Y W For the world coordinate system, O W X' C Y' CLet be the projected coordinate system of the camera coordinate system onto the ground. The angle between the target P and the projection of the optical axis is ε, ∠cO. C O is γ. Target P is on axis O. W Y' C Let A be the projection point on the image, w be the lateral distance from point p to the center point of the image, h be the longitudinal distance from point p to the center point of the image, and O be the longitudinal distance between the target and the UAV. W B is D, the lateral distance BP is K, and the projected distance O is... W If P is L, then:

[0126]

[0127] (5-2) Based on geometric relationships, the three-dimensional projected distance L, lateral distance K, longitudinal distance D, and the distance O between the UAV and the personnel target are obtained. C The calculation process for P is as follows:

[0128] Calculate |O based on geometric relations W A|:

[0129]

[0130] From ΔcO c p~ΔAO c P and Then the distance |AP| is:

[0131]

[0132] The calculated projection distance L is:

[0133]

[0134] The calculated lateral distance K is:

[0135]

[0136] The calculated longitudinal distance D is:

[0137]

[0138] The calculated line distance (OCP) between the UAV and the target is:

[0139]

[0140] In this example, the calculated three-dimensional projected distance L, lateral distance K, longitudinal distance D, and the distance |O between the UAV and the personnel target are obtained. c P| represents: |O cP|=18.38m, L=17.98m, K=2.87m, D=17.75m.

[0141] (6) Figure 7 As shown, the results of personnel target recognition, detection, and ranging are output, and the target position and distance measurement are calibrated in the form of a box, thus completing the recognition, detection, and three-dimensional ranging of personnel targets from the perspective of the UAV.

[0142] This embodiment also provides a monocular ranging system for UAVs and personnel targets based on deep learning, including:

[0143] The personnel target detection unit uses a deep learning target detection network to detect and identify personnel targets.

[0144] The distance measurement unit takes into account the changes in the flight attitude of the UAV and uses ranging theory to measure the three-dimensional distance between the UAV and the personnel target.

[0145] The calibration unit is used to calibrate the personnel target position and measurement distance in the form of a box.

[0146] This invention improves the network by using a small target feature processing module, a feature communication enhancement module, a multi-scale feature fusion network design, and various attention mechanisms, making the network more sensitive to small objects and achieving accurate identification of small targets such as people. By considering the changes in the flight attitude of the UAV and modifying the ranging theory, this invention achieves high accuracy in measuring the three-dimensional distance between the UAV and the target. This invention can identify and detect small targets such as people from the UAV's perspective and perform three-dimensional ranging, calibrating the target position and distance measurement in the form of a bounding box. It has high generalization and robustness, small measurement error, and has wide application scenarios and significant promotional value.

[0147] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A monocular ranging method for UAVs and personnel targets based on deep learning, characterized in that, include: Use a deep learning-based object detection network to detect and identify human targets; Considering the changes in the flight attitude of the UAV, the three-dimensional distance between the UAV and the human target is measured using ranging theory. The location and distance of personnel targets are calibrated in the form of a box, and the identification, detection and three-dimensional ranging of personnel targets from the perspective of the UAV are completed. The specific steps include: (1) Calibrate the intrinsic parameters of the UAV's onboard camera and obtain the camera's focal length f and optical center coordinates (u0, v0); (2) Obtain images frame by frame from the video stream captured by the UAV's onboard camera; (3) Use a deep learning target detection network to identify the target person from frame-by-frame images, and obtain the target person location information and the target ranging feature points p(u, v); (4) Use the fuselage sensors to read the UAV altitude H, UAV attitude angle data pitch angle β and yaw angle δ; (5) Input the camera focal length f, optical center coordinates (u0, v0) obtained in step (1), the ranging feature point p(u, v) obtained in step (3), and the aircraft attitude data obtained in step (4) into the ranging model to obtain the three-dimensional projected distance L, lateral distance K, longitudinal distance D, and line distance O between the UAV and the personnel target. C P; (6) Output the personnel target positioning information obtained in step (3) and the ranging data obtained in step (5), and calibrate the personnel target position and the measurement distance in the form of a box to complete the identification, detection and three-dimensional ranging of personnel targets from the perspective of the UAV. The deep learning object detection network uses the improved YOLOv5s-TinyVision network, based on the YOLOv5s detection algorithm, to perform personnel target detection and feature extraction, output personnel target location information, and convert it into the ranging feature points p(u, v) of the personnel target. The improved YOLOv5s-TinyVision network adopts a small target feature processing module, a feature communication enhancement module, a multi-scale feature fusion module, and multiple attention mechanisms to improve the YOLOv5 network. The deep learning object detection network uses the improved Yolov5s-TinyVision network, based on the Yolov5s detection algorithm, for person object detection and feature extraction, specifically including the following steps: (3-1) Using the drone scene dataset, the optimal anchor box size for personnel targets is obtained based on K-Means clustering and genetic algorithm; (3-2) Input image, use feature extraction network to initially extract target features X; (3-3) Use the small target feature processing module to process the initially extracted target features X to obtain features X that are more relevant to small targets. tiny The small target feature processing module replaces the cross-layer convolutional and pooling layers in the feature extraction network, and consists of a spatial-to-depth layer and a non-stride convolutional layer. (3-4) Using the Feature Communication Enhancement Module to leverage the CBAM attention mechanism for target feature X tiny After further processing, the enhanced personnel target feature X is obtained. tiny-f ; (3-5) Using a feature pyramid network, the enhanced personnel target features X tiny-f Processing is performed to obtain feature X. tiny-r ; (3-6) Use the detection head to target feature X tiny-r The process is performed to obtain relevant information on the target location of personnel; (3-7) The output module obtains the coordinate information (x, y) of the person target bounding box in the image coordinate system based on the person target location information. left ,y top ),(x right ,y bottom The coordinates of the ranging feature point p(u, v) of the personnel target are calculated as follows: u=(x left +x right ) / 2 v=y bottom 。 2. The monocular ranging method for UAVs and personnel targets based on deep learning according to claim 1, characterized in that, Step (1) uses the Zhang Zhengyou calibration method to calibrate the intrinsic parameters of the UAV's airborne camera. The resulting intrinsic parameter matrix is: Among them, f x =f / dx, f y = f / dy, where f is the camera focal length, and dx and dy are inherent camera parameters, representing the physical length of a unit pixel in the x and y directions, respectively.

3. The monocular ranging method for UAVs and personnel targets based on deep learning according to claim 1, characterized in that, The specific process by which the small target feature processing module processes feature X is as follows: X tiny =Concat(CBR(f 0,0 ),CBR(f 0,1 ),...,CBR(f scale-1,scale-1 ))·Sigmoid(CB(X)) In this context, Split(X, Scale) represents downsampling and slicing the feature map X by the scaling factor scale, Concat(·) represents concatenating the sliced ​​features along the channel direction, CBR(·) represents performing convolution, normalization, and activation function operations on the feature map in sequence, Sigmoid(·) represents performing Sigmoid function activation and normalization operations on the feature map, and CB(·) represents performing convolution and normalization operations on the feature map in sequence.

4. The monocular ranging method for UAVs and personnel targets based on deep learning according to claim 1, characterized in that, Step (3-4) specifically includes the following steps: (3-4-1) Use the channel attention mechanism to focus on the target feature X in the channel. tiny Enhancement is performed to obtain the channel attention-enhanced feature X. C The formula for calculating the channel weights for channel attention is: M C (X tiny )=Sigmoid(MLP(AvgPool(X tiny ))+MLP(MaxPool(X tiny ))) X C =M C (X tiny )·X tiny Where AvgPool(·) represents average pooling operation, MaxPool(·) represents max pooling operation, MLP(·) represents multilayer perceptron model, (·) represents element-based dot product, and (+) represents matrix addition. (3-4-2) Obtain the feature X after channel attention enhancement C Then, spatial attention mechanism is used to analyze spatial features X. C Enhancement is performed to obtain the enhanced personnel target feature X. tiny-f for: M S (X C )=Sigmoid(Conv(Concat(AvgPool(X C ),MaxPoll(X C )))) X tiny-f =M S (X C )·X C Here, Conv(·) represents the convolution operation.

5. The monocular ranging method for UAVs and personnel targets based on deep learning according to claim 1, characterized in that, Steps (3-6) specifically include: The three detection head structures of the Yolov5 Head are increased to four, and a small target detection layer is added. The detection head structure is a Transformer detection head structure, which is used to process the features X obtained in steps (3-5). tiny-r The result of the processing is: X trans-r =Conv(Concat(TB(Conv(X tiny-r )),Conv(X tiny-r) )) Wherein, TB(·) refers to the Transformer Block module based on the Transformer Encoder architecture, specifically: TB(X)=MLP(LN(X temp ))+X temp X temp =MutiAttention(LN(Flatten(X)))+X Among them, the Flatten(·) layer flattens the two-dimensional feature vector into a one-dimensional vector, LN(·) represents the normalization operation, and MultiAttention(·) represents the multi-head attention mechanism; Each single-head attention in the multi-head attention mechanism is: Where Q,K,V∈R N×C Q represents the query matrix, key matrix, and value matrix, respectively. i ,K i V i This represents the multiplication of Q, K, V with the weight matrix of the self-attention mechanism, where softmax(·) represents the softmax activation function, and the Attention is obtained after feature mapping for each single-head attention mechanism. i The final output of the multi-head attention mechanism, formed through a multi-scale feature fusion module, is: Attention=Concat(Attention1,Attention2,...,Attention8); After processing, relevant information on personnel target location is obtained.

6. The monocular ranging method for UAVs and personnel targets based on deep learning according to claim 1, characterized in that, Step (5) specifically includes the following steps: (5-1) Design a UAV ranging model, define a coordinate system and corresponding coordinate points, let P be the position of the target to be detected, p be the corresponding position of the target in the imaging plane, and O be the center of the camera target surface. C For the camera optical center, O P uv is the pixel coordinate system, O W X W Y W For the world coordinate system, O W X' C Y' C Let be the projected coordinate system of the camera coordinate system onto the ground; let ε be the angle between the target P and the projection of the optical axis, and let ∠cO be the angle between the target P and the optical axis projection. C O is γ, and the target P is on axis O. W Y' C Let A be the projection point on the image, w be the lateral distance from point p to the center point of the image, h be the longitudinal distance from point p to the center point of the image, and O be the longitudinal distance between the target and the UAV. W B is D, the lateral distance BP is K, and the projected distance O is... W If P is L, then: w=(u-u0)dx h = (v0 - v)dy (5-2) Using geometric relationships, we obtain the projected distance L, lateral distance K, longitudinal distance D between the UAV and the personnel target, and the distance O between the UAV and the target. C P is:

7. A monocular ranging system for UAVs and personnel targets based on deep learning, implementing the method of any one of claims 1-6, characterized in that, include: The personnel target detection unit uses a deep learning target detection network to detect and identify personnel targets. The distance measurement unit takes into account the changes in the flight attitude of the UAV and uses ranging theory to measure the three-dimensional distance between the UAV and the personnel target. The calibration unit is used to calibrate the personnel target position and measurement distance in the form of a box.

Images