Unmanned aerial vehicle and personnel target monocular distance measurement method and system based on deep learning

Abstract

The invention discloses a monocular distance measurement method and system for an unmanned aerial vehicle and a personnel target based on deep learning. The method comprises the following steps: detecting and identifying the personnel target by using a deep learning target detection network; the flight attitude change of the unmanned aerial vehicle is considered, and the three-dimensional distance between the unmanned aerial vehicle and the personnel target is measured through the distance measurement theory; the position of the personnel target and the measurement distance are calibrated in the form of a square frame, and identification detection and three-dimensional distance measurement of the personnel target under the view angle of the unmanned aerial vehicle are completed. According to the invention, identification detection and three-dimensional distance measurement of a personnel target under the view angle of the unmanned aerial vehicle can be realized, and calibration of target position and distance measurement is carried out in a square frame form.

Description

Technical Field

[0001] The present invention relates to the field of target detection and ranging, and in particular to a monocular ranging method and system for unmanned aerial vehicle (UAV) and human targets based on deep learning. Background Art

[0002] Target detection and ranging are key technologies in environmental perception, and can provide an algorithmic basis for emergency obstacle avoidance and path decision-making of UAVs. The premise of target ranging is to identify and locate objects in the image, that is, target detection. Since the environment of the UAV in flight is complex and changeable, it is not only affected by external environmental factors such as lighting, weather, noise and shadow, but also by the imaging quality of the onboard camera itself. The robustness and generalization ability of the algorithm are required to be higher, which greatly increases the difficulty of target detection. The distance between the UAV and the target is the key information for understanding the surrounding environment and making decision-making and control. However, the traditional target recognition algorithm has poor recognition accuracy and robustness in the UAV flight scene, and it is difficult to accurately recognize smaller targets, which is prone to false detection and missed detection; because the traditional monocular ranging method does not consider the change of the UAV flight posture, there is a large measurement error, and it does not perform well in the UAV scene. Therefore, it is of great significance to propose an improved UAV platform personnel detection and monocular ranging method to achieve accurate recognition and ranging of personnel targets.

[0003] In order to solve the problems existing in traditional monocular ranging technology, the present invention obtains the position of the target in the image through deep learning technology. By improving the YOLOv5 algorithm, the multi-scale feature fusion module design and multiple attention mechanisms are used to achieve accurate recognition of human targets; by designing a monocular ranging model that takes into account the attitude angle, the three-dimensional distance data between the drone and the target is obtained, and the accurate recognition and positioning of human targets from the perspective of the drone is achieved, effectively solving the bottlenecks encountered by traditional monocular ranging technology.

[0004] This paper proposes a method for personnel identification and monocular ranging on a UAV platform based on deep learning. By improving the deep learning network and optimizing the ranging theory, accurate identification and ranging of personnel targets from the perspective of the UAV are achieved. This method can provide reliable target information support for the flight of UAVs in complex environments, and has important application prospects and value in improving the accuracy of target ranging and positioning. Summary of the invention

[0005] The present invention provides a monocular ranging method and system for UAV and human targets based on deep learning, which can realize the recognition, detection and three-dimensional ranging of human targets from the perspective of the UAV, and calibrate the target position and distance measurement in the form of a box.

[0006] The technical solution to achieve the purpose of the present invention is:

[0007] A monocular ranging method for UAV and human targets based on deep learning, comprising:

[0008] Use deep learning target detection network to detect and identify human targets;

[0009] Considering the change of UAV flight attitude, the three-dimensional distance between the UAV and the human target is measured by using the ranging theory;

[0010] The position of the personnel target and the measured distance are calibrated in the form of a box to complete the recognition, detection and three-dimensional ranging of the personnel target from the perspective of the drone.

[0011] Furthermore, the method specifically comprises the following steps:

[0012] (1) Calibrate the internal parameters of the drone's onboard camera to obtain the camera's focal length f and optical center coordinates (u0, v0);

[0013] (2) Obtain images frame by frame from the video stream captured by the drone’s onboard camera;

[0014] (3) Use deep learning target detection to identify the target person from each frame of the image and obtain the ranging feature points p(u, v) of the target; use the improved network Yolov5s-TinyVision based on the Yolov5s detection algorithm to perform personnel target detection and feature extraction, output the personnel target positioning information, and convert it into the ranging feature points p(u, v) of the personnel target;

[0015] (4) Use the fuselage sensor to read the drone's altitude H, the drone's attitude angle data pitch angle β, and yaw angle δ;

[0016] (5) The camera focal length f, optical center coordinates (u0, v0) obtained in step (1), the personnel target ranging feature point p (u, v) obtained in step (3), and the aircraft attitude data obtained in step (4) are brought into the optimized ranging model to obtain the three-dimensional projection distance L, lateral distance K, longitudinal distance D between the UAV and the personnel target, and the connecting line distance O between the UAV and the target. C P;

[0017] (6) The results of human target recognition, detection and ranging are output, and the target position and distance measurement are calibrated in the form of a box to complete the recognition, detection and three-dimensional ranging of human targets from the perspective of the drone.

[0018] Furthermore, in step (1), when the drone airborne camera is calibrated with internal parameters, the Zhang Zhengyou calibration method is used, and the calibration board is selected as a black and white chessboard calibration board; the internal parameter matrix obtained by calibration is:

[0019]

[0020] Among them, f x =f / dx,f y =f / dy, f is the focal length of the camera, dx and dy are the intrinsic parameters of the camera, representing the physical length of the unit pixel in the x- and y-axis directions respectively.

[0021] Furthermore, in step (2), the onboard camera may be a camera of any model.

[0022] Furthermore, in step (3), when deep learning target detection is used to identify the target person to be detected from frame-by-frame images and obtain the ranging feature points p (u, v) of the target, the convolutional neural network model used for the deep learning target detection is a Yolov5s-TinyVision network improved based on the Yolov5s detection algorithm, which mainly includes the steps of using the deep learning neural network Yolov5s-TinyVision to extract features and using feature detection positioning information to obtain the ranging feature points p (u, v) of the target person. The deep learning neural network Yolov5s-TinyVision improves the network by using a small target feature processing module, a feature exchange enhancement module, a multi-scale feature fusion module design and a variety of attention mechanisms.

[0023] Furthermore, the step (3) specifically comprises the steps of:

[0024] (3-1) Using the drone scene dataset, the optimal anchor frame size of the human target is obtained based on K-Means clustering and genetic algorithm; the dataset can be an actual shooting dataset, or a network dataset for drone shooting scenes, such as VisDrone2019, VisDrone2021, etc.;

[0025] (3-2) Input the image and use the feature extraction network to preliminarily extract the feature X of the target; the feature extraction network can be any feature extraction network, such as CSPDarknet53, ResNet, etc.;

[0026] (3-3) Use the small target feature processing module to process the initially extracted target feature X to obtain a feature X that focuses more on the small target tiny , the small target feature processing module is used to replace the cross-layer convolution and pooling layers in the feature extraction network, which consists of a space-to-depth (SPD) layer and a no-stride convolution layer;

[0027] (3-4) Use the feature communication enhancement module to process the target feature Xtiny again; use the CBAM attention mechanism to enhance the feature communication of the personnel target information, and obtain the enhanced personnel target feature Xtiny-f ;

[0028] (3-5) Use the feature pyramid network to enhance the target feature X tiny-f Processing to obtain feature X tiny-r ; The feature pyramid structure used can be any feature pyramid structure, such as FPN, PAN, BiFPN, etc.;

[0029] (3-6) Use the detection head to detect feature X tiny-r Processing is performed to obtain relevant information about personnel target positioning;

[0030] (3-7) Output the personnel target positioning information and obtain the coordinate information of the personnel target box in the image coordinate system (x left ,y top ), (x right ,y bottom ); the coordinates of the ranging feature point p(u, v) of the human target are calculated as:

[0031]

[0032] Furthermore, the specific process of using the small target feature processing module to extract feature X is as follows:

[0033]

[0034] Where Split(X, Scale) represents downsampling and slicing the feature map X with the scale factor scale, Concat(·) represents concatenating the sliced ​​features in the channel direction; CBR(·) represents sequentially performing convolution, normalization, and activation function operations on the feature map, Sigmoid(·) represents performing Sigmoid function activation and normalization operations on the feature map, and CB(·) represents sequentially performing convolution and normalization operations on the feature map;

[0035] Furthermore, the specific calculation process of the feature exchange enhancement module is as follows:

[0036] (3-4-1) Use the channel attention mechanism to enhance the personnel target information in the channel; the formula for calculating the weight of each channel by channel attention is:

[0037] M C (X tiny )=Sigmoid(MLP(AvgPool(X tiny ))+MLP(MaxPool(X tiny ))) (3)

[0038] X C =M C (Xtiny )·X tiny (4)

[0039] Among them, AvgPool(·) represents the average pooling operation, MaxPool(·) represents the maximum pooling operation, MLP(·) represents the multi-layer perceptron model, (·) represents the element-based dot product, and (+) represents the matrix addition;

[0040] (3-4-2) Get the feature X after channel attention enhancement C After that, the spatial attention mechanism is used to enhance the spatial personnel target information and obtain the enhanced personnel target feature X tiny-f Here are the steps:

[0041] M S (X C )=Sigmoid(Conv(Concat(AvgPool(X C ), MaxPool(X C )))) (5)

[0042] X tiny-f =M S (X C )·X C (6)

[0043] Where Conv(·) represents the convolution operation;

[0044] Furthermore, the steps (3-6) specifically include:

[0045] The three detection head structures of Yolov5 Head are increased to four detection head structures, and a small target detection layer is added; wherein the detection head structure is a Transformer detection head structure; using the Transformer detection head structure, the feature X obtained in step (3-5) is tiny-r The formula for using Transformer's multiple attention mechanism is:

[0046] X trans-r =Conv(Concat(TB(Conv(X tiny-r )), Conv(X tiny-r ))) (7)

[0047] Among them, TB(·) refers to the Transformer Block module based on the Transformer Encoder architecture. The specific process is:

[0048]

[0049] The Flatten(·) layer flattens the two-dimensional feature vector into a one-dimensional vector, LN(·) represents the normalization operation, and MutiAttention(·) represents the multi-head attention mechanism.

[0050] Furthermore, in step (4), the drone height H is the relative height between the drone and the plane where the target is located, and the drone attitude angle data pitch angle β and yaw angle δ are the fuselage IMU data.

[0051] Furthermore, in step (5), the projection distance L, the lateral distance K, the longitudinal distance D between the drone and the human target, and the line distance O between the drone and the target are obtained. C The specific process of P is as follows:

[0052] (5-1) Input the focal length f of the camera in step (1), the coordinates of the optical center (u0, v0), the characteristic point p (u, v) of the human target distance measurement in step (3), the height H of the drone read by the fuselage sensor in step (4), the pitch angle β, the yaw angle δ and other parameters of the drone attitude angle data.

[0053] (5-2) Design the UAV ranging model and define the coordinate system and corresponding coordinate points, where P is the position of the target to be detected, p is the corresponding position of the target in the imaging plane, O is the center of the camera target surface, and O C is the camera optical center, O P uv is the pixel coordinate system, O W X W Y W is the world coordinate system, O W X' C Y' C is the projection coordinate system of the camera coordinate system on the ground; the angle between the target P and the optical axis projection is ε, ∠cO C O is γ; the target P is on axis O W Y' C The projection point on the image is A, w is the horizontal distance from point p to the center of the image, h is the vertical distance from point p to the center of the image, and the vertical distance between the target and the drone is O. W B is D, lateral distance BP is K, projection distance O W P is L; then:

[0054]

[0055] (5-3) Based on the geometric relationship, we can obtain the projection distance L, lateral distance K, longitudinal distance D between the drone and the human target, and the line distance O between the drone and the target. C The calculation process of P is:

[0056] According to the geometric relationship, calculate |O W A|:

[0057]

[0058] By ΔcO c p~ΔAO c P and Then the distance |AP| is:

[0059]

[0060] The projection distance L is calculated as:

[0061]

[0062] The calculated lateral distance K is:

[0063]

[0064] The longitudinal distance D is calculated as:

[0065]

[0066] Calculate the distance O between the drone and the target C P is:

[0067]

[0068] Furthermore, in step (6), the three-dimensional distance measurement output result of the human target from the perspective of the drone is the projection distance L, the lateral distance K, the longitudinal distance D and the connecting distance O between the target and the drone calculated in step (5). C P.

[0069] A monocular ranging system for UAV and human targets based on deep learning, comprising:

[0070] Personnel target detection unit, which uses deep learning target detection network to detect and identify personnel targets;

[0071] The distance measurement unit takes into account the change of the UAV's flight posture and measures the three-dimensional distance between the UAV and the human target through the distance measurement theory;

[0072] The calibration unit calibrates the personnel target position and the measurement distance in the form of a box.

[0073] Compared with the prior art, the present invention has the following gain effects:

[0074] 1. The improved Yolov5s-TinyVision deep learning network proposed in this invention improves the network by using small target feature processing module, feature communication enhancement module, multi-scale feature fusion module design and multiple attention mechanisms, making the network more sensitive to tiny objects and achieving accurate recognition of human targets;

[0075] 2. The monocular distance measurement model considering the attitude angle proposed in the present invention does not need to determine the reference length, which makes up for the defect that the traditional monocular distance measurement theory can only perform vertical distance measurement. By considering the change of the UAV's flight attitude and modifying the distance measurement theory, the three-dimensional distance measurement between the UAV and the target is achieved with high accuracy.

[0076] 3. The present invention can realize accurate recognition and positioning of human targets from the perspective of drones by improving the deep learning network and monocular ranging model. The method has high generalization and robustness, small measurement error, and has a wide range of application scenarios and significant promotion value. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 It is a schematic diagram of the overall process of an implementation method of the present invention.

[0078] Figure 2 This is a schematic diagram of the improvement of the deep learning neural network Yolov5s-TinyVision in the implementation process of the present invention.

[0079] Figure 3 It is a schematic diagram of a characteristic communication enhancement module in the process of an implementation mode of the present invention.

[0080] Figure 4 It is a schematic diagram of the structure of the detection head in the process of the implementation mode of the present invention.

[0081] Figure 5 This is a diagram of the deep learning neural network detection results in the implementation process of the present invention.

[0082] Figure 6 Schematic diagram of a monocular ranging model in the process of an implementation method of the present invention.

[0083] Figure 7 Schematic diagram of the recognition, detection and ranging results of human targets from the perspective of a drone in the process of an implementation mode of the present invention. DETAILED DESCRIPTION

[0084] The present invention will be further described below in conjunction with the accompanying drawings and specific implementation examples.

[0085] The present invention provides a monocular ranging method for unmanned aerial vehicles and personnel targets based on deep learning, which can realize the recognition, detection and three-dimensional ranging of personnel targets from the perspective of unmanned aerial vehicles, and calibrate the target position and distance measurement in the form of a box, with high accuracy. The unmanned aerial vehicle platform personnel detection and monocular ranging example based on deep learning established in this embodiment is based on the Ubuntu 18.04 operating system, uses the Python 3.8 programming language to build the PyTorch deep learning framework, and uses the CUDA 11.4 and cuDNN 8.3 neural network libraries to improve the model training speed. The computer hardware configuration is as follows: the CPU is Intel (R) Core (TM) i5-6400 CPU @ 2.70 GHz; the GPU is NVIDIA GeForce RTX 2080Ti.

[0086] like Figure 1 As shown, a monocular ranging method for UAV and human targets based on deep learning specifically includes the following steps:

[0087] (1) Calibrate the internal parameters of the drone's airborne camera to obtain the camera's focal length f and optical center coordinates (u0, v0). When calibrating the internal parameters of the drone's airborne camera, the Zhang Zhengyou calibration method is used, and the calibration board is a black and white chessboard calibration board. The internal parameter matrix obtained by calibration is:

[0088]

[0089] Among them, f x =f / dx,f y =f / dy, where f is the focal length of the camera, dx and dy are intrinsic parameters of the camera, representing the physical length of a unit pixel in the x-axis and y-axis directions, respectively. For the camera used in this example, the calibration result is: f x =2816,f y =2816, u0=2016, v0=1520.

[0090] (2) Obtain images frame by frame from the video stream captured by the drone's onboard camera. For the drone's onboard camera in this example, the acquired images are RGB three-channel images with a resolution of 4032*3024.

[0091] (3) Figure 2 As shown, the improved network Yolov5s-TinyVision based on the Yolov5s detection algorithm is used to detect and extract features of personnel targets, obtain the positioning information of personnel targets, and convert it into the ranging feature points p(u, v) of the personnel targets. The specific process is:

[0092] (3-1) Use the drone scene dataset VisDrone2019, which is mainly aimed at drone capture scenes, covers a wide range, and contains annotations of many common targets. The image detection task of this dataset includes 6471 training images, 548 verification images, and 3190 test images. This dataset contains a large number of human target scenes from the perspective of drones. The environments in which people are located are diverse, which can better test the performance of the model and is well suitable for the target detection task of this network. Extract the images and annotation files of the person category, use the pre-trained weights based on Yolov5s.pt, and perform 100 epoch training.

[0093] During the network training process, adaptive anchor boxes, image size scaling and data enhancement techniques are used to enhance the input images. Through genetic algorithms, the results of K-Means clustering of YOLO are mutated to obtain the optimal anchor box size. Among them, the optimal anchor box sizes of the four detection heads obtained based on the VisDrone dataset are: [2, 4, 3, 6, 4, 5, 3, 9], [5, 8, 5, 12, 7, 9, 6, 15], [8, 13, 11, 10, 8, 19, 13, 17], [12, 25, 19, 24, 18, 38, 39, 53]. It can be seen that for the small person target detection task, most people are standing, so most of the target anchor boxes are small-sized narrow rectangular boxes. By designing targeted anchor box sizes, the accuracy of person target detection can be improved.

[0094] (3-2) Input the RGB three-channel image with a resolution of 4032*3024 obtained by the drone's onboard camera, and use the feature extraction network to preliminarily extract the target's feature X. The feature extraction network can be any feature extraction network, such as CSPDarknet53, ResNet, etc. In this example, CSPDarknet53 is used as the feature extraction network, and the number of feature channels it outputs is 1024.

[0095] (3-3) Use the small target feature processing module to process the initially extracted target feature X to obtain a feature X that focuses more on the small target tiny The specific process of the small target feature processing module is as follows:

[0096]

[0097] Among them, Split(X, Scale) represents downsampling and slicing the feature map X with the scale factor scale, and Concat(·) represents concatenating the sliced ​​features in the channel direction. In this operation example, CBR(·) performs 1×1 convolution, normalization, and RELU activation function activation operations on the feature map in sequence, Sigmoid(·) represents the Sigmoid function activation and normalization operation on the feature map, CB(·) represents the 1×1 convolution and normalization operations on the feature map in sequence, and (·) represents the element-by-element dot multiplication operation based on the element.

[0098] In this operation example, scale = 2, then four sub-feature maps f can be obtained by performing small target feature processing on X 0，0 , f 1，0 , f 0，1 , f 1，1 , whose shape is X and is downsampled by 2 times. After obtaining the sub-feature map, it is connected in the channel dimension and a 1×1 convolution filter with stride=1 is performed to implement non-strided convolution to obtain the feature X tiny .

[0099] (3-4) Figure 3 As shown, the feature exchange enhancement module is used to enhance the target feature X tiny The attention mechanism is integrated to enhance the characteristics of personnel target information and obtain the enhanced personnel target characteristics X tiny-f The specific calculation process of the characteristic AC enhancement module is as follows:

[0100] The channel attention mechanism is used to enhance the information of the person target in the channel. The formula for calculating the weight of each channel by channel attention is:

[0101] M C (X tiny )=Sigmoid(MLP(AvgPool(X tiny ))+MLP(MaxPool(X tiny ))) (3)

[0102] X C =M C (X tiny )·X tiny (4)

[0103] Among them, AvgPool(·) represents the average pooling operation, MaxPool(·) represents the maximum pooling operation, MLP(·) represents the multi-layer perceptron model, and (+) represents matrix addition. tinyThe average pooling and maximum pooling operations are performed, and the feature maps after the operations are sent to a two-layer neural network (MLP). Then, the features output by the MLP are summed based on the matrix elements, and then activated by the Sigmoid operation to obtain the MLP processed by the channel attention mechanism. C (X tiny ). Finally, M C (X tiny ) and the input feature map X tiny Perform element-by-element dot multiplication to generate the input features required by the spatial attention mechanism.

[0104] Get the feature X after channel attention enhancement C After that, the spatial attention mechanism is used to enhance the spatial personnel target information and obtain the enhanced personnel target feature X tiny-f Here are the steps:

[0105] M S (X C )=Sigmoid(Conv(Concat(AvgPool(X C ), MaxPool(X C )))) (5)

[0106] X tiny-f =M S (X C )·X C (6)

[0107] Where Conv(·) represents the convolution operation. In this operation example, we first perform convolution on the feature X. C Average pooling and maximum pooling operations are performed, and then the two feature maps are spliced ​​in the channel direction. After a Conv operation, the convolution kernel is selected to be 7×7, the channel dimension is reduced to 1, and then a Sigmoid activation operation is performed to obtain M processed by the channel attention mechanism. S (X C ). Finally, M S (X C ) and the input feature map X C Perform multiplication operation to obtain the final generated feature X tiny-f .

[0108] (3-5) Use the feature pyramid network to enhance the target feature X tiny-f Processing to obtain feature X tiny-r The feature pyramid structure used may be any feature pyramid structure, such as FPN, PAN, BiFPN, etc. This implementation example uses the FPN feature pyramid structure.

[0109] (3-6) Figure 4 As shown, the detection head is used to detect feature X tiny-r Processing is performed to obtain relevant information about personnel target positioning. The specific process is:

[0110] The three detection heads of Yolov5 Head are increased to four detection heads, and a small target detection layer is added. The detection head structure is a Transformer detection head structure. By integrating the Transformer module into the C3 module of the YOLOv5 architecture, a C3TR module is formed, and the feature X obtained in step (3-5) is tiny-r The formula for using Transformer's multiple attention mechanism is:

[0111] X trans-r =Conv(Concat(TB(Conv(X tiny-r )), Conv(X tiny-r ))) (7)

[0112] That is, 1×1 convolution and TB(·) operations are used to process X tiny-r , and concatenate the processed features with the features processed by 1×1 convolution in the channel direction, and then use 1×1 convolution to process the concatenated features. Among them, TB(·) refers to the Transformer Block module based on the Transformer Encoder architecture. The specific process of TB(·) operation is:

[0113]

[0114] The Flatten(·) layer flattens the two-dimensional feature vector into a one-dimensional vector, LN(·) represents the normalization operation, and MutiAttention(·) represents the multi-head attention mechanism. In this example, 8 attention heads are used, and each head is randomly initialized. The formula for each single attention head is:

[0115]

[0116] Among them, Q, K, V ∈ R N×C Represent the query matrix, key matrix and value matrix respectively, Q i , K i , V i represents the multiplication of Q, K, V and the weight matrix of the self-attention mechanism. softmax(·) represents the softmax activation function. The Attention obtained after the feature mapping of each single-head attention i The final output of the multi-head attention mechanism is formed through the multi-scale feature fusion module:

[0117] Attention=Concat(Attention1, Attention2,...,Attention8) (10)

[0118] After the processing is completed, the output result diagram of the personnel target after detection and identification by the deep neural network is obtained, as shown in Figure 5 shown.

[0119] (3-7) Based on the positioning information of the human target box, the coordinate information of the human target box in the image coordinate system is obtained (x left ,y top ), (x right ,y bottom ). For the picture in this example, the coordinate information of the target box is: (2207, 2314), (2256, 2433).

[0120] The coordinates of the ranging feature point p(u, v) of the human target are calculated as follows:

[0121]

[0122] In this example, the distance measurement feature point is calculated to be: (2231.5, 2433).

[0123] (4) Use the fuselage sensor to read the drone's altitude H, and the drone's attitude angle data, the pitch angle β, and the yaw angle δ. In this example, the fuselage sensor reads the drone's altitude as 3.8m, and the drone's attitude angle data, the pitch angle, is 6°, and the yaw angle, is 5°.

[0124] (5) The camera focal length f, optical center coordinates (u0, v0) obtained in step (1), the personnel target ranging feature point p (u, v) obtained in step (3), and the aircraft attitude data obtained in step (4) are brought into the optimized ranging model to obtain the projection distance L, lateral distance K, longitudinal distance D between the UAV and the personnel target, and the connecting line distance O between the UAV and the target. C P. The specific process is as follows:

[0125] (5-1) Figure 6 The UAV ranging model shown in the figure defines the coordinate system and the corresponding coordinate points. Among them, P is the position of the target to be detected, p is the corresponding position of the target in the imaging plane. O is the center of the camera target surface, O C is the camera optical center, O P uv is the pixel coordinate system, O W X W Y W is the world coordinate system, O W X' C Y' Cis the projection coordinate system of the camera coordinate system on the ground. The angle between the target P and the optical axis projection is ε, ∠cO C O is γ. The target P is on the axis O W Y' C The projection point on the image is A, w is the horizontal distance from point p to the center of the image, h is the vertical distance from point p to the center of the image, and the vertical distance between the target and the drone is O. W B is D, lateral distance BP is K, projection distance O W P is L. Then:

[0126]

[0127] (5-2) Based on the geometric relationship, we can obtain the three-dimensional projection distance L, lateral distance K, longitudinal distance D between the UAV and the human target, and the line distance O between the UAV and the target. C The calculation process of P is:

[0128] Based on the geometric relationship, calculate |O W A|:

[0129]

[0130] By ΔcO c p~ΔAO c P and Then the distance |AP| is:

[0131]

[0132] The projection distance L is calculated as:

[0133]

[0134] The calculated lateral distance K is:

[0135]

[0136] The longitudinal distance D is calculated as:

[0137]

[0138] The calculated distance OCP between the drone and the target is:

[0139]

[0140] For this example, the calculated three-dimensional projection distance L, lateral distance K, longitudinal distance D between the drone and the human target and the line distance between the drone and the target are c P| are respectively: |O cP|=18.38m, L=17.98m, K=2.87m, D=17.75m.

[0141] (6) Figure 7 As shown, the results of personnel target recognition, detection and ranging are output, and the target position and distance measurement are calibrated in the form of a box to complete the recognition, detection and three-dimensional ranging of personnel targets from the perspective of the drone.

[0142] This embodiment also provides a monocular ranging system for UAV and human targets based on deep learning, including:

[0143] Personnel target detection unit, which uses deep learning target detection network to detect and identify personnel targets;

[0144] The distance measurement unit takes into account the change of the UAV's flight posture and measures the three-dimensional distance between the UAV and the human target through the distance measurement theory;

[0145] The calibration unit calibrates the personnel target position and the measurement distance in the form of a box.

[0146] The present invention improves the network by using a small target feature processing module, a feature exchange enhancement module, a multi-scale feature fusion network design and a variety of attention mechanisms, so that the network is more sensitive to tiny objects and can accurately identify small targets of people; the present invention considers the change of the flight posture of the UAV, modifies the ranging theory, and realizes the three-dimensional distance measurement between the UAV and the target with high accuracy; the present invention can realize the recognition and detection of small targets of people from the perspective of the UAV and the three-dimensional ranging, and calibrates the target position and distance measurement in the form of a box, has high generalization and robustness, small measurement error, and has a wide range of application scenarios and significant promotion value.

[0147] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A monocular ranging method for UAV and human targets based on deep learning, characterized in that: include: Use deep learning target detection network to detect and identify human targets; Considering the change of UAV flight attitude, the three-dimensional distance between the UAV and the human target is measured by using the ranging theory; The position of the personnel target and the measured distance are calibrated in the form of a box to complete the recognition, detection and three-dimensional ranging of the personnel target from the perspective of the drone.

2. The method for monocular ranging between a drone and a human target based on deep learning according to claim 1, characterized in that: The specific steps include: (1) Calibrate the internal parameters of the drone’s onboard camera to obtain the camera’s focal length f and optical center coordinates (u0, v0); (2) Obtain images frame by frame from the video stream captured by the drone’s onboard camera; (3) Using the deep learning target detection network to identify the target person to be detected from each frame of the image, obtain the target person positioning information and the target's ranging feature points p(u, v); (4) Use the fuselage sensor to read the drone's altitude H, the drone's attitude angle data, the pitch angle β, and the yaw angle δ; (5) The camera focal length f, optical center coordinates (u0, v0) obtained in step (1), the ranging feature point p (u, v) obtained in step (3), and the aircraft attitude data obtained in step (4) are brought into the ranging model to obtain the three-dimensional projection distance L, lateral distance K, longitudinal distance D between the UAV and the human target, and the connecting line distance O between the UAV and the target. C P; (6) The personnel target positioning information obtained in step (3) and the ranging data obtained in step (5) are output, and the personnel target position and the measurement distance are calibrated in the form of a box to complete the recognition, detection and three-dimensional ranging of the personnel target from the perspective of the drone.

3. The method for monocular ranging between a drone and a human target based on deep learning according to claim 2, characterized in that: In step (1), the Zhang Zhengyou calibration method is used to calibrate the intrinsic parameters of the drone airborne camera, and the intrinsic parameter matrix obtained by calibration is: Among them, f x =f / dx,f y =f / dy, f is the focal length of the camera, dx and dy are the intrinsic parameters of the camera, representing the physical length of the unit pixel in the x- and y-axis directions respectively.

4. The method for monocular ranging between a drone and a human target based on deep learning according to claim 2, characterized in that: The deep learning target detection network uses the improved network Yolov5s-TinyVision network based on the Yolov5s detection algorithm to perform personnel target detection and feature extraction, outputs personnel target positioning information, and converts it into the ranging feature points p (u, v) of the personnel target. The improved network Yolov5s-TinyVision network adopts a small target feature processing module, a feature exchange enhancement module, a multi-scale feature fusion module and a variety of attention mechanisms to improve the YOLOv5 network.

5. The method for monocular ranging between a drone and a human target based on deep learning according to claim 4, characterized in that: The deep learning target detection network uses the improved network Yolov5s-TinyVision network based on the Yolov5s detection algorithm to perform personnel target detection and feature extraction, which specifically includes the following steps: (3-1) Using the drone scene dataset, the optimal anchor box size of the human target is obtained based on K-Means clustering and genetic algorithm; (3-2) Input the image and use the feature extraction network to preliminarily extract the target feature X; (3-3) Use the small target feature processing module to process the initially extracted target feature X to obtain a feature X that focuses more on the small target tiny The small target feature processing module replaces the cross-layer convolution and pooling layers in the feature extraction network, and consists of a space-to-depth layer and a non-strided convolution layer; (3-4) Use the feature communication enhancement module to use the CBAM attention mechanism to focus on the target feature X tiny After further processing, the enhanced personnel target feature X is obtained. tiny-f ; (3-5) Use the feature pyramid network to enhance the target feature X tiny-f Processing to obtain feature X tiny-r ; (3-6) Use the detection head to detect feature X tiny-r Processing is performed to obtain relevant information about personnel target positioning; (3-7) The output module obtains the coordinate information of the person target box in the image coordinate system based on the relevant information of the person target positioning (x left ,y top ),(x right ,y bottom ), the coordinates of the ranging feature point p(u, v) of the human target are calculated as: u=(x left +x right ) / 2 v=y bottom 。 6. The method for monocular ranging between a drone and a human target based on deep learning according to claim 5, characterized in that: The specific process of the small target feature processing module processing feature X is as follows: X tiny =concat(CBR(f 0,0 ),CBR(f 0,1 ),...,CBR(f scale-1,scale-1 ))·Sigmoid(CB(X)) Among them, Split(X, Scale) represents downsampling and slicing the feature map X with the scale factor scale, Concat(·) represents concatenating the sliced ​​features in the channel direction, CBR(·) represents sequentially performing convolution, normalization, and activation function operations on the feature map, Sigmoid(·) represents sigmoid function activation and normalization operations on the feature map, and CB(·) represents sequentially performing convolution and normalization operations on the feature map.

7. The method for monocular ranging between a drone and a human target based on deep learning according to claim 5, characterized in that: The step (3-4) specifically includes the steps of: (3-4-1) Use the channel attention mechanism to focus on the target feature X in the channel tiny Enhance it and get the feature X after channel attention enhancement C , the formula for calculating the weight of each channel by channel attention is: M C (X tiny )=Sigmoid(MLP(AvgPool(X tiny ))+MLP(MaxPool(X tiny ))) X C =M C (X tiny )·X tiny Among them, AvgPool(·) represents the average pooling operation, MaxPool(·) represents the maximum pooling operation, MLP(·0 represents the multi-layer perceptron model, (·) represents the element-based dot multiplication, and (+) represents the matrix addition; (3-4-2) Get the feature X after channel attention enhancement C Then, the spatial attention mechanism is used to focus on the spatial feature X C Enhanced, to obtain enhanced personnel target feature X tiny-f for: M S (X C )=Sigmoid(Conv(Concat(AvgPool(X C ),MaxPool(X C )))) X tiny-f =M S (X C )·X C Among them, Conv(·) represents the convolution operation.

8. The method for monocular ranging between a drone and a human target based on deep learning according to claim 5, characterized in that: The steps (3-6) specifically include: The three detection head structures of Yolov5 Head are increased to four detection head structures, and a small target detection layer is added. The detection head structure is a Transformer detection head structure. The Transformer detection head structure is used to detect the feature X obtained in step (3-5). tiny-r Processing results: X trans-r =Conv(Concat(TB(Conv(X tiny-r )),Conv(X tiny-r ))) Among them, TB(·) refers to the Transformer Block module based on the Transformer Encoder architecture, specifically: TB(X)=MLP(LN(X temp ))+X temp X temp =MutiAttention(LN(Flatten(X)))+X The Flatten(·) layer flattens the two-dimensional feature vector into a one-dimensional vector, LN(·) represents the normalization operation, and MutiAttention(·) represents the multi-head attention mechanism. Each single-head attention of the multi-head attention mechanism is: Among them, Q,K,V∈R N×C Represent the query matrix, key matrix and value matrix respectively, Q i ,K i ,V i represents the multiplication of Q, K, V and the weight matrix of the self-attention mechanism, softmax(·) represents the softmax activation function, and the Attention obtained after the feature mapping of each single-head attention i The final output of the multi-head attention mechanism is formed by the multi-scale feature fusion module, which is: Attention=Concat(Attention1,Attention2,...,Attention8); After the processing is completed, the relevant information of personnel target positioning is obtained.

9. The method for monocular ranging between a drone and a human target based on deep learning according to claim 2, characterized in that: The step (5) specifically comprises the steps of: (5-1) Design the UAV ranging model, define the coordinate system and corresponding coordinate points, let P be the position of the target to be detected, p be the corresponding position of the target in the imaging plane, O be the center of the camera target surface, O C is the camera optical center, O P uv is the pixel coordinate system, O W X W Y W is the world coordinate system, O W X' C Y' C is the projection coordinate system of the camera coordinate system on the ground; the angle between the target P and the optical axis projection is ε, ∠cO C O is γ, the target P is on the axis O W Y' C The projection point on the image is A, w is the horizontal distance from point p to the center of the image, h is the vertical distance from point p to the center of the image, and the vertical distance between the target and the drone is O. W B is D, lateral distance BP is K, projection distance O W P is L; then: w=(u-u0)dx h=(v0-v)dy (5-2) Based on the geometric relationship, we can obtain the projection distance L, lateral distance K, longitudinal distance D between the drone and the human target, and the line distance O between the drone and the target. C P is:

10. A monocular ranging system for UAV and human targets based on deep learning to implement any of the methods described in claims 1-9, characterized in that: include: Personnel target detection unit, which uses deep learning target detection network to detect and identify personnel targets; The distance measurement unit takes into account the change of the UAV's flight posture and measures the three-dimensional distance between the UAV and the human target through the distance measurement theory; The calibration unit calibrates the personnel target position and the measurement distance in the form of a box.

Images