Key point detection methods, devices, electronic equipment and storage media for unmanned aerial vehicles
By introducing a 3D voxel model to assist the key point detection neural network in UAV key point detection, the problem of decreased detection performance caused by occlusion and image blurring is solved, and the detection accuracy is improved.
Patent Information
- Application Number
- CN202410743445.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-11
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2044-06-11
AI Technical Summary
Existing key point detection methods for drones perform poorly in situations involving occlusion and blurred images, making it difficult to accurately identify key points on drones.
By introducing a 3D voxel model of a drone as prior knowledge, and integrating it into the reasoning and prediction process of a keypoint detection neural network through feature extraction, the keypoint detection results can be assisted or corrected.
It improves the accuracy of key point detection for drones, especially when key points are occluded or the image is blurry, significantly enhancing detection performance.
Smart Images

Figure CN118609000B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision technology, and in particular to a method, apparatus, electronic device, and storage medium for detecting key points on a drone. Background Technology
[0002] In recent years, with the increasing maturity of drone technology and the rapid growth of the drone application market, drones have been widely used in various fields of production and daily life, such as precision irrigation and pesticide spraying in agriculture, last-mile delivery in extreme geographical environments in logistics, and fire scene surveying and high-altitude firefighting in firefighting. While the application of drones has brought many conveniences to production and daily life, it has also brought significant safety hazards, such as frequent occurrences of illegal intrusion, illegal delivery, and illegal flight, threatening public safety. Therefore, the monitoring and management of drone targets is essential.
[0003] The key point detection task for drones involves detecting the pixel coordinates of key or vulnerable parts of a drone target in a video or image, typically the centroid of the key part. This task helps drone defense systems locate specific parts of the target for precision strikes and also provides information on the positional relationships between key points to assist in attitude estimation and trajectory prediction of the drone target, thus making it of great significance.
[0004] Keypoint detection originated from human keypoint detection tasks, which involve detecting the coordinates of key nodes of a human target in an image, such as the head, shoulders, arms, and knees. With the rise of deep neural networks and deep learning, current human keypoint detection is primarily implemented using deep learning techniques. The principle involves using convolutional neural networks to extract image features, and then using deep neural networks for keypoint coordinate regression or keypoint heatmap prediction. Keypoint coordinate regression methods directly regress the coordinate values of keypoints through the network; heatmap prediction methods generate heatmaps corresponding to keypoints through the network. The pixel value at each location in the heatmap represents the probability that that location is a keypoint, and the location with the highest probability is used as the predicted coordinates of the keypoint.
[0005] Traditional methods for keypoint detection on unmanned aerial vehicles (UAVs) primarily utilize target detection networks to obtain image regions containing UAV targets. Then, the target region is cropped from the original image and input into a keypoint regression network based on cross-level connections and depthwise separable convolutions to output regression values of keypoint coordinates. Alternatively, images containing infrared targets are input into an improved Mask R-CNN network to achieve keypoint regression prediction. Another approach uses the OpenPose network as the base network model for keypoint detection, modifying the convolutional network structure to improve multi-scale feature extraction capabilities. Existing methods generally use two-dimensional images as input and employ convolutional neural networks for keypoint detection. However, UAV target imaging often exhibits blurring and occlusion of key parts due to UAV maneuverability, preventing convolutional network models from extracting effective keypoint image features. Figure 1 As shown, in a series of video frames, the left front wing is occluded by the drone itself, resulting in disordered and significantly deviated keypoint positions output by the convolutional keypoint detection model. Therefore, keypoint detection methods that use images as a single input struggle to handle issues such as occlusion and blurring in images, leading to poor keypoint detection performance. Summary of the Invention
[0006] This invention provides a key point detection method for unmanned aerial vehicles (UAVs). The method can introduce the three-dimensional voxel model of the UAV as prior knowledge and integrate it into the inference and prediction process of the key point detection neural network through feature extraction. The aim is to use the three-dimensional voxel model of the UAV to assist or correct the key point detection results, especially in cases where key points are occluded or the image is blurred, thereby improving the key point detection performance and the accuracy of UAV key point detection.
[0007] This invention provides a method for detecting key points on a UAV, the method comprising:
[0008] The system acquires an image of the UAV to be detected and a three-dimensional voxel model of the UAV, which is obtained by voxelizing the three-dimensional model of the UAV.
[0009] The image to be detected and the three-dimensional voxel model are input into a trained keypoint detection model to perform keypoint detection, thereby obtaining the keypoint detection results of the UAV, which include the keypoints of the UAV.
[0010] Optionally, the step of acquiring the image to be detected from the drone includes:
[0011] Acquire raw images of the drone;
[0012] The original image is input into a trained target detection model to detect drone targets and obtain drone target bounding boxes.
[0013] Based on the drone target bounding box, the image region containing the drone target is extracted from the original image to obtain the drone image to be detected.
[0014] Optionally, before inputting the original image into the trained target detection model to perform UAV target detection and obtain the UAV target bounding box, the method further includes:
[0015] A convolutional neural network with a feature fusion mechanism is used as the target detection model. The input of the target detection model is a full-size image, and the output is the target box information (x,y,w,h) of the UAV, where (x,y) are the coordinates of the center of the target box in the image, and w and h are the width and height of the target box.
[0016] The object detection model is trained to obtain a trained object detection model, wherein the first loss function in the training process is as follows:
[0017] L(w) = bbox(w) + conf(w)
[0018]
[0019]
[0020]
[0021] Where L(w) is the loss function, w are the parameters of the object detection network model, f is the network model, and I is the input image; the loss function consists of two parts: localization loss bbox(w) and confidence loss conf(w). In the localization loss bbox(w), A represents the target bounding box predicted by the model, B represents the ground truth target bounding box, and A represents the bounding box containing the target. The minimum region bounding box of B; in the confidence loss conf(w), K*K is the size of the output feature map, and M is the number of prior boxes. and As an indicator variable, when the j-th prior bounding box of the i-th grid in the input feature map contains a UAV target, and on the contrary, and λ noobj Here is a hyperparameter representing the weight of negative samples. C represents the confidence level of the model prediction, and C represents the true confidence level.
[0022] Optionally, the step of obtaining the three-dimensional voxel model of the UAV includes:
[0023] Obtain the three-dimensional geometric model of the UAV;
[0024] The three-dimensional geometric model is subjected to point cloud analysis to obtain a three-dimensional point cloud model;
[0025] The three-dimensional point cloud model is converted into a three-dimensional voxel model at a preset resolution.
[0026] Optionally, the step of converting the 3D point cloud model into a 3D voxel model at a preset resolution includes:
[0027] The length, width, and height of a single voxel are determined to be L. x ×L y ×L z ;
[0028] The resolution is set to N;
[0029] Divide the three-dimensional cube into N*N*N L-shaped segments. x ×L y ×L z 3D mesh;
[0030] Map the three-dimensional point cloud model to the three-dimensional mesh, assign the value 1 to the three-dimensional mesh containing at least one three-dimensional coordinate point in the three-dimensional point cloud model to obtain the first voxel unit, and assign the value 0 to the three-dimensional mesh that does not contain any three-dimensional coordinate points in the three-dimensional point cloud model to obtain the second voxel unit.
[0031] A three-dimensional voxel model is obtained based on the first voxel unit and the second voxel unit.
[0032] Optionally, the keypoint model includes an image encoding network, a pose prediction network, a voxel encoding network, a feature fusion network, and a decoding network. The step of inputting the image to be detected and the three-dimensional voxel model into the trained keypoint detection model for keypoint detection to obtain the keypoint detection results of the UAV includes:
[0033] Image features are extracted from the image to be detected using the image coding network to obtain a multi-channel feature map of the image;
[0034] The pose prediction network is used to perform pose prediction processing on the multi-channel feature map of the image to obtain the affine transformation matrix.
[0035] The affine transformation matrix is used to perform an affine transformation on the three-dimensional voxel model to obtain the affine transformed three-dimensional voxel model.
[0036] Voxel features are extracted from the affine-transformed 3D voxel model using the voxel encoding network to obtain a voxel multi-channel feature map.
[0037] The image multi-channel feature map and the voxel multi-channel feature map are fused using a feature fusion network to obtain a fused multi-channel feature map.
[0038] The key point detection results of the UAV are obtained by decoding the fused multi-channel feature map through the decoding network.
[0039] Optionally, before inputting the image to be detected and the three-dimensional voxel model into the trained keypoint detection model for keypoint detection, the method further includes:
[0040] The keypoint detection model is trained to obtain a trained target detection model, wherein the second loss function during the training process is as follows:
[0041]
[0042] Where θ represents all learnable parameters of the network model. The first part is the keypoint heatmap predicted by the network model, while the second part is the actual heatmap. The actual heatmap is generated based on manual annotation of keypoints from real-world footage and a Gaussian function. The specific formula is as follows:
[0043]
[0044] heatmap i Let (x, y) represent the heatmap corresponding to the i-th real key point, and (x, y) represent any coordinates in the heatmap. i ,y i Let be the coordinates of the i-th real key point in the corresponding heatmap, σ be the standard deviation of the Gaussian kernel, and e be the natural logarithm.
[0045] This invention also provides a key point detection device for unmanned aerial vehicles (UAVs), the key point detection device comprising:
[0046] The acquisition module is used to acquire the image to be detected of the UAV and to acquire the three-dimensional voxel model of the UAV, wherein the three-dimensional voxel model is obtained by voxelization processing based on the three-dimensional model of the UAV.
[0047] The processing module is used to input the image to be detected and the three-dimensional voxel model into a trained key point detection model to perform key point detection, and obtain the key point detection result of the UAV, wherein the key point detection result includes the key points of the UAV.
[0048] The present invention also provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the UAV key point detection method as described in any one of the embodiments of the present invention.
[0049] The present invention also provides a computer storage medium on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the UAV key point detection method as described in any one of the embodiments of the present invention.
[0050] The key point detection method for UAVs of the present invention can introduce the three-dimensional voxel model of the UAV as prior knowledge and integrate it into the inference and prediction process of the key point detection neural network through feature extraction. It aims to use the three-dimensional voxel model of the UAV to assist or correct the key point detection results, especially in cases where key points are occluded or images are blurred, thereby improving the key point detection performance and improving the accuracy of UAV key point detection. Attached Figure Description
[0051] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0052] Figure 1 This is an example image of a key point being occluded.
[0053] Figure 2 This is a schematic diagram of a process for detecting key points of an unmanned aerial vehicle (UAV) according to an embodiment of the present invention;
[0054] Figure 3 This is a flowchart illustrating a method for detecting key points of a UAV that integrates a three-dimensional geometric model, provided in an embodiment of the present invention.
[0055] Figure 4 This is a schematic diagram illustrating the definition of key points of a drone according to an embodiment of the present invention;
[0056] Figure 5 This is a schematic diagram of the target detection model provided in an embodiment of the present invention;
[0057] Figure 6 This is a visualization diagram of a UAV's three-dimensional geometric model, point cloud model, and voxel model provided in an embodiment of the present invention;
[0058] Figure 7This is a schematic diagram of the key point detection model provided in an embodiment of the present invention;
[0059] Figure 8 This is a schematic diagram of the structure of a key point detection device for unmanned aerial vehicles provided in an embodiment of the present invention;
[0060] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation
[0061] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0062] like Figure 2 As shown, Figure 2 This is a flowchart illustrating a method for detecting key points on a drone according to an embodiment of the present invention. The method for detecting key points on a drone includes the following steps:
[0063] 201. Obtain the image of the UAV to be detected, and obtain the three-dimensional voxel model of the UAV.
[0064] In this embodiment of the invention, the three-dimensional voxel model is obtained by voxelizing the three-dimensional model of the UAV.
[0065] The three-dimensional geometric model of a specific target is voxelized to obtain a three-dimensional voxel model of the UAV. The three-dimensional voxel model can be represented as a three-dimensional matrix in mathematical form, which can be used as input to a neural network.
[0066] For the image of the key points to be detected (or a video frame for the video), drone target detection is performed to obtain the drone target area image as the drone's image to be detected.
[0067] 202. Input the image to be detected and the 3D voxel model into the trained key point detection model to perform key point detection and obtain the key point detection results of the UAV.
[0068] In this embodiment of the invention, the key point detection results include the key points of the UAV.
[0069] The overall implementation process is as follows: Figure 3As shown, firstly, the 3D geometric model of a specific target is voxelized to obtain a UAV voxel model (3D voxel model). The 3D voxel model can be represented as a 3D matrix in mathematical form, which can be used as input to a neural network. Secondly, UAV target detection is performed on the image of the key points to be detected (or a video frame for video), resulting in a UAV target region image (obtained by cropping the original image or video frame using a target detection bounding box) as the UAV's image to be detected. Finally, the UAV target region image, combined with the 3D voxel model, is input into a key point detection model that integrates the 3D voxel model. The key point detection model outputs a key point heatmap, which is then transformed to obtain the final coordinate positions of the key points, thus yielding the key point detection result.
[0070] This invention provides a specific definition of the task of detecting key points of UAV targets: for any image containing UAV targets, detect multiple key points of the UAV targets and mark the position of each key point in the image.
[0071] like Figure 4 As shown, this invention uses the DJI Mavic Pro 2 quadcopter drone as an example of the drone type to be tested, and uses the following 5 key points as the key points to be tested:
[0072] 1) The center point of the drone, i.e. the geometric center of the frame, is usually where the battery is located, and is represented by the number 0;
[0073] 2) The location of the motor on the left front wing of the drone is indicated by the number 1;
[0074] 3) The location of the motor on the right front wing of the drone is indicated by the number 2;
[0075] 4) The location of the motor on the right rear wing of the drone is indicated by the number 3;
[0076] 5) The location of the motor on the left rear wing of the drone is indicated by the number 4.
[0077] The above five key points cover the main key points of UAV targets, which are sufficient to provide strong support for UAV precision strike missions, attitude estimation missions, etc., but do not mean that the method proposed in this invention can only detect quadcopter UAVs and the above five key points. The method proposed in this invention is also applicable to other UAVs (such as hexacopter, octocopter, and even fixed-wing UAVs), and is applicable to scenarios where key points are added or removed, such as adding a key point located on the airborne camera.
[0078] In this embodiment of the invention, the three-dimensional voxel model of the UAV can be introduced as prior knowledge and integrated into the inference and prediction process of the key point detection neural network through feature extraction. The aim is to use the three-dimensional voxel model of the UAV to assist or correct the key point detection results, especially in cases where key points are occluded or images are blurred, thereby improving the key point detection performance and thus improving the accuracy of key point detection of the UAV.
[0079] Optionally, the steps for obtaining the drone image to be detected include: obtaining the original image of the drone; inputting the original image into a trained target detection model to detect the drone target and obtain the drone target bounding box; and extracting the image region containing the drone target from the original image based on the drone target bounding box to obtain the drone image to be detected.
[0080] In this embodiment of the invention, a convolutional neural network (e.g., YOLO v5 model) with an introduced feature fusion mechanism can be used as the target detection model, which can adapt to multi-scale UAV targets. The input is a UAV target image, and the output is the target bounding box information of the UAV target, i.e. (x,y,w,h), where (x,y) are the coordinates of the center of the target bounding box in the original image, and w and h are the width and height of the target bounding box.
[0081] like Figure 5 As shown, the object detection network model consists of four parts: input, backbone, neck, and prediction. The input is responsible for data augmentation, adaptive image scaling, and other input image processing operations. The backbone, composed of multiple convolutional layers, pooling layers, and residual layers, is responsible for extracting rich information features from the input image. The neck, composed of a Feature Pyramid Network (FPN) and a Path Aggregation Network (PAN), is responsible for upsampling, downsampling, and lateral connection of the features generated by the backbone to achieve a feature fusion mechanism. The prediction part, composed of convolutional layers, is responsible for classifying and locating feature maps at three scales. It outputs multiple bounding boxes that may contain drone targets and their corresponding confidence scores, then uses Non-Maximum Suppression (NMS) to filter out highly overlapping bounding boxes, ultimately obtaining accurate bounding boxes containing drone targets. Because of the feature fusion mechanism, this model can effectively extract target regions at multiple scales; for example, large-scale target regions can be extracted from deep feature maps, while small-scale targets can be extracted from shallow feature maps.
[0082] Optionally, before inputting the original image into the trained object detection model for UAV target detection to obtain the UAV target bounding box, the method further includes: using a convolutional neural network with a feature fusion mechanism as the object detection model, wherein the input of the object detection model is a full-size image, and the output is the UAV target bounding box information (x, y, w, h), where (x, y) are the coordinates of the center of the target bounding box in the image, and w and h are the width and height of the target bounding box; training the object detection model to obtain a trained object detection model, wherein the first loss function in the training process is as follows:
[0083] L(w) = bbox(w) + conf(w)
[0084]
[0085]
[0086]
[0087] Where L(w) is the loss function, w are the parameters of the object detection network model, f is the network model, and I is the input image; the loss function consists of two parts: localization loss bbox(w) and confidence loss conf(w). In the localization loss bbox(w), A represents the target bounding box predicted by the model, B represents the ground truth target bounding box, and A represents the bounding box containing the target. The minimum region bounding box of B; in the confidence loss conf(w), K*K is the size of the output feature map, and M is the number of prior boxes. and As an indicator variable, when the j-th prior bounding box of the i-th grid in the input feature map contains a UAV target, and on the contrary, and λ noobj Here is a hyperparameter representing the weight of negative samples. C represents the confidence level of the model prediction, and C represents the true confidence level.
[0088] Based on the above target detection network model, this invention uses the following steps to achieve UAV target detection and obtain UAV target region images:
[0089] First, the above loss function is used as the objective function to minimize it. Gradient descent is used to learn the network parameters w until the network weights converge.
[0090] Next, using the pre-trained object detection network model, the input image is the target image, and the model outputs several bounding boxes (x, y, w, h) that may contain the drone target, along with their corresponding confidence scores (conf). Bounding boxes with confidence scores greater than a certain threshold (e.g., 0.5) are retained, and non-maximum suppression (NMS) is used to filter out highly overlapping boxes, ultimately yielding accurate bounding boxes containing the drone target. Note that if the confidence scores of all bounding boxes output by the model are not greater than the threshold, it is considered that no drone target exists in the current image.
[0091] Finally, the target bounding boxes obtained from the detection model are adjusted and appropriately enlarged to ensure that the drone target is completely contained within the bounding boxes. For example, the length and width of the target bounding box are both set to max(w,h)*1.2, resulting in a new, enlarged, square target bounding box. The original image is then cropped based on the new target bounding box to obtain the drone target region image. Note that if the new target bounding box extends beyond the edge of the original image, zeros are padded to the area outside the original image before cropping. To ensure a consistent input image size for subsequent keypoint detection models, the size of the drone target region image also needs to be uniformly adjusted to 256*256 using interpolation.
[0092] Based on the above steps, the final image of the UAV target area, img∈R, is obtained. 256×256×3 .
[0093] Optionally, the steps for obtaining the three-dimensional voxel model of the UAV include: obtaining the three-dimensional geometric model of the UAV; performing point cloud analysis on the three-dimensional geometric model to obtain a three-dimensional point cloud model; and converting the three-dimensional point cloud model into a three-dimensional voxel model at a preset resolution.
[0094] In this embodiment of the invention, the three-dimensional geometric model of the UAV can be designed and generated using the three-dimensional modeling software Autodesk 3D Max. A point cloud model can be parsed from the three-dimensional geometric model, which contains the three-dimensional coordinates of all vertices in the three-dimensional geometric model. Voxelization refers to the process of converting the three-dimensional geometric model into a voxel model from the point cloud model while maintaining a certain resolution. A voxel is a structured three-dimensional data representation, a basic unit constituting three-dimensional space, i.e., a single three-dimensional mesh. Combined with... Figure 6 The relationship between the 3D model, point cloud model, and voxel model can be clearly seen.
[0095] Optionally, the steps of converting a 3D point cloud model into a 3D voxel model at a preset resolution include: determining the length, width, and height of a single voxel as L. x ×L y ×L z ; Determine the resolution as N; Divide the 3D cube into N*N*N L-shaped segments. x ×L y×L z A three-dimensional mesh is obtained; the three-dimensional point cloud model is mapped to the three-dimensional mesh, and the three-dimensional mesh containing at least one three-dimensional coordinate point in the three-dimensional point cloud model is assigned a value of 1 to obtain the first voxel unit, and the three-dimensional mesh that does not contain any three-dimensional coordinate points in the three-dimensional point cloud model is assigned a value of 0 to obtain the second voxel unit; based on the first voxel unit and the second voxel unit, a three-dimensional voxel model is obtained.
[0096] In this embodiment of the invention, the following steps are used to convert the three-dimensional geometric model of the UAV into a voxel model with a resolution of 128*128*128.
[0097] First, a point cloud model is obtained by analyzing the 3D geometric model. Then, the coordinate range of the voxel model's X, Y, and Z axes is determined by traversing each point in the point cloud model, i.e., [x...]. min ,x max ],[y min ,y max ],[z min ,z max ], then the voxel model will be distributed in (x min ,y min ,z min ) and (x max ,y max ,z max In a three-dimensional cube with diagonal vertices ), .
[0098] Secondly, determine the length, width, and height (L) of a single voxel. x ×L y ×L z The specific formula is as follows:
[0099]
[0100] Where N is the resolution of 128, the 3D cube is divided into 128*128*128 L-shaped blocks. x ×L y ×L z A three-dimensional mesh.
[0101] Finally, the 128*128*128 three-dimensional meshes are filled with values of 0 or 1. Specifically, the three-dimensional meshes are mapped to the point cloud model. Elements of the voxel model (i.e., the 128*128*128 three-dimensional matrix) corresponding to a three-dimensional mesh containing one or more three-dimensional coordinate points in the point cloud model are set to 1, while matrix elements corresponding to three-dimensional meshes without any three-dimensional coordinate points are set to 0. For example, using a three-dimensional coordinate point (x... i ,y i ,z i Obtain the corresponding 3D mesh coordinates (g) x,g y ,g z The formula is as follows:
[0102]
[0103] Where floor is the floor function.
[0104] Based on the above steps, the final UAV target voxel model vox∈R is obtained. 128×128×128 .
[0105] Optional, such as Figure 7 As shown, the keypoint detection model includes an image encoding network, a pose prediction network, a voxel encoding network, a feature fusion network, and a decoding network. The steps for inputting the image to be detected and the 3D voxel model into the trained keypoint detection model to obtain the UAV keypoint detection results are as follows: The image encoding network extracts image features from the image to be detected, obtaining a multi-channel feature map; the pose prediction network performs pose prediction processing on the multi-channel feature map to obtain an affine transformation matrix; the affine transformation matrix is used to perform an affine transformation on the 3D voxel model, obtaining an affine-transformed 3D voxel model; the voxel encoding network extracts voxel features from the affine-transformed 3D voxel model, obtaining a voxel multi-channel feature map; the feature fusion network fuses the multi-channel feature map of the image and the multi-channel feature map of the voxels, obtaining a fused multi-channel feature map; and the decoding network decodes the fused multi-channel feature map to obtain the UAV keypoint detection results.
[0106] In this embodiment of the invention, in order to enable the model to better handle poor image quality conditions such as image blurring and keypoint occlusion, the method proposed in this invention uses the target region image img∈R. 256×256×3 Joint target voxel model based on 3D geometric model vox∈R 128×128×128 Both are used as input to the keypoint detection network, which outputs a keypoint heatmap. The keypoint detection network structure of this invention is shown in Figure 7. The algorithm for keypoint detection implemented by this network model is as follows:
[0107] Model Input: The keypoint detection network designed in this invention has two inputs, namely the target region image img∈R. 256×256×3 and the target voxel model based on the three-dimensional geometric model vox∈R 128×128×128 .
[0108] Image feature extraction: target region image img∈R 256×256×3 Feature extraction is performed using an image coding network to obtain the multi-channel feature map img_ft∈R. 2048×8×8This is used in subsequent target pose prediction networks and feature map fusion. The image encoding network can be a common convolutional-based image extraction network; this invention uses ResNet18.
[0109] Target pose prediction: using multi-channel feature maps of the image, img_ft∈R 2048×8×8 The input is the affine transformation matrix T = [R|t] ∈ R, which is output by the pose prediction network. 3×4 The affine transformation matrix T is represented by R ∈ R. 3×3 And denote the translation transformation vector t∈R 3×1 Obtained by splicing. The rotation-stretching transformation matrix R∈R 3×3 The translation vector t ∈ R represents the rotation and stretching of the UAV target (relative to the body coordinate system) in the input image. 3×1 This represents the translation of the UAV target (relative to the origin of the body coordinate system) in the input image. An ideal affine transformation matrix T can perform rotation, stretching, and translation operations on the voxel model, aligning the voxel model with the UAV target in the input image after the affine transformation. The pose prediction network consists of a single 2D convolutional network and two fully connected layers. The network computation process is as follows:
[0110] ft1=Conv2D(2048,1024,k=8)(img_ft)
[0111] ft2=W2*Relu(W1*Relu(Flatten(ft1))+b1)+b2
[0112] T = Reshape(ft2)
[0113] Where img_ft∈R 2048×8×8 The input to the pose estimation network is the multi-channel feature map of the image; Conv2D is a 2D convolutional layer with 2048 input channels, 1024 output channels, and an 8*8 kernel size; ft1∈R 1024×1×1 This is the output of the convolutional layer; Flatten is the feature flattening operation, which transforms ft1 into R. 1024×1 eigenvectors; W1∈R 128×1024 and b1∈R 128×1 Here are the weights and biases of the first fully connected layer; W2∈R 12×128 and b2∈R 12×1 The weights and biases of the second fully connected layer; ft2∈R 12×1 The output of the two fully connected layers is represented by ReLU, which is the activation function. Reshape is the feature dimension transformation operation, which transforms ft2 into an affine transformation matrix T∈R. 3×4 .
[0114] Voxel model encoding: First, implement the encoding of the voxel model vox∈R. 128×128×128 The affine transformation is applied, and the resulting voxel model will be aligned with the UAV target in the input image as much as possible in terms of pose and size. The specific formula is as follows:
[0115]
[0116] in and These are the affine transformation precursor voxel models vox∈R 128×128×128 voxel model after affine transformation T ∈R 128×128×128 The coordinates of the i-th voxel grid, T∈R 3×4 This is the predicted affine transformation matrix. The voxel model `vox` after the affine transformation can be obtained by iterating through i = 0, 1, ..., 128*128*128-1. T ∈R 128×128×128 Secondly, using the affine-transformed voxel model as input, voxel features are extracted through a voxel encoding network to obtain a voxel multi-channel feature map vox_ft∈R. 2048×8×8 This is used for subsequent feature map fusion. The voxel encoding network consists of two 3D convolutional layers, and the network's computation process is as follows:
[0117] ft1 = Conv3D1(1,32,k=3)(vox T )
[0118] ft1 = Relu(ft1)
[0119] ft2=Conv3D2(32,32,k=3)(ft1)
[0120] ft2 = Relu(ft2)
[0121] ft3 = Maxpool3D(k = 2)(ft2)
[0122] vox _ ft = Reshape(ft3)
[0123] Among them, vox T ∈R 128×128×128 ft1, ft2 ∈ R is the input to the voxel encoding network; Conv3D1 is the first 3D convolutional layer with 1 input channel, 32 output channels, and a 3*3*3 kernel size; Conv3D2 is the second 3D convolutional layer with 32 input and 32 output channels and a 3*3*3 kernel size; ft1, ft2 ∈ R 32*64*64*64These are the outputs of Conv3D1 and Conv3D2, respectively; ReLU is the activation function; Maxpool3D is a 3D max pooling layer with a 2*2*2 kernel size; ft3∈R 32*16*16*16 The pooled features are represented by the Reshape operation, which transforms ft3 into a voxel multichannel feature map vox_ft∈R. 2048×8×8 .
[0124] Heatmap prediction: First, the multi-channel feature map img_ft∈R is generated. 2048×8×8 voxel multichannel feature map vox_ft∈R 2048×8×8 The summation yields the fused multi-channel feature map merge_ft∈R 2048×8×8 Secondly, the multi-channel feature map is used as input, and a key point heatmap is generated through a decoding network. The prediction is as follows. The decoding network consists of three deconvolutional layers and one convolutional layer. The deconvolutional layers are responsible for upsampling the fused multi-channel feature map. After three deconvolutional layers, the size of the feature map can be increased from 8*8 to 64*64. The convolutional layer is responsible for converting the number of channels in the feature map to the same number of channels as the number of keypoints. Therefore, each keypoint i corresponds to a heatmap. Used for keypoint detection. The computation process of the decoding network is as follows:
[0125] ft1=DeConv2D1(2048,256,k=4)(merge_ft)
[0126] ft1 = Relu(ft1)
[0127] ft2=DeConv2D2(256,256,k=4)(ft1)
[0128] ft2 = Relu(ft2)
[0129] ft3=DeConv2D3(256,256,k=4)(ft2)
[0130] ft3 = Relu(ft3)
[0131]
[0132] Where merge_ft∈R 2048×8×8 The input to the decoding network is DeConv2D1, which is the first deconvolutional layer with 2048 input channels, 256 output channels, and a 4x4 kernel size. DeConv2D2 and DeConv2D3 are the second and third deconvolutional layers, respectively, with 256 input and output channels and a 4x4 kernel size. ft3∈R 256×64×64The output is the feature map of three deconvolutional layers; Conv2D is a convolutional layer with 256 input channels, 5 output channels, and a kernel size of 1*1; ReLU is the activation function. This is the output of the decoding network, which is a heatmap corresponding to the 5 key points.
[0133] After the above algorithm steps of the keypoint detection network that integrates the 3D geometric model, a heatmap corresponding to each keypoint is obtained. The coordinates of the keypoints with the highest probability values in the heatmap are then transformed to obtain the final coordinates of the detected keypoints. The formula for coordinate transformation is as follows:
[0134]
[0135] Where (x) i ,y i (x) represents the coordinate with the highest probability value in the i-th heatmap. org_i ,y org_i ) represents the coordinates of the i-th key point in the input image, and scale represents the scaling factor used to adjust the input UAV target area image to a size of 256*256.
[0136] Optionally, before inputting the image to be detected and the 3D voxel model into the trained keypoint detection model for keypoint detection, the method further includes: training the keypoint detection model to obtain a trained target detection model, wherein the second loss function in the training process is as follows:
[0137]
[0138] Where θ represents all learnable parameters of the network model. The first part is the keypoint heatmap predicted by the network model, while the second part is the actual heatmap. The actual heatmap is generated based on manual annotation of keypoints from real-world footage and a Gaussian function. The specific formula is as follows:
[0139]
[0140] heatmap i Let (x, y) represent the heatmap corresponding to the i-th real key point, and (x, y) represent any coordinates in the heatmap. i ,y i Let be the coordinates of the i-th real key point in the corresponding heatmap, σ be the standard deviation of the Gaussian kernel, and e be the natural logarithm.
[0141] The embodiments of the present invention have been experimentally verified using real-world video data as follows:
[0142] Dataset: This invention selects the DJI Mavic Pro 2 quadcopter drone as the target. After the drone takes off, images and videos of the target are acquired using ground cameras and aerial drones. After data filtering, cleaning, and manual annotation, a dataset containing 14,392 images and corresponding annotations is finally obtained.
[0143] Evaluation Metric: This invention uses PCK as the evaluation metric. PCK is defined as the proportion of correctly detected keypoints out of all keypoints, that is, the percentage of keypoints whose distance from their true labeled location is less than a set threshold in the keypoint detection results. The specific calculation formula is as follows:
[0144]
[0145]
[0146] Where d ij d represents the Euclidean distance between the j-th model-predicted keypoint and its corresponding ground truth keypoint in the i-th sample (image or video frame); i_thres The distance threshold set for the i-th sample is 1 / 100 of the diagonal length of the actual target bounding box of the UAV in the i-th sample, which is selected as the distance threshold in the experiment; n is the total number of samples, and k is the number of key points.
[0147] Comparison Method: This invention uses the Simple Baselines network model [6] as the baseline method for comparison. Simple Baselines uses ResNet as the backbone of the key point feature extraction network and makes lightweight improvements to the upsampling structure part of resolution restoration. While simplifying the network structure, it effectively ensures the high accuracy of key point detection. The key point detection network model that integrates the three-dimensional geometric model proposed in this invention has certain similarities with Simple Baselines in terms of network structure, such as the image encoding network and decoding network. However, the structure of voxel model input, pose prediction network and voxel encoding network is unique to the method proposed in this invention. Therefore, comparing the effect with the Simple Baselines network model can highlight the effectiveness of the method proposed in this invention.
[0148] The final experimental results are shown in Table 1, where PCK0 to PCK4 represent the five keypoint PCK indices, and PCKmean is the average PCK index of all keypoints. It can be seen that the method proposed in this invention significantly improves upon the Simple Baselines model in both single keypoint detection and all keypoint detection. The final experimental results are shown in Table 1.
[0149] Table 1
[0150] Comparison Methods <![CDATA[PCK0]]> <![CDATA[PCK1]]> <![CDATA[PCK2]]> <![CDATA[PCK3]]> <![CDATA[PCK4]]> <![CDATA[PCK mean ]]> Simple Baselines 71.795% 69.369% 69.023% 69.468% 69.023% 69.536% The method proposed in this invention 74.498% 74.775% 74.290% 74.498% 73.597% 74.331%
[0151] The key point detection method for UAVs of the present invention can introduce the three-dimensional voxel model of the UAV as prior knowledge and integrate it into the inference and prediction process of the key point detection neural network through feature extraction. It aims to use the three-dimensional voxel model of the UAV to assist or correct the key point detection results, especially in cases where key points are occluded or images are blurred, thereby improving the key point detection performance and improving the accuracy of UAV key point detection.
[0152] like Figure 8 As shown, Figure 8 This is a schematic diagram of a key point detection device for unmanned aerial vehicles (UAVs) provided in an embodiment of the present invention. The key point detection device for UAVs includes:
[0153] The acquisition module 801 is used to acquire the image to be detected of the UAV and to acquire the three-dimensional voxel model of the UAV, wherein the three-dimensional voxel model is obtained by voxelization processing based on the three-dimensional model of the UAV.
[0154] The processing module 802 is used to input the image to be detected and the three-dimensional voxel model into a trained key point detection model to perform key point detection, and obtain the key point detection result of the UAV, wherein the key point detection result includes the key points of the UAV.
[0155] Optionally, the acquisition module 801 is further configured to:
[0156] Acquire raw images of the drone;
[0157] The original image is input into a trained target detection model to detect drone targets and obtain drone target bounding boxes.
[0158] Based on the drone target bounding box, the image region containing the drone target is extracted from the original image to obtain the drone image to be detected.
[0159] Optionally, the device further includes:
[0160] The module is used to use a convolutional neural network with a feature fusion mechanism as an object detection model. The input of the object detection model is a full-size image, and the output is the target box information (x,y,w,h) of the UAV, where (x,y) are the coordinates of the center of the target box in the image, and w and h are the width and height of the target box.
[0161] The first training module is used to train the object detection model to obtain a trained object detection model. The first loss function during the training process is as follows:
[0162] L(w) = bbox(w) + conf(w)
[0163]
[0164]
[0165]
[0166] Where L(w) is the loss function, w are the parameters of the object detection network model, f is the network model, and I is the input image; the loss function consists of two parts: localization loss bbox(w) and confidence loss conf(w). In the localization loss bbox(w), A represents the target bounding box predicted by the model, B represents the ground truth target bounding box, and A represents the bounding box containing the target. The minimum region bounding box of B; in the confidence loss conf(w), K*K is the size of the output feature map, and M is the number of prior boxes. and As an indicator variable, when the j-th prior bounding box of the i-th grid in the input feature map contains a UAV target, and on the contrary, and λ noobj Here is a hyperparameter representing the weight of negative samples. C represents the confidence level of the model prediction, and C represents the true confidence level.
[0167] Optionally, the acquisition module 801 is further configured to:
[0168] Obtain the three-dimensional geometric model of the UAV;
[0169] The three-dimensional geometric model is subjected to point cloud analysis to obtain a three-dimensional point cloud model;
[0170] The three-dimensional point cloud model is converted into a three-dimensional voxel model at a preset resolution.
[0171] Optionally, the acquisition module 801 is further configured to:
[0172] The length, width, and height of a single voxel are determined to be L. x ×L y ×L z ;
[0173] The resolution is set to N;
[0174] Divide the three-dimensional cube into N*N*N L-shaped segments. x ×L y ×L z 3D mesh;
[0175] Map the three-dimensional point cloud model to the three-dimensional mesh, assign the value 1 to the three-dimensional mesh containing at least one three-dimensional coordinate point in the three-dimensional point cloud model to obtain the first voxel unit, and assign the value 0 to the three-dimensional mesh that does not contain any three-dimensional coordinate points in the three-dimensional point cloud model to obtain the second voxel unit.
[0176] A three-dimensional voxel model is obtained based on the first voxel unit and the second voxel unit.
[0177] Optionally, the keypoint model includes an image encoding network, a pose prediction network, a voxel encoding network, a feature fusion network, and a decoding network, and the processing module 802 is further used for:
[0178] Image features are extracted from the image to be detected using the image coding network to obtain a multi-channel feature map of the image;
[0179] The pose prediction network is used to perform pose prediction processing on the multi-channel feature map of the image to obtain the affine transformation matrix.
[0180] The affine transformation matrix is used to perform an affine transformation on the three-dimensional voxel model to obtain the affine transformed three-dimensional voxel model.
[0181] Voxel features are extracted from the affine-transformed 3D voxel model using the voxel encoding network to obtain a voxel multi-channel feature map.
[0182] The image multi-channel feature map and the voxel multi-channel feature map are fused using a feature fusion network to obtain a fused multi-channel feature map.
[0183] The key point detection results of the UAV are obtained by decoding the fused multi-channel feature map through the decoding network.
[0184] Optionally, the device further includes:
[0185] The second training module is used to train the keypoint detection model to obtain a trained target detection model. The second loss function during the training process is shown below:
[0186]
[0187] Where θ represents all learnable parameters of the network model. The first part is the keypoint heatmap predicted by the network model, while the second part is the actual heatmap. The actual heatmap is generated based on manual annotation of keypoints from real-world footage and a Gaussian function. The specific formula is as follows:
[0188]
[0189] heatmap i Let (x, y) represent the heatmap corresponding to the i-th real key point, and (x, y) represent any coordinates in the heatmap. i ,y i Let be the coordinates of the i-th real key point in the corresponding heatmap, σ be the standard deviation of the Gaussian kernel, and e be the natural logarithm.
[0190] The UAV key point detection device provided in this embodiment of the invention can implement all the processes implemented by the UAV key point detection method in the above method embodiments, and can achieve the same beneficial effects. To avoid repetition, it will not be described again here.
[0191] See Figure 9 , Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention, such as... Figure 6 As shown, it includes: a memory 902, a processor 901, and a computer program for a UAV key point detection method stored in the memory 902 and executable on the processor 901, wherein:
[0192] The processor 901 is used to call the computer program stored in the memory 902 and execute the following steps:
[0193] The system acquires an image of the UAV to be detected and a three-dimensional voxel model of the UAV, which is obtained by voxelizing the three-dimensional model of the UAV.
[0194] The image to be detected and the three-dimensional voxel model are input into a trained keypoint detection model to perform keypoint detection, thereby obtaining the keypoint detection results of the UAV, which include the keypoints of the UAV.
[0195] Optionally, the step of acquiring the image to be detected of the UAV performed by the processor 901 includes:
[0196] Acquire raw images of the drone;
[0197] The original image is input into a trained target detection model to detect drone targets and obtain drone target bounding boxes.
[0198] Based on the drone target bounding box, the image region containing the drone target is extracted from the original image to obtain the drone image to be detected.
[0199] Optionally, before inputting the original image into the trained target detection model for UAV target detection to obtain the UAV target bounding box, the method executed by the processor 901 further includes:
[0200] A convolutional neural network with a feature fusion mechanism is used as the target detection model. The input of the target detection model is a full-size image, and the output is the target box information (x,y,w,h) of the UAV, where (x,y) are the coordinates of the center of the target box in the image, and w and h are the width and height of the target box.
[0201] The object detection model is trained to obtain a trained object detection model, wherein the first loss function in the training process is as follows:
[0202] L(w) = bbox(w) + conf(w)
[0203]
[0204]
[0205]
[0206] Where L(w) is the loss function, w are the parameters of the object detection network model, f is the network model, and I is the input image; the loss function consists of two parts: localization loss bbox(w) and confidence loss conf(w). In the localization loss bbox(w), A represents the target bounding box predicted by the model, B represents the ground truth target bounding box, and A represents the bounding box containing the target. The minimum region bounding box of B; in the confidence loss conf(w), K*K is the size of the output feature map, and M is the number of prior boxes. and As an indicator variable, when the j-th prior bounding box of the i-th grid in the input feature map contains a UAV target, and on the contrary, and λ noobj Here is a hyperparameter representing the weight of negative samples. C represents the confidence level of the model prediction, and C represents the true confidence level.
[0207] Optionally, the step of acquiring the three-dimensional voxel model of the UAV performed by the processor 901 includes:
[0208] Obtain the three-dimensional geometric model of the UAV;
[0209] The three-dimensional geometric model is subjected to point cloud analysis to obtain a three-dimensional point cloud model;
[0210] The three-dimensional point cloud model is converted into a three-dimensional voxel model at a preset resolution.
[0211] Optionally, the step of converting the 3D point cloud model into a 3D voxel model at a preset resolution, performed by the processor 901, includes:
[0212] The length, width, and height of a single voxel are determined to be L. x ×L y ×L z ;
[0213] The resolution is set to N;
[0214] Divide the three-dimensional cube into N*N*N L-shaped segments. x ×L y ×L z 3D mesh;
[0215] Map the three-dimensional point cloud model to the three-dimensional mesh, assign the value 1 to the three-dimensional mesh containing at least one three-dimensional coordinate point in the three-dimensional point cloud model to obtain the first voxel unit, and assign the value 0 to the three-dimensional mesh that does not contain any three-dimensional coordinate points in the three-dimensional point cloud model to obtain the second voxel unit.
[0216] A three-dimensional voxel model is obtained based on the first voxel unit and the second voxel unit.
[0217] Optionally, the keypoint model includes an image encoding network, a pose prediction network, a voxel encoding network, a feature fusion network, and a decoding network. The step executed by the processor 901 of inputting the image to be detected and the three-dimensional voxel model into the trained keypoint detection model for keypoint detection to obtain the keypoint detection results of the UAV includes:
[0218] Image features are extracted from the image to be detected using the image coding network to obtain a multi-channel feature map of the image;
[0219] The pose prediction network is used to perform pose prediction processing on the multi-channel feature map of the image to obtain the affine transformation matrix.
[0220] The affine transformation matrix is used to perform an affine transformation on the three-dimensional voxel model to obtain the affine transformed three-dimensional voxel model.
[0221] Voxel features are extracted from the affine-transformed 3D voxel model using the voxel encoding network to obtain a voxel multi-channel feature map.
[0222] The image multi-channel feature map and the voxel multi-channel feature map are fused using a feature fusion network to obtain a fused multi-channel feature map.
[0223] The key point detection results of the UAV are obtained by decoding the fused multi-channel feature map through the decoding network.
[0224] Optionally, before inputting the image to be detected and the three-dimensional voxel model into the trained keypoint detection model for keypoint detection, the method executed by the processor 901 further includes:
[0225] The keypoint detection model is trained to obtain a trained target detection model, wherein the second loss function during the training process is as follows:
[0226]
[0227] Where θ represents all learnable parameters of the network model. The first part is the keypoint heatmap predicted by the network model, while the second part is the actual heatmap. The actual heatmap is generated based on manual annotation of keypoints from real-world footage and a Gaussian function. The specific formula is as follows:
[0228]
[0229] heatmap i Let (x, y) represent the heatmap corresponding to the i-th real key point, and (x, y) represent any coordinates in the heatmap. i ,y i Let be the coordinates of the i-th real key point in the corresponding heatmap, σ be the standard deviation of the Gaussian kernel, and e be the natural logarithm.
[0230] The electronic device provided in this embodiment of the invention can implement all the processes of the UAV key point detection method in the above method embodiments, and can achieve the same beneficial effects. To avoid repetition, it will not be described again here.
[0231] This invention also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the various processes of the UAV key point detection method provided in this invention and achieves the same technical effect. To avoid repetition, it will not be described again here.
[0232] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The computer-readable storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.
[0233] The above description discloses only preferred embodiments of the present invention and should not be construed as limiting the scope of the present invention. Therefore, equivalent variations made in accordance with the claims of the present invention are still within the scope of the present invention.
Claims
1. A method for detecting key points on a UAV, characterized in that, The key point detection method for UAVs includes: The system acquires an image of the UAV to be detected and a three-dimensional voxel model of the UAV, which is obtained by voxelizing the three-dimensional model of the UAV. The image to be detected and the three-dimensional voxel model are input into a trained key point detection model to perform key point detection, and the key point detection result of the UAV is obtained. The key point detection result includes the key points of the UAV. The steps for acquiring the image to be detected from the UAV include: Acquire raw images of the drone; The original image is input into a trained target detection model to detect drone targets and obtain drone target bounding boxes. Based on the drone target bounding box, the image region containing the drone target is extracted from the original image to obtain the drone detection image; The keypoint model includes an image encoding network, a pose prediction network, a voxel encoding network, a feature fusion network, and a decoding network. The step of inputting the image to be detected and the three-dimensional voxel model into the trained keypoint detection model to perform keypoint detection and obtain the keypoint detection results of the UAV includes: Image features are extracted from the image to be detected using the image coding network to obtain a multi-channel feature map of the image; The pose prediction network is used to perform pose prediction processing on the multi-channel feature map of the image to obtain the affine transformation matrix. The affine transformation matrix is used to perform an affine transformation on the three-dimensional voxel model to obtain the affine transformed three-dimensional voxel model. Voxel features are extracted from the affine-transformed 3D voxel model using the voxel encoding network to obtain a voxel multi-channel feature map. The image multi-channel feature map and the voxel multi-channel feature map are fused using a feature fusion network to obtain a fused multi-channel feature map. The key point detection results of the UAV are obtained by decoding the fused multi-channel feature map through the decoding network.
2. The UAV key point detection method as described in claim 1, characterized in that, Before inputting the original image into the trained target detection model to perform UAV target detection and obtain the UAV target bounding box, the method further includes: A convolutional neural network with a feature fusion mechanism is used as the target detection model. The input of the target detection model is a full-size image, and the output is the target box information (x,y,w,h) of the UAV, where (x,y) are the coordinates of the center of the target box in the image, and w and h are the width and height of the target box. The object detection model is trained to obtain a trained object detection model, wherein the first loss function in the training process is as follows: L(w) = bbox(w) + conf(w) Where L(w) is the loss function, w is the object detection network model parameter, f is the network model, and I is the input image; the loss function consists of two parts: localization loss bbox(w) and confidence loss conf(w); in the localization loss bbox(w), A represents the target bounding box predicted by the model, B represents the ground truth target bounding box, and A represents the bounding box containing the target. The minimum region bounding box of B; in the confidence loss conf(w), K*K is the size of the output feature map, and M is the number of prior boxes. and As an indicator variable, when the j-th prior bounding box of the i-th grid in the input feature map contains a UAV target, and on the contrary, and λ noobj Here is a hyperparameter representing the weight of negative samples. C represents the confidence level of the model prediction, and C represents the true confidence level.
3. The UAV key point detection method as described in claim 1, characterized in that, The step of obtaining the three-dimensional voxel model of the UAV includes: Obtain the three-dimensional geometric model of the UAV; The three-dimensional geometric model is subjected to point cloud analysis to obtain a three-dimensional point cloud model; The three-dimensional point cloud model is converted into a three-dimensional voxel model at a preset resolution.
4. The UAV key point detection method as described in claim 3, characterized in that, The step of converting the 3D point cloud model into a 3D voxel model at a preset resolution includes: determining the length, width and height of the individual voxels as L x x L y x L z ; The resolution is set to N; Divide the three-dimensional cube into N*N*N L-shaped segments. x ×L y ×L z 3D mesh; Map the three-dimensional point cloud model to the three-dimensional mesh, assign the value 1 to the three-dimensional mesh containing at least one three-dimensional coordinate point in the three-dimensional point cloud model to obtain the first voxel unit, and assign the value 0 to the three-dimensional mesh that does not contain any three-dimensional coordinate points in the three-dimensional point cloud model to obtain the second voxel unit. A three-dimensional voxel model is obtained based on the first voxel unit and the second voxel unit.
5. The UAV key point detection method as described in claim 1, characterized in that, Before inputting the image to be detected and the three-dimensional voxel model into the trained keypoint detection model for keypoint detection, the method further includes: The keypoint detection model is trained to obtain a trained target detection model, wherein the second loss function during the training process is as follows: Where θ represents all learnable parameters of the network model. The first part is the keypoint heatmap predicted by the network model, while the second part is the actual heatmap. The actual heatmap is generated based on manual annotation of keypoints from real-world footage and a Gaussian function. The specific formula is as follows: heatmap i Let (x, y) represent the heatmap corresponding to the i-th real key point, and (x, y) represent any coordinates in the heatmap. i ,y i Let be the coordinates of the i-th real key point in the corresponding heatmap, σ be the standard deviation of the Gaussian kernel, and e be the natural logarithm.
6. A key point detection device for unmanned aerial vehicles (UAVs), characterized in that, The UAV key point detection device adopts the UAV key point detection method as described in any one of claims 1 to 5; The key point detection device for the unmanned aerial vehicle includes: The acquisition module is used to acquire the image to be detected of the UAV and to acquire the three-dimensional voxel model of the UAV, wherein the three-dimensional voxel model is obtained by voxelization processing based on the three-dimensional model of the UAV. The processing module is used to input the image to be detected and the three-dimensional voxel model into a trained key point detection model to perform key point detection, and obtain the key point detection result of the UAV, wherein the key point detection result includes the key points of the UAV.
7. An electronic device, characterized in that, include: The device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of the UAV key point detection method as described in any one of claims 1 to 5.
8. A computer storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the UAV key point detection method as described in any one of claims 1 to 5.
Citation Information
Patent Citations
Three-dimensional target detection method based on multi-scale heterogenous feature adaptive fusion
CN114495089A
Face occlusion detection method and system, device, and storage medium
WO2022134337A1
