An unmanned aerial vehicle tracking method based on skeleton feature extraction and fitting

Abstract

This invention discloses a drone tracking method based on skeleton feature extraction and fitting, belonging to the field of image processing and target tracking. The method includes: acquiring and preprocessing images from a sensor; extracting and optimizing the single-pixel skeleton of the drone target; locating skeleton endpoints and intersections as key feature points; performing feature matching in subsequent frames to achieve tracking; when matching fails, performing brightness abrupt changes or occlusion detection; if an interference state is determined, using successfully matched feature points to perform geometric connection fitting, using the intersections as temporary tracking positions to maintain tracking continuity under interference conditions where target features are missing; if no interference state is present or fitting times out, the tracking process is re-initialized. This invention utilizes the skeleton structural features of the target for tracking, overcoming the shortcomings of traditional methods that rely on volatile appearance features, and can achieve stable, accurate, and continuous tracking of key drone positions under interference such as strong light, smoke, and partial occlusion.

Description

Technical Field

[0001] This invention belongs to the field of image processing and target tracking technology, specifically relating to a method for target feature extraction and stable tracking of unmanned aerial vehicles (UAVs), which is particularly suitable for precise tracking of specific parts of UAVs under laser interference conditions. Background Technology

[0002] With the widespread application of drones, achieving stable and accurate tracking of drones has become increasingly important. In various practical application scenarios, tracking systems often face challenges from complex environments. For example, when a drone is subjected to interference such as strong light, partial obstruction, or smoke, its image characteristics will change drastically.

[0003] Most existing photoelectric tracking methods rely on matching and tracking based on the target's overall appearance, grayscale, or texture features. When encountering such interference, these methods can lead to distortion or interference with the target's optical signal, resulting in tracking point drift or target loss. The root cause is that these features are susceptible to environmental changes and lack utilization of the target's stable structural information. Existing technologies primarily focus on improving tracking robustness after interference occurs by optimizing matching algorithms, but they do not fundamentally address how to lock onto the target's key structural points. This makes it difficult to achieve continuous and reliable tracking of critical parts of the UAV under strong interference conditions, limiting the overall performance and reliability of the tracking system. Summary of the Invention

[0004] The purpose of this invention is to solve the problem of tracking drift and loss caused by existing UAV tracking methods relying on target appearance features under interference conditions. This invention proposes a tracking method based on skeleton feature extraction and fitting. By solving and tracking the target's internal structural skeleton, and fitting the method using the geometric relationship of feature points under interference conditions, stable and continuous tracking of key positions of the UAV can be achieved.

[0005] To achieve the above objectives, the technical solution provided by this invention is:

[0006] A drone tracking method based on skeleton feature extraction and fitting is provided, including the following steps:

[0007] Step 1, Image Acquisition and Preprocessing: Acquire video images containing the UAV target from the sensor, perform binarization and noise removal on the acquired images, and fill in the holes in the target area to form a connected binarized target image;

[0008] Step 2, skeleton extraction and optimization: Extract single-pixel skeletons from the binarized target image obtained in Step 1, and remove redundant branches on the skeleton;

[0009] Step 3, Feature point localization: Locate the skeleton endpoints and the intersection points of the skeleton on the skeleton optimized in Step 2, which are used as the key feature points of the target, and number the key feature points according to the preset rules.

[0010] Step 4, Tracking and Matching: Acquire subsequent video image frames from the sensor and perform feature point matching within the neighborhood of key feature points;

[0011] Step 5, Interference Judgment and Branch Processing: Determine whether the feature point matching in Step 4 was successful;

[0012] If the number of successfully matched feature points exceeds the set threshold, the tracking is deemed reliable, the tracking position is updated, and the process returns to step 4 to continue processing subsequent frames.

[0013] If the match fails, then a sudden change in brightness or occlusion detection is performed on the image region where the feature point is located.

[0014] Step 6, Fitting and Tracking and Reinitialization: Branching is performed based on the detection results from Step 5;

[0015] If a sudden change in brightness or occlusion is detected, it is determined to be in an interference state. Using the currently matched key feature points, geometric connection fitting is performed according to their numbers. The calculated intersection point is used as a temporary tracking position, and the process returns to step 4.

[0016] If no brightness change or occlusion is detected, or if the temporary tracking position fails to restore the match after a preset time, then step 1 is restarted to obtain a new tracking position.

[0017] Furthermore, the binarization of the image in step 1 specifically includes: determining the image segmentation threshold using a dual thresholding method, wherein the dual thresholds are adaptively determined by maximizing the inter-class variance of the image.

[0018] Furthermore, filling the holes in the target area in step 1 specifically includes:

[0019] Traverse the binarized image and mark points with a pixel value of 0 as unprocessed.

[0020] By iteratively traversing, black pixels are distinguished as background points or suspected hole points based on the relationship between the pixel and its neighborhood and the image boundary.

[0021] Modify the pixel value of the suspected hole point that was finally determined to be a real hole to 255 to complete the hole filling.

[0022] Furthermore, in step 2, the extraction of single-pixel skeletons is achieved using an iterative thinning algorithm, and the removal of redundant branches on the skeleton is achieved using a skeleton pruning algorithm.

[0023] Furthermore, the skeleton pruning algorithm includes:

[0024] Scan the skeleton pixels in column coordinate order and count the number of skeleton pixels in each column;

[0025] If the current column contains only one skeleton pixel, then that pixel is retained;

[0026] If there are multiple skeleton pixels in the current column, calculate the distance between each pixel and the pixels retained in the previous column, and retain the closest pixel as the main skeleton pixel;

[0027] Repeat the above steps until all columns have been processed, thereby eliminating redundant branches on the skeleton.

[0028] Furthermore, step 3, locating the skeleton endpoints and intersections, includes:

[0029] Calculate the connectivity of the 8-neighbor pixels of the skeleton point;

[0030] Points with a connectivity of 1 are identified as endpoints, and points with a connectivity of 3 or more are identified as intersections.

[0031] Furthermore, in step 4, a normalized cross-correlation algorithm is used to perform feature matching in the neighborhood of the key feature points.

[0032] Furthermore, step 5 involves detecting brightness abrupt changes or occlusions in the image region where the feature points are located, including:

[0033] Calculate the average brightness of the current frame within the neighborhood of the feature point and the previous frame. The difference in average brightness across frames;

[0034] If the difference is greater than the preset threshold, it is determined that a sudden change in brightness or occlusion has been detected.

[0035] Furthermore, the geometric connection fitting in step 6 specifically involves:

[0036] Number The endpoints and numbers are The endpoints are paired and connected to generate A line connecting the feature points, among which The total number of endpoints that have been successfully matched is calculated, and the intersection of these lines is used as a temporary tracking location.

[0037] The advantages of this invention are:

[0038] The UAV tracking method based on skeleton feature extraction and fitting provided by this invention overcomes the inherent defects of traditional methods that rely on volatile appearance features by extracting and tracking the skeleton structural features of the UAV target. It significantly improves the tracking robustness and stability under complex interference such as strong light, smoke, and partial occlusion. By accurately locating the skeleton endpoints and intersections, it achieves precise tracking of key positions such as the UAV body and arms, meeting the requirements of high-precision applications. At the same time, by introducing brightness change and occlusion detection mechanisms, it can intelligently identify the cause of tracking failure. After confirming the interference state, it uses the unoccluded endpoints to fit the occluded key points through geometric connections, achieving continuous and uninterrupted tracking. This forms a complete closed loop from normal tracking, interference response to self-recovery, and ultimately achieves high-precision, high-reliability, and stable tracking of key positions of the UAV in complex environments. Attached Figure Description

[0039] The above and / or other features and advantages of the present invention will become more readily understood from the following description with reference to the accompanying drawings, in which:

[0040] Figure 1 This is a flowchart of the target feature extraction and tracking solution framework provided in the embodiments of the present invention;

[0041] Figure 2 This is a schematic diagram of the original image of the input UAV target provided in an embodiment of the present invention;

[0042] Figure 3 This is a flowchart of the binarized image hole filling process provided in an embodiment of the present invention;

[0043] Figure 4 These are comparison images of binarized images before and after hole filling provided in this embodiment of the invention, where (a) is the binarized image before hole filling and (b) is the binarized image after hole filling.

[0044] Figure 5 This is a flowchart of the skeleton pruning process provided in an embodiment of the present invention;

[0045] Figure 6 These are comparison diagrams of the skeleton before and after pruning provided in the embodiments of the present invention, wherein (a) is the skeleton before pruning and (b) is the skeleton after pruning.

[0046] Figure 7 This is a schematic diagram of skeleton feature location (endpoints and intersections) extraction provided in an embodiment of the present invention;

[0047] Figure 8 This is a schematic diagram of target tracking position fitting provided in an embodiment of the present invention. Detailed Implementation

[0048] The present invention will now be described in detail with reference to the accompanying drawings and exemplary embodiments thereof. It should be noted that the following detailed description of the present invention is for illustrative purposes only and is not intended to limit the scope of the invention.

[0049] Reference Figure 1 The UAV tracking method based on skeleton feature extraction and fitting, as an exemplary embodiment of the present invention, includes the following steps:

[0050] Step S1: Image Acquisition and Preprocessing

[0051] First, a video sequence containing the drone target is acquired from sensors such as photoelectric sensors, and one frame is read as the input image, such as... Figure 2 As shown.

[0052] Next, the input image is binarized. This embodiment employs a dual-threshold method, where the dual thresholds are adaptively determined by maximizing the inter-class variance of the image. Specifically, pixels in the input image are divided into three classes: dark background (A), bright background (B), and target (T). and These are the lower and higher thresholds in a dual-threshold approach, i.e., the segmentation thresholds. The grayscale values ​​are then... The pixels within are classified as B, and the grayscale value is in The pixels within are classified as B, and the grayscale value is in The pixels within are classified into T. Then the proportions of the three types of pixels to the total number of pixels are respectively... , , The average grayscale values ​​of these three types of pixels are respectively , , Then the global mean The expression for the variance between classes is: ,make To obtain the maximum value and To determine the optimal threshold, grayscale values ​​greater than [a certain threshold] are [selected]. The pixel value is set to 255, which is less than Set the pixel value to 0.

[0053] Then, noise removal is performed on the binarized image. The binarized image is initialized as follows:

[0054]

[0055] In the formula, For binarized image values, These are the initialized image values. The binarized image pixels... The neighborhood is divided into four neighborhoods: top, bottom, left, and right. and diagonal neighborhood Randomly select a pixel from the image Define contribution function and , respectively representing the neighboring pixels that make The effects of darkening and brightening are represented as follows:

[0056]

[0057]

[0058] In the formula, , , These are the influence coefficients of the initial grayscale value, the four neighboring pixels, and the diagonal neighboring pixels on the result, respectively, which are set to 10, 5, and 2. Then, according to... and The size relationship will The grayscale value is updated to , is represented as:

[0059]

[0060] The binarized image is denoised using the method described above to eliminate isolated points in the background and burrs on the target edges.

[0061] Finally, refer to Figure 3 Fill the "holes" in the target of the binarized image using the following process to avoid the skeleton from generating unnecessary branches:

[0062] 1) For the denoised binarized image, all pixels with a value of 0 are marked as "no-flag," i.e., unprocessed. For each "no-flag" pixel, first check if there are any pixels with a value of 0 in its upper, lower, left, and right neighboring pixels. If so, this pixel is directly set as a "background pixel," and the process moves to the next black pixel for further evaluation. If not, check if there are any white pixels starting from this pixel and extending to the upper, lower, left, and right boundaries of the image. If there are white pixels in all four directions of this pixel, this pixel may be located in a "hole" in the target image, and this pixel is marked as a "suspected hole pixel." After the first traversal, all black pixels are marked as either "background pixels" or "suspected hole pixels."

[0063] 2) Perform a second traversal. For each "suspected hole point", check if there are "background points" in its upper, lower, left, and right neighboring points. If they exist, set the "suspected hole point" as a "background point". If they do not exist, keep the "suspected hole point". If there is a change from "suspected hole point" to "background point" during the traversal, perform another traversal until no "suspected hole point" becomes a "background point" during the traversal. Then the second traversal ends. After the second traversal, all the remaining "suspected hole points" are actually the real hole points.

[0064] 3) Set the pixel value of all "suspected hole points" to 255. At this point, the holes inside the UAV target are eliminated, forming a complete and connected binarized target image. The binarized images before and after hole filling are shown below. Figure 4 .

[0065] Step S2, skeleton extraction and optimization, includes the following sub-steps:

[0066] Step S21: For the binarized target image obtained in step S1, a thinning algorithm is used to extract single-pixel skeletons. The specific method is as follows:

[0067] If we define white pixels as 1 and black pixels as 0, then the input image is a matrix composed of 1s and 0s. The 8-neighborhood pixels of the center pixel P1 are defined as follows:

[0068]

[0069] In this table, P1 is the center pixel currently being detected and judged; P2, P3, P4, P5, P6, P7, P8, and P9 are the eight neighboring pixels surrounding the center pixel P1. They are arranged in a clockwise order, but can also be arranged counterclockwise, starting from the top.

[0070] Define A(P1) as the number of transitions from black to white in the sequence {P2, P3, P4, P5, P6, P7, P8, P9, P2}, and B(P1) as the number of white pixels in the neighborhood of P1. The pixels other than the skeleton are gradually removed through the following two iterative processes:

[0071] 1) Test each pixel in the image, and set the pixels that simultaneously meet the following conditions to black, until no pixels are set, then end the current iteration:

[0072] The pixel is white and has 8 neighboring pixels;

[0073] 2 <B(P1)<6;

[0074] A(P1) = 1;

[0075] At least one of P2, P4, and P6 is black;

[0076] At least one of P4, P6, and P8 is black;

[0077] 2) Test each pixel of the image adjusted in the previous iteration again, setting pixels that simultaneously meet the following conditions to black, until no pixels are set, ending the current iteration:

[0078] The pixel is white and has 8 neighboring pixels;

[0079] 2 <B(P1)<6;

[0080] A(P1) = 1;

[0081] At least one of P2, P4, and P8 is black;

[0082] At least one of P2, P6, and P8 is black.

[0083] Step S22: The skeleton lines obtained after the above operations have branches, which will affect subsequent calculations. Therefore, it is necessary to remove redundant branches on the skeleton (i.e., skeleton pruning). The process is as follows: Figure 5 As shown. The specific method is:

[0084] 1) Record skeleton curve The set of coordinates of all pixels, sorted in ascending order of column coordinates. Scan and count the elements in the middle. Total number of columns in the pixel distribution ;

[0085] 2) Statistics Number of pixels in each column , ;

[0086] 3) Judgment Is it 1? If =1, then the first The unique pixel coordinates of the column Store in a new collection ;like If >1, then calculate the first... Columns of all pixels and Previously stored pixel coordinates The distance, and the pixel coordinates corresponding to the minimum distance. deposit ;

[0087] 4) Judgment and Are they equal? Then the output is the set of pixel coordinates of the processed skeleton curve. ;otherwise, Increment by 1, and repeat steps 2) to 4).

[0088] After removing the burrs, all branches on the skeleton except the main line are removed. A comparison of the skeleton before and after pruning is shown below. Figure 6 As shown.

[0089] Step S3: Feature Point Localization

[0090] On the skeleton optimized in step S2, the skeleton endpoints and their intersections are located as key feature points of the target. The type of skeleton points is determined by connectivity numbers, and the feature points are numbered and labeled sequentially. The specific steps are as follows:

[0091] 1) Take the 8-neighborhood of the point on the skeleton line that needs to be judged and number it. Here, the numbering is taken from the top left corner counterclockwise.

[0092] 2) The method for calculating the connected number of this point is as follows:

[0093]

[0094] In the formula, The value is assigned to the corresponding pixel number (0 or 1). Since the pixel values ​​within the same area are the same, the difference is 0, which does not increase the value of the calculation. If the pixel values ​​cross areas, the difference is 1, and such changes will occur at the beginning and end of an area, so we divide by 0.5.

[0095] 3) Determine the attributes of the pixel:

[0096] Connectivity is 1: endpoints;

[0097] Connectivity is 2: interior points;

[0098] Connection number 3: branch point;

[0099] The connectivity is 4: intersections;

[0100] The detected endpoints and intersections are numbered as follows: and ,like Figure 7 As shown.

[0101] Step S4, Tracking and Matching

[0102] The latest frame of video image is acquired from the sensor. Then, feature matching is performed within the neighborhood of key feature points. This embodiment uses the Normalized Cross-Correlation (NCC) algorithm for matching.

[0103] Step S5: Interference Judgment and Branch Processing

[0104] First, determine whether the feature point matching in step S4 was successful. In this embodiment, the threshold is set to require that the number of matched points exceeds half of the total number of intersection points.

[0105] Branch 1 (Successful Match): If the number of successfully matched feature points exceeds a set threshold, the tracking is deemed reliable, and the tracking position is updated. Subsequently, the process returns to step S4 to continue processing subsequent frames.

[0106] Branch 2 (Match Failure): If the match fails, then abrupt brightness changes or occlusion detection is performed on the image region where the feature point is located.

[0107] The specific method for detecting brightness abrupt changes or occlusion is as follows: Calculate the difference between the average brightness of the current frame in the neighborhood of the feature point and the average brightness of the previous few frames, such as the previous 10 frames. If the variance of the difference is greater than a preset threshold (e.g., 1000), it is determined that an anomaly has been detected.

[0108] Step S6: Fitting and Tracking, and Reinitialization

[0109] This step executes the corresponding branch based on the detection results of step S5.

[0110] Branch A (Interference State): If a sudden change in brightness or occlusion is detected, the system is determined to be in an interference state. In this case, the currently successfully matched key feature points, i.e., endpoints, are utilized. Geometric connection fitting is performed based on their numbers. The specific rule is: [The text abruptly ends here, likely due to an incomplete sentence or missing information.] The endpoints and numbers are Pair the endpoints and connect them to generate Draw a straight line from two feature points and obtain the intersection of the lines connecting the two feature points. The calculated intersection point is used as a temporary tracking position to continue tracking. See the schematic diagram of this fitting process. Figure 8 The process then returns to step S4, attempting to restore a normal match in subsequent frames.

[0111] Branch B (Reinitialization): If no sudden brightness change or occlusion is detected (indicating that the target may have undergone non-interference large deformation or displacement), or if the temporary tracking position continues for more than the preset time, such as 3 seconds, and the tracking reference point still cannot be restored. If a match is found, step S1 is re-executed to obtain a new tracking position.

[0112] Therefore, as described above, this invention overcomes the inherent defects of traditional methods that rely on volatile appearance features by extracting and tracking the skeletal structural features of the UAV target, significantly improving the tracking robustness and stability under complex interference such as strong light, smoke, and partial occlusion. By accurately locating the skeleton endpoints and intersections, it achieves precise tracking of key positions such as the UAV body and arms, meeting the requirements of high-precision applications. At the same time, by introducing brightness change and occlusion detection mechanisms, it can intelligently identify the cause of tracking failure. After confirming the interference state, it uses the unoccluded endpoints to fit the occluded key points through geometric connections, achieving continuous and uninterrupted tracking, thus forming a complete closed loop from normal tracking, interference response to self-recovery, and finally achieving high-precision, high-reliability, and stable tracking of key positions of the UAV in complex environments.

[0113] Finally, it should be noted that the features mentioned and / or shown in the above description of exemplary embodiments of the present invention can be combined in the same or similar manner with one or more other embodiments, combined with or substituted for corresponding features in other embodiments. These combined or substituted technical solutions should also be considered to be included within the scope of protection of the present invention.

Claims

1. A UAV tracking method based on skeleton feature extraction and fitting, characterized in that, Includes the following steps: Step 1, Image Acquisition and Preprocessing: Acquire video images containing the UAV target from the sensor, perform binarization and noise removal on the acquired images, and fill in the holes in the target area to form a connected binarized target image; Step 2, skeleton extraction and optimization: Extract single-pixel skeletons from the binarized target image obtained in Step 1, and remove redundant branches on the skeleton; Step 3, Feature point localization: Locate the skeleton endpoints and the intersection points of the skeleton on the skeleton optimized in Step 2, which are used as the key feature points of the target, and number the key feature points according to preset rules; Step 4, Tracking and Matching: Acquire subsequent video image frames from the sensor and perform feature point matching within the neighborhood of the key feature points; Step 5, Interference Judgment and Branch Processing: Determine whether the feature point matching in Step 4 was successful; If the number of successfully matched feature points exceeds the set threshold, the tracking is deemed reliable, the tracking position is updated, and the process returns to step 4 to continue processing subsequent frames. If the match fails, then a sudden change in brightness or occlusion detection is performed on the image region where the feature point is located. Step 6, Fitting and Tracking and Reinitialization: Branching is performed based on the detection results from Step 5; If a sudden change in brightness or occlusion is detected, it is determined to be in an interference state. Using the currently matched key feature points, geometric connection fitting is performed according to their numbers. The calculated intersection point is used as a temporary tracking position, and the process returns to step 4. If no brightness change or occlusion is detected, or if the temporary tracking position fails to restore the match after a preset time, then step 1 is restarted to obtain a new tracking position.

2. The UAV tracking method based on skeleton feature extraction and fitting according to claim 1, characterized in that, The binarization of the image in step 1 specifically includes: determining the image segmentation threshold using a dual thresholding method, wherein the dual thresholds are adaptively determined by maximizing the inter-class variance of the image.

3. The UAV tracking method based on skeleton feature extraction and fitting according to claim 1 or 2, characterized in that, The filling of holes in the target area in step 1 specifically includes: Traverse the binarized image and mark points with a pixel value of 0 as unprocessed. By iteratively traversing, black pixels are distinguished as background points or suspected hole points based on the relationship between the pixel and its neighborhood and the image boundary. Modify the pixel value of the suspected hole point that was finally determined to be a real hole to 255 to complete the hole filling.

4. The UAV tracking method based on skeleton feature extraction and fitting according to claim 1 or 2, characterized in that, In step 2, the extraction of single-pixel skeletons is achieved using an iterative thinning algorithm, and the removal of redundant branches on the skeleton is achieved using a skeleton pruning algorithm.

5. The UAV tracking method based on skeleton feature extraction and fitting according to claim 4, characterized in that, The skeleton pruning algorithm includes: Scan the skeleton pixels in column coordinate order and count the number of skeleton pixels in each column; If the current column contains only one skeleton pixel, then that pixel is retained; If there are multiple skeleton pixels in the current column, calculate the distance between each pixel and the pixels retained in the previous column, and retain the closest pixel as the main skeleton pixel; Repeat the above steps until all columns have been processed, thereby eliminating redundant branches on the skeleton.

6. The UAV tracking method based on skeleton feature extraction and fitting according to claim 1 or 2, characterized in that, Step 3, locating the skeleton endpoints and intersections, includes: Calculate the connectivity of the 8-neighbor pixels of the skeleton point; Points with a connectivity of 1 are identified as endpoints, and points with a connectivity of 3 or more are identified as intersections.

7. The UAV tracking method based on skeleton feature extraction and fitting according to claim 1 or 2, characterized in that, In step 4, a normalized cross-correlation algorithm is used to perform feature matching in the neighborhood of key feature points.

8. The UAV tracking method based on skeleton feature extraction and fitting according to claim 1 or 2, characterized in that, Step 5, which involves detecting brightness abrupt changes or occlusions in the image region where the feature points are located, includes: Calculate the average brightness of the current frame within the neighborhood of the feature point and the previous frame. The difference in average brightness across frames; If the difference is greater than a preset threshold, it is determined that a sudden change in brightness or occlusion has been detected.

9. The UAV tracking method based on skeleton feature extraction and fitting according to claim 1 or 2, characterized in that, The geometric connection fitting in step 6 specifically involves: Number The endpoints and numbers are The endpoints are paired and connected to generate A line connecting the feature points, among which The total number of endpoints that have been successfully matched is calculated, and the intersection of these lines is used as a temporary tracking location.

Images