Moving object gray level image ORB feature point detection method
Through the dual-field detection method, improved ORB feature point detection, inter-frame differential method combined with FLANN feature point matching, the difficulty of tracking of air targets under high-speed motion and occlusion is solved, and stable and accurate tracking of air targets is achieved. and repositioning improve detection speed and accuracy.
Patent Information
- Application Number
- CN202411107159.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-11-29
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2043-11-29
AI Technical Summary
The existing aerial target detection and tracking algorithms are difficult to achieve real-time and accurate target tracking when facing high-speed motion, occlusion, background interference and light changes, and focal length control leads to the loss of targets, lacking effective stable tracking methods.
Using the dual-field detection method, the target is initially captured and algorithmically analyzed by a large field of view camera, providing the reference position for the small field of view camera for accurate tracking. The improved ORB feature point detection and inter-frame differential method combined with FLANN feature point matching is achieved. Stable tracking and relocation of air targets.
The speed and accuracy of target monitoring are improved, target loss and error detection are avoided, tracking capabilities for high-speed moving targets are enhanced, detection time is reduced, and detection speed and accuracy are improved.
Smart Images

Figure CN119068020A_ABST
Abstract
Description
[0001] Related Applications
[0002] This application is a divisional application of the Chinese invention patent with application date of November 29, 2023 and application number 202311605753.0. Technical Field
[0003] The invention belongs to the technical field of aerial target detection and tracking, and in particular relates to a grayscale image ORB feature point detection method for a moving target. Background Art
[0004] When tracking aerial targets such as airplanes, in order to evaluate the current operating attitude of the target aircraft, the camera needs to increase its focal length to make attitude judgments. However, aerial targets such as airplanes have the characteristics of fast movement speed and large scale changes. Blindly increasing the focal length during the tracking process may cause problems such as aircraft occlusion and background interference. When the target has high maneuvers, the camera with an increased focal length is likely to cause the target to be lost from the field of view, and it is impossible to complete the normal detection and tracking task.
[0005] In addition, the commonly used target detection and tracking algorithms have huge computational complexity and poor robustness, making it difficult to meet the real-time and accuracy requirements for aerial target tracking. When these algorithms face challenges such as occlusion and disappearance of aerial targets, they cannot effectively combine global features, resulting in weak ability to distinguish between background and target, and produce erroneous tracking results.
[0006] At present, target detection and tracking have made great progress both at home and abroad. However, whether it is the improvement of various basic methods or the generation of new algorithms, they will face some common problems, namely, the problem of sunlight intensity and light changes caused by weather, the interference of moving targets in the scene, the disappearance of moving targets after the target is blocked, the shadow problem of the detected target, and the real-time problem of the moving target detection and tracking algorithm. Different methods also have different targets for these problems. Due to the complexity of target detection and tracking, it is unrealistic to establish a universal target detection and tracking method suitable for all situations. It is necessary to establish a system that meets the actual conditions according to the specific situation. For aerial targets, due to the high maneuverability of aerial targets, it is easy to cause the target to be lost from the field of view. In addition, aerial targets are long-distance targets. In the case of single-field tracking, a focus control algorithm is needed to control the single-field camera, which increases the probability of target loss. At the same time, unpredictable environmental factors such as lighting, obstructions, poor air quality, and low visibility in the background of the sky will also cause great interference to the detection and tracking of aerial targets. Therefore, there is currently no good method or system that can continuously and stably track aerial targets. Summary of the invention
[0007] In order to solve the above problems existing in the prior art, the present invention provides a method for detecting ORB feature points of a grayscale image of a moving target. The technical problem to be solved by the present invention is achieved by the following technical solutions:
[0008] A method for detecting ORB feature points of a grayscale image of a moving target, comprising:
[0009] Acquire the grayscale image of the moving target;
[0010] Constructing a multi-scale image expression pyramid for the grayscale image to obtain multiple scale images;
[0011] Calculate the FAST corner points of each layer for each scale image;
[0012] The FAST corner points are optimized by using a non-maximum suppression algorithm;
[0013] The grayscale centroid method is used to calculate the centroid of the FAST corner points after optimization;
[0014] Obtaining the direction of a direction vector by connecting the FAST corner point and the centroid;
[0015] The BEBLID descriptor is used to calculate the BEBLID descriptor of the optimized FAST corner points to obtain the improved ORB feature points at a certain scale;
[0016] The improved ORB feature points at all scales are fused to obtain the final ORB feature points of the grayscale image.
[0017] In one embodiment of the present invention, the step of calculating the FAST corner points of each layer for each scale image includes:
[0018] Take a pixel of each scale image as the center point P, and its gray value is I p ;
[0019] Set the gray threshold F for determining FAST corner points p , is chosen to be 20 percent of the pixels;
[0020] Compare the grayscale value of the center point P with the grayscale value of all the pixels around it with a distance of 3. If there are 12 consecutive pixels with grayscale values less than (I P -F P ) or greater than (I P +F P ), then the center point P is the FAST corner point;
[0021] The above three steps are iterated to calculate the FAST corner points in the entire image.
[0022] In one embodiment of the present invention, the method of calculating the centroid of the FAST corner points after optimization by using the grayscale centroid method includes:
[0023] The grayscale centroid method is used to select a rectangular area with the selected FAST corner point as the center, and the image moment of the rectangular area is calculated according to the following formula;
[0024]
[0025] Where: I(x,y) is the pixel value at pixel (x,y);
[0026] Use the following formula to calculate the center of mass O of the rectangular area;
[0027]
[0028] In one embodiment of the present invention, the obtaining the direction of the direction vector by connecting the FAST corner point and the centroid includes:
[0029] Connect the FAST corner point and the centroid to get the direction vector, and the direction θ of the direction vector is obtained by the following formula;
[0030]
[0031] In one embodiment of the present invention, the step of using the BEBLID descriptor to calculate the BEBLID descriptor from the optimized FAST corner points to obtain the improved ORB feature points at a certain scale includes:
[0032] Using the BEBLID descriptor, a square neighborhood x with a side length of 32 pixels is constructed with the selected FAST corner point as the center; sampling is performed in the neighborhood to obtain a window pair of size l, represented by R(p1,l) and R(p2,l) respectively, and the difference f(x) of the average gray value of the pixels is calculated by the following formula;
[0033]
[0034] Where: I(q) and I(r) represent the grayscale values of pixels in R(p1,l) and R(p2,l) respectively;
[0035] Substitute f(x) into the following formula, normalize it to get the binary descriptor, and get the improved ORB feature points at a certain scale;
[0036]
[0037] Where: T(x) is the gray value threshold at the neighborhood x.
[0038] Compared with the prior art, the present invention has the following beneficial effects:
[0039] The ORB feature point detection method for grayscale images of moving targets proposed in an embodiment of the present invention is more uniform and complete than the feature points detected by the original ORB algorithm. In addition to detecting feature points of various scales, it also detects feature points missed by the original ORB algorithm, thereby improving the speed and accuracy of target monitoring. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] Figure 1 It is a flowchart of a dual-field-of-view aerial target detection and tracking method provided by an embodiment of the present invention;
[0041] Figure 2 is a comparison diagram of detection effects between the improved ORB feature point detection method according to an embodiment of the present invention and the original ORB feature point detection method;
[0042] Figure 3 is a comparison diagram of the detection effect of the improved inter-frame difference method of the embodiment of the present invention and the original inter-frame difference method on the moving target;
[0043] Figure 4 It is a process diagram of controlling and compensating the PTZ by the PID control algorithm provided by the embodiment of the present invention;
[0044] Figure 5 It is the outdoor aircraft detection and tracking results of large and small fields of view in the third frame on a cloudy day;
[0045] Figure 6 This is the outdoor aircraft detection and tracking result of large and small fields of view in frame 15 on a cloudy day;
[0046] Figure 7 This is the outdoor aircraft detection and tracking result of large and small fields of view at frame 62 on a cloudy day;
[0047] Figure 8 This is the outdoor aircraft detection and tracking result of large and small fields of view at frame 298 on a cloudy day;
[0048] Fig. 9 This is the outdoor aircraft detection and tracking result of large and small fields of view at frame 600 on a cloudy day;
[0049] Fig.10 This is the outdoor aircraft detection and tracking result of large and small fields of view at frame 700 on a cloudy day;
[0050] Fig.11 This is the outdoor aircraft detection and tracking result of large and small fields of view at frame 970 on a cloudy day;
[0051] Fig.12 This is the outdoor aircraft detection and tracking result of large and small fields of view in the 9th frame on a sunny day;
[0052] Fig.13 This is the outdoor aircraft detection and tracking result of large and small fields of view at frame 156 on a sunny day;
[0053] Fig.14 This is the outdoor aircraft detection and tracking result of large and small fields of view at frame 264 on a sunny day;
[0054] Fig.15 This is the outdoor aircraft detection and tracking result of large and small fields of view at frame 284 on a sunny day;
[0055] Fig.16 This is the outdoor aircraft detection and tracking result of large and small fields of view at frame 491 on a sunny day;
[0056] Fig.17 This is the outdoor aircraft detection and tracking result of large and small fields of view at frame 506 on a cloudy day;
[0057] Fig.18 These are the outdoor aircraft detection and tracking results for large and small fields of view at frame 563 on a cloudy day. DETAILED DESCRIPTION
[0058] The present invention is further described in detail below with reference to specific embodiments, but the embodiments of the present invention are not limited thereto.
[0059] The main problem of being unable to continuously detect and track a single aerial moving target is that the aerial target will undergo a series of complex changes during the tracking process. These complex changes will cause uncertainty in the position of the aerial target, making it difficult for the detection and tracking algorithm to accurately detect and track the target. The complex changes of aerial targets include scale deformation of aerial targets, sudden disappearance of aerial targets, background interference, and rapid movement of aerial targets. In addition to these existing problems, human needs will also increase the difficulty of detecting and tracking aerial targets. For example, when evaluating the flight status of the target aircraft, it is necessary to monitor and track the details of the aircraft's attitude. However, due to the long distance of aerial targets such as aircraft, a camera zoom strategy is often required in a single field of view to observe the detailed attitude of the aircraft in detail. When the camera zooms, the camera's field of view will be reduced from a large field of view to a small field of view. When the aircraft moves at high speed or is blocked, the small field of view will cause the target to be easily detached.
[0060] Based on this, the present invention proposes a dual-field-of-view aerial target detection and tracking method. The inventive concept of the present invention is: first, real-time video is acquired through large and small field-of-view cameras, and then a specific large field-of-view target detection algorithm is used to preliminarily capture the aerial moving target in the motion scene, and the corresponding algorithm analysis is performed. Since the large field of view has a large visual range, the target is not easy to leave the scene, so the target can be initially positioned and provided to the small field of view. After the small field of view obtains the reference position given by the large field of view, the posture characteristics of the aerial target can be clearly captured, and the small field-of-view aerial target detection and recognition algorithm can be used to detect whether the target needs to be tracked. If this target is a target that needs to be tracked, the precise positioning and tracking of the aerial target can be achieved through the subsequent target tracking algorithm. When the aerial target is blocked, the background is disturbed, or it moves quickly away from the small field-of-view camera, the large field-of-view camera can provide the small field-of-view camera with the repositioning position of the target, so that the small field of view can recapture the target without special processing, and complete the re-tracking task of the aerial target.
[0061] See also Figure 1 , Figure 1 1 is a flow chart of a dual-view aerial target detection and tracking method provided by an embodiment of the present invention. The dual-view aerial target detection and tracking method provided by an embodiment of the present invention comprises the following steps:
[0062] S1. Use the large field of view moving target detection algorithm to detect the moving target in the air.
[0063] The large field of view moving target detection algorithm in this step specifically includes the following steps:
[0064] S101, obtaining a grayscale image of the moving target;
[0065] S102, using an improved ORB feature point detection method to detect the ORB feature points of the grayscale image, wherein the improved ORB feature point detection method is to obtain multiple scale images by constructing a multi-scale image expression pyramid for the grayscale image, then calculating the FAST corner points of each layer for each scale image, and using a non-maximum suppression algorithm to optimize the FAST corner points, and then using a grayscale centroid method to calculate the centroid of the optimized FAST corner points, and connecting the FAST corner points with the centroid to obtain the direction of the direction vector; then using a BEBLID descriptor to calculate a BEBLID descriptor for the optimized FAST corner points to obtain improved ORB feature points at a certain scale; and fusing the improved ORB feature points at all scales to obtain the final improved ORB feature points of the grayscale image;
[0066] The specific calculation steps based on the improved ORB feature point detection method in this step are as follows:
[0067] S1021, randomly construct an image multi-scale expression pyramid with a grayscale image, and obtain multiple scale images N i ;
[0068] S1022, sequentially perform N i Perform the following steps S1023-S1032;
[0069] S1023, taking a certain pixel of the image as the center point P, its gray value is I p ;
[0070] S1024, set the gray threshold F for determining FAST corner points p , is chosen to be 20 percent of the pixels;
[0071] S1025, compare the gray value of the center point P with the gray values of all the pixels around it with a distance of 3. If there are 12 consecutive pixels with gray values less than (I P -F P ) or greater than (I P +F P ), then the center point P is the FAST corner point;
[0072] S1026, iterate the first three steps to calculate the FAST corner points in the entire image;
[0073] S1027, using a non-maximum suppression algorithm to optimize the FAST corner points, and retaining the FAST corner points with the largest response values;
[0074] S1028, using the grayscale centroid method to select a rectangular area with the FAST corner point obtained in S1027 as the center, and calculating the image moment of the rectangular area according to the following formula;
[0075]
[0076] Where: I(x,y) is the pixel value at pixel (x,y);
[0077] S1029, using the following formula to calculate the centroid O of the rectangular area;
[0078]
[0079] S1030, connecting the FAST corner point and the centroid to obtain a direction vector, and obtaining the direction θ of the direction vector by the following formula;
[0080]
[0081] S1031, using the BEBLID descriptor, construct a square neighborhood x with a side length of 32 pixels with the FAST corner point in step S1027 as the center; sampling is performed in the neighborhood to obtain a window pair of size l, represented by R(p1,l) and R(p2,l) respectively, and the difference f(x) of the average gray value of the pixels is calculated by the following formula;
[0082]
[0083] Where: I(q) and I(r) represent the grayscale values of pixels in R(p1,l) and R(p2,l) respectively;
[0084] S1032, substituting f(x) into the following formula, normalizing to obtain a binary descriptor, and obtaining an improved ORB feature point at a certain scale;
[0085]
[0086] Where: T(x) is the gray value threshold at the neighborhood x;
[0087] S1033, fusing the improved ORB feature points at all scales of the image to obtain the final improved ORB feature points of the original grayscale image.
[0088] The improved ORB feature point detection method proposed in the embodiment of the present invention and the original ORB feature point detection method are used to perform feature point detection on the same image. The effect is as follows: Figure 2 As shown, Figure 2 (a) is the detection effect diagram of the original ORB feature point detection method. Figure 2 (b) is a detection effect diagram of the improved ORB feature point detection method; it can be seen that the feature points detected by the improved ORB feature point detection method proposed in the embodiment of the present invention are more uniform and complete than those detected by the original ORB algorithm. In addition to detecting feature points of various scales, feature points missed by the original ORB algorithm are also detected.
[0089] S103, using a FLANN-based feature point matching strategy to perform feature point matching on the detected ORB feature points to determine corresponding point pairs;
[0090] For the improved ORB feature points obtained above, the BEBLID descriptor they have is a binary descriptor. For feature points with binary descriptors, the similarity of feature points is generally represented by the Hamming distance between two feature points, as shown in formula (6).
[0091]
[0092] Where: O r1 , O r2These are two improved ORB feature points. The smaller the value of H is, the more similar the feature points are.
[0093] In FLANN, similarity search can be performed for the improved ORB feature point pairs using FLANN's local sensitive hashing method based on Hamming distance. The principle of calculating the similarity probability of two feature points using the local sensitive hashing function based on Hamming distance is shown in formula (7).
[0094]
[0095] Where: d is the length of the binary descriptor of the feature point, H is the value of the Hamming distance, and c is a parameter and is greater than 1.
[0096] In formula (7), when H≤d, the probability that two feature points are similar is at least P1; when H≥cd, the probability that two feature points are similar is at most P2. In this way, when two feature points are exactly the same and the Hamming distance is 0, the probability that their features are equal after being mapped by the hash function is 1; when the binary descriptors of the two feature points are completely different and the Hamming distance is the length of all descriptors, the probability that their features are the same after being mapped is 0. Through this process, whether the feature points are similar can be distinguished according to probability. In the actual calculation process, FLANN feature point matching uses multiple hash functions for mapping to cascade to form a hash key. The descriptors in the database are indexed by the hash key, and a hash table is established to achieve fast matching. In order to improve the accuracy of matching, multiple hash tables are used to increase the probability of finding approximate nearest neighbors, distinguish similar feature points, and finally achieve the rapid generation of nearest neighbor samples of feature points.
[0097] After obtaining the nearest neighbor sample, since the descriptor of a feature point may have multiple matching nearest neighbor sample descriptors, it is impossible to determine which descriptor is the best. Therefore, the embodiment of the present invention selects to add the KNN algorithm after the FLANN similar feature point search to perform feature point matching screening. The specific method includes:
[0098] Select the two closest neighbor points in the FLANN similar feature point search sample. Among these two feature points, if the following formula is met, this pair of matching points is accepted;
[0099]
[0100] Where: d1 is the distance to the nearest feature point, d2 is the distance to the next nearest feature point, and Ratio is the threshold, which is selected as 0.7.
[0101] S104, after filtering out the point pairs on the moving target, perform background compensation calculation to obtain a compensation map;
[0102] This step may include:
[0103] S1041, obtaining positions of all matched feature point pairs in the grayscale image;
[0104] S1042, calculating the distance difference between the feature point and the edge of the grayscale image;
[0105] S1043, separating background point pairs from target point pairs, and filtering out target point pairs according to a threshold value;
[0106] S1044, eliminating errors from the matching points after filtering out the target point pair, to obtain selected point pairs;
[0107] S1045, calculating the global motion parameter matrix for the selected point pairs;
[0108] S1046: After obtaining the affine matrix of two adjacent frames of grayscale images, obtain a compensation map of the previous frame relative to the next frame according to the affine transformation.
[0109] Furthermore, the above method specifically comprises the following steps:
[0110] S10411, obtaining positions of all feature point pairs after matching in the image;
[0111] S10422, respectively calculate the shortest distance difference dw between the feature point and the length and width edges of the image according to the following formula i and dh i ;
[0112]
[0113] Where: x i ,y i are the horizontal and vertical coordinates of the feature point, w and h are the length and width of the image, and i is the number of feature points;
[0114] S10433, according to the following formula, the background point pairs and the target point pairs are separated to obtain the background feature points after the target is filtered out;
[0115] dw i ≤T∪dh i ≤T (10);
[0116] Where: T is the threshold value, which is 100;
[0117] S10444, using an error elimination algorithm to eliminate errors from the matching point pairs after the target is filtered out;
[0118] S10455. Solve the following formula to calculate the matrix H by eliminating and filtering the three groups of matching point pairs.
[0119]
[0120] Where: x l ,y l is the coordinate value of the matching point in the previous frame image; x r ,y r is the coordinate value of the matching point in the next frame image;
[0121] S10466. After obtaining the affine matrix H of two adjacent frames of images, obtain a compensation image of the previous frame relative to the next frame according to the affine transformation; the affine transformation calculation formula is as follows:
[0122]
[0123] Where: x′, y′ is the coordinate value after image transformation, and x, y is the coordinate value before image transformation.
[0124] Through the above steps, an accurate compensation map of the current image relative to the next frame can be obtained.
[0125] S105, performing frame difference on two adjacent frames before and after the grayscale image and the compensation image thereof, respectively, to obtain two frame difference images, and performing AND operation on the two frame difference images to obtain a final result image, thereby realizing moving target detection;
[0126] Specifically, this step includes:
[0127] S1051, first, according to the following formula, the two frames of image I are respectively k+1 ,I k+2 and compensation diagram I′ k , I′ k+1 Apply the frame difference method to obtain two differential images and
[0128]
[0129] Where: θ is the frame difference threshold, which is 20;
[0130] S1052: According to the following formula, the two differential images are and Perform morphological corrosion and expansion processing respectively;
[0131]
[0132] Where: A, B are sets of structural elements, x, y are the coordinates of the B structure position;
[0133] S1053, according to the following formula, the two differential images and Perform binarization processing respectively;
[0134]
[0135] Where: A(x, y) is the element of each point in the image, src(x, y) is the gray value of each point in the image, T is the threshold, which is 40;
[0136] S1054, according to the following formula, and Perform AND operation to obtain the final improved inter-frame difference method binary image D k ;
[0137]
[0138] The improved inter-frame difference method proposed in the embodiment of the present invention is used for calculation, and the effect obtained is as follows: Figure 3 As shown, Figure 3 (a) is the detection effect of the original frame difference method on the moving target. Figure 3 (b) is the improved inter-frame difference method for detecting moving targets. It can be seen that this method perfectly avoids the "ghosting" problem in the difference results compared to the original inter-frame difference method, and solves the problem of false detection of fast moving targets.
[0139] The large-field-of-view moving target detection method proposed in the embodiment of the present invention quickly obtains reliable global feature points by improving ORB feature point detection and FLANN feature point matching on grayscale images, thereby solving the problems of slow feature point extraction and high mismatch rate of the original algorithm; in background compensation, an operation of filtering target feature points is proposed to make the obtained motion compensation image more refined than the original algorithm; the present invention also improves the inter-frame difference method in the original algorithm, thereby solving the problem that the original algorithm will produce "ghosts" when facing high-speed moving objects. According to comparative test analysis, the large-field-of-view moving target detection method proposed in the present invention realizes the detection of aerial moving targets under dynamic backgrounds, avoids misdetection and wrong detection caused by local background motion of the video, and greatly reduces the detection time. While maintaining the accuracy of aerial target detection unchanged and avoiding the "ghost" problem, the detection speed is also improved. The speed is increased by 65.31% on high-resolution videos and 13.12% on medium-resolution videos, and can be used for large-field-of-view target processing in dual-field-of-view aerial target detection and tracking systems.
[0140] S2. Using a small field of view target detection and recognition algorithm to identify the moving target and determine whether tracking is required.
[0141] In step S2, the small field of view target detection and recognition algorithm is the YoloV7-tiny target detection algorithm, and the YoloV7-tiny target detection algorithm is obtained by training the YoloV7-tiny model using a transfer learning method based on the task target of the small field of view.
[0142] In this step, the YoloV7-tiny model can be trained by the YoloV7-tiny model training strategy based on transfer learning, which can solve the problem of low training accuracy caused by insufficient data set data. Further, by using the K-Means clustering algorithm to cluster the aircraft images in the data set of the present invention to obtain the optimal anchor box width and height parameters, the YoloV7-tiny target detection algorithm can improve the detection ability of certain aircraft targets of unknown size and shape.
[0143] S3: If it is determined to be a target that needs to be tracked, a target tracking algorithm is used to continuously track the target.
[0144] After completing large-field-of-view target detection and small-field-of-view target recognition, in order to continuously track aerial moving targets and make control decisions, a dual-field-of-view target tracking algorithm is required to track moving targets.
[0145] The Kernel Correlation Filter Tracking Algorithm (KCF) is a discriminative tracking algorithm based on correlation filtering. The principle is to find the target in the neighboring area of the next frame by training a local area target detector in each frame of video. When it finds the target in the neighboring area of the next frame, it will also update the target detector with the new target result. On a theoretical level, the KCF algorithm transforms the target tracking problem by proposing a classification problem for the target and the background. And it uses the ridge regression equation to transform this low-dimensional nonlinear classification problem into a high-dimensional linear space to simplify the calculation. In addition, the KCF algorithm also uses the circulant matrix and accelerates the calculation speed through frequency domain calculation. At the same time, combined with the fast Fourier transform, its detection speed can reach hundreds of frames per second.
[0146] The above existing KCF algorithm is based on the assumption that the target can be tracked correctly in each frame after the algorithm selects the target in the first frame. However, in reality, the KCF algorithm may lose the tracking target due to the large-scale changes and rapid movement of the aircraft during flight; in addition, the KCF algorithm will also have the same tracking failure problem as large-scale changes in the target when the target moves rapidly or is blocked.
[0147] Based on this, an embodiment of the present invention proposes a new method for solving the target tracking failure of the KCF algorithm, that is, the target tracking algorithm that integrates KCF and YoloV7-tiny in this step: the YoloV7-tiny target detection algorithm is used to initialize the sampling frame obtained by the kernel correlation filtering algorithm, generate a Gaussian response graph, and perform real-time tracking response; and by setting a response threshold, and judging whether the response peak of the real-time tracking response is lower than the response threshold, if it is not lower than the response threshold, the target tracking result is output; if it is lower than the response threshold, the YoloV7-tiny target detection algorithm is used to reinitialize the sampling frame obtained by the kernel correlation filtering algorithm, so as to achieve re-tracking.
[0148] Specifically, the target tracking algorithm in step S3 includes:
[0149] S301, using Yolov7-tiny target detection algorithm to perform target detection on the input image, obtain the position calculation relationship of the target frame, and according to the position calculation relationship, obtain the center coordinate position of the target and the size of the target frame to determine the sampling target area of the tracked target;
[0150] S302, the sampling target area is enlarged by two times to form a HOG sampling area, and within the sampling area, the image information is represented as: x i =[x1,x2,x3,…,x n ] T ;
[0151] S303, introducing a permutation matrix I of cyclic shift:
[0152]
[0153] S304, through {I u x i |u=0,1,...,n-1} to get the circulant matrix X:
[0154]
[0155] S305, construct the ridge regression function equation using the following formula:
[0156] f(x i )=w T x i (19);
[0157] Where: w is the weight coefficient;
[0158] S306, obtain the distance between the next frame's true target position and the minimized sampling data by the following formula:
[0159]
[0160] Where: {y i} is the sample feature {x i} is the regression label value; λ is the regularization parameter to ensure the generalization ability of the model;
[0161] S307. By using the formulas in S305 and S306 and setting the partial derivative to 0, solve the following formula to obtain the selection range of the weight coefficient w:
[0162] w=(X H X+λI) -1 X H y (21);
[0163] Where: X H represents the complex conjugate transposed matrix;
[0164] S308, using the diagonal property of the circulant matrix X, obtain the frequency domain representation of w under Fourier transform
[0165]
[0166] 309. The kernel function matrix K is obtained through the Gaussian kernel function φ(x), so that the high-dimensional weight parameter α replaces the solution to w:
[0167] K=φ(x)·φ(x) H , α=(K+λI) -1 y (23);
[0168] Where: The kernel function matrix K is a circulant matrix;
[0169] S310, further simplify the high-dimensional weight parameter α to obtain
[0170]
[0171] Where: is the Fourier transform of the first row of the kernel function matrix K;
[0172] S311. Definition of K Z To represent the matrix of similarity between training samples and sampling samples, the training samples and sampling samples are processed to obtain the response value expressed in the frequency domain
[0173]
[0174] Where: is the kernel matrix K Z The first line of
[0175] S312, setting the response peak threshold η, if The target is lost, the calculation process is interrupted and re-enters process 1. Then enter S313;
[0176] S313, the frequency domain The Fourier transform is converted to the time domain f max , through the time domain f max The corresponding position finds the target position:
[0177]
[0178] S314, using bilinear interpolation and combining the previous detection parameters to update the following model parameters, so as to realize continuous looping of the entire algorithm process and complete the continuous tracking of the aerial target and the re-tracking after occlusion:
[0179]
[0180] Where: θ is the update coefficient; t is the tth frame of the video sequence; is the target model; is a high-dimensional weight parameter.
[0181] The target tracking algorithm based on the fusion of KCF and YoloV7-tiny proposed in the embodiment of the present invention avoids the inapplicability of the existing KCF algorithm and the problem of algorithm speed reduction caused by adjustment of the sampling detection frame through the principle of repositioning the sampling detection frame, and avoids the idea of using brute force full-scale operations.
[0182] The target tracking algorithm was evaluated and tested using a computer with an Intel-Core(TM) i7-7700 CPU, a main frequency of 2.5GHz, and a memory of 16GB. The performance of the algorithm was tested using a real outdoor scene aircraft video. During the experiment, the target tracking algorithm proposed in the present invention was compared with the traditional BOOSTING, CSRT, KCF, and TLD target tracking algorithms. The results are shown in Table 1. The comprehensive tracking performance of each algorithm was fully compared through a total of 350 frames of aircraft video, and the results were displayed in specific frames.
[0183] Table 1 Tracking performance indicators of various target tracking algorithms
[0184] Comparing the target tracking algorithm based on the fusion of KCF and YoloV7-tiny proposed in the present invention with the traditional BOOSTING, CSRT, KCF and TLD target tracking algorithms, it can be seen that the target tracking algorithm proposed in the present invention is better in all aspects and has the ability of re-tracking, and can continuously track the target.
[0185] S4. During the continuous tracking process, if the target is accelerated away, blocked, or interfered by the background, return to steps S1 to S3 to re-detect the moving target and continue tracking.
[0186] When the target is accelerated, occluded, or interfered by background, the process returns to step S1 to reposition and capture the target through the large field of view motion target detection algorithm, and continues to track the target through steps S2 and S3. By continuously looping the entire algorithm process, the continuous tracking of aerial targets and re-tracking after occlusion can be completed, ensuring that the tracking algorithm based on the fusion of KCF and YoloV7-tiny will not fail to track aerial targets with large-scale changes, fast movement, and occlusion.
[0187] In the above-mentioned embodiment of the present invention, in order to realize continuous tracking of aerial targets and make control judgments, it is also necessary to perform real-time tracking and control of the gimbal according to a control algorithm.
[0188] The principle of the present invention for tracking and controlling the pan-tilt is as follows: according to the PID (proportion integration differentiation) formula, the PID control algorithm is used to control the pitch, rotation direction and speed of the pan-tilt, so as to ensure that when an aerial target deviates from the center of the field of view, its deviation can be compensated by rotating the pan-tilt, so that the large and small field of view cameras on the pan-tilt and the aerial targets such as airplanes in the actual scene can maintain a relatively static state, and the offset distance of the aerial target position is small or even zero.
[0189] The control algorithm includes:
[0190] S3101, obtain the offset distance Q and offset d of the aerial target position in the current field of view x and d y ;
[0191] S3102. Discretize the PID formula, adopt position PID control, and obtain the response output calculation formula:
[0192]
[0193] Where: u(k) is the response output, e(k) is the offset, e(k-1) is the previous offset, ∑ i=0e(i) is the accumulated offset, K p , K i , K d is the proportionality coefficient;
[0194] S3103, using two PID controllers to output the control corresponding to the pan / tilt and rotation respectively;
[0195] S3104, for the pan / tilt control, according to the target position offset d y The size of the proportional coefficient K of the proportional control method in the PID algorithm is selected p , when d y >200, K p =0.15; when d y When ≤200, K p =0.1; then select the integral proportional coefficient as K i =0, select the differential proportional coefficient as K d =0.12, let e(k)=d y , substitute into the response output formula to obtain the corresponding control direction and speed compensation value u1(k);
[0196] S3105, for the pan / tilt rotation control, according to the target position offset d x The size of the proportional coefficient K of the proportional control method in the PID algorithm is selected p , when d x >200, K p =0.15; when d x When ≤200, K p =0.1; then select the integral proportional coefficient as K i =0, select the differential proportional coefficient as K d =0.12, let e(k)=d x , and substitute it into the response output formula to obtain the corresponding control direction and speed compensation value u2(k); so far, the entire PID control algorithm process is continuously cycled to control the gimbal to complete the continuous tracking of the aerial target.
[0197] like Figure 4As shown in the figure, it is a process diagram of the PID control algorithm controlling and compensating the gimbal. When the target aircraft is 248.8 pixels away from the center of the field of view, and the vertical and horizontal coordinate offsets are 239 pixels and -69 pixels respectively, the two PID controllers control the gimbal to rotate right at a speed of 6.9 degrees per second and rotate downward at a speed of 35.9 degrees per second, so that the center of the field of view quickly approaches the aircraft. When the target aircraft is 9.2 pixels away from the center of the field of view, and the vertical and horizontal coordinate offsets are 9 pixels and 2 pixels respectively, the two PID controllers control the gimbal to rotate left at a speed of 0.2 degrees per second and rotate downward at a speed of 0.9 degrees per second, so that the center of the field of view slowly approaches the aircraft.
[0198] From the above content, we can see that as the target shifts, the PID control algorithm effectively feeds back the direction and speed compensation values of the gimbal pitch control and rotation control, and sends instructions for control through these compensation values, ultimately aligning the camera field of view with the target for continuous tracking.
[0199] The following aircraft detection and tracking results in a real outdoor scene are used to prove that the dual-field-of-view aerial target detection and tracking method proposed in the present invention can effectively detect and track aerial targets.
[0200] The present invention adopts a notebook computer with a CPU of Intel-Core(TM) i7-7700, a main frequency of 2.5GHz, and a memory of 16GB and hardware equipment built according to the dual-field-of-view aerial target detection and tracking method to carry out a field experiment of the dual-field-of-view aerial target detection and tracking method.
[0201] During the entire outdoor field experiment, the proposed dual-field-of-view aerial target detection and tracking method was used to detect and track actual aircraft in the air, and interference factors such as the aircraft leaving the small field of view and blocking the target were artificially added to verify the results of the dual-field-of-view aerial target detection and tracking method in specific video frames for detecting and tracking aircraft in actual scenes on cloudy and sunny days.
[0202] from Figures 5 to 11 From the cloudy outdoor aircraft detection and tracking results, we can see that the dual-field aerial target detection and tracking method can effectively detect and track outdoor aircraft on cloudy days.
[0203] See also Figure 5 and Figure 6 ,in, Figure 5 (a) is the large field of view aircraft detection and tracking result of the third frame image. Figure 5 (b) is the small field of view aircraft detection and tracking result of the third frame image; Figure 6 (a) is the large field of view aircraft detection and tracking result of the 15th frame image. Figure 6(b) is the small field of view aircraft detection and tracking result of the 15th frame image. It can be seen that in the cloudy day 3rd frame image ( Figure 5 ) and the cloudy 15th frame image ( Figure 6 ), due to the small field of view, when capturing an aerial moving target in the large field of view ( Figure 5 (a) Figure 6 (a)), the small field of view cannot capture the tracking target in time ( Figure 5 (b)). The dual-field-of-view aerial target detection and tracking method uses coarse positioning in the large field of view and PID algorithm to control the gimbal, so that the target gradually appears in the small field of view. When the target appears in the small field of view ( Figure 6 (b)), the dual-field-of-view aerial target detection and tracking method determines whether the target is a target that needs to be tracked by detecting and identifying the target.
[0204] See also Figure 7 and Figure 8 ,in, Figure 7 (a) is the large field of view aircraft detection and tracking result of the 62nd frame image. Figure 7 (b) is the small field of view aircraft detection and tracking result of the 62nd frame image; Figure 8 (a) is the large field of view aircraft detection and tracking result of the 298th frame image. Figure 8 (b) is the small field of view aircraft detection and tracking result of the 298th frame image. Figure 7 ) and the 298th frame image ( Figure 8 ), the dual-field-of-view aerial target detection and tracking method controls the gimbal through the tracking algorithm and the PID algorithm to make the tracked aircraft target located in the center of the dual-field-of-view.
[0205] See also Fig. 9 , Fig.10 and Fig.11 ,in, Fig. 9 (a) is the large field of view aircraft detection and tracking result of the 600th frame image. Fig. 9 (b) is the small field of view aircraft detection and tracking result of the 600th frame image; Fig.10 (a) is the large field of view aircraft detection and tracking result of the 770th frame image. Fig.10 (b) is the small field of view aircraft detection and tracking result of the 770th frame image; Fig.11 (a) is the large field of view aircraft detection and tracking result of the 970th frame image. Fig.11 (b) is the small field of view aircraft detection and tracking result of the 970th frame image. Fig. 9 )、770th frame image( Fig.10 ) and the 970th frame image ( Fig.11), which are the three postures of the aircraft target approaching gradually, passing overhead and moving away gradually. It can be seen that the dual-field-of-view aerial target detection and tracking method can accurately track and detect it. And through the small field-of-view image of the dual-field-of-view aerial target detection and tracking method, the details of the aircraft target can be clearly identified, such as whether the landing gear is lowered, whether the flight attitude is stable, etc.
[0206] from Figures 12 to 18 It can be seen from the sunny outdoor aircraft detection and tracking results that the dual-field-of-view aerial target detection and tracking method also effectively completes the detection and tracking tasks. In addition, when there is human interference and short-term target occlusion, the method is still not affected.
[0207] See also Fig.12 and Fig.13 ,in, Fig.12 (a) is the large field of view aircraft detection and tracking result of the 9th frame image. Fig.12 (b) is the small field of view aircraft detection and tracking result of the 9th frame image; Fig.13 (a) is the large field of view aircraft detection and tracking result of the 156th frame image. Fig.13 (b) is the small field of view aircraft detection and tracking result of frame 156. The 9th and 156th frames on a sunny day are the process of capturing the aerial moving target with the large field of view and the small field of view at the beginning, and the process of controlling the gimbal through the dual-field of view aerial target detection and tracking method so that the tracked aircraft target is located in the center of the dual field of view.
[0208] See also Fig.14 and Fig.15 ,in, Fig.14 (a) is the large field of view aircraft detection and tracking result of the 264th frame image. Fig.14 (b) is the small field of view aircraft detection and tracking result of the 264th frame image; Fig.15 (a) is the large field of view aircraft detection and tracking result of the 284th frame image. Fig.15 (b) is the small field of view aircraft detection and tracking result of frame 284. The 264th and 284th frames on a sunny day are the process in which the aircraft target is out of the field of view center due to human interference, but is re-tracked through the self-adjustment of the dual-field of view aerial target detection and tracking method.
[0209] See also Fig.16 , Fig.17 and Fig.18 ,in, Fig.16 (a) is the large field of view aircraft detection and tracking result of the 491st frame image. Fig.16 (b) is the small field of view aircraft detection and tracking result of the 491st frame image; Fig.17 (a) is the large field of view aircraft detection and tracking result of the 506th frame image. Fig.17(b) is the small field of view aircraft detection and tracking result of the 506th frame image; Fig.18 (a) is the large field of view aircraft detection and tracking result of the 563rd frame image. Fig.18 (b) is the small field of view aircraft detection and tracking result of the 563rd frame. The images of the 491st, 506th and 563rd frames on a sunny day show the three situations in which the aircraft target is blocked by leaves, gradually escapes from the blockage and moves away from the blockage. It can be seen that when the aircraft target is suddenly blocked by leaves and the small field of view target is lost, the large field of view makes it difficult for the target to leave the scene, and the large field of view of the dual-field of view aerial target detection and tracking method can correctly provide the small field of view with the target relocation position. When the aircraft target gradually escapes from the blockage, no special processing is required, and the small field of view successfully captures the target again. With the small field of view recaptures the target, the entire system enters a stable tracking state and completes the continuous tracking process of the aircraft target.
[0210] In summary, the dual-field-of-view aerial target detection and tracking method effectively completes the task of detecting and tracking outdoor aircraft in cloudy, sunny, and occluded conditions. The aircraft detection and tracking results of the dual-field-of-view aerial target detection and tracking method in real outdoor scenes prove the effectiveness and rationality of the method. It proves that the dual-field-of-view aerial target detection and tracking method proposed in the present invention can complete the continuous detection and tracking of aerial moving targets, and solves some problems faced by aerial targets during the tracking process.
[0211] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.
Claims
1. A grayscale image ORB feature point detection method for a moving target, characterized in that: include: Acquire the grayscale image of the moving target; Constructing a multi-scale image expression pyramid for the grayscale image to obtain multiple scale images; Calculate the FAST corner points of each layer for each scale image; The FAST corner points are optimized by using a non-maximum suppression algorithm; The grayscale centroid method is used to calculate the centroid of the FAST corner points after optimization; Obtaining the direction of a direction vector by connecting the FAST corner point and the centroid; The BEBLID descriptor is used to calculate the BEBLID descriptor of the optimized FAST corner points to obtain the improved ORB feature points at a certain scale; The improved ORB feature points at all scales are fused to obtain the final ORB feature points of the grayscale image.
2. The method for detecting ORB feature points of a grayscale image of a moving target according to claim 1, characterized in that: The calculation of the FAST corner points of each layer for each scale image includes: Take a pixel of each scale image as the center point P, and its gray value is I p ; Set the gray threshold F for determining FAST corner points p , is chosen to be 20 percent of the pixels; Compare the grayscale value of the center point P with the grayscale value of all the pixels around it with a distance of 3. If there are 12 consecutive pixels with grayscale values less than (I P -F P ) or greater than (I P +F P ), then the center point P is the FAST corner point; The above three steps are iterated to calculate the FAST corner points in the entire image.
3. The method for detecting ORB feature points of a grayscale image of a moving target according to claim 1, characterized in that: The method of calculating the centroid of the FAST corner points after optimization by using the grayscale centroid method includes: The grayscale centroid method is used to select a rectangular area with the selected FAST corner point as the center, and the image moment of the rectangular area is calculated according to the following formula; Where: I(x,y) is the pixel value at pixel (x,y); Use the following formula to calculate the center of mass O of the rectangular area; 4. The method for detecting ORB feature points of a grayscale image of a moving target according to claim 1, characterized in that: The direction of the direction vector obtained by connecting the FAST corner point and the centroid includes: Connect the FAST corner point and the centroid to get the direction vector, and the direction θ of the direction vector is obtained by the following formula; 5. The method for detecting ORB feature points of a grayscale image of a moving target according to claim 1, characterized in that: The method of using the BEBLID descriptor to calculate the BEBLID descriptor of the optimized FAST corner points to obtain the improved ORB feature points at a certain scale includes: Using the BEBLID descriptor, a square neighborhood x with a side length of 32 pixels is constructed with the selected FAST corner point as the center; sampling is performed in the neighborhood to obtain a window pair of size l, represented by R(p1,l) and R(p2,l) respectively, and the difference f(x) of the average gray value of the pixels is calculated by the following formula; Where: I(q) and I(r) represent the grayscale values of pixels in R(p1,l) and R(p2,l) respectively; Substitute f(x) into the following formula, normalize it to get the binary descriptor, and get the improved ORB feature points at a certain scale; Where: T(x) is the gray value threshold at the neighborhood x.
Citation Information
Patent Citations
Method for designing monocular vision odometer with light stream method and feature point matching method integrated
CN103325108A
Moving vehicle detection method based on background updating and particle swarm optimization algorithm
CN105069441A
Binary description method of triangular feature
CN107229935A
Feature matching method for binocular image splicing of mobile inspection robot
CN110246168A
Feature point extraction and matching method of improved ORB algorithm
CN113554036A
