An image tracking method based on feature point correction

Abstract

The application discloses an image tracking method based on feature point correction and belongs to the field of computer vision measurement and target tracking. The method first selects a region of interest on the surface of a tracking object, matches feature points in the region in a first frame image with continuous video frame images, then calculates an affine transformation matrix according to the affine transformation relationship between the matched feature points and identifies matching error feature points existing in the matching process, next corrects the identified matching error feature points according to the affine transformation matrix, and finally calculates the centroid of the feature points in the region of interest and takes the centroid as a tracking point to perform target tracking. The application identifies and corrects the feature points with matching errors and inconsistent positions, effectively improving the precision and stability of image tracking.

Description

Technical Field

[0001] This invention belongs to the field of computer vision measurement and target tracking, and particularly relates to an image tracking method based on feature point correction. Background Technology

[0002] In structural health monitoring of large swaying towers (such as wind turbine towers, transmission towers, communication towers, and high-rise building towers), obtaining their dynamic displacement response under wind loads, earthquakes, or mechanical vibrations is crucial for assessing structural safety. Traditional displacement measurement methods mainly employ contact sensors such as accelerometers, laser displacement gauges, or GPS. However, these methods suffer from drawbacks including complex equipment installation, high cost, susceptibility to electromagnetic interference, and difficulty in achieving multi-point synchronous measurements.

[0003] In recent years, image tracking technology based on computer vision has been increasingly applied to the vibration and displacement monitoring of tower structures due to its advantages such as non-contact operation, high precision, and full-field measurement. This type of method selects natural textures or artificial markers on the tower surface as tracking targets, and uses feature point matching algorithms (such as SIFT, SURF, ORB, etc.) to identify and locate the targets in continuous video images, thereby obtaining the vibration and displacement time history curve of the tower.

[0004] However, in real-world environments, image acquisition of oscillating towers faces numerous challenges. First, the complex background of the tower (e.g., sky, clouds, buildings behind) easily leads to mismatches between feature points and the background environment. Second, changes in lighting conditions over time (e.g., sunrise, sunset, shadow movement) alter the surface texture of the tower, further reducing the stability of feature point matching. Furthermore, the tower's own three-dimensional oscillation causes rotation, scaling, and non-uniform deformation of the tracked target, resulting in inconsistencies in the positions of matched feature points across different frames. Directly using feature points with matching errors or positional deviations for displacement calculations will introduce significant tracking errors and may even distort the vibration response curve.

[0005] While existing feature point matching algorithms can improve matching robustness to some extent, they have not yet effectively solved the problem of automatic identification and position correction of mismatched feature points, especially lacking a dedicated tracking method for large, flexible structures like swaying towers during dynamic deformation. Therefore, there is an urgent need for an image tracking method capable of identifying and correcting mismatched and inconsistent feature points to improve the accuracy of vibration displacement measurement of tower structures in complex environments. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention provides an image tracking method based on feature point correction, comprising the following steps: Feature point matching is performed between the selected region of interest in the first frame and the consecutive video frames. Calculate the affine transformation matrix based on the affine transformation relationship between the matched feature points; Identify matching error feature points that exist during the matching process; The identified mismatched feature points are corrected based on the affine transformation matrix. Based on the corrected feature points, the centroid of the feature points within the region of interest is calculated, and the centroid is used as the tracking point for target tracking.

[0007] Optionally, in the method of matching feature points between the selected region of interest in the first frame image and consecutive video frame images, the region of interest in the first frame image is used as the tracking region, and feature points are matched with each subsequent frame image.

[0008] Optionally, in the step of calculating the affine transformation matrix based on the affine transformation relationship between the matched feature points, the affine transformation matrix between the matched feature points in each pair of matched images is calculated based on the two-dimensional affine transformation theory.

[0009] Optionally, in the step of identifying matching error feature points in the matching process, the maximum inter-class variance theory is used in combination with the error minimization algorithm to identify matching error feature points and positionally inconsistent feature points.

[0010] Optionally, in the step of correcting the identified mismatched feature points according to the affine transformation matrix, the optimal affine transformation matrix is ​​obtained by optimizing the correctly matched feature points, and the optimal affine transformation matrix is ​​applied to the feature points in the reference target to correct the mismatched feature points and inconsistent position feature points identified in the matching target.

[0011] Optionally, in the step of calculating the centroid of feature points within the region of interest and using the centroid as the tracking point for target tracking, the centroid coordinates of feature points within the region of interest in each frame of the image are calculated according to statistical principles.

[0012] Optionally, in the step of calculating the centroid coordinates of feature points within the region of interest in each frame of an image based on statistical principles, the centroid of the feature points within the region of interest in the first frame of the image is used as the reference point, and the centroid of the corresponding feature points within the region of interest in consecutive frames of the image is used as the tracking target point of the current frame.

[0013] Optionally, in the step of matching feature points between the selected region of interest in the first frame image and the consecutive video frame images, the SIFT, SURF, or ORB feature point matching algorithm is used for matching.

[0014] Compared with the prior art, the present invention has the following advantages and technical effects: This invention provides an image tracking method based on feature point correction, applied to the structural vibration displacement monitoring of swaying towers. It effectively identifies and corrects erroneous matching points and inconsistencies in position that occur during feature point matching in video images, significantly reducing tracking errors caused by complex backgrounds, lighting variations, or the tower's own three-dimensional swaying. By calculating the centroid of the corrected feature point set as the tracking target, this invention maintains stable tracking performance even under non-ideal conditions such as large tower swaying, partial occlusion, or deformation, avoiding abnormal jumps in the displacement curve caused by deviations of individual erroneous feature points in traditional methods. Ultimately, this invention can output continuous, smooth, and high-precision tower vibration time-history displacement curves, providing a reliable data foundation for subsequent modal analysis and structural safety assessment. Attached Figure Description

[0015] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a schematic diagram of the method flow according to an embodiment of the present invention. Detailed Implementation

[0016] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0017] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0018] Example 1 like Figure 1 As shown, this embodiment provides an image tracking method based on feature point correction, including the following steps: Feature point matching is performed between the selected region of interest in the first frame and the consecutive video frames. Calculate the affine transformation matrix based on the affine transformation relationship between the matched feature points; Identify matching error feature points that exist during the matching process; The identified mismatched feature points are corrected based on the affine transformation matrix. Based on the corrected feature points, the centroid of the feature points within the region of interest is calculated, and the centroid is used as the tracking point for target tracking.

[0019] Specifically, the following steps are included: S1. First, select a Region of Interest (ROI) on the surface of the object being tracked as the tracking region. Then, perform feature point matching using the first frame image and subsequent video frames. Specifically, use the ROI in the first frame image as the tracking region and perform feature point matching with each subsequent frame image. Common matching algorithms can be used, such as SIFT, SURF, ORB, and other feature point matching algorithms.

[0020] S2. When using visual methods for target tracking, the relative positions between points on the structural surface remain unchanged, and the motion of such targets can be approximated as a two-dimensional rigid body transformation. Therefore, in consecutive video frames, the target matching feature points also exhibit a two-dimensional rigid body transformation relationship. Based on the two-dimensional rigid body transformation theory, the affine transformation matrix between matching feature points in every two matching images is calculated. Specifically, based on the two-dimensional affine transformation theory, the affine transformation matrix between matching feature points in every two matching images is calculated.

[0021] S2 specifically includes the following steps: i. Assume that the set of feature points within the circular reference target selected in the image of the structure being measured is... The set of points matching the target feature points is .in, n < m And point set p , q The number of interior points is no less than 3. According to the principle of rigid body transformation, the point set... p Points within can be accessed via rotation matrices. R Translation matrix T Convert to point set q Points within. To find the points between two sets of points. R and T This problem can be modeled as: (1) In the formula, wi >0 represents the weight of each point in the point set. Because the point set... q The number of points within the set is less than the number of points in the point set. p The number of interior points. Therefore, a point set with a smaller number of points is used. q The correspondence between interior points is established, so the number of points selected in equation (1) is... n .

[0022] ii. To compute the optimal rotation matrix for a pair of points. R The translation matrix must be eliminated first. T Due to the influence of this, the point sets are re-centered, with the center points of the two point sets being... , The expressions are as follows: (2) (3) iii. After eliminating the influence of the translation matrix, the corresponding point set moves towards the center, forming a new corresponding point set: (4) Differentiating equation (1) with T as the independent variable, we get: (5) iv. Divide both sides of equation (5) by Then you can get, (6) v. Substituting equations (2) and (3) into equation (6), we obtain, (7) (8) Substituting equation (8) into equation (1) yields an equivalent formula. (9) ⅵ. Expanding equation (9) in matrix form, we can obtain, (10) vii. Equation (9) can be extended according to equation (10): (11) ⅷ. Furthermore, because in equation (11), and Since it is independent of the rotation matrix, equation (11) can be transformed into: (12) In the matrix, (13) make , , Then equation (13) can be transformed into, (14) make Then equation (14) can be transformed into, (15) In the formula, V , U , R All are orthogonal matrices, therefore It is also an orthogonal matrix.

[0023] ⅸ. According to the properties of orthogonal matrices, in a matrix... S In, each column vector Sk All Therefore, it has the property of... Therefore, we have: (16) According to equation (12), R To obtain the maximum value, we have: (17) Calculate R Then, according to equation (8), we can calculate... T .

[0024] x. Since camera imaging is an affine transformation process, the algorithm in this section considers not only the object's rotation and translation, but also scaling and shearing transformations to improve matching accuracy. Scaling transformation enlarges or reduces each point in two-dimensional space according to a given scaling factor after rotation and translation. q ( x , y The scaling transformation of ) can be expressed as: (18) In the formula, ( x ′, y ′) are the coordinates of the point after scaling transformation. kx , ky They are respectively x shaft and y The scaling factor along the axis. Its matrix representation is as follows: (19) The shearing transform moves a point in one direction along another, thus changing the shape of an image without changing its size. q ( x , y The shear transform of ) can be expressed as: (20) In the formula, ( tx ′, ty ′) are the coordinates of the point after the shear transformation. kxy It is along x Axial shear y The scaling factor of the axis, kyx It is along y Axial shear x The scaling factor of the axis. Its matrix representation is as follows: (twenty one) An affine transformation matrix that integrates rotation, translation, scaling, and shearing matrices to determine the transformation relationships between feature points in a two-dimensional plane. TR It can be represented as: (twenty two) In the formula, tr 11 and tr 22 respectively represent x shaft and y Scaling factor along the axis, tr 12 and tr 21 represents the rotation and shearing factors, respectively. tr 13 and tr 23 is the translation factor, indicating that in x shaft and y The amount of movement along the axis. The last row of the matrix [0, 0, 1] is the normalized row in homogeneous coordinates, used to maintain the consistency of matrix multiplication.

[0025] S3. Identification of incorrectly matched feature points during image matching; S3 specifically includes the following steps: Specifically, by utilizing the Otsu's inter-class variance theory and combining it with the error minimization algorithm, we can identify mismatched feature points and feature points with inconsistent positions.

[0026] To correct mismatched feature points, it is necessary to determine in advance whether there are any mismatched feature points in the matching image. The specific principle is as follows: assuming the set of feature points in the reference target is... The set of feature points in the target is , n The number of points. Based on the transformation relationships between the feature points in two-dimensional matching, the point set can be calculated. pi and qi The transformation relationship between matching point sets, i.e., the transformation matrix. TR Set the points pi pass TR Convert to qi ′, under ideal conditions, point set qi 'and qi They should perfectly overlap, meaning every corresponding point should be in the same position. However, due to matching errors, the calculated transformation matrix... TR The transformation matrix is ​​not optimal, causing the corresponding points in the two point sets to not completely overlap, and the distance between the two sets of points will have some error. To determine if there is a matching error, the point sets are calculated. q and q The set of distances between all corresponding pairs of points in triangle ' is denoted as ∆. di ,in i =(1, 2, 3, …,n ), and set a threshold. th If ∆ di > th If the match is incorrect, then it is considered that there are incorrect matching pairs in this group of matching points.

[0027] To identify mismatching points, a mismatching feature point identification algorithm is proposed based on the principle of the maximum inter-class variance distance algorithm. The specific process of the algorithm is as follows: First, calculate the set of distances ∆ between all matching feature points between the reference target and the matching target. d i Then, a histogram of the distance sets is constructed, dividing the data into multiple intervals, assuming the interval values ​​range from [0, ... T ], η j The representative interval is j The probability of the data appearing, assuming there is a threshold. t The intervals can be divided into correctly matched intervals. a and matching error b Two parts of data, then a , b The probability of each part accounting for the whole. ω a , ω b They are respectively: (twenty three) (twenty four) In the formula, ω t The representative interval is t The probability of previous data points relative to the total data. (Array) a and b Partial average μ a and μ b They are respectively: (25) (26) In the formula, ​ t Is the threshold value t This is the average value of the data at that time. At this point, it represents the average value of the entire histogram data. ​ The expression is: (27) Therefore, the problem of identifying mismatches can be transformed into the problem of maximizing the inter-class variance, with the objective function being: (28) Finally, when the maximum inter-class variance reaches its maximum value,t The distance corresponding to the value is the optimal threshold. Based on this threshold, correctly matched and incorrectly matched feature point pairs can be distinguished, thus achieving the identification of incorrectly matched feature points.

[0028] S4. Correct the mismatched feature points identified in S3 based on the affine transformation matrix calculated in S2. Specifically, optimize the affine transformation matrix based on the correctly matched feature points, and apply the optimal affine transformation matrix to the feature points in the reference target to correct the mismatched feature points and inconsistent position feature points identified in the matching target.

[0029] S4 specifically includes the following steps: i. After identifying and deleting erroneous matching points in the target match, the feature points in the reference target remain unchanged and are still [missing information]. The set of points in the target that correctly match the feature points is then... Because there may be incorrectly matched feature points that are then deleted, there are... To correct matching errors, the deleted incorrect matching points can be recalculated using an affine transformation matrix. Finding the optimal affine transformation matrix can be formalized as an optimization problem. The core objective is to minimize the distance error between the corresponding point obtained by mapping a feature point in the reference target through this transformation matrix and the corresponding correctly matching feature point in the matching target. The transformation matrix that minimizes the distance error is the optimal matrix, expressed as follows: (29) in, ​ It is a transformation matrix. ​ ′( ​ ) indicates applying the transformation matrix to ​ , m To correctly match the number of feature point logs, for L The 2-norm is used to calculate the Euclidean distance between the transformed point and the matching target point, and is the transformation matrix that makes equation (29) hold. ​ ′ is the optimal transformation matrix.

[0030] ii. Because some feature points of the matching target are removed during the process of eliminating mismatched points, and to ensure that the feature points in the reference target and the feature points in the matching target of the continuous video images belong to the same set of feature points, thus maintaining the centroid position of the reference target unchanged, an optimal transformation matrix can be applied to the feature points of the reference target. ​ The algorithm calculates the deleted feature points and the matching feature points with inconsistent positions, thereby correcting the feature points in the matching target. Let... ​ The expression for ′ is: (30) iii. Matching the target k Feature points that were removed or feature points with inconsistent positions. According to the corresponding first in the reference target k Points Applying affine transformation matrix ​ Supplemental correction: (31) S5. Finally, based on statistical principles, the centroid of feature points in each ROI is calculated, and the centroid is used as the tracking point to track the target. Specifically, based on statistical principles, the centroid coordinates of feature points within the region of interest in each frame of the image are calculated.

[0031] S5 specifically includes the following steps: The vibration tracking algorithm based on feature point matching correction utilizes the centroids of feature points in the target region to achieve target tracking in continuous video images. The core idea of ​​this method is to use the target region in the first frame as a reference target and set the centroid coordinates of the feature points within the reference target as the starting reference point for tracking. Based on this, the centroid coordinates of the corresponding regions in consecutive video frames are calculated as the tracking target point for the current frame. The algorithm further calculates the Euclidean distance between the starting reference point and the tracking target point in each frame, and plots the time-history displacement curve of the target vibration with time as the x-axis and the target's movement distance in each frame as the y-axis. Furthermore, for time-history displacement curves exhibiting periodic characteristics, frequency domain transformation techniques can be used to calculate the spectrum of the tracked target vibration.

[0032] The main process of the vibration tracking algorithm based on feature point matching correction is as follows: First, feature points are matched between the reference target and the matching target image, and any possible mismatched feature points are corrected. Then, for each frame, the centroid coordinates of the feature points in the matching target corresponding to the reference target are calculated. Let the number of consecutive image frames be... m And the number of feature point pairs between the reference target and the matching target is n , ( ​ i , ​ i ) indicates the first reference target i The position coordinates of each feature point, ( ​ i , ​ i ) indicates the first target in the current frame. i The position coordinates of each feature point. It is worth noting that in this algorithm, the centroid of the feature points in the reference target of the first frame image is used as the reference point for the entire tracking process. Reference point The expression is as follows: (32) (33) The first in a continuous image l Tracking target points in matching target images The expression is: (34) (35) Finally, based on the obtained reference target and all matching target centroids, the distances between consecutive centroids and the reference target centroid are calculated. l The displacement expression between the centroid of the matching target and the centroid of the reference target is shown below: (36) in, l =1, 2, ..., m By continuously calculating the distance between the reference target and the matched target, the pixel-level temporal displacement of the continuous distance can be plotted on the time axis. Furthermore, by combining the camera calibration results, the actual temporal displacement of the continuous distance can be calculated.

[0033] Example 2 A method for tracking swaying tower images based on feature point correction includes the following steps: Feature point matching is performed between the selected region of interest on the surface of the swinging tower in the first frame image and the continuous video frame images; Calculate the affine transformation matrix based on the affine transformation relationship between the matched feature points; Identify matching error feature points that exist during the matching process; The identified mismatched feature points are corrected based on the affine transformation matrix. Based on the corrected feature points, the centroid of the feature points within the region of interest is calculated, and the centroid is used as the tracking point to track the swinging tower.

[0034] In practical image tracking of swaying towers, a region with clear texture and stable lighting is first selected on the tower surface as the region of interest (ROI). This region is typically located in the upper middle part of the tower to avoid bottom occlusion. The ROI from the first frame is used as the tracking region, and feature point matching is performed with each subsequent frame. Common feature point matching algorithms such as SIFT, SURF, or ORB can be used. Because swaying towers undergo significant swaying under wind loads or earthquakes, local feature points on the tower surface may rotate, scale, or temporarily move out of view between consecutive frames. Therefore, it is necessary to ensure that the selected ROI has sufficient feature point density to maintain the stability of the matching.

[0035] For oscillating tower structures, to achieve accurate tracking of their vibration displacement, the centroid coordinates of feature points within the region of interest (ROI) in each frame of the image are calculated based on statistical principles. Specifically, the centroid of the ROI feature points in the first frame is used as the reference point, and the centroids of the corresponding ROI feature points in subsequent frames are used as the tracking target points for the current frame. By continuously calculating the Euclidean distance between the centroid and the reference point in each frame, the displacement time-history curve of the tower on the time axis can be plotted. This centroid tracking method can effectively suppress tracking jumps caused by mismatches of individual feature points or local occlusion, making the output tower vibration response curve smoother and more reliable, and providing accurate displacement data for the structural health assessment of the tower.

[0036] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image tracking method based on feature point correction, characterized in that, Includes the following steps: Feature point matching is performed between the selected region of interest in the first frame and the consecutive video frames. Calculate the affine transformation matrix based on the affine transformation relationship between the matched feature points; Identify matching error feature points that exist during the matching process; The identified mismatched feature points are corrected based on the affine transformation matrix. Based on the corrected feature points, the centroid of the feature points within the region of interest is calculated, and the centroid is used as the tracking point for target tracking.

2. The method according to claim 1, characterized in that, In the method of matching feature points between the region of interest selected in the first frame image and the subsequent video frame images, the region of interest in the first frame image is used as the tracking region, and feature points are matched with each subsequent frame image.

3. The method according to claim 1, characterized in that, In the step of calculating the affine transformation matrix based on the affine transformation relationship between the matched feature points, the affine transformation matrix between the matched feature points in each pair of matched images is calculated based on the two-dimensional affine transformation theory.

4. The method according to claim 1, characterized in that, In the step of identifying matching error feature points in the matching process, the maximum inter-class variance theory is used in combination with the error minimization algorithm to identify matching error feature points and positionally inconsistent feature points.

5. The method according to claim 1, characterized in that, In the step of correcting the identified mismatched feature points based on the affine transformation matrix, the optimal affine transformation matrix is ​​obtained by optimizing the correctly matched feature points, and the optimal affine transformation matrix is ​​applied to the feature points in the reference target to correct the mismatched feature points and inconsistent position feature points identified in the matching target.

6. The method according to claim 1, characterized in that, In the step of calculating the centroid of feature points within the region of interest and using the centroid as the tracking point for target tracking, the centroid coordinates of feature points within the region of interest in each frame of the image are calculated according to statistical principles.

7. The method according to claim 6, characterized in that, In the step of calculating the centroid coordinates of feature points within the region of interest in each frame of an image based on statistical principles, the centroid of the feature points within the region of interest in the first frame of the image is used as the reference point, and the centroid of the corresponding feature points within the region of interest in consecutive frames of the image is used as the tracking target point of the current frame.

8. The method according to claim 1, characterized in that, In the step of matching feature points between the selected region of interest in the first frame image and the continuous video frame images, the SIFT, SURF or ORB feature point matching algorithm is used for matching.

Images