A multi-target tracking method based on ROI and homogenous matrix

Abstract

The application discloses a multi-target tracking method based on ROI and a homologous matrix, adopts a binocular camera to acquire video images of a to-be-tracked target in real time, matches each candidate circular marker in the first frame images of left and right videos, takes the successfully matched candidate circular marker as an effective tracking target, allocates an identity ID to the effective tracking target one by one, and acquires a corresponding region of interest ROI; according to each effective tracking target in the last frame image, the corresponding region of interest ROI of the current frame image is intercepted, candidate circular markers in each region of interest ROI are detected again, if there are candidate circular markers, the candidate circular markers are updated as current candidate circular markers, and the corresponding identity ID is inherited; if there are no candidate circular markers, current candidate circular markers are estimated, a displacement amount of the candidate circular markers corresponding to the last frame image before being blocked is calculated, short-time blocking or long-time blocking processing is triggered, and three-dimensional coordinates and a displacement amount of the inherited identity ID are calculated.

Description

Technical Field

[0001] This invention belongs to the technical field of image processing, specifically relating to a multi-target tracking method based on ROI and common source matrix. Background Technology

[0002] In high-frequency video measurement, vision-based 3D displacement measurement relies on high-precision center localization and continuous tracking of the target (usually a circular marker). This technology typically employs a binocular or multi-view stereo vision system. First, sub-pixel-level coordinates are acquired from a 2D image. Then, stereo matching and triangulation are used to convert these coordinates into 3D coordinates in physical space, thereby obtaining the target's 3D trajectory over a continuous time series. The system can accurately extract the dynamic displacement of the target, providing a crucial data foundation for structural vibration analysis, deformation monitoring, and safety assessment.

[0003] Common detection-tracking (MOT) algorithms include DeepSORT (based on depth features), ByteTrack (based on enhanced correlation capabilities), and OC-SORT (based on observation center momentum). These methods perform well in pedestrian or vehicle tracking (open-world scenarios), but they have the following problems when it comes to structural monitoring such as buildings (closed-loop world, strongly geometrically constrained scenarios): 1. Visual homogeneity leads to the failure of appearance feature (Re-ID) Phenomenon: In Structural Health Monitoring (SHM) scenarios, a large number of circular markers with completely identical appearance, texture, and size must be used.

[0004] Mechanistic flaws: DeepSORT and other algorithms rely on deep appearance feature extraction networks (CNNs). When processing these markers, the extracted feature vectors are highly clustered in the latent space, failing to distinguish individual differences. In this case, feature similarity calculation is mainly dominated by image noise rather than the texture differences of the target itself. Once local occlusion or nearby movement occurs, identity swapping is very likely to occur.

[0005] 2. Nonlinear structural dynamics cause Kalman filter divergence. Phenomenon: The structure undergoes nonlinear motion such as rotation and high-frequency reciprocating vibration. Mechanism flaw: Standard Kalman filtering is based on the "linear Gaussian Markov" assumption. However, the vibration of a structure usually includes high-frequency sinusoidal or rotational components. When the sampling rate is insufficient to cover the complete waveform or when abrupt motion occurs, the linear predictor will produce severe "overshoot" or "hysteresis," causing the real target to fall outside the predicted search gate, thus leading to tracking loss.

[0006] 3. "Amnesia" and fragmented trajectory caused by prolonged occlusion Phenomenon: In large displacement tests, marker points are often obscured by cables or components for more than a hundred frames.

[0007] Mechanism flaws: Existing algorithms such as ByteTrack or OC-SORT only set a limited "maximum retention time" (Max Age, usually 30-60 frames). Once the target disappears for longer than this threshold, the algorithm determines that the trajectory has terminated and clears the memory state. When the target reappears, due to the lack of memory and utilization of the overall geometric rigidity of the structure, the algorithm cannot associate it with the historical trajectory and can only initialize it as a new target. Summary of the Invention

[0008] This invention provides a multi-target tracking method and inverter parallel structure based on ROI and homogeneous matrix. It is designed for binocular video streams acquired synchronously from left and right, and aims to solve the technical problems of target loss and identity confusion caused by visual homogeneity of artificial markers, nonlinear motion and long-term occlusion in the structural health monitoring (SHM) scenario of existing visual measurement technology.

[0009] To achieve the above objectives, the present invention provides the following technical solution: A multi-target tracking method based on ROI and common origin matrix is ​​proposed. This method uses a binocular camera to acquire real-time video images of the target to be tracked and performs target tracking according to the following steps: Step 1: Match each candidate circular marker in the first frame of the left and right videos, take the successfully matched candidate circular markers as valid tracking targets, assign an identity ID to each one, and obtain the corresponding region of interest (ROI). Step 2: Based on each valid tracked target in the previous frame image, extract the Region of Interest (ROI) corresponding to the current frame image, and then re-detect the candidate circular markers in each ROI. If there are any, update them as the current candidate circular markers and inherit the corresponding identity ID; if not, estimate the current candidate circular markers and calculate the displacement between them and the candidate circular markers in the last frame image before occlusion. Step 3: If the displacement is not greater than a set threshold, short-term occlusion processing is performed; if it is greater than the set threshold, long-term occlusion processing is performed. Simultaneously, using the current frame images from the left and right videos, the three-dimensional coordinates and corresponding displacement of the valid tracking target that has inherited the identity ID are calculated.

[0010] Furthermore, in the matching process of step one, the center point coordinates of the candidate circular markers in the two first frame images are detected first, and the matching is performed accordingly. The candidate circular markers corresponding to the successfully matched point pairs are taken as effective tracking targets, and an identity ID is assigned to each of them. At the same time, the corresponding region of interest (ROI) is obtained. In step two, based on the center point coordinates of each effective tracked target in the previous frame image, the region of interest (ROI) corresponding to the current frame image is extracted, and the center point coordinates of candidate circular markers in each ROI are re-detected. If they exist, the center point coordinates of the current candidate circular markers are updated accordingly, and the corresponding identity ID is inherited. If they do not exist, the center point coordinates of the current candidate circular markers are estimated, and the displacement between the center point coordinates of the candidate circular markers in the last frame image before occlusion and the center point coordinates of the candidate circular markers is calculated.

[0011] Furthermore, for short-term occlusion processing, the geometric center coordinates of the inherited candidate circular markers in each frame of the image within the sliding time window are first calculated. and the coordinates of these geometric center points Perform mean-removal processing to obtain a zero-mean sequence. The sliding time window is set to include the previous frame image, including the current frame image. Frame image; Then, for the zero-mean sequence Perform a Fourier transform to obtain the frequency domain response. Search for the maximum value of the spectral modulus to determine the dominant frequency. and the corresponding peak amplitude If the peak amplitude If the value is not less than the activation threshold, then the vibration compensation increment is calculated using the Hilbert transform. The coordinates of the center point of the occluded candidate circular marker in the current frame image are calculated using the following formula. , in, This represents the confidence weighting coefficient. , All of these represent preset parameters.

[0012] Furthermore, the vibration compensation increment is calculated using the following formula. , in, This indicates the preset activation threshold. Represents the Hilbert transform. This indicates the time interval between two adjacent frames.

[0013] Furthermore, using the following formula, the coordinates of these geometric center points are... Perform mean-removal processing to obtain a zero-mean sequence. , in, Represents the coordinates of the geometric center point of all frames within the sliding time window. The arithmetic mean.

[0014] Furthermore, for long-term occlusion processing, based on the inherited identity ID, all matching point pairs between the last frame image before occlusion and the first frame image are obtained to construct an unclassified point set. ; Then use the RANSAC algorithm to classify the unclassified point set. Verification was performed to separate the interior point sets that conform to the same motion model. They belong to the same motion group, for the set of exterior points that does not conform to this motion model. The RANSAC algorithm is used again for verification, and this process is repeated until the remaining number of points is insufficient to solve the planar model, thus eliminating the unclassified point set. Divided into multiple independent motion groups; Then, based on the identity ID of the occluded candidate circular marker in the current frame image, the motion group to which it belongs is determined. The coordinates are recovered by using the geometric constraints of the remaining point pairs in the motion group or the image features of the occluded candidate circular marker itself, thereby obtaining the coordinates of the center point of the occluded candidate circular marker in the current frame image.

[0015] Furthermore, if the number of remaining point pairs in the motion group Then, based on the center point coordinates of the remaining point pairs' identity IDs in the current frame image and the last frame image before occlusion, the homography matrix H is calculated. Then, using the homography matrix H, based on the center point coordinates of the occluded candidate circular marker in the last frame image, the center point coordinates of the occluded candidate circular marker in the current frame image are deduced. If the number of remaining pairs of points in the motion group satisfies Then, based on the coordinates of the center points of the remaining point pairs' IDs in the current frame and the last frame before occlusion, the transformation matrix is ​​calculated using the affine transformation principle. Then use the transformation matrix Based on the center point coordinates of the occluded candidate circular marker in the last frame image, the center point coordinates of the occluded candidate circular marker in the current frame image are deduced. If the number of remaining point pairs in the motion group Then, the LightGlue deep feature matching network model is invoked to obtain feature matching point pairs between the current frame image and the last frame image before occlusion, and the homography matrix is ​​calculated based on the feature matching point pairs. Then, using the homography matrix Based on the center point coordinates of the occluded candidate circular marker in the last frame before it was occluded, the center point coordinates of the occluded candidate circular marker in the current frame are deduced.

[0016] Furthermore, a linear triangulation method based on SVD is used to calculate the three-dimensional coordinates of the valid tracking targets that have inherited the identity ID in the current frame images of the left and right videos, and to calculate the displacement of these valid tracking targets relative to the previous frame image.

[0017] Compared with the prior art, the beneficial effects of the present invention are: 1. Solving the identity interchange problem caused by visual homogeneity: To address the defect of depth appearance feature (Re-ID) failure caused by highly similar circular markers, this invention aims to transform the complex global multi-object tracking (MOT) problem into a spatially isolated parallel single-object tracking (SOT) task by using stereo vision geometric verification initialization and dynamic ROI (region of interest) strategy. By using geometric position constraints to replace appearance features, the identity ambiguity is fundamentally eliminated.

[0018] 2. Solving the prediction divergence problem caused by nonlinear high-frequency motion: In view of the problem that structural vibration and rotational motion violate the linear assumption of standard Kalman filtering, this invention aims to introduce a frequency domain vibration analysis and compensation mechanism to correct the predicted trajectory in real time under short-term occlusion, and prevent high-frequency vibration from causing the target to escape from the search area.

[0019] 3. Solving the problem of trajectory fragmentation under long-term occlusion: In view of the problem that existing algorithms lack long-term memory and cannot recover identity after long-term occlusion, this invention aims to use the spatial rigidity constraint of marker groups and use the homography matrix to map the historical topology to the current frame to achieve high-precision identity recapture after long-term occlusion.

[0020] 4. Achieve high-precision 3D displacement measurement based on binocular vision: The aim is to utilize the corresponding feature points successfully tracked in the left and right video images, combined with camera calibration parameters, to construct a stereo triangulation measurement equation, transforming the two-dimensional pixel coordinates into three-dimensional coordinates in physical space, thereby obtaining the accurate displacement curve of the measured structure in real three-dimensional space, meeting the sub-millimeter level engineering measurement requirements. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the overall process of the present invention; Figure 2 This is a schematic diagram of the short-term occlusion processing and long-term occlusion processing of the present invention; Figure 3 This is a schematic diagram showing the occlusion changes of each candidate circular marker over a period of time in an embodiment of the present invention. Detailed Implementation

[0022] To make the technical means, creative features, objectives and effects of the present invention easier to understand, the following embodiments, in conjunction with the accompanying drawings, specifically illustrate the inverter circuit structure of the present invention. It should be noted that the description of these embodiments is for the purpose of helping to understand the present invention, but does not constitute a limitation of the present invention.

[0023] like Figure 1 As shown, this invention provides a multi-target tracking method based on ROI and common origin matrix for real-time acquisition of video images of the target to be tracked using a binocular camera. First, candidate circular markers in the first frame of the left and right videos are matched, and successfully matched candidate circular markers are taken as valid tracking targets, each assigned an identity ID. Then, continuous real-time monitoring is performed, using the positions of candidate circular markers in the previous frame to generate dynamic ROI regions, which are used as processing objects. Once a valid tracking target is lost, short-term occlusion processing and long-term occlusion processing are adaptively selected based on the accumulated displacement to obtain the position coordinates of the occluded valid tracking target. Finally, a linear triangulation method based on SVD decomposition is used to solve for sub-millimeter-level three-dimensional coordinates and displacement. This invention achieves highly robust three-dimensional coordinate calculation and displacement measurement by establishing a precise correspondence between the left and right views.

[0024] Specifically as follows: 1. First frame processing The process involves matching each candidate circular marker in the first frame of both left and right video images. Successfully matched circular markers are identified as valid tracking targets, and each is assigned an identification ID. Simultaneously, the corresponding Region of Interest (ROI) is obtained. The details are as follows: 1) Global Object Detection: Obtain the first full-resolution image from the synchronously acquired left and right views, and input it into a pre-trained global detection model. This model can be a detection model built based on the YOLOv12-OBB framework, outputting the oriented bounding boxes (OBBs) and center point coordinates of all candidate circular markers. .

[0025] 2) Stereo geometric verification: In order to eliminate false detections due to background noise and establish a three-dimensional correspondence, this invention introduces the LightGlue feature matching algorithm to perform cross-view matching on the center points of candidate circular markers in the two first frame images. Only point pairs that are successfully matched in both the left and right views are retained, and their corresponding candidate circular markers are recorded as "valid tracking targets". Any isolated detection points that cannot be paired in the stereo view are regarded as false detection noise and are eliminated.

[0026] 3) ROI Initialization and ID Assignment: Based on the center point coordinates of the validated valid tracking targets, the system initializes a fixed-size Region of Interest (ROI) for each valid tracking target i, denoted as . At the same time, a unique and permanent identity ID is assigned to each pair of valid tracking targets i that have successfully matched in 3D.

[0027] 2. Real-time ROI processing: Based on the valid tracked targets in the previous frame, the Region of Interest (ROI) corresponding to the current frame is extracted. Candidate circular markers in each ROI are then re-detected. If found, this is updated as the current candidate circular marker and inherits its corresponding ID. If not found, the current candidate circular marker is estimated, and the displacement between it and the corresponding candidate circular marker in the previous frame is calculated, as follows: 1) Dynamic ROI generation: Based on the center point position of the effective tracking target i in the previous frame image, the corresponding region of interest (ROI) is extracted in the current frame image. The ROIs of all targets are concatenated into a tensor batch and processed uniformly to speed up the processing. Localized precise detection: Tensor batches are input into a dedicated detection model to accurately obtain the center point coordinates of each candidate circular marker.

[0028] 2) Identity Inheritance: Due to ROI regions The target was physically isolated in space. The system determined that: [anything within...] High-confidence targets detected within the range, i.e., candidate circular markers, directly inherit the identity ID bound to the candidate circular marker. This step uses spatial geometric constraints to replace the similarity comparison based on appearance features in traditional methods, fundamentally eliminating the ID switching problem caused by the consistency of marker texture (visual homogeneity).

[0029] 3) Dynamic ROI Updates and Closed Loop: If in A valid target was successfully detected and tracked. The newly detected center point coordinates were used. Update the center point coordinates within the current ROI region for use in the next frame capture; if in If no target is detected, the effective tracking target is immediately determined to be in a "lost state" and occlusion processing is triggered to complete the task of predicting the center point position of the effective tracking target.

[0030] 3. Obstruction treatment It aims to solve the problem of target loss. For example... Figure 2 As shown, the cumulative displacement of the effectively tracked target is monitored in real time by observing the occluded area. And compare it with a set threshold m based on the ROI size (defined as the minimum side length of the ROI). 1 / 10 of The system performs a comparison and then adaptively activates either the "short-term occlusion processing" or "long-term occlusion processing" mechanism based on the comparison results.

[0031] a) Short-term occlusion The process involves performing FFT spectral analysis, calculating the instantaneous phase, and applying quasi-harmonic vibration compensation to prevent high-frequency vibrations from causing target escape. The entire process no longer relies solely on linear assumptions but decomposes the motion into linear trend terms and high-frequency vibration terms, which are processed separately. Details are as follows: (1) Group motion feature extraction: definition It is the set of all visible candidate circular markers in the current frame. Given the total number of candidate circular markers in the set. Calculate the population at the current time. That is, the geometric center coordinates of the current frame image. As a raw signal characterizing the overall movement trend of the structure: in, Represents a set The Middle The candidate circular markers at the current time The coordinates.

[0032] (2) Frequency Domain Vibration Analysis (FFT): In order to accurately extract vibration parameters from the overall motion, the system performs the following signal processing procedure. First, the length of the sliding time window is set to... (Frame number), including the current frame image, and then using step (1) to calculate the geometric center coordinates of each frame image within the sliding time window, forming a geometric center coordinate sequence, and then using the following formula to perform mean removal processing to obtain a zero-mean sequence { }; in, It is the arithmetic mean of the geometric center coordinate signals within the sliding time window.

[0033] Then, using the following formula, the zero-mean sequence { Perform a Fast Fourier Transform (FFT) to obtain the frequency domain response. , Within the effective frequency range ( Search for the maximum value of the spectral modulus to determine the dominant frequency. and the corresponding peak amplitude ,in, The camera sampling frame rate; (3) Instantaneous phase and harmonic increment calculation: When the detected peak amplitude If the structure is determined not to have experienced significant vibration, then a forced vibration compensation increment is imposed. ;like If the structure is in a state of significant vibration, then the vibration compensation increment needs to be calculated. .

[0034] Specifically, using the Hilbert transform (denoted as Hilbert transform) ) Instantaneous phase of the analytical signal To accurately capture the current state of the vibration waveform: Based on the calculated parameters Construct the vibration compensation increment for the current frame. , in, The preset activation threshold, This represents the time interval between two adjacent frames (i.e.) = 1 / fps).

[0035] (4) Momentum fusion and final position prediction: predefined This is the linear momentum of the target itself (i.e., the displacement vector of the target in the previous frame). This represents the average momentum of the group (i.e., the displacement vector at the group center). The final current frame position prediction. A weighted fusion strategy is adopted: in, This indicates the coordinates of the center point of the occluded candidate circular marker in the previous frame image. This is the confidence weighting coefficient, used to adjust the ratio of self-inertia to group constraint.

[0036] b) Long-term occlusion: The RANSAC-based motion segmentation algorithm RBMS is launched, and the translation and rotation motion layers are iteratively stripped. The homography matrix is ​​used to accurately map the historical topology to the current frame to restore the target.

[0037] When the cumulative displacement caused by the occluded effective tracking target Exceeding the set threshold In cases of short-term occlusion failure, this step utilizes the geometric rigidity of the structure's surface to accurately map the geometric topology in the reference frame to the current frame by calculating a transformation matrix. Addressing the non-rigid composite motion within the structure (i.e., multiple independent rigid moving parts simultaneously existing in the scene) and the potential feature point sparsity problem, this invention proposes a comprehensive recovery strategy that integrates RBMS (Random Access Module Segmentation based on RANSAC) and AGCR (Adaptive Geometric Correspondence Reconstruction). Specifically: 1) Based on the inherited identity ID, obtain all matching point pairs between the last frame image before occlusion and the first frame image, i.e., subsequent circular markers with the same identity ID in the two frames, and construct an unclassified point set in this way. ; definition The last frame before it was occluded. Using the first frame image as an example, based on the identity ID that the effectively tracked target maintains throughout continuous monitoring, all matching point pairs between the two frames are extracted to construct the complete set. ,in, The coordinates of the identity ID in the first frame of the image. Initialize the unclassified point set based on the coordinates of the identity ID in the last frame of the image before it was occluded. .

[0038] 2) Iterative motion layer stripping of RBMS: Using the RANSAC algorithm to strip unclassified point sets Verification was performed to separate the interior point sets that conform to the same motion model. They belong to the same motion group, for the set of exterior points that does not conform to this motion model. The RANSAC algorithm is used again for verification, and this process is repeated until the remaining number of points is insufficient to solve the planar model, thus eliminating the unclassified point set. The motion field is divided into multiple independent motion groups. In this way, the mixed motion field can be separated into multiple independent motion models, thereby improving the classification of unclassified point sets. The internal matching point pairs are classified.

[0039] I) Dominant model fitting: This involves fitting the unclassified set... Input the data into the RANSAC algorithm, select the motion model with the most interior points as the dominant model, and separate the interior point set belonging to this motion model. and the set of external points that do not conform to the motion model ; II) Residual secondary utilization: without discarding the outer point set Instead, it was identified as a potential "secondary motion structure observation point" that needed to be separated again; Ⅲ) Iterative loop: Convert the outer point set... As a new Repeat the above processes I) and II) until the remaining number of points is insufficient to solve the planar model, thereby classifying the unclassified set. Divided into A separate sports group .

[0040] 3) Adaptive Geometry Solution Strategy (AGCR): Based on the identity ID of the occluded candidate circular marker in the current frame image, determine the motion group to which it belongs, and use the geometric constraints of the remaining point pairs in the motion group or the image features of the occluded candidate circular marker itself to recover the coordinates of the center point of the occluded candidate circular marker in the current frame image.

[0041] Based on the number of remaining matching pairs in the current motion group The adaptive adjustment model for solving the transformation matrix (DoF) is used to address feature degradation caused by sparse textures or occlusion. I) Fully Monography Model ( When there are enough matching point pairs, the complete 8-DOF homography matrix can be solved. Based on the IDs of the remaining point pairs, extract their coordinate sets in the current frame image. and the set of coordinates of the last frame before it was occluded The homography matrix is ​​calculated using the least squares method. : in, As the normalization factor, - Transformation parameters used to describe planar rotation, translation, and perspective distortion.

[0042] II) Affine Degradation Model ( When there are fewer than 4 but at least 3 feature points, the system automatically downgrades to a 6-DOF affine transformation model. In this case, the perspective term of the forced constraint matrix is ​​0 to ensure solution stability in the case of fewer points. Similar to step I, the coordinate set of the remaining point pairs in the current frame image is extracted based on their IDs. and the set of coordinates of the last frame before it was occluded The following affine transformation matrix is ​​solved using the least squares method. : in, For rotation and scaling parameters, These represent the translation amounts in the horizontal and vertical directions, respectively.

[0043] In both of the above scenarios, first determine the occluded target. sports group And obtain the optimal transformation matrix corresponding to this group. (i.e., the result obtained above) or Subsequently, the matrix was used to determine the original coordinates of the target in the last frame before it was occluded. Perform coordinate mapping and calculate its restored coordinates in the current frame image. : III) Feature reconnection and forced matching When a motion group has fewer than 3 matching point pairs, the above geometric constraints are completely invalidated. The system triggers a feature reconnection mechanism: using the center point of the last effective tracking target as the center, the cropping radius is expanded to construct a reference image, which is then matched with the complete image of the current frame using LightGlue depth features. Since the range of the reference image has been expanded and the LightGlue depth feature matching algorithm can extract a large number of depth texture features, a sufficient number of feature point pairs (usually far more than 4) can be obtained at this time, and the homography matrix can be directly calculated. And use this matrix to determine the coordinates of the occluded target in the last frame. Mapped to the current frame, we get If the target is still not detected after performing the above steps for k consecutive frames (e.g., k=30), the system will automatically switch to global retrieval mode. That is, every k frames, global LightGlue depth feature matching is performed on the complete image of the current frame and the complete image of the last frame before occlusion. The feature correspondence between the two frames is used to force the recapture of the target and correct the accumulated drift.

[0044] 4. Three-dimensional displacement calculation module This step aims to transform the two-dimensional subpixel image coordinates output by the aforementioned module into high-precision three-dimensional world coordinates through a rigorous photogrammetric model. To eliminate the impact of lens optical distortion on measurement accuracy, this module strictly follows the "correction first, reconstruction later" processing flow.

[0045] a) Construction of stereo projection matrix: Based on the extrinsic parameter calibration results of the binocular system (rotation matrix) Translation vector ), respectively construct the left and right cameras Projection matrix and : b) Construction of linear triangulation equations: matching point pairs using the distorted left and right views. and Based on the collinearity constraint of projection (i.e., the image point vector and the reprojected point vector are parallel, and their cross product is zero), the fundamental equations are established as follows: in, Represents the coordinate vector of a homogeneous image. Indicates 3 A 4x4 camera projection matrix, Represents the homogeneous 3D world coordinate vector to be solved. ,symbol This represents the vector cross product operation.

[0046] Applying the above constraints to the left and right cameras respectively, unfolding and combining them, we can construct a three-dimensional spatial point... overdetermined linear equations The specific expansion form of the coefficient matrix A is as follows: in, , and , These are the pixel coordinates of the target in the left and right views, respectively; Represents the projection matrix of the left camera The i-th row vector (i=1,2,3); Represents the projection matrix of the right camera The i-th row vector.

[0047] c) SVD optimization solution: for the coefficient matrix Perform Singular Value Decomposition (SVD): Homogeneous solution of three-dimensional world coordinates Take as a matrix The right singular vector corresponding to the minimum singular value (i.e. The last column), then homogeneous normalization is performed to obtain Euclidean coordinates: d) Calculation of dynamic displacement: based on the initial time ( That is, the static coordinates of the first frame image. Using this as a baseline, calculate the current time. Three-dimensional displacement vector This outputs a displacement curve over continuous time. like Figure 3 As shown, to demonstrate the robustness of this invention in complex dynamic scenarios, this embodiment constructed an experimental system simulating a real structural monitoring environment. The experimental object was a structure undergoing a combined translational and rotational motion, with a long-term occlusion zone set along its motion trajectory.

[0048] During the test, the marker rotated with the disk, and marker point 0 was completely invisible for a prolonged period from 28 to 38 seconds. During this time, due to the simultaneous rotation and translation of the structure, the actual trajectory of the marker exhibited complex nonlinear characteristics. For conventional algorithms, this nonlinear motion would cause the predicted trajectory based on linear assumptions to diverge rapidly, making it impossible to retrieve the target upon reconstruction.

[0049] To address this challenge, this system, upon detecting a long-term target loss, does not blindly rely on inertial prediction but instead utilizes the geometric rigidity of the structure's surface. By analyzing the correspondence between remaining visible markers in the current frame and historical reference frames, the system calculates the geometric transformation parameters describing the deformation of the local area (primarily affine transformation constraints in this specific embodiment). Based on these constraints, the system successfully deduces the nonlinear motion path of the occluded marker within the invisible region. When the marker finally reappears, the predicted ROI accurately covers its true location, immediately achieving automatic identity reconnection. Experimental results demonstrate that even in the absence of visual observation data, this invention can maintain tracking continuity through geometric topology analysis, effectively solving the problem of long-term occlusion recovery under nonlinear motion.

[0050] To verify the feasibility of this invention, it was rigorously validated on a proprietary dataset customized for structural health monitoring (SHM). This dataset covers real-world engineering scenarios such as controlled occlusion, nonlinear motion (rotation / vibration), and high frame rate acquisition, ensuring the authenticity, validity, and engineering applicability of performance metrics (ORR, Re-ID) and avoiding misjudgments on general public datasets.

[0051] Meanwhile, under prolonged complete occlusion (occlusion exceeding 200 frames), Circle-Tracker still maintains an occlusion recovery rate of over 92%, while mainstream MOT trackers (DeepSORT, ByteTrack, OC-SORT) all fail under this condition (0% ORR), demonstrating the robustness of identity reconstruction under long-term occlusion. The occlusion recovery rate (ORR) is significantly improved, and the inheritance accuracy of the target's identity ID is as high as 95.6%, effectively solving the identity switching problem caused by visual homogeneity. Furthermore, it can achieve high-precision displacement measurement, with the final 3D displacement measurement accuracy reaching sub-millimeter level.

[0052] It is important to note that the schemes and arrangements of this application shown in the exemplary embodiments are merely exemplary. Although only a few embodiments are described in detail in this disclosure, those who consult this disclosure will readily understand that many modifications are possible (e.g., variations in various parameter values ​​(temperature, power, humidity, etc.), installation arrangements, names, colors, logical orders, etc.) without substantially departing from the novel teachings and advantages of the subject matter described in this application. Therefore, all such modifications are also included within the scope of the invention, and the order or sequence of any process or method steps may be changed or rearranged according to alternative embodiments. In the claims, any "apparatus plus function" clause is intended to cover the structure described herein for performing the function, and not only structurally equivalent but also equivalent in structure. Other substitutions, modifications, alterations, and omissions may be made in the design, operation, and arrangement of the exemplary embodiments without departing from the scope of the invention. Therefore, the invention is not limited to the particular embodiments but extends to a variety of modifications that still fall within the scope of the appended claims.

[0053] Furthermore, in order to provide a concise description of exemplary embodiments, not all features of actual embodiments (i.e., those features that are not relevant to the best mode of carrying out the invention as currently considered, or those features that are not relevant to implementing the invention) may be omitted.

[0054] It should be understood that numerous specific implementation decisions can be made during the development of any practical implementation, such as in any engineering or design project. Such development efforts may be complex and time-consuming, but for those skilled in the art who benefit from this disclosure, the development effort will be a routine work of design, manufacturing, and production without requiring much experimentation.

[0055] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A multi-target tracking method based on ROI and homology matrix, characterized in that: Using a binocular camera to acquire real-time video images of the target to be tracked, target tracking is performed according to the following steps. Step 1: Match each candidate circular marker in the first frame of the left and right videos, take the successfully matched candidate circular markers as valid tracking targets, assign an identity ID to each one, and obtain the corresponding region of interest (ROI). Step 2: Based on each valid tracked target in the previous frame image, extract the Region of Interest (ROI) corresponding to the current frame image, and then re-detect the candidate circular markers in each ROI. If found, update this as the current candidate circular marker and inherit the corresponding identity ID. If none, estimate the current candidate circular marker and calculate the displacement between the candidate circular marker and the last frame image before occlusion; Step 3: If the displacement is not greater than a set threshold, short-term occlusion processing is performed; if it is greater than the set threshold, long-term occlusion processing is performed. Simultaneously, using the current frame images from the left and right videos, the three-dimensional coordinates and corresponding displacement of the valid tracking target that has inherited the identity ID are calculated.

2. The multi-target tracking method based on ROI and homogeneous matrix according to claim 1, characterized in that: In the matching process of step one, the center point coordinates of the candidate circular markers in the two first frame images are detected first, and then matched. The candidate circular markers corresponding to the successfully matched point pairs are taken as effective tracking targets, and an identity ID is assigned to each of them. At the same time, the corresponding region of interest (ROI) is obtained. In step two, based on the center point coordinates of each effective tracked target in the previous frame image, the region of interest (ROI) corresponding to the current frame image is extracted, and then the center point coordinates of the candidate circular markers in each ROI are re-detected. If they are found, the center point coordinates of the current candidate circular markers are updated accordingly, and the corresponding identity ID is inherited. If not, estimate the center point coordinates of the current candidate circular marker and calculate the displacement between the center point coordinates of the candidate circular marker and the center point coordinates of the last frame image before occlusion.

3. The multi-target tracking method based on ROI and homogeneous matrix according to claim 2, characterized in that: For short-term occlusion processing, first calculate the coordinates of the geometric center points of the inherited candidate circular markers in each frame image within the sliding time window. and the coordinates of these geometric center points Perform mean-removal processing to obtain a zero-mean sequence. The sliding time window is set to include the previous frame image, including the current frame image. Frame image; Then, for the zero-mean sequence Perform a Fourier transform to obtain the frequency domain response. Search for the maximum value of the spectral modulus to determine the dominant frequency. and the corresponding peak amplitude If the peak amplitude If the value is not less than the activation threshold, then the vibration compensation increment is calculated using the Hilbert transform. The coordinates of the center point of the occluded candidate circular marker in the current frame image are calculated using the following formula. , in, This represents the confidence weighting coefficient. , All of these represent preset parameters.

4. The multi-target tracking method based on ROI and homogeneous matrix according to claim 3, characterized in that: Calculate the vibration compensation increment using the following formula. , in, This indicates the preset activation threshold. Represents the Hilbert transform. This indicates the time interval between two adjacent frames.

5. The multi-target tracking method based on ROI and homogeneous matrix according to claim 3, characterized in that: Using the following formula, the coordinates of these geometric center points... Perform mean-removal processing to obtain a zero-mean sequence. , in, Represents the coordinates of the geometric center point of all frames within the sliding time window. The arithmetic mean.

6. The multi-target tracking method based on ROI and homogeneous matrix according to claim 2, characterized in that: For long-term occlusion processing, first, based on the inherited identity ID, obtain all matching point pairs between the last frame image before occlusion and the first frame image, and use this to construct an unclassified point set. ; Then use the RANSAC algorithm to classify the unclassified point set. Verification was performed to separate the interior point sets that conform to the same motion model. They belong to the same motion group, for the set of exterior points that does not conform to this motion model. The RANSAC algorithm is used again for verification, and this process is repeated until the remaining number of points is insufficient to solve the planar model, thus eliminating the unclassified point set. Divided into multiple independent motion groups; Then, based on the identity ID of the occluded candidate circular marker in the current frame image, the motion group to which it belongs is determined. The coordinates are recovered by using the geometric constraints of the remaining point pairs in the motion group or the image features of the occluded candidate circular marker itself, thereby obtaining the coordinates of the center point of the occluded candidate circular marker in the current frame image.

7. The multi-target tracking method based on ROI and homogeneous matrix according to claim 6, characterized in that: If the number of remaining point pairs in the motion group Then, based on the center point coordinates of the remaining point pairs' identity IDs in the current frame image and the last frame image before occlusion, the homography matrix H is calculated. Then, using the homography matrix H, based on the center point coordinates of the occluded candidate circular markers in the last frame image, the center point coordinates of the occluded candidate circular markers in the current frame image are deduced. If the number of remaining pairs of points in the motion group satisfies Then, based on the center point coordinates of the remaining point pairs' IDs in the current frame and the last frame before occlusion, the transformation matrix is ​​calculated using the affine transformation principle. Then use the transformation matrix Based on the center point coordinates of the occluded candidate circular marker in the last frame image, the center point coordinates of the occluded candidate circular marker in the current frame image are deduced. If the number of remaining point pairs in the motion group Then, the LightGlue deep feature matching network model is invoked to obtain feature matching point pairs between the current frame image and the last frame image before occlusion, and the homography matrix is ​​calculated based on the feature matching point pairs. Then, using the homography matrix Based on the center point coordinates of the occluded candidate circular marker in the last frame before it was occluded, the center point coordinates of the occluded candidate circular marker in the current frame are deduced.

8. The multi-target tracking method based on ROI and homogeneous matrix according to claim 1, characterized in that: A linear triangulation method based on SVD is used to calculate the three-dimensional coordinates of valid tracking targets with inherited identity IDs in the current frame images of the left and right videos, and to calculate the displacement of these valid tracking targets relative to the previous frame image.

Images