A method for tracking trajectory of moving target and measuring physical quantity based on binocular vision

Abstract

The application provides a motion target trajectory tracking and physical quantity measurement method based on binocular vision, and belongs to the technical field of computer vision and image processing, which comprises the following steps: acquiring a stereoscopic image sequence of a motion target collected by a binocular camera; in the starting frame of the sequence, identifying the motion target and determining an initial position through a target detection model; generating a tracking query vector based on the position, tracking the motion target in subsequent frames according to the vector, and outputting a continuous target region; based on pre-calibrated camera parameters, performing epipolar rectification on each frame of image in the stereoscopic image sequence to obtain left and right rectified images; based on the images, calculating the disparity in the continuous target region to obtain disparity data; according to the disparity data and the calibration parameters, obtaining the motion trajectory of the motion target; and calculating the motion physical quantity according to the motion trajectory. Through optimization of the whole process of detection tracking, binocular vision processing and physical quantity measurement, the application realizes real-time measurement of target trajectory tracking and physical quantity in a motion scene.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and image processing technology, and specifically relates to a method for tracking the trajectory of moving targets and measuring physical quantities based on binocular vision. Background Technology

[0002] Computer vision technology has broad prospects for application in fitness and sports training. In various sports, the height, trajectory, and speed of moving targets (such as barbells, balls, and human joints) are core indicators for measuring the standardization of athletes' movements and the efficiency of power output. The accuracy and real-time performance of these measurements directly affect the optimization of training effects and the quality of sports performance evaluation. Existing physical quantity measurement technologies for moving targets are mainly divided into three categories: The first category is manual measurement, which involves coaches observing with their naked eyes or manually recording with simple tools. This method is greatly affected by subjective experience, has low measurement accuracy, cannot achieve continuous recording of dynamic processes, and is difficult to capture instantaneous physical quantities during rapid movements. The second category is measurement technology based on two-dimensional vision, which uses a monocular camera to acquire images and combines them with target detection algorithms to estimate physical quantities. However, it lacks depth information, is easily affected by shooting angle and background interference, and has large measurement errors, failing to meet the needs of high-precision training. The third category is measurement technology based on binocular vision, which uses the parallax principle of binocular cameras to obtain three-dimensional spatial information. Compared with monocular vision, it has higher measurement accuracy and has been gradually applied to three-dimensional measurement of sports scenes.

[0003] However, existing binocular vision measurement technology still has many shortcomings in sports training applications: First, the frame rate and resolution of general-purpose binocular cameras are difficult to balance, making it unsuitable for real-time tracking of fast-moving targets and prone to trajectory loss or measurement delays. Sports training targets often exhibit high-speed motion characteristics, placing high demands on both the camera's frame rate and image resolution. Second, stereo matching is a core component of binocular vision systems, and optimizing the stereo matching algorithm is crucial for improving the accuracy of 3D target recognition. Existing disparity calculation algorithms mostly employ traditional local or global matching algorithms. Local matching algorithms are prone to mismatches in areas with sparse texture on the target surface, while global matching algorithms have high computational complexity, failing to meet real-time requirements and exhibiting poor robustness in scenarios with athlete occlusion or complex backgrounds. Third, the camera calibration and image correction processes lack optimization for sports scenarios, leading to accumulated calibration errors that affect the accuracy of depth mapping and physical quantity measurement. In sports scenarios, camera deployment positions and shooting angles vary greatly, making conventional calibration methods difficult to adapt to the accuracy requirements of different scenarios. Fourth, existing technologies lack a comprehensive collaborative optimization scheme encompassing "detection and tracking - disparity calculation - depth mapping - physical quantity measurement." Independent operation of each module can easily lead to data gaps, reducing the overall system's stability and measurement accuracy. Detection and tracking results cannot effectively guide the optimization of disparity calculation range, and disparity calculation results also fail to provide feedback to the tracking module to improve robustness in occluded scenarios.

[0004] Therefore, there is an urgent need for a motion target detection, tracking, and physical quantity measurement technology that is adaptable to complex motion scenarios and combines real-time performance with high accuracy, in order to overcome the shortcomings of existing technologies and meet the needs of professional sports training and evaluation. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention proposes a method for tracking moving target trajectories and measuring physical quantities based on binocular vision. By optimizing the entire process of detection and tracking, binocular vision processing, and physical quantity calculation, high-precision tracking of target trajectories and real-time calculation of key physical quantities in motion scenarios are achieved.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] In a first aspect, the present invention proposes a method for tracking the trajectory of a moving target and measuring physical quantities based on binocular vision, comprising the following steps:

[0008] Acquire a sequence of stereo images of a moving target captured by a binocular camera;

[0009] In the initial frame of the stereo image sequence, a moving target is identified and its initial position is determined by a target detection model; a tracking query vector is generated based on the initial position, and the moving target is tracked in subsequent frames according to the tracking query vector. The bounding box of the moving target in the image coordinate system is output frame by frame as a continuous target region.

[0010] Based on pre-calibrated binocular camera parameters, epipolar correction is performed on each frame of the stereo image sequence to obtain epipolar-aligned left and right corrected images; based on the left and right corrected images, disparity is calculated in the continuous target region to obtain disparity data corresponding to the moving target.

[0011] Based on the parallax data and the calibration parameters of the binocular camera, the motion trajectory of the moving target in three-dimensional space is obtained; based on the motion trajectory, the motion physical quantities of the moving target are calculated.

[0012] Furthermore, after acquiring the stereo image sequence of the moving target captured by the binocular camera, the process also includes:

[0013] The stereo image sequence is preprocessed to obtain the frame rate and image height from the metadata of the stereo image sequence, and the original image frames are sampled in the time dimension at a preset sampling rate to obtain the sampled image frames.

[0014] Furthermore, in the initial frame of the stereo image sequence, a moving target is identified and its initial position is determined using a target detection model; a tracking query vector is generated based on the initial position, and the moving target is tracked in subsequent frames according to the tracking query vector. The bounding box of the moving target in the image coordinate system is output frame by frame as a continuous target region, specifically:

[0015] The initial frame of the image frame is input into the target detection model; the target detection model outputs the initial bounding box of the moving target, which is characterized by the coordinates of the center point of the initial bounding box and the size of the initial bounding box;

[0016] A tracking query vector is generated based on the center point coordinates of the initial bounding box. The tracking query vector contains the initial position of the moving target in the starting frame and the starting frame identifier.

[0017] The tracking query vector is input into the tracking model, which tracks the moving target frame by frame in the subsequent frames of the stereo image sequence based on the tracking query vector to obtain the position vector of the tracking point;

[0018] Based on the position vector, the tracking bounding box of the moving target in the image coordinate system is output frame by frame. The tracking bounding box is characterized by the coordinates of the center point of the tracking bounding box and the size of the tracking bounding box. The tracking bounding box output frame by frame is used as the continuous target region of the moving target.

[0019] Furthermore, based on the pre-calibrated binocular camera parameters, epipolar correction is performed on each frame of the stereo image sequence to obtain epipolar-aligned left-right corrected images, specifically:

[0020] Based on the distortion coefficients of the binocular camera, distortion is eliminated from the left and right images in the stereo image to obtain distortion-free left and right images; the distortion coefficients include radial distortion coefficients. and tangential distortion coefficient ;in, Indicates the second-order radial distortion coefficient; Indicates the fourth-order radial distortion coefficient; Represents the sixth-order radial distortion coefficient; Indicates the first tangential distortion coefficient; Indicates the second tangential distortion coefficient;

[0021] Based on the intrinsic and extrinsic parameters of the binocular cameras, stereo correction is performed on the distortion-free left and right images to obtain epipolar-aligned corrected images; the intrinsic parameters include the focal lengths of the left and right cameras. and principal point coordinates The extrinsic parameters include the rotation matrix. Translation vector .

[0022] Furthermore, the distortion removal process is as follows:

[0023] For any pixel in the left and right images, let the pixel coordinates in the original distorted image be... Convert its pixel coordinates to normalized image coordinates with the principal point as the origin and the focal length as the unit. ;

[0024] Calculate the coordinates after radial distortion correction based on the radial distortion coefficient. , is represented as:

[0025] ;

[0026] in, ; The distance from any pixel to the origin;

[0027] Calculate the normalized coordinates after tangential distortion correction based on the tangential distortion coefficient. , is represented as:

[0028] ;

[0029] Based on the intrinsic parameters, the normalized coordinates after tangential distortion correction are... Convert to corrected pixel coordinates , is represented as:

[0030] ;

[0031] in, The camera is in shaft and Pixel focal length along the axial direction; Principal point coordinates;

[0032] Based on the corrected pixel coordinates Interpolation methods are used to extract the original distorted image. By sampling pixel values, a distortion-free image is reconstructed.

[0033] Furthermore, the stereo correction process is as follows:

[0034] Based on rotation matrix Translation vector Calculate the correction rotation matrix for the left and right images. , and the corrected projection matrix , and depth mapping matrix ;

[0035] Based on the correction rotation matrix of the left and right images , and projection matrix , Combined with intrinsic parameters, a pixel mapping table for the left and right images is generated;

[0036] According to the pixel mapping table, the distortion-free left and right images are resampled to obtain epipolar aligned corrected left and right images, in which the corresponding pixels of the left and right images are located on the same horizontal scan line.

[0037] Furthermore, based on the corrected left and right images, disparity is calculated within the continuous target region to obtain disparity data corresponding to the moving target, specifically as follows:

[0038] Horizontal Sobel operator edge enhancement processing is applied to the local images corresponding to the continuous target regions in the left and right corrected images;

[0039] On the edge-enhanced local image, a semi-global block matching algorithm is used to calculate the initial disparity. The matching cost calculation window size of this algorithm is a preset window size, and the smoothing penalty coefficient is set to... and ;and The matching cost is accumulated through dynamic programming along multiple preset directions, and the disparity with the minimum total cost is selected as the initial disparity of the pixel.

[0040] The initial disparity is subjected to left-right consistency checks, uniqueness checks, and mismatch removal to generate optimized disparity data corresponding to the moving target. .

[0041] Furthermore, based on the parallax data and the calibration parameters of the binocular camera, the motion trajectory of the moving target in three-dimensional space is obtained, specifically as follows:

[0042] According to parallax data Baseline distance to binocular cameras Calculate the depth information of each point on the moving target. ,in satisfy:

[0043] ;

[0044] in, This is the equivalent focal length of the camera; Indicates the parallax offset;

[0045] Combined with the depth information Coordinates of the moving target in the image coordinate system Convert it to three-dimensional coordinates in the camera coordinate system. The transformation relation satisfies:

[0046] ;

[0047] in, Indicates the coordinates of the principal point; Indicates pixel focal length;

[0048] 3D coordinates acquired frame by frame Connecting them in chronological order, they form the trajectory of the moving target in three-dimensional space.

[0049] Furthermore, based on the motion trajectory, the motion physical quantities of the moving target are calculated, specifically: obtaining the pre-calibrated reference height of the binocular camera center relative to the reference plane. And extracting the three-dimensional coordinates of the center point of the moving target from the motion trajectory. ; Calculate the height of the moving target , satisfy:

[0050] ;

[0051] in, This represents the coordinate components in the vertical direction of the camera coordinate system, with downward being the positive direction;

[0052] Extract the vertical coordinates of the center point of the moving target from the motion trajectory for two consecutive frames. and ,

[0053] Obtain the time interval between adjacent frames ,in , Set the frame rate for the stereo camera; calculate the ascent velocity of the moving target. , satisfy:

[0054] ;

[0055] Height calculated frame by frame and rate of ascent Arranged in chronological order, generate curves showing changes in altitude and velocity of the moving target.

[0056] Secondly, the present invention also proposes a device for tracking the trajectory of a moving target and measuring physical quantities based on binocular vision, characterized in that it includes:

[0057] Binocular camera, used to acquire stereo image sequences of moving targets;

[0058] Memory, used to store executable instructions;

[0059] The processor is connected to both the binocular camera and the memory, and is used to execute executable instructions stored in the memory to implement the aforementioned method for tracking the trajectory of moving targets and measuring physical quantities based on binocular vision.

[0060] The effects described in the invention are merely those of the embodiments, and not all the effects of the invention. One of the above technical solutions has the following advantages or beneficial effects:

[0061] This invention proposes a method for tracking the trajectory of a moving target and measuring physical quantities based on binocular vision, belonging to the field of computer vision and image processing technology. The method includes the following steps: acquiring a sequence of stereo images of a moving target captured by a binocular camera; identifying the moving target and determining its initial position in the first frame of the stereo image sequence using a target detection model; generating a tracking query vector based on the initial position; tracking the moving target in subsequent frames according to the tracking query vector; outputting the bounding box of the moving target in the image coordinate system frame by frame as a continuous target region; performing epipolar correction on each frame of the stereo image sequence based on pre-calibrated binocular camera parameters to obtain epipolar-aligned left-right corrected images; calculating disparity within the continuous target region based on the left-right corrected images to obtain disparity data corresponding to the moving target; obtaining the motion trajectory of the moving target in three-dimensional space based on the disparity data and the calibration parameters of the binocular camera; and calculating the motion physical quantities of the moving target based on the motion trajectory. This invention achieves high-precision tracking of target trajectories and real-time measurement of key physical quantities in motion scenarios by optimizing the entire process of detection and tracking, binocular vision processing, and physical quantity measurement.

[0062] This invention addresses the technical pain points of existing general-purpose binocular cameras, which cannot simultaneously achieve high frame rates and high resolutions, are difficult to adapt to fast-moving targets, and suffer from error accumulation due to the lack of scene optimization in the calibration process. This solution uses a high frame rate, high resolution binocular camera as the hardware core and combines it with an optimized calibration method. It provides high frame rate hardware support for fast motion tracking and eliminates the effects of lens distortion and parameter error accumulation through accurate calibration, achieving coordinated adaptation of high frame rate acquisition and low error modeling.

[0063] This invention addresses the problems of existing algorithms being unsuitable for scenarios with sparse textures of moving targets, athlete occlusion, and complex backgrounds, which easily lead to mismatches and high computational complexity. This solution employs multiple customized optimizations, including edge enhancement processing, optimized matching windows and penalty coefficients, and multi-directional dynamic programming, to accurately compensate for the matching deficiencies of traditional algorithms in weak texture regions. At the same time, it takes into account real-time performance and anti-interference capabilities, and can stably output high-quality parallax data in scenarios with fast-moving targets and partial occlusion.

[0064] This invention addresses the problems in existing technologies where detection and tracking, disparity calculation, and physical quantity measurement modules operate independently, leading to data transmission gaps, trajectory loss, and poor system stability in occluded scenarios. This solution constructs a collaborative framework encompassing video preprocessing, target detection and tracking, binocular vision processing, and physical quantity measurement. The detection and tracking module provides precise target regions for binocular vision processing, while binocular correction and disparity calculation results feed back into the tracking module, optimizing target prediction and relocation accuracy in occluded scenarios. This maintains tracking continuity and measurement stability in typical motion scenarios involving athlete-occluded targets and background interference.

[0065] This invention addresses the problems of existing binocular vision technology, such as numerous holes, high noise, and inaccurate coordinate transformation in depth maps, which lead to large errors in target height calculation and make it unsuitable for professional training assessment. This solution adopts a multi-step depth map optimization strategy, including hole filling, bilateral filtering smoothing, and outlier removal. Combined with precise transformation of the camera coordinate system, using the ground as the reference plane, the camera installation angle and optical center height are simultaneously calibrated, significantly reducing calculation errors. It can accurately output core physical quantities such as target height and ascent speed, providing coaches with objective and quantitative motion assessment data to meet the refined needs of professional sports training. Attached Figure Description

[0066] Figure 1 This is a flowchart of a method for tracking the trajectory of a moving target and measuring physical quantities based on binocular vision, as proposed in Embodiment 1 of the present invention.

[0067] Figure 2 This is a framework diagram of the method proposed in Embodiment 1 of the present invention in a weightlifting training scenario;

[0068] Figure 3 This is a schematic diagram of a device for tracking the trajectory of a moving target and measuring physical quantities based on binocular vision, as proposed in Embodiment 2 of the present invention. Detailed Implementation

[0069] To clearly illustrate the technical features of this solution, the invention will be described in detail below through specific embodiments and in conjunction with the accompanying drawings. The following disclosure provides many different embodiments or examples for implementing different structures of the invention. To simplify the disclosure of the invention, components and arrangements of specific examples are described below. Furthermore, reference numerals and / or letters may be repeated in different examples. This repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed. It should be noted that the components illustrated in the drawings are not necessarily drawn to scale. Descriptions of well-known components, processing techniques, and processes are omitted in this invention to avoid unnecessarily limiting the invention.

[0070] Example 1

[0071] Embodiment 1 of this invention proposes a method for tracking the trajectory of moving targets and measuring physical quantities based on binocular vision, which is used to solve the technical problems of binocular vision technology in motion scenes, such as difficulty in balancing high frame rate and high resolution, easy mismatch in disparity calculation in weak texture areas, easy loss of tracking in occluded scenes, and poor system stability caused by independent operation of each processing module.

[0072] Figure 1 This is a flowchart of a method for tracking the trajectory of a moving target and measuring physical quantities based on binocular vision, as proposed in Embodiment 1 of the present invention.

[0073] In step S1, a stereo image sequence of the moving target is acquired by the binocular camera. The stereo image sequence contains continuous time frames, each frame of which has a timestamp information. There is a fixed time interval between adjacent frames (determined by the camera frame rate). These characteristics provide a time reference for subsequent motion tracking and velocity calculation.

[0074] This invention processes stereo image sequences acquired by a binocular camera to achieve three-dimensional trajectory tracking of moving targets and real-time measurement of key physical quantities, and can be applied to the following scenarios:

[0075] Weightlifting Training: In weightlifting training, binocular cameras are deployed on the side of the training area to capture real-time 3D image sequences of the athlete lifting the barbell. By processing the image sequences, the three-dimensional trajectory of the barbell is tracked in real time, and physical quantities such as the barbell's height and lifting speed are measured. Coaches can use this quantitative data to evaluate the athlete's movement accuracy and power efficiency, providing a scientific basis for training optimization.

[0076] Track and field training: In track and field events such as high jump and long jump, binocular cameras are deployed on the side of the exercise area to capture a sequence of stereoscopic images of the athlete's entire process of takeoff, flight, and landing. By processing the image sequence, the three-dimensional motion trajectory of the athlete's center of gravity or specific joints (such as the knee and ankle joints) can be tracked in real time, and key parameters such as flight height, flight time, and landing angle can be calculated to assist coaches in analyzing technical movements.

[0077] Ball sports: In ball sports such as basketball, football, and volleyball, binocular cameras are deployed to the side or above the court to capture a sequence of stereo images of the ball's movement. By processing the image sequences, the ball's trajectory is tracked in real time, and parameters such as flight altitude, speed, landing point, and rotation speed are calculated, providing data support for tactical analysis and training optimization.

[0078] Strength training in fitness settings: In the strength training area of ​​a gym, binocular cameras are deployed on the sides of the training equipment to capture a sequence of 3D images of users performing exercises such as squats, bench presses, and deadlifts. By processing the image sequences, the 3D motion trajectory of the barbell or user's body parts is tracked in real time, and indicators such as range of motion, speed of motion, and symmetry of the motion trajectory are calculated to help users correct their movements and prevent sports injuries.

[0079] Aerobic exercise in fitness settings: During aerobic exercises such as running and skipping rope, binocular cameras are deployed on the side of the exercise area to capture a sequence of stereoscopic images of the user's movement. By processing the image sequence, the system tracks the user's limb movement trajectory and calculates parameters such as stride frequency, stride length, and flight time to provide personalized exercise suggestions to the user.

[0080] Gait analysis in a biomechanical analysis scenario: In a gait analysis laboratory, binocular cameras are deployed around a walking path to acquire stereoscopic image sequences of subjects walking. By processing the image sequences, the three-dimensional motion trajectories of various joints in the lower limbs (hip, knee, ankle) are tracked in real time, and parameters such as stride length, gait speed, and joint angle changes are calculated to provide quantitative evidence for clinical diagnosis and rehabilitation assessment.

[0081] The scope of protection of this invention is not limited to the specific scenarios listed in Example 1, and those skilled in the art can make reasonable selections based on the actual situation.

[0082] Embodiment 1 of the present invention uses a barbell in weightlifting training as an example to illustrate the technical solution of the present invention in detail. Figure 2 This is a framework diagram of the method proposed in Embodiment 1 of the present invention in a weightlifting training scenario;

[0083] In Example 1, a ZED 2i binocular camera is used. This camera is equipped with two high-resolution cameras, left and right, with a fixed baseline distance B between them. The camera simultaneously captures scene images and generates depth information using the principle of binocular parallax. The camera is deployed to the side of the weightlifting training area to ensure that the barbell movement is within the camera's field of view.

[0084] In weightlifting training scenarios, video data of the weightlifting motion process is continuously collected by binocular cameras, corresponding to a stereo image sequence.

[0085] Those skilled in the art should understand that video is essentially a time series composed of continuous image frames, and each frame of video captured by a binocular camera contains two images, one on the left and one on the right, thus forming a stereoscopic image sequence.

[0086] After acquiring the stereo image sequence of the moving target captured by the binocular camera, the process further includes: preprocessing the stereo image sequence, obtaining the frame rate and image height from the metadata of the stereo image sequence, and sampling the original image frames in the time dimension at a preset sampling rate to obtain the sampled image frames.

[0087] Therefore, in a weightlifting training scenario, the task objective is to process a length of... The weightlifting video, through real-time tracking of the barbell's trajectory and combined with the principle of binocular parallax, generates depth information, and then calculates the precise height parameter data during the target's movement.

[0088] Given a video to be processed, basic information such as frame rate and frame height are extracted from metadata to ensure consistency and accuracy in subsequent processing. (Original video frames) A sampling operation will be performed, with the sampling rate being [missing information]. The process is performed over time to reduce computational complexity and improve processing efficiency, ultimately resulting in sampled video frames. .

[0089] In step S2, in the starting frame of the stereo image sequence, a moving target is identified and its initial position is determined by a target detection model; a tracking query vector is generated based on the initial position, and the moving target is tracked in subsequent frames according to the tracking query vector. The bounding box of the moving target in the image coordinate system is output frame by frame as a continuous target region.

[0090] The specific process is as follows: the starting frame of the image frame is input into the target detection model; the target detection model outputs the initial bounding box of the moving target, which is characterized by the coordinates of the center point of the initial bounding box and the size of the initial bounding box;

[0091] A tracking query vector is generated based on the center point coordinates of the initial bounding box. The tracking query vector contains the initial position of the moving target in the starting frame and the starting frame identifier.

[0092] The tracking query vector is input into the tracking model, which tracks the moving target frame by frame in the subsequent frames of the stereo image sequence based on the tracking query vector to obtain the position vector of the tracking point;

[0093] Based on the position vector, the tracking bounding box of the moving target in the image coordinate system is output frame by frame. The tracking bounding box is characterized by the coordinates of the center point of the tracking bounding box and the size of the tracking bounding box. The tracking bounding box output frame by frame is used as the continuous target region of the moving target.

[0094] In a weightlifting training scenario, input The object detection model is used to detect the weight plates on the left and right sides of the barbell to obtain the coordinates of the center point of the initial bounding box. and length and width dimensions The barbell detection network used is an improved target detection model based on YOLOv8n, which adopts a BiFormer-based backbone network. In order to adapt to barbell plates of different sizes, a BiFPN structure is introduced for bidirectional cross-scale feature fusion.

[0095] Then, based on the coordinates of the center point of the initial bounding box when the moving target was first detected... Generate tracking query vectors The query vector contains the initial positions of the center points of the left and right barbell plates to be tracked and the starting frame identifier. The tracking model uses the previously generated query vector... The tracking query vector is obtained frame by frame in the video. The tracking model adopts the CoTracker model, which is an advanced point tracking model based on the Transformer architecture.

[0096] In step S3, based on the pre-calibrated binocular camera parameters, epipolar correction is performed on each frame of the stereo image sequence to obtain epipolar-aligned left and right corrected images; based on the left and right corrected images, disparity is calculated in the continuous target area to obtain disparity data corresponding to the moving target.

[0097] The parameters of a binocular camera include intrinsic parameters, extrinsic parameters, and distortion coefficients.

[0098] Internal parameters include the focal length of the left and right cameras. Units are pixels; principal point coordinates , which is the pixel coordinate of the intersection of the imaging plane and the optical axis; intrinsic parameters describe the optical characteristics of the camera;

[0099] Extrinsic parameters include rotation matrix Translation vector The external parameters reflect the spatial relative positional relationship between the two cameras;

[0100] Distortion coefficients include radial distortion coefficients and tangential distortion coefficient ;in, Indicates the second-order radial distortion coefficient; Indicates the fourth-order radial distortion coefficient; Represents the sixth-order radial distortion coefficient; This represents the first tangential distortion coefficient, used to describe the tangential distortion component of the image in the horizontal direction; This represents the second tangential distortion coefficient, used to describe the tangential distortion component of the image in the vertical direction. The distortion coefficient is used to correct image distortion caused by lens optical defects (such as edge pixel stretching, tilting, etc.). This invention primarily employs the Zhang Zhengming calibration method to calibrate the binocular camera.

[0101] The core purpose of binocular image correction is to eliminate image distortion and ensure that corresponding pixels captured by the left and right cameras are located on the same horizontal scan line (i.e., epipolar alignment). This simplifies the pixel matching problem in a two-dimensional plane to a one-dimensional horizontal search, significantly improving the efficiency and accuracy of subsequent disparity calculations. The process mainly consists of two steps: distortion correction and stereo correction, both based on the camera parameters obtained from the first-stage calibration.

[0102] Due to the limitations of camera lens optical characteristics, original images will have varying degrees of distortion, such as straight lines becoming curved or object edges being deformed. Without correction, this will cause pixel position deviations between corresponding objects in the left and right images, severely affecting matching accuracy. The essence of distortion correction is to map pixels in the distorted image to their corresponding positions in the ideal, distortion-free image using calibrated distortion coefficients.

[0103] The distortion removal process is as follows: For any pixel in the left and right images, let the pixel coordinates in the original distorted image be... Convert its pixel coordinates to normalized image coordinates with the principal point as the origin and the focal length as the unit. ;

[0104] Calculate the coordinates after radial distortion correction based on the radial distortion coefficient. , is represented as:

[0105] ;

[0106] in, ; The distance from any pixel to the origin;

[0107] Calculate the normalized coordinates after tangential distortion correction based on the tangential distortion coefficient. , is represented as:

[0108] ;

[0109] Based on the intrinsic parameters, the normalized coordinates after tangential distortion correction are... Convert to corrected pixel coordinates , is represented as:

[0110] ;

[0111] in, The camera is in shaft and Pixel focal length along the axial direction; Principal point coordinates;

[0112] Based on the corrected pixel coordinates Interpolation methods are used to extract the original distorted image. By sampling pixel values, a distortion-free image is reconstructed.

[0113] The core of stereo correction is to make the optical axes of the left and right cameras parallel through coordinate transformation, and to make the epipolar lines of the left and right images (i.e., the lines connecting the projection points of a point in space onto the left and right images) consistent with the horizontal direction of the images.

[0114] The stereo correction process is as follows: based on the rotation matrix Translation vector Calculate the correction rotation matrix for the left and right images. , and the corrected projection matrix , and depth mapping matrix ;

[0115] Based on the correction rotation matrix of the left and right images , and projection matrix , Combined with intrinsic parameters, the cv::initUndistortRectifyMap function generates pixel mapping tables for the left and right images (including joint transformations for distortion correction and stereo correction).

[0116] The cv::remap function resamples the distortion-free left and right images according to the pixel mapping table to obtain epipolar aligned corrected left and right images, in which the corresponding pixels of the left and right images are located on the same horizontal scan line.

[0117] In weightlifting scenarios, the effectiveness of 3D correction directly impacts the success rate of barbell pixel matching. In the corrected left and right images, the corresponding point of each pixel on the barbell in the other image only needs to be searched on the same horizontal line. This avoids the increased computational load and matching ambiguities caused by 2D search, ensuring efficient and accurate pixel matching even when the barbell is moving rapidly.

[0118] The time difference data calculation process is as follows: Horizontal Sobel edge enhancement processing is performed on the local images corresponding to the continuous target region in the left and right corrected images; on the edge-enhanced local images, a semi-global block matching algorithm is used to calculate the initial disparity. The matching cost calculation window size of this algorithm is a preset window size, and the smoothing penalty coefficient is set to... and ;and The matching cost is accumulated through dynamic programming along multiple preset directions, and the disparity with the minimum total cost is selected as the initial disparity of the pixel. The initial disparity is then subjected to left-right consistency checks, uniqueness checks, and mismatch removal to generate optimized disparity data corresponding to the moving target. .

[0119] Parallax calculation is a core step in binocular vision depth measurement. Essentially, it involves pixel-level matching of the epipolar-aligned left and right corrected images to find the horizontal offset of corresponding pixels. (i.e., parallax data). Addressing the challenges of sparse barbell texture, high-speed movement, complex backgrounds, and susceptibility to athlete occlusion in weightlifting scenarios, an adaptively optimized semi-global block matching (SGBM) algorithm is employed for parallax calculation. First, the corrected left and right images undergo horizontal Sobel edge enhancement processing to strengthen the barbell's edge contour texture and compensate for weak texture defects. Then, core parameters are customized based on scene characteristics, setting the parallax range to a value adapted to the barbell's movement range. The matching block size is set to 15×15 to balance texture capture capability and computational efficiency, and the smoothing penalty coefficient is set to... , To ensure the smoothness and edge sharpness of the disparity data, a 15×15 matching block is used in conjunction with the SAD function to calculate the matching cost of corresponding pixels in the left and right images, reducing interference from illumination changes. Dynamic programming is performed along 16 preset directions to accumulate the matching cost, and the disparity with the minimum total cost is selected as the final disparity of the pixel. Finally, mismatched points are eliminated through left-right consistency checks (maximum disparity difference value = 5) and uniqueness checks (threshold = 15). A 100×100 window size is used to filter isolated noise spots in the background (maximum disparity change = 16). Combined with the target area output by the barbell detection and tracking module mentioned above, the focus is on the barbell range to optimize the disparity calculation accuracy and reduce invalid calculations in the background area, ultimately obtaining high-quality disparity data to provide reliable input for subsequent depth mapping.

[0120] In step S4, the motion trajectory of the moving target in three-dimensional space is obtained based on the parallax data and the calibration parameters of the binocular camera; the motion physical quantities of the moving target are calculated based on the motion trajectory.

[0121] Depth mapping is a crucial step in converting disparity data into physical depth. Its core principle is to utilize the geometric relationships of binocular vision, using calibrated camera parameters (focal length, baseline) to map pixel-level disparity. Mapped to the physical depth of a spatial point from the camera This provides a foundation for 3D coordinate transformation. Depth calculation in binocular vision is based on the principle of triangulation: the optical centers of the left and right cameras are respectively... , Baseline distance ; spatial point The projection points on the left and right images are respectively , Due to polar alignment, Parallax .

[0122] According to the principle of similar triangles, there exists a geometric relationship:

[0123] ;

[0124] According to parallax data Baseline distance to binocular cameras Calculate the depth information of each point on the moving target. ,in satisfy:

[0125] ;

[0126] in, This is the equivalent focal length of the camera; This represents the disparity offset, which is obtained by converting the minimum disparity value minDisparity, as shown in Example 1. Since the value is 0.02mm, it can be ignored, so the simplified formula is: ; This is the optimized effective disparity value (with invalid pixels and noise removed).

[0127] Directly calculated depth maps may contain noise and holes (corresponding to invalid pixels due to parallax), requiring targeted optimization based on the weightlifting scenario: Hole filling: For hole pixels in the barbell area (such as local areas obscured by the athlete), a distance-based interpolation method is used: Centering on the hole pixel, search for effective depth values ​​within a 3×3 neighborhood, and fill with a distance-weighted average (closer distances have higher weights) to ensure the continuity of barbell depth; Depth smoothing: A bilateral filter is used to smooth the depth map with a 5×5 kernel size, a color similarity sigma value of 10, and a spatial distance sigma value of 5. This filter preserves abrupt changes in barbell edge depth while suppressing background noise (such as minor fluctuations in ground depth); Outlier removal: Based on the depth range of the weightlifting scenario (0.5m~3.0m), depth values ​​exceeding this range (such as depth values ​​from distant walls or ceilings) are removed to avoid interfering with barbell depth extraction.

[0128] Combined with the depth information Coordinates of the moving target in the image coordinate system Convert it to three-dimensional coordinates in the camera coordinate system. The transformation relation satisfies:

[0129] ;

[0130] in, Indicates the coordinates of the principal point; Indicates pixel focal length;

[0131] 3D coordinates acquired frame by frame Connecting them in chronological order, they form the trajectory of the moving target in three-dimensional space.

[0132] Obtain the pre-calibrated reference height of the binocular camera center relative to the reference plane. And extracting the three-dimensional coordinates of the center point of the moving target from the motion trajectory. ; Calculate the height of the moving target , satisfy:

[0133] ;

[0134] in, Represents the coordinate components in the vertical direction in the camera coordinate system;

[0135] Extract the vertical coordinates of the center point of the moving target from the motion trajectory for two consecutive frames. and ;

[0136] Obtain the time interval between adjacent frames ,in , Set the frame rate for the stereo camera; calculate the ascent velocity of the moving target. , satisfy:

[0137] ;

[0138] Height calculated frame by frame and rate of ascent Arranged in chronological order, generate curves showing changes in altitude and velocity of the moving target.

[0139] Based on the actual needs of weightlifting scenarios, the barbell height measurement process is as follows:

[0140] Extract the depth value of the barbell center from the depth map. Substituting the values ​​into the 3D coordinate transformation formula, we obtain the coordinates in the camera coordinate system. The reference height is calculated from the ground to the center of the binocular camera. The center height of the barbell is .

[0141] This invention achieves high-precision tracking of target trajectories and real-time measurement of key physical quantities in weightlifting scenarios by optimizing the entire process of detection and tracking, binocular vision processing, and physical quantity measurement.

[0142] Example 2

[0143] This invention also proposes a device for tracking the trajectory of moving targets and measuring physical quantities based on binocular vision. Figure 3 This is a schematic diagram of a device for tracking the trajectory of a moving target and measuring physical quantities based on binocular vision, as proposed in Embodiment 2 of the present invention. At the hardware level, the electronic device 300 includes: a binocular camera for acquiring a sequence of stereo images of the moving target; and a processor 310, optionally including an internal bus 320, a network interface 330, and a memory. The processor 310 is connected to both the binocular camera and the memory. The memory may include RAM, such as high-speed random-access memory (RAM), and may also include non-volatile memory, such as at least one disk storage device. Of course, this electronic device may also include other hardware required for other business operations.

[0144] For example, the device may also include a display unit for real-time display of the detection and tracking results of the moving target, its three-dimensional motion trajectory, and the change curves of physical quantities such as height and speed. It may also include a communication unit for transmitting measurement data to external devices (such as mobile phones, tablets, and computers) for further analysis and display.

[0145] The processor 310, network interface 330, and memory can be interconnected via an internal bus 320. This internal bus 320 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be categorized as an address bus, data bus, control bus, etc. For ease of illustration, only a single bidirectional arrow is used in this diagram, but this does not imply that there is only one bus or one type of bus. The memory is used to store programs. Specifically, the program can include program code, which includes computer operation instructions. The memory can include main memory 340 and non-volatile memory 350, and provides instructions and data to the processor 310.

[0146] Processor 310 reads the corresponding computer program from non-volatile memory 350 into memory 340 and then runs it, forming a device for locating the target user at the logical level. Processor 310 executes the program stored in memory and specifically performs the following:

[0147] In step S1, a sequence of stereo images of the moving target is acquired by the binocular camera;

[0148] In step S2, in the starting frame of the stereo image sequence, a moving target is identified and its initial position is determined by a target detection model; a tracking query vector is generated based on the initial position, and the moving target is tracked in subsequent frames according to the tracking query vector. The bounding box of the moving target in the image coordinate system is output frame by frame as a continuous target region.

[0149] In step S3, based on the pre-calibrated binocular camera parameters, epipolar correction is performed on each frame of the stereo image sequence to obtain epipolar-aligned left and right corrected images; based on the left and right corrected images, disparity is calculated in the continuous target area to obtain disparity data corresponding to the moving target.

[0150] In step S4, the motion trajectory of the moving target in three-dimensional space is obtained based on the parallax data and the calibration parameters of the binocular camera; the motion physical quantities of the moving target are calculated based on the motion trajectory.

[0151] Figure 1 It can be applied to processor 310, or implemented by processor 310. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit in the hardware of the processor or by instructions in the form of software. The processor mentioned above can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0152] Of course, in addition to software implementation, the electronic device of this application does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. In other words, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0153] Example 3

[0154] The present invention also proposes a readable storage medium on which a computer program is stored, wherein the computer program, when executed by a processor, implements the following method steps:

[0155] In step S1, a sequence of stereo images of the moving target is acquired by the binocular camera;

[0156] In step S2, in the starting frame of the stereo image sequence, a moving target is identified and its initial position is determined by a target detection model; a tracking query vector is generated based on the initial position, and the moving target is tracked in subsequent frames according to the tracking query vector. The bounding box of the moving target in the image coordinate system is output frame by frame as a continuous target region.

[0157] In step S3, based on the pre-calibrated binocular camera parameters, epipolar correction is performed on each frame of the stereo image sequence to obtain epipolar-aligned left and right corrected images; based on the left and right corrected images, disparity is calculated in the continuous target area to obtain disparity data corresponding to the moving target.

[0158] In step S4, the motion trajectory of the moving target in three-dimensional space is obtained based on the parallax data and the calibration parameters of the binocular camera; the motion physical quantities of the moving target are calculated based on the motion trajectory.

[0159] Embodiment 3 of this application also provides a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, such as a memory that stores a computer program, which can be executed by a processor to complete the steps described in the aforementioned method. The computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disc, or CD-ROM.

[0160] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks. Alternatively, if the integrated units of this application are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0161] The present application provides a device for tracking the trajectory of a moving target and measuring physical quantities based on binocular vision in embodiment 2. The description of the relevant part of the storage medium for tracking the trajectory of a moving target and measuring physical quantities based on binocular vision in embodiment 3 of the present application can be found in the detailed description of the corresponding part of the method for tracking the trajectory of a moving target and measuring physical quantities based on binocular vision in embodiment 1 of the present application, and will not be repeated here.

[0162] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that the elements inherent in a process, method, article, or apparatus that includes a list of elements are included. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. Additionally, portions of the technical solutions provided in the embodiments of this application that are consistent with the implementation principles of corresponding technical solutions in the prior art have not been described in detail to avoid excessive elaboration.

[0163] While specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art can make other modifications or variations based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for tracking the trajectory of a moving target and measuring physical quantities based on binocular vision, characterized in that, Includes the following steps: Acquire a sequence of stereo images of a moving target captured by a binocular camera; In the initial frame of the stereo image sequence, a moving target is identified and its initial position is determined using a target detection model; A tracking query vector is generated based on the initial position. The moving target is tracked in subsequent frames according to the tracking query vector, and the bounding box of the moving target in the image coordinate system is output frame by frame as a continuous target region. Specifically, the starting frame of the image frame is input to the target detection model; the target detection model outputs the initial bounding box of the moving target, which is characterized by the coordinates of the center point of the initial bounding box and the size of the initial bounding box; a tracking query vector is generated based on the coordinates of the center point of the initial bounding box, which contains the initial position of the moving target in the starting frame and the starting frame identifier. The tracking query vector is input into the tracking model, which tracks the moving target frame by frame in the subsequent frames of the stereo image sequence according to the tracking query vector to obtain the position vector of the tracking point; according to the position vector, the tracking bounding box of the moving target in the image coordinate system is output frame by frame. The tracking bounding box is characterized by the coordinates of the center point of the tracking bounding box and the size of the tracking bounding box. The tracking bounding box output frame by frame is used as the continuous target region of the moving target. Based on pre-calibrated binocular camera parameters, epipolar correction is performed on each frame of the stereo image sequence to obtain epipolar-aligned left and right corrected images; based on the left and right corrected images, disparity is calculated in the continuous target region to obtain disparity data corresponding to the moving target; specifically, horizontal Sobel operator edge enhancement processing is performed on the local images in the left and right corrected images corresponding to the continuous target region. On the edge-enhanced local image, a semi-global block matching algorithm is used to calculate the initial disparity. The matching cost calculation window size of this algorithm is a preset window size, and the smoothing penalty coefficient is set to... and ;and The matching cost is accumulated through dynamic programming along multiple preset directions, and the disparity with the minimum total cost is selected as the initial disparity of the pixel. The initial disparity is subjected to left-right consistency checks, uniqueness checks, and mismatch removal to generate optimized disparity data corresponding to the moving target. ; Based on the parallax data and the calibration parameters of the binocular camera, the motion trajectory of the moving target in three-dimensional space is obtained; based on the motion trajectory, the motion physical quantities of the moving target are calculated.

2. The method for tracking moving targets and measuring physical quantities based on binocular vision according to claim 1, characterized in that, After acquiring the stereo image sequence of the moving target captured by the binocular camera, the process also includes: The stereo image sequence is preprocessed to obtain the frame rate and image height from the metadata of the stereo image sequence, and the original image frames are sampled in the time dimension at a preset sampling rate to obtain the sampled image frames.

3. The method for tracking moving targets and measuring physical quantities based on binocular vision according to claim 1, characterized in that, Based on pre-calibrated binocular camera parameters, epipolar correction is performed on each frame of the stereo image sequence to obtain epipolar-aligned left-right corrected images, specifically: Based on the distortion coefficients of the binocular camera, distortion is eliminated from the left and right images in the stereo image to obtain distortion-free left and right images; the distortion coefficients include radial distortion coefficients. and tangential distortion coefficient ;in, Indicates the second-order radial distortion coefficient; Indicates the fourth-order radial distortion coefficient; Represents the sixth-order radial distortion coefficient; Indicates the first tangential distortion coefficient; Indicates the second tangential distortion coefficient; Based on the intrinsic and extrinsic parameters of the binocular cameras, stereo correction is performed on the distortion-free left and right images to obtain epipolar-aligned corrected images; the intrinsic parameters include the focal lengths of the left and right cameras. and principal point coordinates The extrinsic parameters include the rotation matrix. Translation vector .

4. The method for tracking moving target trajectories and measuring physical quantities based on binocular vision according to claim 3, characterized in that, The distortion removal process is as follows: For any pixel in the left and right images, let the pixel coordinates in the original distorted image be... Convert its pixel coordinates to normalized image coordinates with the principal point as the origin and the focal length as the unit. ; Calculate the coordinates after radial distortion correction based on the radial distortion coefficient. , is represented as: ； in, ; The distance from any pixel to the origin; Calculate the normalized coordinates after tangential distortion correction based on the tangential distortion coefficient. , is represented as: ； Based on the intrinsic parameters, the normalized coordinates after tangential distortion correction are... Convert to corrected pixel coordinates , is represented as: ； in, The camera is in shaft and Pixel focal length along the axial direction; Principal point coordinates; Based on the corrected pixel coordinates Interpolation methods are used to extract the original distorted image. By sampling pixel values, a distortion-free image is reconstructed.

5. The method for tracking moving target trajectories and measuring physical quantities based on binocular vision according to claim 4, characterized in that, The stereo correction process is as follows: Based on rotation matrix Translation vector Calculate the correction rotation matrix for the left and right images. , and the corrected projection matrix , and depth mapping matrix ; Based on the correction rotation matrix of the left and right images , and projection matrix , Combined with intrinsic parameters, a pixel mapping table for the left and right images is generated; According to the pixel mapping table, the distortion-free left and right images are resampled to obtain epipolar aligned corrected left and right images, in which the corresponding pixels of the left and right images are located on the same horizontal scan line.

6. The method for tracking moving target trajectories and measuring physical quantities based on binocular vision according to claim 1, characterized in that, Based on the parallax data and the calibration parameters of the binocular camera, the motion trajectory of the moving target in three-dimensional space is obtained, specifically as follows: According to parallax data Baseline distance to binocular cameras Calculate the depth information of each point on the moving target. ,in satisfy: ； in, This is the equivalent focal length of the camera; Indicates the parallax offset; Combined with the depth information Coordinates of the moving target in the image coordinate system Convert it to three-dimensional coordinates in the camera coordinate system. The transformation relation satisfies: ； in, Indicates the coordinates of the principal point; Indicates pixel focal length; 3D coordinates acquired frame by frame Connecting them in chronological order, they form the trajectory of the moving target in three-dimensional space.

7. The method for tracking moving targets and measuring physical quantities based on binocular vision according to claim 6, characterized in that, Based on the motion trajectory, the motion physical quantities of the moving target are calculated, specifically: obtaining the pre-calibrated reference height of the binocular camera center relative to the reference plane. And extracting the three-dimensional coordinates of the center point of the moving target from the motion trajectory. ; Calculate the height of the moving target , satisfy: ； in, This represents the coordinate components in the vertical direction of the camera coordinate system, with downward being the positive direction; Extract the vertical coordinates of the center point of the moving target from the motion trajectory for two consecutive frames. and , Obtain the time interval between adjacent frames ,in , Set the frame rate for the stereo camera; calculate the ascent velocity of the moving target. , satisfy: ； Height calculated frame by frame and rate of ascent Arranged in chronological order, generate curves showing changes in altitude and velocity of the moving target.

8. A device for tracking the trajectory of a moving target and measuring physical quantities based on binocular vision, characterized in that, include: Binocular camera, used to acquire stereo image sequences of moving targets; Memory, used to store executable instructions; The processor is connected to the binocular camera and the memory respectively, and is used to execute executable instructions stored in the memory to implement a method for tracking the trajectory of a moving target and measuring physical quantities based on binocular vision as described in any one of claims 1 to 6.

Images