A method and device for reconstructing a 3D trajectory of a tennis ball based on monocular vision

Abstract

A tennis 3D trajectory reconstruction method and device based on monocular vision mainly include: real-time acquisition of monocular motion video; extracting the 2D pixel bounding box of the tennis in each frame, and splicing out the complete tennis 2D motion trajectory; taking the tennis space motion physical model containing gravitational acceleration and air resistance as the constraint condition, constructing the re-projection error function from the 3D space coordinate to the 2D pixel coordinate, taking the initial 3D position and the initial velocity vector of the tennis as the joint optimization variable, and adopting the nonlinear optimization algorithm for iterative fitting, so that the re-projection error between the coordinates of the theoretical 3D trajectory re-projected to the 2D plane and the actual tennis 2D motion trajectory is minimized, thereby inversely solving the accurate 3D trajectory. Through the 3D reconstruction method based on 2D trajectory fitting and projection iterative optimization, the tracking and depth estimation problem of high-speed tennis in the terminal with limited computing power can be effectively solved.

Description

Technical Field

[0001] This invention relates to the field of machine vision technology, and in particular to a method and apparatus for reconstructing the 3D trajectory of a tennis ball based on monocular vision. Background Technology

[0002] With the development of smart sports and computer vision, 3D reconstruction of motion trajectories has been widely used in sports event refereeing assistance (such as the "Hawk-Eye" system), athlete training analysis, and mass sports entertainment.

[0003] Tennis balls are not only small in size, but also fly at extremely high speeds (serving speeds can exceed 200 km / h). When they are moving at high speeds, the images captured by cameras are often blurry pixelation with motion blur, and they are easily confused with the background of the court (such as white lines and stands). This is a typical problem of "high-speed, small target detection and tracking".

[0004] Existing technological solutions mainly include:

[0005] Multi-view / binocular vision systems (such as traditional Hawk-Eye technology): use multiple (usually 6-10) high-speed cameras to capture images simultaneously from different angles. Through extremely rigorous camera joint calibration, the 3D spatial coordinates of the tennis ball are calculated using the principles of stereo matching and triangulation.

[0006] Traditional monocular vision systems use ordinary background subtraction or basic 2D object detection algorithms (such as the YOLO version) to extract the 2D pixel coordinates of the tennis ball, and then rely solely on physical parabolic models and simple ground plane assumptions (such as assuming that the tennis ball only moves on a specific plane or must hit the ground to be measured) to roughly infer the 3D trajectory.

[0007] Disadvantages of existing technology:

[0008] Excessive hardware and computing costs: Binocular / multi-view systems require multiple expensive high-speed industrial cameras, strict clock synchronization hardware, and high-performance GPU workstations to process massive multi-view image data, making it impossible to scale up to ordinary edge computing devices (such as smartphones and lightweight computing boxes).

[0009] Deployment and calibration are extremely complicated: multi-camera systems require harsh on-site installation environments, and once a camera undergoes even a slight displacement, complex joint calibration must be performed again.

[0010] Monocular reconstruction suffers from low accuracy and poor robustness: Existing monocular technology cannot effectively solve the problem of missing depth information, and when faced with background interference and high-speed blurring, 2D detection is prone to missing detection, causing subsequent 3D fitting to completely fail. Summary of the Invention

[0011] To address the aforementioned issues, this disclosure provides a monocular vision tennis 3D trajectory reconstruction system and method. Through a 3D reconstruction method based on 2D trajectory fitting and projection iterative optimization, it can solve the problems of high-speed small target detection, tracking, and depth estimation while significantly reducing hardware costs, computing power thresholds, and deployment difficulties. This achieves high-precision tennis 3D trajectory reconstruction, velocity measurement, and landing point calculation comparable to multi-view systems.

[0012] The monocular vision tennis ball 3D reconstruction method disclosed herein mainly includes the following steps:

[0013] S1, perform a one-time offline calibration before running;

[0014] S2 uses a monocular image acquisition device on the application terminal to acquire real-time tennis video.

[0015] S3, extract the 2D pixel bounding box of the tennis ball in each frame, and stitch it together on the 2D image plane to form the complete 2D motion trajectory of the tennis ball;

[0016] S4 uses a tennis ball spatial motion physics model that includes gravitational acceleration and air resistance as constraints, and constructs a reprojection error function from 3D spatial coordinates to 2D pixel coordinates. The initial 3D position and initial velocity vector of the tennis ball are used as joint optimization variables. A nonlinear optimization algorithm is used for iterative fitting to minimize the reprojection error between the coordinates of the theoretical 3D trajectory reprojected onto the 2D plane and the actual 2D motion trajectory of the tennis ball extracted in step S3, thereby solving for the accurate 3D trajectory.

[0017] Furthermore, step S1 specifically includes:

[0018] The intrinsic parameter matrix K of the monocular camera, including the focal length f, was obtained in advance using Zhang Zhengyou's calibration method combined with a checkerboard pattern. x f y and principal point coordinates c x c y and distortion coefficient;

[0019] Using the standard geometric dimensions of a tennis court, the corresponding pixel corners in the image are extracted. The PnP algorithm is used to calculate the camera's extrinsic parameters relative to the tennis court, including the rotation matrix R and translation vector T, to establish a unified 3D world coordinate system.

[0020] Furthermore, step S3 specifically includes:

[0021] A lightweight object detection algorithm is used to extract the 2D pixel bounding boxes of tennis balls in each frame;

[0022] By combining multi-target tracking algorithms, a complete 2D trajectory of a tennis ball is stitched together on a 2D image plane.

[0023] Furthermore, step S4 specifically includes:

[0024] S41, Establish the equations for the perspective projection model: In the formula, (u,v) are 2D pixel coordinates, (X,Y,Z) are 3D spatial coordinates, K is the intrinsic parameter, and [R|t] is the extrinsic parameter.

[0025] S42, Add physical constraints: Introduce a spatial motion physics model of tennis balls as a constraint condition, in which the parabolic deformation caused by gravitational acceleration and air resistance is considered;

[0026] S43, Nonlinear optimization: Input the known 2D trajectory (u, v, t) of consecutive frames to construct the reprojection error function;

[0027] The Levenberg-Marquardt algorithm or Gauss-Newton method is used to iteratively optimize and fit the initial velocity vector and initial 3D position of the tennis ball. This minimizes the error between the coordinates of the 3D theoretical trajectory derived from the tennis ball's spatial motion physics model and the coordinates projected onto the 2D plane through the perspective projection model equation and the actual detected 2D trajectory, thus solving for the accurate 3D trajectory.

[0028] Furthermore, the method also includes the following steps:

[0029] The 2D trajectory obtained in step S3 is filled and smoothed using cubic spline interpolation or Kalman filtering.

[0030] Furthermore, the method also includes the following steps:

[0031] Based on the obtained accurate 3D trajectory, the instantaneous velocity and landing point of the tennis ball are calculated.

[0032] Furthermore, specific methods for calculating instantaneous velocity and impact point include:

[0033] (1) Instantaneous velocity calculation: Instantaneous velocity is calculated using a time sliding window. Assume the video frame rate is FPS and the time interval is... The instantaneous spatial velocity at frame t The calculation formula is:

[0034]

[0035] Indicates the number of frames contained in the sliding window;

[0036] (2) Calculation of landing point:

[0037] The method of Z-axis extreme value detection combined with fitting is used to find the frame tt with the lowest z-axis, and 3D curve fitting is performed on the trajectory of the m frames before and n frames after tt respectively. The intersection point obtained by the two curve fittings is taken as the landing point.

[0038] A monocular vision-based 3D tennis trajectory reconstruction device applying the above method mainly includes:

[0039] The camera and site offline calibration module is used to perform a one-time offline calibration before the system is run.

[0040] The real-time tennis video acquisition module is used to acquire real-time tennis motion videos through a monocular image acquisition device.

[0041] The 2D trajectory acquisition module is used to extract the 2D pixel bounding boxes of the tennis ball in each frame and stitch them together on the 2D image plane to form a complete 2D motion trajectory of the tennis ball.

[0042] 3D trajectory reconstruction module: Based on the perspective projection relationship between two-dimensional and three-dimensional planes, as well as the spatial motion constraints of tennis balls, iteratively optimizes the derived and fitted 3D trajectory to obtain an accurate 3D trajectory.

[0043] A storage medium containing a computer program for performing the above-described method.

[0044] The main technical approaches of this solution include:

[0045] (1) System architecture: It relies solely on a monocular camera, combined with offline site / camera calibration, and uses high-precision binocular data as ground truth to train the end-to-end monocular 3D reconstruction network;

[0046] (2) Core algorithm and lightweight: Under limited computing power, iterative optimization based on perspective projection relationship is used to solve the "tracking" problem of high-speed and weak targets such as tennis balls and establish 3D trajectory;

[0047] (3) Temporal features to solve blur and occlusion: The temporal features of multiple consecutive frames of images are used to solve the problem of severe motion blur and occlusion of tennis balls caused by high-speed motion in a single frame.

[0048] Compared with the prior art, the beneficial effects of this disclosure are:

[0049] ① Extremely low hardware cost and computing power threshold: Compared with binocular / multi-camera systems, it eliminates the need for multiple expensive cameras and computing power equipment. Only a single ordinary camera is needed. Through the lightweight design of the algorithm, it can run smoothly on ordinary edge computing devices with limited computing power.

[0050] ② Extremely simple deployment method: It avoids the complexities of binocular joint calibration. Only a simple monocular camera and field line calibration is required during system initialization, which greatly improves the tolerance to site environment and installation location.

[0051] ③ Breaking through the ceiling of monocular accuracy: Compared with traditional monocular algorithms, this disclosure utilizes a 3D reconstruction method based on 2D trajectory fitting and projection iterative optimization, whose monocular 3D reconstruction accuracy far exceeds that of traditional methods that rely on simple physical assumptions. Attached Figure Description

[0052] The above and other objects, features and advantages of this disclosure will become more apparent from the more detailed description of exemplary embodiments of this disclosure taken in conjunction with the accompanying drawings, in which the same reference numerals generally represent the same components.

[0053] Figure 1 This is a flowchart of a monocular tennis ball 3D trajectory reconstruction method according to the present disclosure. Detailed Implementation

[0054] Preferred embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While preferred embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure will be thorough and complete, and will fully convey the scope of the present disclosure to those skilled in the art.

[0055] This disclosure provides a method for reconstructing the 3D trajectory of tennis balls based on monocular vision. It mainly addresses the technical problems of existing high-precision tennis ball 3D trajectory reconstruction and speed measurement systems (such as binocular / multi-view vision systems) in digital scenarios of sports (tennis) events or daily training, which are characterized by high hardware costs, demanding deployment environments, and extremely high requirements for computing resources (unable to run on edge devices or devices with limited computing power).

[0056] Meanwhile, for monocular vision systems, this disclosure solves the technical problems that traditional algorithms are prone to motion blur, occlusion, and severe loss of depth information when facing high-speed, small moving targets such as tennis balls, which leads to low 3D reconstruction accuracy and easy loss of trajectory tracking.

[0057] In one exemplary implementation:

[0058] The overall architecture of a monocular tennis ball 3D reconstruction system / method according to this disclosure is attached. Figure 1 As shown. Specifically, it includes the following key modules and steps:

[0059] 1. Offline camera and site positioning module

[0060] Implementation: The system performs a one-time offline calibration before operation.

[0061] First, the intrinsic parameter matrix K (including focal lengths f_x, f_y and principal point coordinates c_x, c_y) and distortion coefficients of the monocular camera are obtained in advance using the Zhang Zhengyou calibration method combined with the chessboard grid.

[0062] Secondly, using the standard geometric dimensions of the tennis court (such as the physical length and intersection of the service line and sidelines), the corresponding pixel corner points in the image are extracted. The PnP (Perspective-n-Point) algorithm is used to calculate the external parameters of the camera relative to the tennis court (rotation matrix R and translation vector T) to establish a unified 3D world coordinate system.

[0063] 2. Robust 2D Detection and Tracking: Lightweight target detection algorithms (such as the YOLO series) are used to extract the 2D pixel bounding boxes of the tennis ball in each frame. Combined with multi-target tracking algorithms (such as DeepSORT or ByteTrack), the complete 2D motion trajectory of the tennis ball is stitched together on the 2D image plane.

[0064] 3. 2D trajectory smoothing and interpolation: For missed frames caused by blurring, cubic spline interpolation or Kalman filtering is used to fill and smooth the 2D trajectory.

[0065] 4. Iterative optimization solution based on perspective projection relationship:

[0066] (1) Establish the equations for the perspective projection model:

[0067] Where (u,v) are 2D pixel coordinates, (X,Y,Z) are 3D spatial coordinates, K is the intrinsic parameter, and [R|t] is the extrinsic parameter;

[0068] S is the scale factor, representing the depth Zc in the camera coordinate system. This scale factor is needed because the monocular camera loses depth. 1 is a homogeneous coordinate constant added to meet the dimension requirements of matrix multiplication.

[0069] Typically, after calculating the right side of the equation to obtain a 3-dimensional vector, dividing the vector by the value of its third dimension (i.e., s) will give us [u, v, 1]. T .

[0070] (2) Incorporating physical constraints: Since the depth Z is lost in monocular vision, the above formula alone cannot solve the problem. A spatial motion physics model of tennis is introduced, which considers the parabolic deformation caused by gravitational acceleration and air resistance as constraints.

[0071] (3) Nonlinear optimization: Input the known 2D trajectory (u, v, t) of consecutive frames to construct the reprojection error function. Use the Levenberg-Marquardt (LM) algorithm or Gauss-Newton method to iteratively optimize and fit the initial velocity vector and initial 3D position of the tennis ball, so that the coordinates of the 3D trajectory derived from the physical model and the coordinates projected back onto the 2D plane have the smallest error with the actual detected 2D trajectory, thus solving for the accurate 3D trajectory.

[0072] The reprojection error function is a cost function that involves "differentiation". In this embodiment:

[0073] Error function = ||(actually detected 2D pixel coordinates) - (theoretical 2D pixel coordinates calculated based on the physical model and perspective projection equations)|| 2

[0074] The goal of iterative optimization in this step is to minimize this difference (error).

[0075] 5. Based on the timestamp and the difference between the 3D coordinates, the flight trajectory of the tennis ball is calculated using smoothing algorithms such as Kalman filtering to obtain the instantaneous speed and landing point after the continuous 3D coordinate points are output.

[0076] (1) Instantaneous velocity calculation: Since single-frame calculation at high frame rates is easily affected by small noise, this method adopts a time sliding window. Calculate the instantaneous velocity (e.g., at intervals of 3-5 frames). Assume the video frame rate is FPS, and the time interval... The instantaneous spatial velocity at frame t. The calculation formula is:

[0077]

[0078] (2) High-precision calculation of impact point (point of impact):

[0079] The actual landing point of a tennis ball often occurs between two frames, and simply taking the frame with the lowest Z-axis value as the landing point results in a large error. This system uses a method of Z-axis extreme value detection combined with fitting to find the frame tt with the lowest Z-axis value, and performs 3D curve fitting on the trajectory of the m frames before and n frames after tt respectively. The intersection point obtained from the two curve fittings is taken as the landing point.

[0080] In this embodiment, under the condition of having only "one eye" and "limited brain computing power" (constrained computing devices), a 3D reconstruction method based on 2D trajectory fitting and projection iterative optimization is used. The pure 2D pixel trajectories of all known continuous frames are input to construct a global "reprojection error cost function". Through the Levenberg-Marquardt (LM) algorithm or Gauss-Newton method, nonlinear iterative gradient descent is performed on position and velocity in multidimensional space to find the global optimal solution that minimizes the overall 2D reprojection error, thereby accurately obtaining the 3D spatial position of the sphere.

[0081] The above technical solutions are merely exemplary embodiments of the present invention. For those skilled in the art, based on the application methods and principles disclosed in the present invention, it is easy to make various types of improvements or modifications, and not limited to the methods described in the specific embodiments of the present invention. Therefore, the methods described above are merely preferred and not restrictive.

Claims

1. A method for reconstructing the 3D trajectory of a tennis ball based on monocular vision, characterized in that, Includes the following steps: S1, perform offline calibration before running; S2 acquires real-time tennis video through a monocular image acquisition device; S3, extract the 2D pixel bounding box of the tennis ball in each frame, and stitch it together on the 2D image plane to form the complete 2D motion trajectory of the tennis ball; S4 uses a tennis ball spatial motion physics model that includes gravitational acceleration and air resistance as constraints, and constructs a reprojection error function from 3D spatial coordinates to 2D pixel coordinates. The initial 3D position and initial velocity vector of the tennis ball are used as joint optimization variables. A nonlinear optimization algorithm is used for iterative fitting to minimize the reprojection error between the coordinates of the theoretical 3D trajectory reprojected onto the 2D plane and the actual 2D motion trajectory of the tennis ball extracted in step S3, thereby solving for the accurate 3D trajectory.

2. The method according to claim 1, characterized in that, Step S1 specifically includes: The intrinsic parameter matrix K of the monocular camera, including the focal length f, was obtained in advance using Zhang Zhengyou's calibration method combined with a checkerboard pattern. x f y and principal point coordinates c x c y and distortion coefficient; Using the standard geometric dimensions of a tennis court, the corresponding pixel corners in the image are extracted. The PnP algorithm is used to calculate the camera's extrinsic parameters relative to the tennis court, including the rotation matrix R and translation vector T, to establish a unified 3D world coordinate system.

3. The method according to claim 1, characterized in that, Step S3 specifically includes: A lightweight object detection algorithm is used to extract the 2D pixel bounding boxes of tennis balls in each frame; By combining multi-target tracking algorithms, a complete 2D trajectory of a tennis ball is stitched together on a 2D image plane.

4. The method according to claim 1, characterized in that, Step S4 specifically includes: S41, Establish the equations for the perspective projection model: In the formula, (u,v) are 2D pixel coordinates, (X,Y,Z) are 3D spatial coordinates, K is the intrinsic parameter, and [R|t] is the extrinsic parameter. S42, Add physical constraints: Introduce a spatial motion physics model of tennis balls as a constraint condition, in which the parabolic deformation caused by gravitational acceleration and air resistance is considered; S43, Nonlinear optimization: Input the known 2D trajectory (u, v, t) of consecutive frames to construct the reprojection error function; The Levenberg-Marquardt algorithm or Gauss-Newton method is used to iteratively optimize and fit the initial velocity vector and initial 3D position of the tennis ball. This minimizes the error between the coordinates of the 3D theoretical trajectory derived from the tennis ball's spatial motion physics model and the coordinates projected onto the 2D plane through the perspective projection model equation and the actual detected 2D trajectory, thus solving for the accurate 3D trajectory.

5. The method according to claim 1, characterized in that, It also includes the following steps: The 2D trajectory obtained in step S3 is filled and smoothed using cubic spline interpolation or Kalman filtering.

6. The method according to claim 1, characterized in that, It also includes the following steps: Based on the obtained accurate 3D trajectory, the instantaneous velocity and landing point of the tennis ball are calculated.

7. The method according to claim 6, characterized in that, Specific methods for calculating instantaneous velocity and impact point include: (1) Instantaneous velocity calculation: Instantaneous velocity is calculated using a time sliding window. Assume the video frame rate is FPS and the time interval is... The instantaneous spatial velocity at frame t The calculation formula is: Indicates the number of frames contained in the sliding window; (2) Calculation of landing point: The method of Z-axis extreme value detection combined with fitting is used to find the frame tt with the lowest z-axis, and 3D curve fitting is performed on the trajectory of the m frames before and n frames after tt respectively. The intersection point obtained by the two curve fittings is taken as the landing point.

8. A monocular vision-based 3D trajectory reconstruction device for tennis balls using the method described in any one of claims 1-7, characterized in that, include: The camera and site offline calibration module is used to perform offline calibration before the system is run. The real-time tennis video acquisition module is used to acquire real-time tennis motion videos through a monocular image acquisition device. The 2D trajectory acquisition module is used to extract the 2D pixel bounding boxes of the tennis ball in each frame and stitch them together on the 2D image plane to form a complete 2D motion trajectory of the tennis ball. 3D trajectory reconstruction module: Based on the perspective projection relationship between two-dimensional and three-dimensional planes, as well as the spatial motion constraints of tennis balls, iteratively optimizes the derived and fitted 3D trajectory to obtain an accurate 3D trajectory.

9. A storage medium containing a computer program for performing the method of any one of claims 1-7.

Images