A Method for Reconstructing and Predicting 3D Trajectory of Tennis Balls Based on Monocular Video

Abstract

This invention relates to a method for reconstructing and predicting the 3D trajectory of a tennis ball based on monocular video. The method includes: matching key points on a 2D court with corresponding points on a 3D standard court to calculate the camera's intrinsic and extrinsic parameters, establishing a 3D world coordinate system based on the 3D standard court; using the known 3D coordinates of racket key points, combined with their 2D coordinates, calculating the racket's 3D depth and pose within the monocular video frame to determine the position of the hitting point in 3D space; and setting hitting point and landing point constraints, adjusting the input initial velocity vector and spin vector, and using physical motion equations to construct a simulated 3D trajectory passing through the hitting and landing points, then projecting the simulated 3D trajectory onto a 2D plane and fitting it with a 2D pixel trajectory to construct the optimal 3D trajectory. This method can reconstruct the 3D trajectory and output the ball's true velocity, net clearance height, and landing depth, meeting the needs of professional quantitative analysis.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method for reconstructing and predicting the three-dimensional trajectory of a tennis ball based on monocular video. Background Technology

[0002] Ball sports (such as table tennis, badminton, tennis, volleyball, and football) are typically played on standard courts of specified dimensions, involving high-intensity competition and characterized by high ball speed and varied trajectories. In recent years, with the development of computer vision and image processing technologies, video-based motion analysis has been widely applied in sports research and training support, becoming an important means of improving technical and tactical skills.

[0003] Taking tennis as an example, existing visual analysis systems are mainly divided into two categories: one is a high-end 3D acquisition system based on multi-view vision, and the other is a low-cost 2D analysis solution based on a monocular camera. However, both of these solutions still have the following significant drawbacks in practical applications: Lack of physical depth information: Existing monocular video analysis methods typically only acquire pixel coordinates on the image plane, and cannot directly measure the ball's actual flight speed, net clearance height, or impact depth. This lack of physical measurement in determining "fast" and "slow" cannot meet the needs of professional quantitative analysis.

[0004] There is visual perspective ambiguity: In a monocular viewpoint, the loss of depth information leads to multiple interpretations of trajectory judgment. For example, a trajectory that appears to be vertically ascending in an image could be either a high-reaching ball or a flat-hit ball flying towards the camera. Existing models, relying solely on the two-dimensional trajectory shape, are prone to misjudgment, resulting in significant deviations in landing point prediction.

[0005] Unable to resolve rotational characteristics: The core of tennis lies in rotation (topspin, backspin, sidespin). In two-dimensional images, the trajectory curvature caused by rotation is easily confused with the perspective distortion of a wide-angle lens. Existing two-dimensional models cannot distinguish between the physical Magnus effect and visual distortion.

[0006] Poor occlusion resistance: In 2D tracking, once the tennis ball is obstructed by the net, player's body, or racket, tracking immediately stops. Simple linear interpolation cannot simulate the non-linear flight of the tennis ball affected by gravity and drag, resulting in large frame interpolation errors.

[0007] Lack of utilization of the racket: Most existing solutions only focus on the ball itself, ignoring the racket as a key interactive medium. In fact, the ball's motion is entirely determined by the racket's posture at the moment of impact; ignoring the racket's three-dimensional posture is equivalent to losing the most crucial causal information. Summary of the Invention

[0008] To address at least some of the problems mentioned above in the prior art, the present invention provides a method for reconstructing and predicting the three-dimensional trajectory of a tennis ball based on monocular video, comprising: Two-dimensional pixel coordinates of the tennis ball's center point are extracted from multiple consecutive monocular video frames to form a two-dimensional pixel trajectory, and two-dimensional racket border, two-dimensional racket key points, two-dimensional court key points, and court lines are extracted. By matching the key points of the 2D court with the corresponding points of the 3D standard court, the intrinsic and extrinsic parameters of the camera are calculated, and a 3D world coordinate system based on the 3D standard court is established. Using the known 3D coordinates of key points on the racket in 3D, combined with the 2D coordinates of key points in 2D, the 3D depth and pose of the racket in a monocular video frame are calculated to determine the position of the hitting point in 3D space; and By setting constraints on the hitting point and landing point, and adjusting the input initial velocity vector and spin vector, a simulated three-dimensional trajectory passing through the hitting point and landing point is constructed using the physical motion equations. The simulated three-dimensional trajectory is then projected onto a two-dimensional plane and fitted with a two-dimensional pixel trajectory to construct the optimal three-dimensional trajectory.

[0009] Furthermore, it also includes: obtaining the three-dimensional position of the tennis ball and the three-dimensional posture and position of the racket over a period of time based on the reconstructed optimal three-dimensional trajectory, constructing multimodal temporal information as input, and using a temporal deep neural network containing an attention mechanism to predict the three-dimensional position and final landing point of the tennis ball over a future period of time.

[0010] Furthermore, using the known 3D coordinates of the racket's key points in 3D, combined with the 2D coordinates of the racket's key points in 2D, a perspective N-point localization algorithm is used to calculate the racket's 3D depth and pose in a monocular video frame, thereby determining the position of the hitting point in 3D space, including: Assuming the three-dimensional coordinates of key points on a standard racket are in the local object coordinate system of the racket, It is known that the two-dimensional coordinates of the key points of the two-dimensional racket detected in a monocular video frame are: According to the principle of perspective projection, the relationship between the 3D coordinates of the key points on a 3D racket and the 2D coordinates of the key points on a 2D racket is as follows: , in For the camera intrinsic parameter matrix, and , respectively, are the rotation and translation matrices of the racket relative to the camera, and s represents the scale factor; Due to the three-dimensional coordinates of key points on a three-dimensional racket and the corresponding two-dimensional coordinates of the key points on the two-dimensional racket Given that the rotation matrix of the racket relative to the camera can be solved using a nonlinear optimization method. Translation matrix This allows us to obtain the three-dimensional position of the racket in each frame, and thus the three-dimensional position of the three-dimensional hitting point.

[0011] Furthermore, the striking point constraint includes: at the moment of impact The three-dimensional center point of a tennis ball Located on the three-dimensional racket face plane fitted by the three-dimensional key points of the racket, and within the physical boundary of the racket face; in simplified calculation, at the moment of impact, the Euclidean space distance between the three-dimensional center point of the tennis ball and the geometric center of the three-dimensional racket face reaches a minimum value. The landing point constraint is at the landing time The vertical height of the ball It is 0.

[0012] Furthermore, the three-dimensional position of the tennis ball It follows the physical equation of motion: , Where m is mass and g is gravity. This represents the three-dimensional acceleration vector of a tennis ball. Represents the spin vector. Represents the velocity vector; The air drag coefficient, The Magnus force coefficient; Numerical integration of the physical equations of motion yields the result over time. and initial state vector The deterministic three-dimensional trajectory function of evolution, denoted as ; The initial state vector to be solved is ,in Let be the initial velocity vector. is the spin vector.

[0013] Furthermore, the initial velocity vector and spin vector are iteratively adjusted using an optimization algorithm. A simulated three-dimensional trajectory is generated through the physical motion equations. The simulated three-dimensional trajectory is then projected onto a two-dimensional plane and fitted to the two-dimensional pixel trajectory. The comprehensive loss function value is calculated, and the optimal three-dimensional trajectory and the corresponding optimal initial velocity vector and optimal spin vector are solved with the goal of minimizing the loss function.

[0014] Furthermore, the comprehensive loss function includes reprojection error and boundary constraint error: , in This represents the reprojection error from the 3D trajectory to the 2D pixel plane. The boundary penalty term is the landing point height. For the penalty item at the point of impact, These are the coordinates of the geometric center of the three-dimensional racket face where the Euclidean distance between the racket face and the three-dimensional center point of the tennis ball reaches a minimum at the moment of impact. and Indicates weight, This represents the three-dimensional center point coordinates of the tennis ball at the moment of impact, where K represents the camera intrinsic parameters. Represent the two-dimensional coordinates of the tennis ball at time t. The coordinates of the simulated three-dimensional center point of the tennis ball at time t are projected onto a two-dimensional plane.

[0015] Furthermore, the two-dimensional racket key points and three-dimensional racket key points include the apex of the racket head, the center of the racket throat, the left edge of the racket face, the right edge of the racket face, and the bottom of the racket handle; The key points of the two-dimensional court include at least two of the following: the corner point of the service line, the corner point of the baseline, the corner point of the singles sideline, the corner point of the doubles sideline, the intersection of the center line and the service line, the intersection of the center line and the baseline, and the corner point of the service area formed by the intersection of the service line and the singles sideline. The court lines include the baseline, singles / doubles sidelines, service line, and center line. The service area is the area enclosed by the center line, service line, net, and sidelines. Each half of the court is divided into left and right service areas.

[0016] Furthermore, the multimodal timing information includes the three-dimensional position sequence of the tennis ball and the three-dimensional pose and position sequence of the racket.

[0017] Furthermore, based on the reconstructed optimal 3D trajectory, a set historical time window is extracted. By analyzing the three-dimensional position of the tennis ball, the three-dimensional posture of the racket, and the three-dimensional position of the racket, a three-dimensional position sequence of the tennis ball is obtained. Three-dimensional posture and position sequence of the racket ; The 3D position sequence of the tennis ball and the 3D pose and position sequence of the racket are input into the feature encoder of a temporal deep neural network. The feature encoder is then used to process... and Perform independent encoding and output three-dimensional features of tennis history. Three-dimensional features of racket history ; The three-dimensional features of tennis history are received by the cross-attention fusion layer of the temporal deep neural network. Three-dimensional features of racket history It employs a cross-attention mechanism for feature fusion, outputting high-dimensional contextual features; High-dimensional context features In the autoregressive decoder of the input temporal deep neural network, decoding is performed in an autoregressive manner, and the output is a future fixed time window. Predicted 3D position sequence of tennis balls in frames And the three-dimensional position of the tennis ball in the last frame. This is used as a prediction of the final landing point of the tennis ball.

[0018] The present invention has at least the following beneficial effects: The present invention provides a method for reconstructing the three-dimensional trajectory of tennis balls based on monocular video. This method extracts key points of the two-dimensional tennis court from monocular video frames, matches these key points with corresponding points on a three-dimensional standard tennis court, calculates the camera's intrinsic and extrinsic parameters, and establishes a three-dimensional world coordinate system based on the three-dimensional standard tennis court. This establishes an absolute spatial reference, giving subsequent calculations of position and velocity the physical meaning of actual units (meters, kilometers per hour). Utilizing a three-dimensional standard tennis court as a natural calibration reference eliminates the need for tedious manual calibration using tools such as checkerboards before shooting, enabling immediate use after shooting. The tennis three-dimensional trajectory reconstruction method based on monocular video of the present invention solves the problems of inaccurate trajectory reconstruction and unclear physical meaning caused by the lack of depth information in traditional monocular video; Compared to traditional two-dimensional methods, the method of this invention can output the ball's true speed, net clearance height, and landing depth, meeting the needs of professional quantitative analysis; The tennis ball 3D trajectory reconstruction method based on monocular video of the present invention eliminates the dependence on expensive multi-view camera arrays, radar or depth cameras. It can calculate the accurate 3D coordinates of the ball in the 3D world coordinate system with just an ordinary camera, which greatly reduces the system deployment cost. This invention utilizes the three-dimensional coordinates of key points on the racket as known prior knowledge, and combines them with the two-dimensional coordinates of the key points on the racket to calculate the three-dimensional depth and pose of the racket in a monocular video frame, so as to determine the position of the hitting point in three-dimensional space. Since the racket pose is reconstructed at the same time, the system can analyze the "hitting point deviation" (whether the ball lands in the center or on the edge of the racket face) and the "relationship between the swing trajectory and the ball path", providing a deeper dimension of evidence for technical analysis. The tennis 3D trajectory reconstruction method based on monocular video of the present invention uses the hitting point (coordinates of the geometric center of the 3D racket face) and the landing point (height 0) as two strongly constrained anchor points in 3D space to compress the infinitely possible trajectory solution space into a unique physically feasible solution. The reconstructed 3D trajectory must pass through the hitting point and the landing point to eliminate visual depth ambiguity. The present invention provides a method for reconstructing the three-dimensional trajectory of tennis balls based on monocular video. It uses physical motion equations that include gravity, air resistance, and Magnus force to construct a simulated three-dimensional trajectory, ensuring that the trajectory conforms to the laws of physics. Even in doubles matches or when the ball is obstructed by a player's body, causing visual tracking to be interrupted, the position of the ball can still be accurately deduced through physical motion equations, ensuring the continuity of the trajectory. Compared to traditional two-dimensional methods, the method of this invention can output the ball's true speed, net clearance height, and landing depth, meeting the needs of professional quantitative analysis. Attached Figure Description

[0019] To further illustrate the above and other advantages and features of the various embodiments of the present invention, a more specific description of the various embodiments of the present invention will be presented with reference to the accompanying drawings. It is to be understood that these drawings depict only typical embodiments of the invention and are therefore not intended to limit its scope.

[0020] Figure 1 The flowchart of a method for reconstructing a three-dimensional trajectory of a tennis ball based on monocular video according to an embodiment of the present invention is shown. Detailed Implementation

[0021] It should be noted that the components in the accompanying drawings may be shown exaggerated for illustrative purposes and may not be to scale.

[0022] In this invention, the various embodiments are merely intended to illustrate the solutions of the invention and should not be construed as limiting.

[0023] In this invention, unless otherwise specified, the quantifiers “a” and “one” do not exclude scenarios involving multiple elements.

[0024] It should also be noted that, in the embodiments of the present invention, only a portion of the parts or components may be shown for clarity and simplicity. However, those skilled in the art will understand that, under the teachings of the present invention, the required parts or components can be added as needed for specific scenarios.

[0025] It should also be noted that within the scope of this invention, the terms "same", "equal", and "equal to" do not mean that the two values ​​are absolutely equal, but allow for a certain reasonable error. In other words, the terms also cover "substantially the same", "substantially equal", and "substantially equal to".

[0026] It should also be noted that in the description of this invention, the terms "center," "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not explicitly or implicitly suggest that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0027] Furthermore, the embodiments of the present invention describe the process steps in a specific order. However, this is only for the convenience of distinguishing each step, and is not a limitation on the order of each step. In different embodiments of the present invention, the order of each step can be adjusted according to the process.

[0028] Figure 1 The flowchart of a method for reconstructing a three-dimensional trajectory of a tennis ball based on monocular video according to an embodiment of the present invention is shown.

[0029] A method for reconstructing the 3D trajectory of a tennis ball based on monocular video includes the following steps: Step 1: Extract the two-dimensional pixel coordinates of the tennis ball's center point from multiple consecutive monocular video frames to form a two-dimensional pixel trajectory, and extract the two-dimensional racket border, two-dimensional racket key points, two-dimensional court key points, and court lines.

[0030] The two-dimensional pixel trajectory is formed by the two-dimensional pixel coordinates of the center point of the tennis ball extracted from multiple consecutive monocular video frames.

[0031] Specifically, three basic features are extracted in parallel from consecutive monocular video frames using a deep learning network: Tennis detection: Extract the two-dimensional pixel coordinates of the center point of the tennis ball from each monocular video frame containing the tennis ball.

[0032] Racket posture detection: Detect the 2D racket frame and 2D racket key points in each monocular video frame containing the racket. The 2D racket key points can include the top of the racket head, the center of the racket throat, the left edge of the racket face, the right edge of the racket face, and the bottom of the racket handle.

[0033] Court Feature Extraction: Detecting key points and court lines in a 2D court within a monocular video frame. Key points include at least two of the following: service line corners, baseline corners, singles sideline corners, doubles sideline corners, the intersection of the center line and service line, the intersection of the center line and baseline, and the service area corners formed by the intersection of the service line and singles sideline. Service line corners are located at the four points where the service line intersects the baseline, marking the boundaries of the service area at the four corners of the court. Baseline corners are the four points where the baseline intersects the sidelines at both ends of the court, signifying the boundaries of the singles and doubles courts. Court lines include the baseline, singles / doubles sidelines, service line, and center line. The service area is the area enclosed by the center line, service line, net, and sidelines; each half of the court is divided into left and right service areas.

[0034] Step 2: Match the key points of the 2D court with the corresponding points of the 3D standard court to calculate the camera's intrinsic and extrinsic parameters and establish a 3D world coordinate system based on the 3D standard court.

[0035] A three-dimensional standard court is a standard court as defined by the International Tennis Federation.

[0036] Intrinsic parameters determine how light is projected onto the image sensor inside the camera; these parameters include the lens's focal length and distortion level. Extrinsic parameters contain the camera's position and orientation in the three-dimensional world. Obtaining both intrinsic and extrinsic parameters establishes a bridge for converting from a two-dimensional image to three-dimensional space.

[0037] The origin of a three-dimensional world coordinate system can be the center point of the court or any corner point. Establishing a three-dimensional world coordinate system establishes an absolute spatial reference, giving all tennis position calculations physical meaning in the "meter" unit.

[0038] Step 3: Using the 3D coordinates of the key points of the 3D racket as prior knowledge, and combining them with the 2D coordinates of the key points of the 2D racket, the 3D depth and pose of the racket in each monocular video frame containing the racket are calculated using the perspective N-point positioning algorithm to determine the position of the hitting point in 3D space.

[0039] Step 3 involves 3D reconstruction of the racket, using the racket as a rigid body of known dimensions, acting as a "depth sensor." Since the tennis ball must contact the racket to change its trajectory, the racket's 3D position directly determines the initial depth of the ball's trajectory.

[0040] The following section introduces racket 3D reconstruction (based on PnP solution of rigid body model): Although a monocular camera cannot directly measure depth, the geometry of a standard tennis racket is fixed (e.g., the major axis of the racket face is fixed). Assuming the 3D coordinates of key points on a standard tennis racket in the racket's local object coordinate system... Given prior knowledge, the two-dimensional coordinates of the key points of the two-dimensional racket detected in a monocular video frame are: The rotation and translation matrices of the racket can be calculated using the perspective projection (PnP) algorithm, thus transforming it into a three-dimensional world coordinate system.

[0041] According to the principle of perspective projection, the relationship between the 3D coordinates of key points on a 3D racket and the 2D coordinates of key points on a 2D racket is as follows: , in For the camera intrinsic parameter matrix, and Here, are the rotation and translation matrices of the racket relative to the camera, respectively, and 's' represents the scale factor, indicating the depth of the keypoint in space. This is because the 3D coordinates of the keypoints on the 3D racket... (Standard size) and the corresponding 2D coordinates of key points on the 2D racket Given the observed values, solve using a nonlinear optimization method. and This allows us to obtain the precise three-dimensional position of the racket in each frame. This directly provides a known three-dimensional hitting point for the ball's trajectory, that is, the three-dimensional position of the hitting point.

[0042] Step 4: Set the hitting point constraint and landing point constraint. By adjusting the input initial velocity vector and spin vector, construct a simulated three-dimensional trajectory passing through the hitting point and landing point using the physical motion equations. Then, project the simulated three-dimensional trajectory onto a two-dimensional plane and fit it with the two-dimensional pixel trajectory to construct the optimal three-dimensional trajectory.

[0043] This invention transforms the underdetermined back-projection problem into a parameter optimization problem with a unique global optimal solution by introducing dual physical boundary conditions (hitting point constraint and landing point constraint) and dynamic differential equations (physical motion equations).

[0044] According to the physical laws of tennis, under ideal conditions where wind interference is ignored, once the tennis ball leaves the racket, its entire flight trajectory is entirely determined by its initial state at the moment of impact. Let the initial state vector to be solved be... ,in Let be the initial velocity vector. is the spin vector.

[0045] In one embodiment, the three-dimensional position of the ball at the "moment of impact" is set to be equal to the three-dimensional position of the point of impact on the racket; the height of the ball at the "moment of landing" is set to zero. Specifically: Hitting point constraint: at the moment of impact The three-dimensional center point of a tennis ball The center point of the tennis ball lies on the three-dimensional racket face plane fitted by multiple of the racket's three-dimensional key points (such as the apex of the racket head, the center of the throat, the left edge of the racket face, the right edge of the racket face, and the bottom of the handle), and is within the physical boundaries of the racket face; in simplified calculations, at the moment of impact, the three-dimensional center point of the tennis ball... Geometric center of the three-dimensional racket face The Euclidean spatial distance between them reaches a minimum.

[0046] Landing point constraint: at the time of landing The vertical height of the ball It must be 0.

[0047] In one embodiment, the kinematic model is constrained by introducing physical equations of motion that incorporate gravity, air resistance, and Magnus forces (spin forces). The three-dimensional position of the tennis ball... Following the aerodynamic differential equations (physical equations of motion): , Where m is mass and g is gravity. This represents the three-dimensional acceleration vector of a tennis ball. Represents the spin vector. Represents the velocity vector; The air drag coefficient, Let be the Magnus force coefficient. Numerical integration of the above differential equation yields the result as a function of time. and initial state vector The deterministic three-dimensional trajectory function of evolution This indicates that at time t, the initial state vector... The three-dimensional coordinates of the center point of an evolved tennis ball.

[0048] Starting point boundary (hitting point constraint): . It is the three-dimensional center point of the tennis ball at the moment of impact. The coordinates of the geometric center of the three-dimensional racket face where the Euclidean distance between them reaches a minimum. In other words, at the moment of impact, the coordinates of the three-dimensional center point of the tennis ball are the same as the coordinates of the geometric center of the three-dimensional racket face.

[0049] End point boundary (landing point constraint): This planar constraint specifies the termination condition for integration and limits the flight time.

[0050] In one embodiment, an optimization algorithm iteratively adjusts the initial velocity vector and spin vector, generates a simulated three-dimensional trajectory using physical motion equations, projects the simulated three-dimensional trajectory onto a two-dimensional plane, fits it to a two-dimensional pixel trajectory, calculates the comprehensive loss function value, and aims to minimize the loss function to solve for the optimal three-dimensional trajectory and its corresponding optimal initial velocity vector and optimal spin vector. (Comprehensive loss function) Includes reprojection error and boundary constraint error: , in This represents the reprojection error from the 3D trajectory to the 2D pixel plane. The boundary penalty term is the landing point height. For the penalty item at the point of impact, and Indicates weight, This represents the three-dimensional center point coordinates of the tennis ball at the moment of impact, where K represents the camera intrinsic parameters. Represent the two-dimensional coordinates of the tennis ball at time t. The coordinates of the simulated three-dimensional center point of the tennis ball at time t are projected onto a two-dimensional plane.

[0051] In one embodiment, reprojection error The calculation is based on the camera intrinsic parameters of the points simulating the 3D trajectory. The distance between the projection onto the two-dimensional plane and the two-dimensional pixel trajectory point.

[0052] The boundary constraint error comprises the spatial distance error between the predicted position of the simulated 3D trajectory at the moment of impact and the 3D impact point of the racket, and the height error between the predicted position of the simulated 3D trajectory at the moment of landing and the ground plane (Z=0). By adjusting the ball's initial velocity vector and spin vector, the comprehensive loss function is minimized, thereby calculating the true 3D trajectory that conforms to the laws of physics. According to the above impact point constraint, the impact point boundary penalty term is 0.

[0053] In one embodiment, the optimization algorithm may be the Levenberg-Marquardt (LM) algorithm, the Gauss-Newton method, or the gradient descent method.

[0054] Based on the optimal three-dimensional trajectory, the ball's true speed (e.g., a serve speed of 180 km / h), net clearance height, and landing depth can be output.

[0055] In one embodiment, the optimal three-dimensional trajectory conforms to the physical equations of motion, passes through both the point of impact and the point of landing, and its projection on the two-dimensional plane is closest to the two-dimensional pixel trajectory.

[0056] By using the point of impact (the coordinates of the geometric center of the three-dimensional racket face) and the landing point (with a height of 0) as two strongly constrained anchor points in three-dimensional space, the infinitely possible trajectory solution space is compressed into a unique physically feasible solution. The reconstructed three-dimensional trajectory must pass through the point of impact and the landing point, eliminating visual depth ambiguity (for example, a ball with strong sidespin may appear to be flying in a straight line on a two-dimensional screen, but after three-dimensional reconstruction, its true lateral bending trajectory will be accurately restored).

[0057] Optionally, in step 5, the three-dimensional position of the tennis ball and the three-dimensional pose and position of the racket are obtained based on the reconstructed optimal three-dimensional trajectory over a period of time. Multimodal temporal information is constructed as input, and a temporal deep neural network with an attention mechanism is used to predict the three-dimensional position and final landing point of the tennis ball over a future period of time.

[0058] The multimodal temporal information includes the three-dimensional position sequence of the tennis ball and the three-dimensional pose and position sequence of the racket.

[0059] Temporal deep neural networks adopt a sequence-to-sequence architecture, which consists of three core sub-modules connected in series: a feature encoder (such as an LSTM or Transformer encoder), a cross-attention fusion layer, and an autoregressive decoder.

[0060] The specific steps are as follows: Step 5.1, Constructing multimodal temporal information: Based on the reconstructed optimal 3D trajectory, extract the set historical time window. By analyzing the three-dimensional position of the tennis ball, the three-dimensional posture of the racket, and the three-dimensional position of the racket, a three-dimensional position sequence of the tennis ball is obtained. Three-dimensional posture and position sequence of the racket For example, extracting the 3D position sequence of the tennis ball within a historical time window before the point of impact, as well as the 3D pose and position sequence of the racket. The 3D position sequence of the tennis ball and the 3D pose and position sequence of the racket are input into a feature encoder, which then processes the data... and Perform independent encoding, extract historical high-dimensional time-series features from both, and output three-dimensional tennis history features. Three-dimensional features of racket history This process belongs to encoding. The feature encoder encodes the three-dimensional position sequence of the tennis ball, outputting three-dimensional features of the tennis ball's history. The feature encoder encodes the three-dimensional pose and position sequence of the racket, outputting three-dimensional features of the racket's history. Among them, and Each implicitly encodes the temporal kinematics of its respective object.

[0061] Step 5.2, Multimodal temporal fusion based on cross-attention: The three-dimensional features of tennis history are received by the cross-attention fusion layer. Three-dimensional features of racket history A cross-attention mechanism is used for feature fusion to output high-dimensional contextual features: combining the three-dimensional features of tennis history. Mapped to a query vector (Query, denoted as ). Three-dimensional features of racket history Mapped to a key vector (Key, denoted as ) and value vector (Value, denoted as High-dimensional contextual features are calculated using the scaled dot product attention formula. The formula is as follows: , in, Let be the dimension of the key vector. It is a normalized exponential function.

[0062] The cross-attention fusion layer computes the query vector. With key vector The similarity between the two frames is used to adaptively assign attention weights to each frame within a time window, focusing on racket posture frames before and after the shot that decisively influence the current tennis ball trajectory. This mechanism allows the network to automatically focus on the instantaneous racket features that have a decisive physical impact on the current tennis ball trajectory, assigning them extremely high weights, while assigning extremely low weights to redundant swing frames where no physical contact occurs. Finally, the value vector... High-dimensional context features of weighted summation output This implicitly extracts the hitting intention and integrates it with dynamic features, avoiding noise interference caused by simple feature splicing.

[0063] By introducing a cross-attention mechanism, the tennis ball actively queries its historical sequence for relevant racket (key / value) states (3D racket posture and position). During most of the tennis ball's flight, racket posture has no impact on the ball's trajectory. Simply concatenating the tennis ball's 3D position and racket posture features at each time step results in a large amount of useless racket posture data being input into the network, leading to decreased prediction performance. The cross-attention mechanism allows the deep neural network to automatically learn and assign extremely high attention weights to key hitting interactions, while assigning very low weights to racket movements during the ball's flight. This ability to intelligently extract contextual cues truly unlocks the value of racket posture data, improving the prediction accuracy for sudden changes in trajectory after impact.

[0064] Step 5.3, Fixed-length sequence decoding and landing point extraction: fused high-dimensional context features The input is fed into an autoregressive decoder, where it is decoded in an autoregressive manner, and the output is a future fixed-time window. Predicted 3D position sequence of tennis balls in frames Subsequently, the three-dimensional position of the tennis ball in the last frame of the sequence was directly extracted. This serves as the system's prediction of the final landing point of the tennis ball.

[0065] In the training phase (step 5) of the temporal deep neural network, in order to overcome the error accumulation effect in long sequence prediction and improve the accuracy of the landing point, a comprehensive loss function based on time step weighting is designed. The loss function calculates the predicted three-dimensional position of the tennis ball. With true three-dimensional position The distance error between them, and based on the prediction step size Assign dynamic weights: , Among them, the weight of the intermediate frame satisfy That is, the closer the frame is to the current observation history, the greater the weight, forcing the network to prioritize fitting the accurate direction of the initial trajectory; A large value is set for the independent final landing point penalty weight to enforce the target accuracy at the end of the prediction sequence.

[0066] The following demonstrates how the method of the present invention converts ordinary monocular mobile phone video into professional-grade three-dimensional motion data.

[0067] Scene description: A tennis enthusiast wants to analyze the landing point and spin of his serves, but he only has a smartphone. He mounts the phone on a tripod behind the baseline of the court to film a series of his serve practice sessions.

[0068] After a user uploads a video, the system automatically identifies the baseline and service line in the frame and calculates the phone's exact position in the court space based on standard court dimensions (e.g., 2 meters behind the baseline and 1.5 meters high).

[0069] Motion reconstruction: When the user tosses and hits the ball, the system accurately captures five 2D key points of the racket at the moment of impact. Using the standard racket dimensions as a priori, the system calculates that the point of impact is 2.8 meters above the baseline.

[0070] Trajectory Calculation: After the ball is launched, the system tracks the two-dimensional pixel trajectory of the tennis ball. When the ball lands in the service area, the system identifies the landing frame and locks the ball's height at that moment as 0. The system then activates the physics engine (physical motion equations) to find a three-dimensional parabola that starts from a height of 2.8 meters and lands at a specific time, ensuring that its projection on the two-dimensional screen perfectly matches the video.

[0071] Output: The system overlays a 3D virtual trajectory onto the video and provides the following data: "Serve speed 165 km / h, topspin 2500 rpm, landing point in the deep inside corner." Even if the user's body briefly obscures the ball during the follow-through, the generated trajectory remains continuous and smooth.

[0072] While some embodiments of the present invention have been described in this application, those skilled in the art will understand that these embodiments are merely illustrative. Numerous variations, alternatives, and improvements will arise in those skilled in the art under the teachings of this invention without departing from its scope. The appended claims are intended to define the scope of the invention and thereby cover methods and structures within the scope of the claims themselves and their equivalents.

Claims

1. A method for three-dimensional trajectory reconstruction and prediction of a tennis ball based on monocular video, characterized in that, include: Two-dimensional pixel coordinates of the tennis ball's center point are extracted from multiple consecutive monocular video frames to form a two-dimensional pixel trajectory, and two-dimensional racket border, two-dimensional racket key points, two-dimensional court key points, and court lines are extracted. By matching the key points of the 2D court with the corresponding points of the 3D standard court, the intrinsic and extrinsic parameters of the camera are calculated, and a 3D world coordinate system based on the 3D standard court is established. By using the known 3D coordinates of the key points of the 3D racket and combining them with the 2D coordinates of the key points of the 2D racket, the 3D depth and pose of the racket in a monocular video frame are calculated to determine the position of the hitting point in 3D space. as well as By setting constraints on the hitting point and landing point, and adjusting the input initial velocity vector and spin vector, a simulated three-dimensional trajectory passing through the hitting point and landing point is constructed using the physical motion equations. The simulated three-dimensional trajectory is then projected onto a two-dimensional plane and fitted with a two-dimensional pixel trajectory to construct the optimal three-dimensional trajectory.

2. The monocular video-based tennis three-dimensional trajectory reconstruction and prediction method according to claim 1, characterized in that, Also includes: Based on the reconstructed optimal 3D trajectory, the 3D position of the tennis ball and the 3D posture and position of the racket are obtained over a period of time. Multimodal temporal information is constructed as input, and a temporal deep neural network with an attention mechanism is used to predict the 3D position and final landing point of the tennis ball in the future.

3. The monocular video-based tennis three-dimensional trajectory reconstruction and prediction method according to claim 1, characterized in that, Using the known 3D coordinates of key points on the 3D racket, combined with the 2D coordinates of key points on the 2D racket, a perspective N-point localization algorithm is used to calculate the 3D depth and pose of the racket in a monocular video frame, thereby determining the position of the hitting point in 3D space, including: Assuming the three-dimensional coordinates of key points on a standard racket are in the local object coordinate system of the racket, It is known that the two-dimensional coordinates of the key points of the two-dimensional racket detected in a monocular video frame are: According to the principle of perspective projection, the relationship between the 3D coordinates of the key points on a 3D racket and the 2D coordinates of the key points on a 2D racket is as follows: ， in For the camera intrinsic parameter matrix, and , respectively, are the rotation and translation matrices of the racket relative to the camera, and s represents the scale factor; Due to the three-dimensional coordinates of key points on a three-dimensional racket and the corresponding two-dimensional coordinates of the key points on the two-dimensional racket Given that the rotation matrix of the racket relative to the camera can be solved using a nonlinear optimization method. Translation matrix This allows us to obtain the three-dimensional position of the racket in each frame, and thus the three-dimensional position of the three-dimensional hitting point.

4. The method for reconstructing and predicting the three-dimensional trajectory of a tennis ball based on monocular video according to claim 1, characterized in that, The striking point constraint includes: at the moment of impact The three-dimensional center point of a tennis ball Located on the three-dimensional racket face plane fitted by the three-dimensional key points of the racket, and within the physical boundary of the racket face; in simplified calculation, at the moment of impact, the Euclidean space distance between the three-dimensional center point of the tennis ball and the geometric center of the three-dimensional racket face reaches a minimum value. The landing point constraint is at the landing time The vertical height of the ball It is 0.

5. The method for reconstructing and predicting the three-dimensional trajectory of a tennis ball based on monocular video according to claim 1, characterized in that, Three-dimensional position of a tennis ball It follows the physical equation of motion: ， Where m is mass and g is gravity. This represents the three-dimensional acceleration vector of a tennis ball. Represents the spin vector. Represents the velocity vector; The air drag coefficient, The Magnus force coefficient; Numerical integration of the physical equations of motion yields the result over time. and initial state vector The deterministic three-dimensional trajectory function of evolution, denoted as ; The initial state vector to be solved is ,in Let be the initial velocity vector. is the spin vector.

6. The method for reconstructing and predicting the three-dimensional trajectory of a tennis ball based on monocular video according to claim 1, characterized in that, The initial velocity vector and spin vector are iteratively adjusted using an optimization algorithm. A simulated three-dimensional trajectory is generated through the physical motion equations. The simulated three-dimensional trajectory is then projected onto a two-dimensional plane and fitted with the two-dimensional pixel trajectory. The comprehensive loss function value is calculated. With the goal of minimizing the loss function, the optimal three-dimensional trajectory and the corresponding optimal initial velocity vector and optimal spin vector are solved.

7. The method for reconstructing and predicting the three-dimensional trajectory of a tennis ball based on monocular video according to claim 6, characterized in that, The overall loss function includes reprojection error and boundary constraint error: ， in This represents the reprojection error from the 3D trajectory to the 2D pixel plane. The boundary penalty term is the landing point height. For the penalty item at the point of impact, These are the coordinates of the geometric center of the three-dimensional racket face where the Euclidean distance between the racket face and the three-dimensional center point of the tennis ball reaches a minimum at the moment of impact. and Indicates weight, This represents the three-dimensional center point coordinates of the tennis ball at the moment of impact, where K represents the camera intrinsic parameters. Represent the two-dimensional coordinates of the tennis ball at time t. The coordinates of the simulated three-dimensional center point of the tennis ball at time t are projected onto a two-dimensional plane.

8. The method for reconstructing and predicting the three-dimensional trajectory of a tennis ball based on monocular video according to claim 1, characterized in that, The key points of the two-dimensional racket and the key points of the three-dimensional racket include the top of the racket head, the center of the racket throat, the left edge of the racket face, the right edge of the racket face, and the bottom of the racket handle; The key points of the two-dimensional court include at least two of the following: the corner point of the service line, the corner point of the baseline, the corner point of the singles sideline, the corner point of the doubles sideline, the intersection of the center line and the service line, the intersection of the center line and the baseline, and the corner point of the service area formed by the intersection of the service line and the singles sideline. The court lines include the baseline, singles / doubles sidelines, service line, and center line. The service area is the area enclosed by the center line, service line, net, and sidelines. Each half of the court is divided into left and right service areas.

9. The method for reconstructing and predicting the three-dimensional trajectory of a tennis ball based on monocular video according to claim 2, characterized in that, The multimodal temporal information includes the three-dimensional position sequence of the tennis ball and the three-dimensional pose and position sequence of the racket.

10. The method for reconstructing and predicting the three-dimensional trajectory of a tennis ball based on monocular video according to claim 9, characterized in that, Based on the reconstructed optimal 3D trajectory, a set historical time window is extracted. By analyzing the three-dimensional position of the tennis ball, the three-dimensional posture of the racket, and the three-dimensional position of the racket, a three-dimensional position sequence of the tennis ball is obtained. Three-dimensional posture and position sequence of the racket ; The 3D position sequence of the tennis ball and the 3D pose and position sequence of the racket are input into the feature encoder of a temporal deep neural network. The feature encoder is then used to process... and Perform independent encoding and output three-dimensional features of tennis history. Three-dimensional features of racket history ; The three-dimensional features of tennis history are received by the cross-attention fusion layer of the temporal deep neural network. Three-dimensional features of racket history It employs a cross-attention mechanism for feature fusion, outputting high-dimensional contextual features; High-dimensional context features In the autoregressive decoder of the input temporal deep neural network, decoding is performed in an autoregressive manner, and the output is a future fixed time window. Predicted 3D position sequence of tennis balls in frames And the three-dimensional position of the tennis ball in the last frame. This is used as a prediction of the final landing point of the tennis ball.

Images