Method and system for reconstructing badminton or tennis three-dimensional trajectory based on monocular video

Abstract

The application discloses a method and system for reconstructing three-dimensional trajectory of a shuttlecock or a tennis ball based on monocular video, which comprises the following steps: selecting a calibration frame from a shuttlecock or tennis ball sports video, and performing camera calibration to obtain camera parameters; obtaining joint coordinates of a two-dimensional human body posture and two-dimensional shuttlecock or tennis ball coordinates from the sports video to obtain a feature group; constructing a training sample for judging a hitting frame to obtain a shuttlecock or tennis ball hitting frame judgment model; inputting the feature group into the judgment model to identify the hitting frame and determine a hitting round; obtaining a shuttlecock or tennis ball movement plane by combining the coordinates corresponding to the identified hitting frame with the camera parameters; mapping the coordinates of the ball in each hitting round to a three-dimensional space according to the camera parameters to obtain a three-dimensional ray, and combining the three-dimensional ray with the shuttlecock or tennis ball movement plane to obtain three-dimensional coordinates of the ball. The application can restore the three-dimensional coordinates of the ball from the video obtained by a monocular camera, and significantly reduces the cost and difficulty of acquisition.

Description

Technical Field

[0001] This invention belongs to the field of computer vision processing technology, specifically relating to a method and system for reconstructing the three-dimensional trajectory of a badminton or tennis ball based on monocular video. Background Technology

[0002] In recent years, with the development of the computer vision field and the consumer market, badminton or tennis 3D trajectory analysis has played an important role in sports training and analysis, event refereeing and assistance, sports data analysis and prediction, games and entertainment.

[0003] Currently, the commonly used methods for reconstructing the trajectory of a badminton or tennis ball in 3D include the following two:

[0004] (1) A method for reconstructing the trajectory of a badminton or tennis ball based on binocular cameras. The binocular vision method imitates the principle of human binocular vision, capturing the same scene from different angles through two cameras and recovering the three-dimensional information using the principle of parallax. Its core principle is based on geometric relationships and parallax calculation. First, a binocular vision system is formed using two cameras; then, the two cameras are calibrated in a checkerboard pattern, stereo corrected, and stereo matched to obtain a parallax map; finally, the depth information of each camera is calculated from the parallax map to generate a depth map, and the three-dimensional trajectory of the badminton or tennis ball is obtained.

[0005] (2) A method for reconstructing the trajectory of a badminton or tennis ball based on a neural network

[0006] This method uses a neural network to start with a 2D ball position sequence captured by a monocular camera. First, it backprojects each 2D point into a ray in 3D space using known camera parameters. Then, it parameterizes the intersections of these rays with the ground or a vertical plane, making the input features independent of the specific viewpoint. Next, it employs a step-by-step LSTM-based network structure to predict trajectory segments, then predicts the ball's height above the ground frame by frame, and finally performs fine-tuning of the 3D trajectory, thus achieving high-precision 3D ball trajectory reconstruction. However, this method has drawbacks: it assumes the motion sequence begins and ends on the ground (requiring preprocessing and cropping); there is still a gap between simulation training and the real distribution (e.g., ignoring rotation and aerodynamics); it may still be affected by extreme or rare motion trajectories; and it has high computational complexity and slow running speed.

[0007] Currently, methods for reconstructing 3D badminton or tennis ball trajectories based on binocular cameras suffer from high costs, high computational complexity, high calibration requirements, and difficulty in obtaining samples, which limit their generalization performance. Methods for reconstructing 3D badminton or tennis ball trajectories based on neural networks suffer from the need for preprocessing and cropping, high computational complexity, and slow running speed. Summary of the Invention

[0008] The purpose of this invention is to provide a method and system for reconstructing the three-dimensional trajectory of a badminton or tennis ball based on monocular video, so as to realize the purpose of restoring the three-dimensional coordinates of the badminton or tennis ball using only two-dimensional video acquired by a monocular camera, thereby reducing costs and the difficulty of obtaining three-dimensional coordinates.

[0009] To achieve the above objectives, the present invention provides a method for reconstructing the three-dimensional trajectory of a badminton or tennis ball based on monocular video, comprising:

[0010] Step 1: Select calibration frames from badminton or tennis video to perform camera calibration and obtain camera parameters;

[0011] Step 2: Obtain the joint coordinates of the two-dimensional human posture and the coordinates of the two-dimensional badminton or tennis ball from the badminton or tennis video to obtain the feature set;

[0012] Step 3: Construct training samples for judging hit frames, and use these training samples to train the classification model to obtain a badminton or tennis hit frame judgment model;

[0013] Step 4: Input the feature group into the badminton or tennis hit frame judgment model to identify the hit frame and determine the hitting round;

[0014] Step 5: Using the identified joint coordinates corresponding to the hit frames and the two-dimensional badminton or tennis ball coordinates, combined with the camera parameters, obtain the three-dimensional coordinates of the projection points of the badminton or tennis ball on the ground corresponding to the two hit frames in a hitting round, and obtain the badminton or tennis ball motion plane from the two projection point three-dimensional coordinates.

[0015] Step 6: Based on the camera parameters, map the two-dimensional badminton or tennis ball coordinates of each hitting round into three-dimensional space to obtain several three-dimensional rays of the badminton or tennis ball.

[0016] Step 7: Obtain the three-dimensional coordinates of the badminton or tennis ball by finding the intersection of the badminton or tennis ball's motion plane with the three-dimensional ray of the badminton or tennis ball. Based on the three-dimensional coordinates, the three-dimensional motion trajectory of the badminton or tennis ball can be obtained.

[0017] Furthermore, in step 1, the camera parameters include the camera matrix and the camera center; the mapping relationship between two-dimensional pixel coordinates (u,v) and three-dimensional coordinates (x,y,z) is solved based on the calibration frame. The camera matrix P is obtained as follows:

[0018] P=

[0019] The complete expression for the camera matrix P is: K is the camera's 3 The 3 intrinsic parameter matrix describes the camera's own optical / hardware characteristics and is independent of the shooting scene; [R|t] is the camera's 3 The 4 extrinsic parameter matrices describe the camera's pose (R, rotation matrix) and position (t, translation vector) in the 3D world coordinate system, and are related to the shooting scene. After matrix multiplication, the 3×3 K is combined with the 3×4 [R|t] to finally generate the 3×4 camera matrix P.

[0020] The standard form of the intrinsic parameter matrix K is:

[0021]

[0022] in The focal length is the x-axis. The focal length is the y-axis. The x-coordinate of the principal point y is the y-coordinate of the main point, and s is the pixel skew coefficient.

[0023] The rotation matrix R satisfies orthogonality and a right-handed coordinate system, and can be described with only 3 columns of parameters (commonly represented by "Euler angles" or "quaternions"). Its function is to correct the rotational offset of 3D world points relative to the camera. The 3 independent columns of parameters correspond to the rotation angles around the x-axis, y-axis, and z-axis, respectively.

[0024] The translation vector t has three parameters that describe the camera's position offset in the x, y, and z axes of the world coordinate system. Its function is to correct the position offset of the 3D world point relative to the camera. The three parameters represent the x-axis offset, y-axis offset, and z-axis offset, respectively.

[0025] The camera center is obtained through the camera matrix P. :

[0026] .

[0027] Furthermore, in step 2, the joint coordinates of the two-dimensional human posture are obtained using a human posture recognition method, and the coordinates of the two-dimensional badminton or tennis ball are obtained using a two-dimensional target detection algorithm and a two-dimensional target tracking algorithm.

[0028] The data of the feature group includes: the ball's acceleration, velocity angle, rate of change of velocity angle, rate of change of angle of the line connecting the right wrist and right shoulder in each frame, the right wrist, the rate of change of angle of the right wrist and right shoulder with the right elbow as the apex, the distance between the right wrist and the ball, and the distance between the head and the ball.

[0029] The human pose recognition methods include MMPose, Openpose, Alphapose, HRNet, SimpleBaseline, Hourglass, VNect, ViTPose, RTMPose, or DEKR;

[0030] The two-dimensional target detection algorithms include YOLO, RT-DETR, DINO-DETR, Deformable DETR, H-Deformable-DETR, Sparse R-CNN, PP-YOLOE, or TrackNet; the two-dimensional target tracking algorithms include SORT, CRST, DeepSORT, ByteTrack, OC-SORT, BoT-SORT, FairMOT, CenterTrack, TransTrack / MOTR / TrackFormer, or KCF.

[0031] Obtain the joint coordinates of a two-dimensional human body pose, including:

[0032] Using points obtained from camera calibration, the area of ​​the badminton or tennis court and the activity space of the two players are delineated;

[0033] By determining the position of the feet at the identified human joint points, people outside the badminton or tennis court are excluded, and the movement space of the players is used to distinguish which player the identified human joint points on the court belong to.

[0034] Furthermore, the method for obtaining the coordinates of a two-dimensional badminton or tennis ball includes:

[0035] The two-dimensional target detection algorithm is trained in advance using a badminton or tennis ball database obtained through manual annotation, resulting in a visual model for two-dimensional badminton or tennis ball recognition.

[0036] The visual model is used to identify the badminton or tennis ball video, and a two-dimensional target tracking algorithm is used to track the badminton or tennis ball to obtain a two-dimensional badminton or tennis ball coordinate sequence.

[0037] Furthermore, step 4 specifically includes:

[0038] S41: Input the feature group described in step 3 into the badminton or tennis hit frame judgment model to obtain the preliminary hit frame recognition result;

[0039] S42: Read the hit frames obtained from the preliminary hit frame recognition results, and obtain the hitter corresponding to each hit frame by comprehensively comparing the coordinates of the two people's human joint points with the distance of the badminton or tennis ball, and then obtain the hitter sequence.

[0040] S43: If the batter sequence contains two consecutive hits by the same batter, the feature group between these two hits is input again into the badminton or tennis hit frame judgment model to identify a new hit frame, determine the batter, and then insert it into the batter sequence.

[0041] S44: Repeat steps S42 and S43 until there are no more consecutive hits by the same person, indicating that all hit frames have been identified and the final hit frame identification result is obtained.

[0042] Furthermore, step 5 specifically includes: using the two-dimensional badminton or tennis ball coordinates corresponding to the hit frame. Obtain the two-dimensional coordinates of the projection point of the badminton shuttlecock or tennis ball on the ground. ;

[0043] Two-dimensional coordinates of the ground projection point Substitute into the formula And by setting z=0, we obtain the three-dimensional coordinates of the ground projection point of the badminton or tennis shuttlecock; from the three-dimensional coordinates of the ground projection points of the two hit frames within a rally... and Thus, the equation of the plane of motion is obtained: .

[0044] In calculating the ground projection point, it is necessary to consider the coordinates of the two-dimensional badminton or tennis ball hit frames. Draw a vertical line (A line parallel to the center net post of the field);

[0045] connect The midpoint between the two points is obtained. ;

[0046] Passing through the midpoint Draw a perpendicular line to the vertical line. The perpendicular foot is the projection point of the badminton or tennis ball onto the ground, and the coordinates of the projection point are... .

[0047] Furthermore, step 6 specifically includes: mapping the coordinates of a two-dimensional badminton shuttlecock or tennis ball. Substitute into the formula This yields the three-dimensional ray equation for a badminton or tennis shuttlecock, namely:

[0048]

[0049] Where t is an indefinite parameter representing the length of the ray emitted from the center of the camera.

[0050] Furthermore, step 7 specifically includes:

[0051] Combining the equations of motion of the plane and the three-dimensional ray equations:

[0052]

[0053] Solving the equation will give you the three-dimensional coordinates of the badminton or tennis shuttlecock:

[0054]

[0055] in,

[0056] .

[0057] This invention also provides a system for reconstructing the three-dimensional motion trajectory of a badminton or tennis ball based on monocular video information, comprising:

[0058] The data acquisition module is used to acquire badminton or tennis video.

[0059] The camera calibration module is used to select calibration frames from badminton or tennis video to perform camera calibration and obtain camera parameters.

[0060] The two-dimensional coordinate acquisition module is used to acquire the joint coordinates of the two-dimensional human posture and the two-dimensional badminton or tennis ball coordinates from the badminton or tennis ball sports video.

[0061] The feature extraction module is used to extract feature sets from the joint coordinates of the acquired two-dimensional human pose and the two-dimensional badminton or tennis ball coordinates.

[0062] The hit frame judgment model building module is used to train the classification model based on the training samples for judging hit frames, so as to obtain the badminton or tennis hit frame judgment model.

[0063] The first calculation module is used to obtain the badminton or tennis ball hitting frames using the badminton or tennis ball hitting frame judgment model, and to determine the hitter of each hitting frame to obtain the hitting round.

[0064] The second calculation module is used to obtain the three-dimensional coordinates of the projection points of the badminton or tennis ball on the ground corresponding to the two hit frames in a hitting round by using the joint point coordinates and two-dimensional badminton or tennis ball coordinates corresponding to the identified hit frames, combined with the camera parameters, and obtain the badminton or tennis ball motion plane from the two projection point three-dimensional coordinates.

[0065] The third calculation module is used to map the two-dimensional badminton or tennis ball coordinates of each hitting round to three-dimensional space according to the camera parameters, so as to obtain several three-dimensional rays of the badminton or tennis ball.

[0066] The 3D coordinate reconstruction module is used to obtain the 3D coordinates of a badminton or tennis ball by finding the intersection of the badminton or tennis ball's motion plane with the 3D ray of the badminton or tennis ball. Based on the 3D coordinates, the 3D motion trajectory of the badminton or tennis ball can be obtained.

[0067] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for reconstructing the three-dimensional trajectory of a badminton or tennis ball based on monocular video as described in any of the preceding claims.

[0068] In summary, compared with the prior art, the above-described technical solutions conceived by this invention mainly possess the following technical advantages:

[0069] 1. The technical solution of this invention provides a method for reconstructing the 3D trajectory of a badminton or tennis ball based on a monocular camera. It utilizes court information to perform simple camera calibration with low precision requirements; it uses ball coordinate information and human skeleton coordinate information to determine the hitting frame and hitting rally; and it reconstructs the 3D trajectory of the badminton or tennis ball without the need for expensive equipment such as binocular cameras or depth cameras, achieving a high degree of automation. This solves the problems of high cost, complex operation, poor generalization, high data acquisition difficulty, and low automation in current 3D badminton or tennis ball trajectory recovery methods, reducing reliance on equipment and workload, and offering the advantages of good generalization and low cost.

[0070] 2. This invention improves the accuracy of hit frame recognition by constructing a hit frame judgment model. The hit frame is used as a node to determine the hitting round, which facilitates the determination of the starting point and motion plane of the motion trajectory. Thus, the three-dimensional coordinates and motion trajectory are obtained by combining camera parameters. At the same time, the hitter sequence is determined by the hitting round, and the hit frame is corrected and improved according to the hitter sequence, further improving the completeness and accuracy of the motion trajectory. Attached Figure Description

[0071] Figure 1 This is a flowchart illustrating a machine learning-based three-dimensional trajectory reconstruction method for badminton or tennis balls, as provided in Embodiment 1 of the present invention.

[0072] Figure 2 This is a schematic diagram of the planar location of the badminton court camera installation area provided in Embodiment 1 of the present invention, with blue indicating the area where the camera can be installed.

[0073] Figure 3 This is a schematic diagram of the planar location of the tennis court camera installation area provided in Embodiment 1 of the present invention, with blue indicating the area where the camera can be installed.

[0074] Figure 4 This is a schematic diagram of the spatial location of the badminton court camera installation range provided in Embodiment 1 of the present invention. The red shaded area indicates the installation range.

[0075] Figure 5 This is a schematic diagram of the spatial location of the tennis court camera installation range provided in Embodiment 1 of the present invention. The red shaded area indicates the installation range.

[0076] Figure 6 This is a schematic diagram of a camera mounting device adapted to the entire system provided in Embodiment 1 of the present invention.

[0077] Figure 7 This is a schematic diagram of mapping the coordinates of a badminton shuttlecock or tennis ball into a two-dimensional ray, provided in Embodiment 1 of the present invention. Detailed Implementation

[0078] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0079] This invention is applicable to scenarios where, during actual badminton or tennis matches, videos of two players hitting the shuttlecock are filmed. Using the method of this invention, data is collected and processed from the video to reconstruct the three-dimensional trajectory of the shuttlecock or tennis ball on the court. Each player's hit counts as one round, resulting in one trajectory. The period from the start of the hit to the shuttlecock's landing is designated as a recognition node, and the various three-dimensional trajectories within that node are obtained.

[0080] The present invention will now be described with reference to a preferred embodiment.

[0081] Example 1

[0082] like Figure 1 As shown, a method for reconstructing the three-dimensional trajectory of a badminton or tennis ball based on monocular video information includes:

[0083] S1. Set up photography equipment to record videos of badminton or tennis;

[0084] like Figure 2-5 Using the specially designed frame pole of this invention, the photography equipment is set up behind a badminton or tennis court, 1-10 meters away from the back sideline of the court (the side parallel to the center line of the players on both sides of the court). Figure 2 The blue area between d2 and d1), with a height of 1.5-5 meters, and the angle between the edge of the area where the erection point is located and the back line ( Figure 2 and 3The angle between the α (of the angle) and the back line should be less than or equal to 120 degrees, or the equipment should be installed behind the tennis court, 2-10 meters from the back line, at a height of 2-5 meters. The angle between the edge of the area where the equipment is installed and the back line should be 90-120 degrees. The area within a 120-degree angle between the two ends of the back line is considered a suitable area for installation. The shooting frame must ensure the entire badminton or tennis court is within the frame for video recording. A diagram showing the installation locations for badminton court photography equipment is provided below. Figure 2 The blue area and Figure 4 The red shaded area indicates the location of the tennis rack. Figure 3 The blue area and Figure 5 The red shaded area is shown.

[0085] The schematic diagram of the support pole is provided by Figure 6 As shown. The specially designed frame consists of a tripod, monitor, telescopic pole, and mounting platform. It helps to set up photography equipment in a location that meets the above requirements on a badminton or tennis court, and the monitor can be connected to the photography equipment to display the shooting angle of the equipment.

[0086] The fixed platform is used to fix the photography equipment; the telescopic pole is used to control the height of the photography equipment; the monitor is used to display the image on the photography equipment, making it convenient for personnel to adjust the setting position; and the tripod is used to fix the position.

[0087] S2. Read the video and select the calibration frame to perform camera calibration and obtain camera parameters;

[0088] When selecting calibration frames, the principle is to ensure that the points used for calibration are not obstructed or otherwise affect the calibration, and a suitable frame is selected for calibration.

[0089] Three calibration methods are available: using the mouse to mark the two-dimensional coordinates of 4, 6, or 10 specific points on the image. , and the three-dimensional coordinates of these points Correspondingly, i is a positive integer from 4 to 10, representing the i-th point.

[0090] Substituting the above one-to-one correspondence of two-dimensional and three-dimensional coordinates into the formula Solve for the camera matrix P, where P is:

[0091]

[0092] The complete expression for the camera matrix P is: K is the camera's 3×3 intrinsic parameter matrix, describing the camera's optical / hardware characteristics and independent of the shooting scene; [R|t] is the camera's 3×4 extrinsic parameter matrix, describing the camera's pose (R, rotation matrix) and position (t, translation vector) in the 3D world coordinate system, and is related to the shooting scene. After matrix multiplication, the 3×3 K and the 3×4 [R|t] are combined to finally generate a 3×4 camera matrix P.

[0093] The standard form of the intrinsic parameter matrix K is:

[0094]

[0095] in The focal length is the x-axis. The focal length is the y-axis. The x-coordinate of the principal point The y-coordinate of the principal point This is the pixel skew coefficient.

[0096] The rotation matrix R satisfies orthogonality and a right-handed coordinate system, and can be described with only 3 columns of parameters (commonly represented by "Euler angles" or "quaternions"). Its function is to correct the rotational offset of 3D world points relative to the camera. The 3 independent columns of parameters correspond to the rotation angles around the x-axis, y-axis, and z-axis, respectively.

[0097] The translation vector t has three parameters that describe the camera's position offset in the x, y, and z axes of the world coordinate system. Its function is to correct the position offset of the 3D world point relative to the camera. The three parameters represent the x-axis offset, y-axis offset, and z-axis offset, respectively.

[0098] The camera center is obtained through the camera matrix P. :

[0099]

[0100] The solved camera matrix P and camera center C are the camera parameters.

[0101] S3. Using human posture recognition methods, obtain the coordinates of the joints of the two-dimensional human posture; using two-dimensional target detection algorithms and two-dimensional tracking algorithms, obtain the coordinates of the two-dimensional badminton or tennis ball, and extract a series of feature groups from the above data.

[0102] Specifically, using two-dimensional human pose detection methods such as MMPose, Openpose, Alphapose, HRNet, SimpleBaseline, Hourglass, VNect, ViTPose, RTMPose, and DEKR, the joint coordinates of two-dimensional human pose are obtained from video information.

[0103] When extracting the coordinates of the joints of a two-dimensional human body posture, it is necessary to use the points obtained during camera calibration to delineate the area of ​​the badminton or tennis court and the activity space of the two players.

[0104] By determining whether the positions of the feet of all human joints identified by the two-dimensional human posture detection method are within the activity space, people outside the badminton or tennis court can be excluded, and the player's movement space can be used to distinguish which side the human joints identified on the court belong to.

[0105] Two-dimensional target detection algorithms such as YOLO, RT-DETR, DINO-DETR, Deformable DETR, H-Deformable-DETR, Sparse R-CNN, PP-YOLOE, and TrackNet, and two-dimensional tracking algorithms such as SORT, CRST, DeepSORT, ByteTrack, OC-SORT, BoT-SORT, FairMOT, CenterTrack, TransTrack / MOTR / TrackFormer, and KCF are used to obtain two-dimensional badminton or tennis ball coordinates from video information.

[0106] When obtaining the coordinates of a two-dimensional badminton or tennis ball, a database containing more than 2,000 sets of data (i.e., two-dimensional coordinate data of badminton ball positions) with manual annotation is required. Two-dimensional target detection algorithms such as YOLO, RT-DETR, DINO-DETR, Deformable DETR, H-Deformable-DETR, Sparse R-CNN, PP-YOLOE, and TrackNet are trained to obtain a visual model for two-dimensional badminton or tennis ball recognition.

[0107] By using this visual model to identify badminton or tennis videos, a two-dimensional badminton or tennis ball coordinate sequence can be obtained.

[0108] Specifically, when there are frames where the two-dimensional target detection algorithm fails to identify them (when the algorithm does not provide a recognition result or provides a result but the confidence level is less than 0.6), two-dimensional target tracking algorithms such as SORT, CRST, DeepSORT, ByteTrack, OC-SORT, BoT-SORT, FairMOT, CenterTrack, TransTrack / MOTR / TrackFormer, and KCF are used to track badminton or tennis balls to obtain a smooth and complete two-dimensional badminton or tennis ball coordinate sequence.

[0109] In this process, the coordinate sequence of a two-dimensional badminton or tennis ball is smoothed using the least squares method, and feature sets are extracted by combining the joint coordinates of a two-dimensional human posture.

[0110] Specifically, this invention constructs a feature set of smoothed two-dimensional badminton or tennis ball coordinate sequence and human joint coordinates, including: the ball's acceleration, velocity angle (i.e., direction of motion), and velocity angle change rate in each frame; the angle change rate of the line connecting the right wrist and right shoulder; the angle change rate of the right wrist, with the right elbow as the vertex and extending in the direction of the right wrist and right shoulder; the distance between the right wrist and the ball; and the distance between the head and the ball.

[0111] Let the coordinates of the two-dimensional badminton or tennis ball in the nth frame be... The coordinates of the right wrist are The coordinates of the right elbow are The coordinates of the right shoulder are The coordinates of the head are .

[0112] Velocity angle: ;

[0113] Rate of change of velocity angle: ;

[0114] The angle between the line connecting a person's right wrist and right shoulder: ;

[0115] Rate of change of the angle of the line connecting the right wrist and right shoulder: ;

[0116] With the right elbow as the apex, the direction of extension is the angle between the right wrist and the right shoulder:

[0117] α en = ;

[0118] The rate of change of the angle extending from the right elbow as the apex to the right wrist and right shoulder: ;

[0119] Distance between right wrist and ball: ;

[0120] Distance between head and ball: ;

[0121] S4. By manually labeling, construct training samples for judging the hit frames.

[0122] Specifically, frames of athletes hitting the ball in multiple badminton or tennis match videos are manually marked to construct several hit frame sequences. The hit frame sequences and the corresponding feature groups obtained in step S3 are used to form training samples to continuously identify each hit frame in each sample. The hit frames from the start of the hit to before the ball lands form a hit frame sequence.

[0123] S5. Using the training samples from the hit frame judgment, train the classification model to obtain the trained badminton or tennis hit frame judgment model.

[0124] Specifically, classification models such as XGBoost, LightGBM, CatBoost, Random Forest, Decision Tree / Extreme Random Tree, RNN / LSTM / GRU, TabNet, HMM, and TCN can be used. The feature groups (input) obtained in step S3 and the manually labeled hit frame sequence (output) are input for training. The results are tested, the accuracy is calculated and judged, and the training parameters are optimized to obtain satisfactory model data.

[0125] S6. Use the trained classification model to obtain the hitting frames of badminton or tennis, and use the two-dimensional human joint coordinates and two-dimensional badminton or tennis ball coordinates at the time of the hitting frame to determine the hitter and determine the hitting round.

[0126] Specifically, the feature groups of people and balls in the match video that need to be judged, obtained through step S3, are input into the trained badminton or tennis hit frame judgment model to obtain preliminary hit frame recognition results.

[0127] The batting frame obtained from the initial batting frame recognition result is read. By comparing the coordinates of the two people's human joints with the distance to the badminton or tennis ball, the batter corresponding to the batting frame is determined (e.g., the person who is closer to the ball in the current batting frame is the batter corresponding to the batting frame). The operation is then continued to obtain the batter sequence. For example, if the two batters are numbered 1 and 2, the batter sequence should normally be [1,2,1,2,...].

[0128] For the analysis of the batter sequence, if there are two consecutive hits by the same batter, such as the occurrence of [1,1] in the sequence, it indicates that there is a missed detection between the two hit frames. In this case, the feature group between the two hit frames is extracted from the video, and the trained badminton or tennis hit frame judgment model is used to re-judge and obtain the point with the highest probability of hitting the ball, and then judge the batter, thereby supplementing and correcting the hit frame sequence.

[0129] Each two hits constitute a hitting round, alternating until the last hit frame (the last hit before the ball lands).

[0130] S7. Using the coordinates of the human joints of the person hitting the ball in the hit frame and the coordinates of the badminton or tennis ball, obtain the motion plane of the badminton or tennis ball.

[0131] Specifically, it consists of two-dimensional badminton or tennis shuttlecock strike frame coordinates. and the coordinates of the batter's feet in the hit frame Obtain the two-dimensional coordinates of the projection point of the badminton shuttlecock or tennis ball on the ground. ;

[0132] Two-dimensional coordinates of the ground projection point Substitute into the formula By setting z=0, we can obtain the three-dimensional coordinates of the badminton or tennis ball's ground projection point. ; the three-dimensional coordinates of the ground projection points of two strike frames within a round and The plane of motion can then be obtained: .

[0133] In calculating the ground projection point, it is necessary to consider the coordinates of the two-dimensional badminton or tennis ball hit frames. Draw a vertical line (A line parallel to the center net post of the field);

[0134] connect The midpoint between the two points is obtained. ;

[0135] Passing through the midpoint Draw a perpendicular line to the vertical line. The perpendicular foot is the projection point of the badminton or tennis ball onto the ground, and the coordinates of the projection point are... .

[0136] S8. The position of a two-dimensional badminton or tennis ball is mapped to a three-dimensional space using camera parameters.

[0137] Specifically, the coordinates of a two-dimensional badminton or tennis ball... Substitute into the formula This yields a three-dimensional ray for a badminton shuttlecock or tennis ball, namely:

[0138] ;

[0139] Where t is an indefinite parameter representing the length of the ray emanating from the camera center, and the mapping result is as follows: Figure 7 As shown.

[0140] S9. Using the plane of motion of the badminton or tennis ball and the three-dimensional rays of the badminton or tennis ball, obtain the estimated results of the three-dimensional coordinates of the badminton or tennis ball:

[0141] Solve the equations of motion of the plane and the three-dimensional ray equations simultaneously:

[0142]

[0143] Solving the equation will give you the three-dimensional coordinates of the badminton or tennis shuttlecock:

[0144]

[0145] in,

[0146] .

[0147] In summary, this invention addresses the problems of expensive equipment and difficulty in obtaining 3D coordinates in existing 3D badminton or tennis ball coordinate estimation methods. It realizes a method to reconstruct the 3D coordinates of badminton or tennis balls using only 2D video acquired by a monocular RGB camera, reducing costs and the difficulty of obtaining 3D coordinates. This facilitates the analysis of players' technical characteristics and helps players formulate tactics.

[0148] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for reconstructing the three-dimensional trajectory of a badminton or tennis ball based on monocular video, characterized in that, include: Step 1: Select calibration frames from badminton or tennis video to perform camera calibration and obtain camera parameters; Step 2: Obtain the joint coordinates of the two-dimensional human posture and the coordinates of the two-dimensional badminton or tennis ball from the badminton or tennis video to obtain the feature set; Step 3: Construct training samples for judging hit frames, and use these training samples to train the classification model to obtain a badminton or tennis hit frame judgment model; Step 4: Input the feature group into the badminton or tennis hit frame judgment model to identify the hit frame and determine the hitting round; Step 5: Using the identified joint coordinates corresponding to the hit frames and the two-dimensional badminton or tennis ball coordinates, combined with the camera parameters, obtain the three-dimensional coordinates of the projection points of the badminton or tennis ball on the ground corresponding to the two hit frames in a hitting round, and obtain the badminton or tennis ball motion plane from the two projection point three-dimensional coordinates. Step 6: Based on the camera parameters, map the two-dimensional badminton or tennis ball coordinates of each hitting round into three-dimensional space to obtain several three-dimensional rays of the badminton or tennis ball. Step 7: Obtain the three-dimensional coordinates of the badminton or tennis ball by finding the intersection of the badminton or tennis ball's motion plane with the three-dimensional ray of the badminton or tennis ball. Based on the three-dimensional coordinates, the three-dimensional motion trajectory of the badminton or tennis ball can be obtained.

2. The method for reconstructing the three-dimensional trajectory of a badminton or tennis ball based on monocular video according to claim 1, characterized in that, In step 1, the camera parameters include the camera matrix and the camera center; the mapping relationship between two-dimensional pixel coordinates (u,v) and three-dimensional coordinates (x,y,z) is solved based on the calibration frame. The camera matrix P is obtained as follows: P= The parameters in the camera matrix P are obtained by multiplying the intrinsic and extrinsic parameters. Then the camera center is obtained through the camera matrix P. : 。 3. The method for reconstructing the three-dimensional trajectory of a badminton or tennis ball based on monocular video according to claim 1, characterized in that, In step 2, the joint coordinates of the two-dimensional human posture are obtained using a human posture recognition method, and the coordinates of the two-dimensional badminton or tennis ball are obtained using a two-dimensional target detection algorithm and a two-dimensional target tracking algorithm. The data of the feature group includes: the ball's acceleration, velocity angle, rate of change of velocity angle, rate of change of angle of the line connecting the right wrist and right shoulder in each frame, the right wrist, the rate of change of angle of the right wrist and right shoulder with the right elbow as the apex, the distance between the right wrist and the ball, and the distance between the head and the ball.

4. The method for reconstructing the three-dimensional trajectory of a badminton or tennis ball based on monocular video according to claim 3, characterized in that, The method for obtaining two-dimensional badminton or tennis ball coordinates includes: The two-dimensional target detection algorithm is trained in advance using a badminton or tennis ball database obtained through manual annotation, resulting in a visual model for two-dimensional badminton or tennis ball recognition. The visual model is used to identify the badminton or tennis ball video, and a two-dimensional target tracking algorithm is used to track the badminton or tennis ball to obtain a two-dimensional badminton or tennis ball coordinate sequence.

5. The method for reconstructing the three-dimensional trajectory of a badminton or tennis ball based on monocular video according to claim 1, characterized in that, Step 4 specifically includes: S41: Input the feature group described in step 3 into the badminton or tennis hit frame judgment model to obtain the preliminary hit frame recognition result; S42: Read the hit frames obtained from the preliminary hit frame recognition results, and obtain the hitter corresponding to each hit frame by comprehensively comparing the coordinates of the two people's human joint points with the distance of the badminton or tennis ball, and then obtain the hitter sequence. S43: If the batter sequence contains two consecutive hits by the same batter, the feature group between these two hits is input again into the badminton or tennis hit frame judgment model to identify a new hit frame, determine the batter, and then insert it into the batter sequence. S44: Repeat steps S42 and S43 until there are no more consecutive hits by the same person, indicating that all hit frames have been identified and the final hit frame identification result is obtained.

6. The method for reconstructing the three-dimensional trajectory of a badminton or tennis ball based on monocular video according to any one of claims 1-5, characterized in that, Step 5 specifically includes: the two-dimensional badminton or tennis ball coordinates corresponding to the hit frame. Obtain the two-dimensional coordinates of the projection point of the badminton shuttlecock or tennis ball on the ground. ; Two-dimensional coordinates of the ground projection point Substitute into the formula And by setting z=0, we obtain the three-dimensional coordinates of the ground projection point of the badminton or tennis shuttlecock; from the three-dimensional coordinates of the ground projection points of the two hit frames within a rally... and Thus, the equation of the plane of motion is obtained: .

7. The method for reconstructing the three-dimensional trajectory of a badminton or tennis ball based on monocular video according to claim 6, characterized in that, Step 6 specifically includes: setting the coordinates of a two-dimensional badminton shuttlecock or tennis ball. Substitute into the formula This yields the three-dimensional ray equation for a badminton or tennis shuttlecock, namely: Where t is an indefinite parameter representing the length of the ray emitted from the center of the camera.

8. The method for reconstructing the three-dimensional trajectory of a badminton or tennis ball based on monocular video according to claim 7, characterized in that, Step 7 specifically includes: Combining the equations of motion of the plane and the three-dimensional ray equations: Solving the equation will give you the three-dimensional coordinates of the badminton or tennis shuttlecock: in, 。 9. A system for reconstructing the three-dimensional trajectory of a badminton or tennis ball based on monocular video, characterized in that, include: The data acquisition module is used to acquire badminton or tennis video. The camera calibration module is used to select calibration frames from badminton or tennis video to perform camera calibration and obtain camera parameters. The two-dimensional coordinate acquisition module is used to acquire the joint coordinates of the two-dimensional human posture and the two-dimensional badminton or tennis ball coordinates from the badminton or tennis ball sports video. The feature extraction module is used to extract feature sets from the joint coordinates of the acquired two-dimensional human pose and the two-dimensional badminton or tennis ball coordinates. The hit frame judgment model building module is used to train the classification model based on the training samples for judging hit frames, so as to obtain the badminton or tennis hit frame judgment model. The first calculation module is used to obtain the badminton or tennis ball hitting frames using the badminton or tennis ball hitting frame judgment model, and to determine the hitter of each hitting frame to obtain the hitting round. The second calculation module is used to obtain the three-dimensional coordinates of the projection points of the badminton or tennis ball on the ground corresponding to the two hit frames in a hitting round by using the joint point coordinates and two-dimensional badminton or tennis ball coordinates corresponding to the identified hit frames, combined with the camera parameters, and obtain the badminton or tennis ball motion plane from the two projection point three-dimensional coordinates. The third calculation module is used to map the two-dimensional badminton or tennis ball coordinates of each hitting round to three-dimensional space according to the camera parameters, so as to obtain several three-dimensional rays of the badminton or tennis ball. The 3D coordinate reconstruction module is used to obtain the 3D coordinates of a badminton or tennis ball by finding the intersection of the badminton or tennis ball's motion plane with the 3D ray of the badminton or tennis ball. Based on the 3D coordinates, the 3D motion trajectory of the badminton or tennis ball can be obtained.

Images