Intelligent tennis training system

Abstract

The invention discloses an intelligent tennis training system, which comprises a virtual construction assembly, an image acquisition assembly and a control unit, and is characterized in that the virtual construction assembly and the image acquisition assembly are connected with the control unit, and the virtual construction assembly is used for constructing a virtual scene of tennis training on a training field; the image acquisition assembly is used for acquiring relevant information of a ball on a training field, and the relevant information of the ball comprises a running track of the ball; and the control unit is used for acquiring the information of the virtual scene and the relevant information of the ball and intelligently analyzing the subsequent moving track of the ball according to the information. On one hand, the field needed for tennis training is effectively reduced, on the other hand, intelligent analysis and storage can be conducted on tennis operation, immersive training, follow-up relevant data playback and the like of a user are facilitated, and the training efficiency and effect are better improved.

Description

Technical Field

[0001] The present invention relates to the field of sporting goods, and in particular to an intelligent tennis training system. Background Art

[0002] Tennis is a beautiful and exciting sport that has long been loved by many. Tennis courts are mainly divided into four types: grass, hard, clay, and carpet. According to the International Tennis Federation, a standard tennis court is 23.77 meters long and 10.97 meters wide. The distance between the singles sidelines is 8.23 ​​meters, the distance between the doubles sidelines is 10.97 meters, and the service line is 6.4 meters from the net. Therefore, tennis training requires sufficient space for every practice, and individual training cannot accurately determine the ball's landing point. Summary of the Invention

[0003] The object of the present invention is to provide an intelligent tennis training system to solve one or more of the above-mentioned problems in the prior art.

[0004] The present invention proposes an intelligent tennis training system, comprising a virtual construction component, an image acquisition component and a control unit, wherein the virtual construction component and the image acquisition component are both connected to the control unit.

[0005] The virtual construction component is used to construct a virtual scene of tennis training on the training ground;

[0006] The image acquisition component is used to collect relevant information about the ball on the training ground, and the relevant information about the ball includes the running trajectory of the ball;

[0007] The control unit is used to obtain information about the virtual scene and related information about the ball, and analyze the subsequent trajectory of the ball based on the information.

[0008] In some specific implementations,

[0009] The virtual construction component includes a projection device and a screen, wherein the projection device is used to project the virtual image stored in the control unit onto the screen;

[0010] The image acquisition component includes at least a camera system, which is mounted on the top of the screen to capture and record the user's batting process to form a corresponding video and transmit the video to the control unit. The camera system includes at least two cameras.

[0011] In some specific implementations, the intelligent tennis training system has a ball path prediction function, and the execution process of the ball path prediction function includes the following steps:

[0012] Identify a sphere in motion;

[0013] Capture the keyframes of the swing hitting the ball;

[0014] Get several frames of the ball's flight starting from the key frame, and combine them with the key frame to form the data of the shot;

[0015] The position of the ball in each frame of the shot data is calculated and fitted, and the movement trend of the ball in the three-dimensional coordinate system after hitting the curtain is calculated. The movement trend includes the ball's flight direction, flight speed, subsequent flight trajectory and landing point.

[0016] In some specific implementations, the control unit uses Pytorch to train a model of the moving sphere, captures the pixel position of the sphere in each frame of the video through the model, and captures the key frames of the user swinging the racket to hit the sphere.

[0017] In some specific implementations, multiple batting videos are obtained and stored, with a portion of the batting videos used as a training video set and the remaining portion used as a testing video set. The model training includes the following steps:

[0018] S2.1. Convert the batting videos in the training video set into images frame by frame;

[0019] S2.2. Label the pixel locations corresponding to the targets in the image, and the labeled images form the training set.

[0020] S2.3, training, build and apply the training set to train the model;

[0021] S2.4, testing: import the video from the test video set into the model for testing. If the model can recognize the target frame by frame in the video, and the recognition accuracy is greater than the pre-set threshold, it can be determined as accurate recognition and model training is complete; otherwise, it is determined as inaccurate recognition and the process proceeds to steps S2 and S5;

[0022] S2.5. Add a new batting video to the training video collection and then re-enter S2.1.

[0023] In some specific implementations, the camera system includes two cameras, and the control unit calculates the straight-line distance between the sphere and the camera based on the input position difference between the two cameras. The three-dimensional coordinates of the sphere in different frames are calculated through the straight-line distance between the sphere and the camera and the angle formed by the center point of the camera's field of view and the sphere. The three-dimensional coordinates of all the spheres in the current hitting data are fitted, and the movement trend of the sphere in the three-dimensional coordinate system after hitting the curtain can be calculated.

[0024] In some specific implementations, the two cameras are a left camera and a right camera, and the straight-line distance between the sphere and the cameras is calculated as follows:

[0025] Set the distance between the projection centers of the two cameras to b, the three-dimensional space point of the sphere to P, and the imaging point of the left camera to P L , the imaging point of the right camera is P R , the three-dimensional space point P is the intersection of the projection center point of the two cameras and the imaging point, X L and X R are the distances from the imaging points of the left camera and the right camera to their respective imaging surfaces, respectively. The parallax d of the three-dimensional space point P between the two cameras is

[0026] d=|X L -X R |,

[0027] Two imaging points P L To P R The distance P between L P R for

[0028]

[0029] According to the theory of similar triangles

[0030]

[0031] It can be seen that the distance Z from the three-dimensional space point P to the projection plane is

[0032]

[0033] This distance Z is the Z-axis coordinate value Z of the sphere in the camera coordinate system c .

[0034] In some specific implementations,

[0035] The coordinate values ​​of the sphere on the X and Y axes are c and Y c They are:

[0036]

[0037] The coordinate information of the sphere (X c , Y c , Z c ) is converted to the coordinate information in the world coordinate system (X w , Y w , Z w ), the conversion formula is:

[0038]

[0039] In the above formula, R represents the rotation of the coordinate system, and t represents the translation of the coordinate system.

[0040] In some specific implementations, the corresponding coordinate information is passed in using Python's spline interpolation during the fitting process to obtain the flight trajectory of the sphere, Z w When it is 0, the corresponding coordinate information is the coordinate information of the landing point.

[0041] In some specific implementations, the intelligent tennis training system also has one or more functions of detecting serve accuracy, ball speed, and player running distance.

[0042] The execution process of detecting the serve hit rate is as follows: using ResNet50 to detect the user, and placing the bottom midpoint of the detection box in the world coordinate system to determine whether the user is on the left half or the right half of the court. The balls served by the user are counted, and the landing point of each serve is calculated. The number of balls that landed in bounds is divided by the total number of serves to obtain the user's serve hit rate.

[0043] The process for detecting ball speed is as follows: starting with the keyframe where the player hits the tennis ball, the 3D coordinates of the ball in the world coordinate system are recorded for 3-5 frames after this frame. The trajectory of the ball in these frames is treated as a straight line connected from beginning to end. The distance traveled is calculated from the initial and final coordinates. The time taken to travel this distance is calculated based on the camera frame rate. The distance traveled is then divided by the time to obtain the ball speed.

[0044] The execution process of detecting the player's running distance is as follows: each time the player hits the ball, it will be recorded as a keyframe. Each time a keyframe is obtained, the world coordinate system 3D coordinates of the bottom midpoint of the detection box obtained by ResNet50 for the user's detection will be recorded. After the training is completed, the total distance run by the player will be calculated by summing up the standing positions.

[0045] The advantages of the intelligent tennis training system of the present invention are:

[0046] It can effectively reduce the limitation of court size when playing tennis;

[0047] It can intelligently identify the trajectory of the ball, estimate the trajectory of the ball after it hits the screen, know the possible landing points, and save relevant data so that users can view the relevant data of the training later, so that users can enter immersive interactive training. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A flowchart of the operation of the intelligent tennis training system in some embodiments of the present invention;

[0049] Figure 2 A flowchart of model training in some embodiments of the present invention;

[0050] Figure 3Schematic diagram of the relative positions of two cameras and a sphere in some embodiments of the present invention;

[0051] Figure 4 Schematic diagram of imaging of a sphere by a camera in some embodiments of the present invention. DETAILED DESCRIPTION

[0052] The present invention proposes an intelligent tennis training system, comprising a virtual construction component, an image acquisition component and a control unit, wherein the virtual construction component and the image acquisition component are both connected to the control unit.

[0053] The virtual construction component is used to construct a virtual scene of tennis training on the training ground, wherein the virtual construction component may include a projection device and a screen, and the projection device is used to project the virtual image stored in the control unit onto the screen;

[0054] The image acquisition component is used to collect information about the ball on the training ground, including the ball's trajectory. The image acquisition component includes at least a camera system, which is mounted on top of the screen (the camera system is generally installed on the top of the screen to shoot at a slightly downward angle, starting from the horizontal position, and the downward angle is generally 0-90 degrees, mainly based on a clear view of the tennis court. The specific angle can be adjusted manually). It is used to record the user's hitting process to form a corresponding video, and transmit the video to the control unit. The camera system includes at least two cameras. The following content takes a camera system including two cameras as an example, and the two cameras are respectively a left camera and a right camera;

[0055] The control unit is used to obtain information about the virtual scene and related information about the ball, and analyze the subsequent trajectory of the ball based on this information.

[0056] Combine Figure 1 The content shown in the figure shows that the intelligent tennis training system has a ball path prediction function. The execution process of the ball path prediction function includes the following steps:

[0057] Obtaining a ball-hitting video captured by a camera system. Generally, the ball-hitting video includes the video from the moment the racket hits the ball until the ball leaves the camera's field of view.

[0058] Identify a sphere in motion;

[0059] Capture the keyframes of the swing hitting the ball;

[0060] Get several frames of the ball's flight starting from the key frame (at least 3 frames, at most all frames from the key frame until the ball leaves the camera's field of view), and combine them with the key frame to form the data of the shot;

[0061] The position of the ball in each frame of the shot data (e.g., 3D coordinates) is calculated and fitted, and the movement trend of the ball in the 3D coordinate system after hitting the curtain is calculated. The movement trend includes the ball's flight direction, flight speed, subsequent flight trajectory, and landing point.

[0062] During the execution of the ball path prediction function, it is also possible to determine whether the ball falls within the bounds based on the calculated landing point. At the same time, the existing scoring program can be implanted in the control unit to calculate the score based on the location of the landing point. Generally, the Z axis can be pre-set to 0, which is the limit value corresponding to the value range of the X axis and Y axis. When the ball lands (that is, the value corresponding to the Z axis in the current three-dimensional coordinates of the ball is zero), and the corresponding values ​​of the X axis and Y axis are both within the pre-set value range of the X axis and Y axis, the ball is determined to have fallen within the bounds. Otherwise, the ball is determined to have not fallen within the bounds.

[0063] The control unit can use Pytorch to train a model of the moving sphere, capture the pixel position of the sphere in each frame of the video through the model, and capture the key frames of the user swinging the racket to hit the sphere.

[0064] Get multiple batting videos and store them. Some of them are used as training video sets, and the remaining ones can be used as test video sets. The number of batting videos in these two types of video sets can be randomly divided. Figure 2 As shown in the figure, the above model training includes the following steps:

[0065] S2.1. Convert the batting videos in the training video set into images frame by frame;

[0066] S2.2. Label the pixel locations corresponding to the target (e.g., a sphere) in the image. This provides the target image pixel locations for the model in subsequent model training. The labeled images constitute the training set (generally, the training set includes at least 2,000 labeled images).

[0067] S2.3, training, build and apply the training set to train the model;

[0068] S2.4, testing: import the video from the test video set into the model for testing. If the model can recognize the target (i.e., the sphere) in each frame of the video, and the recognition accuracy is greater than a preset threshold (e.g., 90%), it can be determined as accurate recognition and model training is complete. Otherwise, it is determined as inaccurate recognition and the process proceeds to steps S2, 5.

[0069] S2.5. Add a new batting video to the training video collection and then re-enter S2.1.

[0070] The steps of model training may also include: S2.6, saving the image in S2.1, randomly adjusting the brightness of the image in S2.1 and saving it separately, thereby obtaining more materials with different brightness, and then performing S2.2.

[0071] The object whose brightness is adjusted and saved in S2.6 can also be adjusted to the annotated image described in S2.2.

[0072] During the execution of S2.3, a gradient descent optimization algorithm can be used to calculate the gradient of the loss function with respect to the weight, and then the weight is updated in the opposite direction of the gradient, so that the loss function gradually decreases, thereby obtaining a more effective training effect.

[0073] The control unit calculates the straight-line distance between the sphere and the camera based on the position difference between the two cameras, and calculates the 3D coordinates of the sphere in different frames through the straight-line distance between the sphere and the camera and the angle between the camera's view center and the sphere. By fitting the 3D coordinates of all the spheres in the current hitting data, the movement trend of the sphere in the 3D coordinate system after hitting the curtain can be calculated. Figure 3 The calculation process of the straight-line distance between the sphere and the camera is as follows:

[0074] Set the distance between the projection centers of the two cameras to b, the three-dimensional space point of the sphere to P, and the imaging point of the left camera to P L , the imaging point of the right camera is P R , the three-dimensional space point P (i.e. the point of the tennis ball in the three-dimensional space) is the intersection of the projection center point of the two cameras and the line connecting the imaging point, X L and X R are the distances from the imaging points of the left camera and the right camera to their respective imaging surfaces, respectively. The parallax d of the three-dimensional space point P between the two cameras is

[0075] d=|X L -X R |,

[0076] Two imaging points P L To P R The distance P between L P R for

[0077]

[0078] According to the theory of similar triangles

[0079]

[0080] In the above formula, f represents the focal length of the camera.

[0081] It can be seen that the distance Z from the three-dimensional space point P to the projection plane is

[0082]

[0083] This distance Z can be used as the Z-axis coordinate value Z of the sphere in the camera coordinate system c .

[0084] The image of the sphere in the camera is as follows Figure 4 As shown, the coordinate values ​​of the sphere on the X-axis and Y-axis in the camera coordinate system are c and Y c They are:

[0085]

[0086] The coordinate information of the sphere (X c , Y c , Z c ) is converted to coordinate information in the world coordinate system (X w , Y w , Z w ), to facilitate subsequent data processing, the specific conversion formula is:

[0087]

[0088] In the above formula, R represents the rotation of the coordinate system, and t represents the translation of the coordinate system.

[0089] The rotation amount is a matrix, which rotates the camera coordinate system around the X, Y, and Z axes by a certain angle so that it points in the same direction as the three axes of the world coordinate system. Taking the rotation around the Z axis as an example, the process of calculating the Z-axis rotation matrix R1 is as follows:

[0090]

[0091] In this formula, x, y, and z are the coordinate information of the camera coordinate system respectively. ′ 、y ′ 、z ′ is the corresponding coordinate information in the world coordinate system, θ is the angle of rotation along the Z axis, z=z ′ The rotation matrices R2 and R3 of the X-axis and Y-axis can also be calculated, R = R1*R2*R3.

[0092] The translation is still a matrix, which represents the relative position of the origin of the two coordinate axes. These calculation processes can be directly implemented using existing technologies, so they will not be described in detail here.

[0093] The two data R and t are determined by the position and rotation angle of the left camera and the right camera.

[0094] In the above fitting process, the corresponding coordinate information can be passed in using Python's spline interpolation to obtain the flight trajectory of the sphere and the function corresponding to the trajectory in the world coordinate system, Z w The coordinate information corresponding to 0 is the coordinate information of the landing point, and the flight trajectory can also be used to determine the flight direction and speed of the ball at a certain point based on the position of the curtain. The solution involved can be directly implemented using existing technology, so it will not be described in detail here.

[0095] Using Unity Engine's 3D modeling technology, a virtual tennis training scene can be constructed and modeled to create a corresponding 3D model. The 3D model will receive the ball's flight trajectory and display it on the 3D model. The projection device can directly provide the image provided by the 3D model, so the ball's trajectory and landing point can be displayed on the screen in a timely manner. This technology can be directly implemented using existing technologies, so it will not be detailed here.

[0096] In addition, the intelligent tennis training system also has one or more functions of detecting serve hit rate, ball speed, and player running distance.

[0097] The execution process of detecting the serve hit rate is as follows: using ResNet50 to detect the user (the detected person swinging the racket, i.e., the player), and placing the bottom midpoint of the detection box in the world coordinate system to determine whether the user is on the left half or the right half of the court. The balls served by the user are counted, and the landing point of each serve is calculated. The number of balls that landed in bounds is divided by the total number of serves to obtain the user's serve hit rate.

[0098] The process for detecting ball speed is as follows: starting with the key frame where the player hits the tennis ball, extract 3-5 frames after this frame, where the interval between two adjacent frames is the same. The 3D coordinates of the ball in the world coordinate system are obtained in each frame. The trajectory of the ball in these frames is treated as a straight line connected from beginning to end. The distance traveled is calculated from the initial and final coordinates. The time taken to travel this distance is calculated based on the camera's frame rate. The distance traveled is then divided by the time to obtain the ball speed.

[0099] The execution process of detecting the player's running distance is as follows: each time the player hits the ball, it will be recorded as a keyframe. Each time a keyframe is obtained, the world coordinate system 3D coordinates of the bottom midpoint of the detection box obtained by ResNet50 for the user's detection will be recorded. After the training is completed, the total distance run by the player will be calculated by summing up the standing positions.

[0100] The control unit can be connected to external devices, including smart terminals such as mobile phones and laptops, and forward the information received, calculated, analyzed, and acquired by the control unit to these external devices for storage and display. The control unit can also be equipped with a separate storage unit to store the information received, calculated, analyzed, and acquired.

[0101] The intelligent tennis training system also has random point training. The implementation process of random point training can be as follows: the intelligent tennis training system divides the projection area on the screen into several partitions, and highlights a certain point randomly or according to a pre-set highlight processing rule (this rule can be considered a setting change and can be directly implemented using existing technology, so it will not be described here). The value of the ball reaching the plane of the three-dimensional coordinates of the screen is calculated through the ball trajectory, that is, the point hitting the screen is calculated through the ball trajectory, and the point hitting judgment is made accordingly. If the calculated hitting position is in the scoring point, it is recorded as a score. The implementation process of random point training can also be achieved by installing one or more cameras that can capture images of the entire area of ​​the screen opposite the screen. The intelligent tennis training system divides the projection area on the screen into several partitions, and highlights a certain partition randomly or according to a pre-set highlight processing rule (this rule can be considered a setting change and can be directly implemented using existing technology, so it will not be described here). The camera captures the image of the ball hitting the screen, and analyzes whether the impact position of the ball coincides with the highlighted partition. If it coincides, it is recorded as a score.

[0102] The intelligent tennis training system can also automatically determine the player's training level based on the scoring situation (for example, pre-setting the scoring standard value, and dividing the player's training level by comparing the current score with the scoring standard value), and provide appropriate training difficulty (for example, changing the size of the highlighted partition and the frequency of the highlighted partition change).

[0103] The above undisclosed matters can be directly implemented using existing technologies and are therefore not described in detail here.

[0104] The above-mentioned intelligent tennis training system can also be directly applied to other ball training. It is only necessary to change the detection object from tennis to the corresponding sphere.

[0105] The above is only a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several similar modifications and improvements without departing from the creative concept of the present invention, and these should also be considered to be within the scope of protection of the present invention.

Claims

1. An intelligent tennis training system, characterized in that: It includes a virtual construction component, an image acquisition component and a control unit, wherein the virtual construction component and the image acquisition component are both connected to the control unit. The virtual construction component is used to construct a virtual scene of tennis training on the training ground; The image acquisition component is used to collect relevant information about the ball on the training ground, and the relevant information about the ball includes the running trajectory of the ball; The control unit is used to obtain information about the virtual scene and related information about the ball, and analyze the subsequent trajectory of the ball based on the information.

2. An intelligent tennis training system according to claim 1, wherein: The virtual construction component includes a projection device and a screen, wherein the projection device is used to project the virtual image stored in the control unit onto the screen; The image acquisition component includes at least a camera system, which is mounted on the top of the screen to capture and record the user's batting process to form a corresponding video and transmit the video to the control unit. The camera system includes at least two cameras.

3. An intelligent tennis training system according to claim 2, wherein: The intelligent tennis training system has a ball path prediction function, and the execution process of the ball path prediction function includes the following steps: Identify a sphere in motion; Capture the keyframes of the swing hitting the ball; Get several frames of the ball's flight starting from the key frame, and combine them with the key frame to form the data of the shot; The position of the ball in each frame of the shot data is calculated and fitted, and the movement trend of the ball in the three-dimensional coordinate system after hitting the curtain is calculated. The movement trend includes the ball's flight direction, flight speed, subsequent flight trajectory and landing point.

4. An intelligent tennis training system according to claim 3, wherein: The control unit uses Pytorch to train a model of the moving sphere, captures the pixel position of the sphere in each frame of the video through the model, and captures the key frames of the user swinging the racket to hit the sphere.

5. An intelligent tennis training system according to claim 4, wherein: A plurality of batting videos are obtained and stored, with a portion of the batting videos used as a training video set and the remaining portion used as a testing video set. The model training includes the following steps: S2.

1. Convert the batting videos in the training video set into images frame by frame; S2.

2. Label the pixel locations corresponding to the targets in the image, and the labeled images form the training set. S2.3, training, build and apply the training set to train the model; S2.4, testing: import the video from the test video set into the model for testing. If the model can recognize the target frame by frame in the video, and the recognition accuracy is greater than the pre-set threshold, it can be determined as accurate recognition and model training is complete; otherwise, it is determined as inaccurate recognition and the process proceeds to steps S2 and S5; S2.

5. Add a new batting video to the training video collection and then re-enter S2.

1.

6. An intelligent tennis training system according to claim 3, wherein: The camera system includes two cameras. The control unit calculates the straight-line distance between the sphere and the camera based on the input position difference between the two cameras. The three-dimensional coordinates of the sphere in different frames are calculated using the straight-line distance between the sphere and the camera and the angle formed by the center point of the camera's field of view and the sphere. The three-dimensional coordinates of all the spheres in the current hitting data are fitted to calculate the movement trend of the sphere in the three-dimensional coordinate system after hitting the curtain.

7. An intelligent tennis training system according to claim 6, wherein: The two cameras are the left camera and the right camera. The straight-line distance between the sphere and the camera is calculated as follows: Set the distance between the projection centers of the two cameras to b, the three-dimensional space point of the sphere to P, and the imaging point of the left camera to P L , the imaging point of the right camera is P R , the three-dimensional space point P is the intersection of the projection center point of the two cameras and the imaging point, X L and X R are the distances from the imaging points of the left camera and the right camera to their respective imaging surfaces, then the parallax d of the three-dimensional space point P between the two cameras is d = |X L -X R |, Two imaging points P L To P R The distance P between L P R for According to the theory of similar triangles It can be seen that the distance Z from the three-dimensional space point P to the projection plane is This distance Z is the Z-axis coordinate value Z of the sphere in the camera coordinate system c .

8. An intelligent tennis training system according to claim 7, wherein: The coordinate values ​​of the sphere on the X and Y axes are c and Y c They are: The coordinate information of the sphere (X c , Y c , Z c ) is converted to the coordinate information in the world coordinate system (X w , Y w , Z w ), the conversion formula is: In the above formula, R represents the rotation of the coordinate system, and t represents the translation of the coordinate system.

9. An intelligent tennis training system according to claim 8, wherein: During the fitting process, the corresponding coordinate information is passed in by using Python's spline interpolation to obtain the flight trajectory of the sphere. w When it is 0, the corresponding coordinate information is the coordinate information of the landing point.

10. An intelligent tennis training system according to claim 9, wherein: The intelligent tennis training system also has one or more functions of detecting serve accuracy, ball speed, and player running distance. The execution process of detecting the serve hit rate is as follows: using ResNet50 to detect the user, and placing the bottom midpoint of the detection box in the world coordinate system to determine whether the user is on the left half or the right half of the court. The balls served by the user are counted, and the landing point of each serve is calculated. The number of balls that landed in bounds is divided by the total number of serves to obtain the user's serve hit rate. The process for detecting ball speed is as follows: starting with the keyframe where the player hits the tennis ball, the 3D coordinates of the ball in the world coordinate system are recorded for 3-5 frames after this frame. The trajectory of the ball in these frames is treated as a straight line connected from beginning to end. The distance traveled is calculated from the initial and final coordinates. The time taken to travel this distance is calculated based on the camera frame rate. The distance traveled is then divided by the time to obtain the ball speed. The execution process of detecting the player's running distance is as follows: each time the player hits the ball, it will be recorded as a keyframe. Each time a keyframe is obtained, the world coordinate system 3D coordinates of the bottom midpoint of the detection box obtained by ResNet50 for the user's detection will be recorded. After the training is completed, the total distance run by the player will be calculated by summing up the standing positions.

Images