Automatic intelligent training method and system based on ball serving robot

Abstract

According to the automatic intelligent training method and system based on the serving robot, camera visual perception is matched with the serving robot to achieve automatic intelligent training of a ball training scene, manual ball feeding of a coach or partner training is replaced, and manpower is saved; computer vision and artificial intelligence technologies are utilized to realize visual perception of a ball hitting track of a trainer, a ball serving combination of the ball serving robot is automatically triggered or adjusted by utilizing a visual perception result, intelligent training is realized, a complicated ball path programming process is avoided, training is closer to actual combat, and training efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method for performing intelligent training using a serving robot. Background Art

[0002] The current ball training usually adopts the following methods:

[0003] Option 1: The coach or sparring partner feeds the ball manually.

[0004] Option 2: Program the ball machine to preset the ball path for semi-automatic training.

[0005] The main disadvantages of these methods are:

[0006] Option 1: The coach or sparring partner feeds the ball manually, which requires an experienced athlete or coach;

[0007] Solution 2: Program the ball machine with preset ball paths for semi-automatic training. The preset ball paths for the ball machine can only set fixed ball paths or ball path combinations, and cannot intelligently match the trainer's ability and batting rhythm, resulting in the ball path of the ball machine being out of the trainer's ability range or not being consistent with actual combat (such as being unable to connect the trainer's serve for training). If you want to program a preset ball path that matches the trainer's ability and rhythm, you need to repeatedly program and try, and the process is complicated. Summary of the invention

[0008] The present disclosure provides a method and system for intelligent training using a serving robot, which uses camera visual perception to cooperate with the serving robot to realize automated intelligent training in ball training scenarios, and can replace the manual feeding of balls by coaches or sparring partners, saving manpower. The method uses computer vision and artificial intelligence technology to realize visual perception of the hitting trajectory of the trainer, and then uses the visual perception result to automatically trigger or adjust the serving combination of the serving robot, realizing automated intelligent training, avoiding the complicated ball path programming process, and the training is closer to actual combat.

[0009] The automated intelligent training method based on the serving robot provided in the present disclosure mainly comprises the following steps:

[0010] S1, set the serve training mode;

[0011] S2, the serving robot serves the first ball and waits for the trainer to return the ball; or the serving robot waits for the trainer to serve the first ball;

[0012] S3, obtaining a partial trajectory of the trainer's hitting the ball through a visual perception device, and performing subsequent trajectory prediction calculation based on the segment of the trajectory to obtain a complete trajectory of the trainer's hitting the ball and the remaining time;

[0013] S4, the serving robot determines whether the trajectory of the ball hit by the trainee given by the visual perception device is a valid ball;

[0014] S5, if it is a valid ball, serve the ball to the training personnel according to the set training mode, wherein:

[0015] In the first training mode, the serving robot serves the first ball in the set serving combination when the remaining time of the trainee's hitting trajectory ends, and then serves the remaining balls in the set serving combination in sequence at the set time intervals;

[0016] In the second training mode, after the serving robot serves the ball, it continuously repeats steps S3-S5, that is, each time it judges that the ball is a valid ball based on the complete hitting trajectory of the trainee output by the visual perception device, it serves the next ball to the trainee according to the preset training mode.

[0017] Furthermore, the training mode includes:

[0018] The first type of training mode includes: a serving connection mode, in which after the serving robot serves the first ball of the set built-in combination, no matter how the trainer returns the ball, it will serve the balls of the set built-in combination in sequence according to the built-in fixed serving frequency;

[0019] The second type of training mode includes: rhythm coordination mode and simulated combat mode, among which:

[0020] In the rhythm coordination mode, after serving each ball, the serving robot will use the visual perception device to obtain the trainer's effective return before serving the next ball, so that the serving frequency matches the trainer's return frequency, and the set built-in combination of balls is served in sequence;

[0021] In the simulated combat mode, the serving robot automatically selects the possible return paths of the opponents in actual combat according to the trainees' return balls, and matches the trainees' return frequency, so that the serving frequency and ball path of the serving robot simulate the real return balls of the opponents in actual combat.

[0022] Furthermore, the step S3 specifically includes:

[0023] S31, using binocular cameras to respectively take pictures of the ball practice scene in real time, and respectively detect the ball targets in the images; respectively track the ball target detection results in real time; and obtain the trajectory of the ball by matching and binocular 3D reconstruction of the binocular tracking results;

[0024] S32, performing subsequent trajectory prediction calculation based on the trajectory obtained in step S31 to obtain a complete training personnel's hitting trajectory and remaining time T.

[0025] Furthermore, the subsequent trajectory prediction and calculation step specifically includes:

[0026] The x, y, and z coordinates of the input ball trajectory are fitted with the time t respectively, and the complete ball trajectory and the remaining flight time T are directly obtained through the fitting coefficient and the prior landing point z value;

[0027] Alternatively, a neural network model is built to predict the subsequent trajectory and remaining flight time of the ball.

[0028] Furthermore, the determination conditions of the valid ball in step S4 include:

[0029] In the serve connection mode and the simulated actual combat mode, the following conditions must be met at the same time: the ball's flight direction is from the trainer to the serve machine, the ball has no contact with the court before flying over the net, and the first contact point with the court after flying over the net is within the serving area.

[0030] In the rhythm coordination mode, the effective ball means: the flight direction of the ball is from the trainer to the ball machine.

[0031] Furthermore, when the simulated combat training mode is adopted, the serving robot serves the ball each time according to the serving parameters selected by the probability distribution function; the serving parameters include: one or more of the ball speed, landing point, rotation direction, rotation speed, and net height;

[0032] Methods for selecting serve parameters through probability distribution functions include:

[0033] First, determine whether the parameters to be selected are the parameters of the first serve. If so, directly and randomly generate a set of serve parameters of the serve machine without input through the serve probability distribution model;

[0034] If not, the complete trajectory and the remaining time T received from the visual perception device are input into the ball return probability distribution model to obtain a set of serving parameters of the serving machine.

[0035] Furthermore, the method for constructing the serve probability distribution model and the return probability distribution model includes:

[0036] 1) Collect several professional ball game videos as a dataset;

[0037] 2) All rounds from a certain serve to the nearest next serve are called a "score", and the game video is divided into several video clips according to the "score", and each video clip is a complete score;

[0038] 3) For each video clip, the AI ​​ball detection model is used to automatically detect the ball in the game video and track the flying ball. Each shot in the score is divided according to the flight direction of the ball to form a record similar to "athlete A serves -> athlete B receives the serve -> athlete A returns the ball -> athlete B returns the ball -> ... until the last shot of the score". At the same time, the start and end time of each shot, the complete trajectory information of this shot and its corresponding ball parameters, and the remaining trajectory flight time information after the ball flies to the net. The ball parameters include: one or more of the ball speed, landing point, rotation direction, rotation speed, and net height;

[0039] The start and end time of each ball shot are obtained by tracking the flying ball and judging it based on the condition of the reverse direction of flight. The complete trajectory information is obtained by dividing the ball detection and tracking results according to the start and end time of each ball shot. The ball speed, landing point, net height, and the remaining trajectory flight time after the ball flies to the net are calculated by fitting the complete trajectory information. The rotation direction and rotation speed are marked by experienced experts.

[0040] 4) The complete ball-out parameters corresponding to the first ball-out of each video clip are used as a serving data set, and the serving data set is clustered to obtain several groups of "serving parameters" and their corresponding weights, i.e., the frequencies of occurrence in the serving data set; these clustered "serving parameters" and their weights can be regarded as a serving probability distribution model;

[0041] 5) Taking the complete trajectory of the previous ball in all adjacent bouncing balls in each video clip and the remaining trajectory flight time T after the ball flies to the net as input and the ball parameters of the next ball in the adjacent bouncing balls as output, a return ball data set is formed, and a neural network model is established to train the return ball data set, and the trained neural network model is obtained as the return ball probability distribution model.

[0042] Furthermore, the method further comprises the following steps:

[0043] Before starting the automated intelligent training for the first time, or after the visual perception device is moved, the visual perception device is visually calibrated to establish a conversion relationship between the camera coordinate system and the practice field coordinate system.

[0044] An automated intelligent training system based on a serving robot using the method of the claim, mainly comprising:

[0045] A serving robot, a camera-based visual perception module, and an application interaction module, including:

[0046] An application interaction module is used to send an automated intelligent training start instruction to the serving robot and the visual perception module; and to set a training mode to control the serving robot to serve the trainer;

[0047] A visual perception module is used to obtain a portion of the trainer's hitting trajectory, and perform subsequent trajectory prediction calculation based on the segment of the trajectory to obtain the trainer's hitting trajectory and remaining time;

[0048] The serving robot is used to determine whether the trajectory of the trainee's hitting ball given by the visual perception device is a valid ball; and is used to serve the ball to the trainee according to the set training mode, wherein:

[0049] In the first training mode, the serving robot serves the first ball in the set serving combination when the remaining time of the trainee's hitting trajectory ends, and then serves the remaining balls in the set serving combination in sequence according to the set time intervals;

[0050] In the second training mode, after each ball is served, the serving robot judges that the ball is a valid ball based on the complete hitting trajectory of the trainee output by the visual perception device, and then serves the next ball to the trainee according to the preset training mode. Furthermore, the visual perception module includes the following submodules:

[0051] Visual calibration submodule: used to calibrate the visual perception of the camera before starting the automated intelligent training for the first time or after the camera is moved, and to establish a conversion relationship between the camera coordinate system and the practice court coordinate system. This coordinate conversion relationship is used to convert the image coordinates of the ball recognized by the camera into the practice court coordinate system;

[0052] Real-time recognition submodule: using binocular cameras to simulate binocular vision, binocular cameras respectively take pictures of the ball practice scene in real time, and detect the ball targets in the image respectively; then the ball target detection results are respectively tracked in real time; the tracking results of the binocular cameras are matched and binocular 3D reconstructed to obtain the trajectory of the ball;

[0053] Trajectory prediction submodule: It is used to perform subsequent trajectory prediction calculations based on the trajectory obtained by the real-time recognition submodule, and output the complete ball trajectory and the remaining time T.

[0054] Compared with the prior art, the beneficial effects of the present invention are: (1) using camera visual perception in conjunction with a serving robot to achieve automated intelligent training in training scenarios, replacing the manual feeding of balls by coaches or sparring partners, saving manpower; (2) using computer vision and artificial intelligence technology to achieve visual perception of the hitting trajectory of trainers, and using the visual perception results to automatically trigger or adjust the serving combination of the serving robot, achieving intelligent training, avoiding the complicated ball path programming process, and making the training closer to actual combat. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The above and other objects, features and advantages of the present disclosure will become more apparent through a more detailed description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, wherein like reference numerals generally represent like components throughout the exemplary embodiments of the present disclosure.

[0056] Figure 1 is a training flow chart in the serve connection mode according to the present disclosure;

[0057] Figure 2 This is a training flow chart in rhythm coordination mode;

[0058] Figure 3 This is a training flow chart simulating actual combat mode;

[0059] Figure 4 Flowchart of the method for selecting serving parameters through probability distribution function. DETAILED DESCRIPTION

[0060] The preferred embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the preferred embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0061] The present disclosure provides a method and system for intelligent training using a serving robot, which utilizes computer vision and artificial intelligence technology to achieve visual perception of a trainee's hitting trajectory, and utilizes the visual perception results to automatically trigger or adjust the serving combination of the serving robot, thereby achieving automated intelligent training.

[0062] The method and system can be applied to various ball games such as tennis, badminton, volleyball, etc. The present disclosure is further explained below using tennis training as an example.

[0063] In this example, the camera visual perception is used in conjunction with a tennis serving robot to achieve automated intelligent training in tennis training scenarios. The automated intelligent training system consists of three parts: a serving robot, camera visual perception, and application interaction software.

[0064] The automated intelligent training methods that the system can provide include: "serve connection", "rhythm coordination" and "simulated combat" modes. These modes can be set using the application interactive software before starting the automated intelligent training.

[0065] Specific methods of visual perception that are common across modalities include:

[0066] 1. Visual calibration: Before starting the automated intelligent training for the first time or after the camera is moved, the visual perception module of the camera needs to be visually calibrated. The camera takes a picture of the tennis court environment, manually specifies or automatically identifies the intersection of the standard tennis court lines (you can also choose the built-in calibration reference, or other standard feature points of the court such as the net, court edge, etc.), and combines the real court coordinates corresponding to the intersection of the standard tennis court lines in the identified image to calibrate the conversion relationship between the camera coordinate system and the tennis court coordinate system. This coordinate conversion relationship is used to convert the image coordinates of the tennis ball identified by the camera to the tennis court coordinate system, which is conducive to the calculation of the 3D points, trajectory status and related parameters of the tennis ball trajectory required for automated intelligent training.

[0067] 2. Real-time recognition through visual perception: Use binocular cameras to take pictures of tennis scenes in real time, apply deep learning tennis target detection algorithms to the 2D images of the two cameras respectively, and obtain the tennis target detection results in their respective images; respectively track the high-frame rate tennis target detection results in the binocular images in real time to obtain the binocular tennis tracking results, and obtain the tennis trajectory through matching and binocular 3D reconstruction of the binocular tennis tracking results.

[0068] 3. Trajectory prediction: Predict the output tennis trajectory and remaining time T through the input tennis trajectory. One method is to fit the x, y, and z coordinates of the input tennis trajectory with the time t respectively, and directly obtain the output tennis trajectory and remaining time T through the fitting coefficient and the prior landing point z value; another method is to establish a neural network model to predict the tennis trajectory and remaining time.

[0069] The following explains the automated intelligent training processes of the three modes: "serve connection", "rhythm coordination" and "simulated combat".

[0070] 1. Serve Connection Mode

[0071] The serve connection mode is mainly used to train the trainee's own serve connection. After the camera visual perception module recognizes the trainee's effective serve, it triggers the serving robot to serve the set serve connection built-in combination (such as: one shot topspin + one shot volley + one shot high pressure + five consecutive topspin, or twenty consecutive topspins at two points on the left and right of the backcourt, or ten consecutive rounds of one-after-one serve combination, etc. How the built-in combination is set does not affect the technical implementation scheme of the present invention).

[0072] The overall process is as follows Figure 1 The detailed process is as follows:

[0073] (1) First, ensure that the camera visual perception module is visually calibrated before starting the automated intelligent training for the first time or after the camera is moved. The specific method is as described above.

[0074] (2) Then, the training mode is set to the serve connection mode using the application interactive software, and the built-in combination of the serve machine is set, such as the built-in combination of serve connection mentioned above, to start the automated intelligent training;

[0075] (3) After the interactive software is used to start the automated intelligent training, the serving robot and the camera visual perception module will both start and perform corresponding initialization, and then wait for the trainer to serve. Specifically, after the camera visual perception module receives the command to start the automated intelligent training, it will first start visual perception, turn on the camera video stream and the corresponding recognition algorithm, and then perceive and recognize the scene in real time; after the serving robot receives the command to start the automated intelligent training, it will first start the serving machine and enable the motors of each joint; then, according to the set built-in combination, it will prepare for serving (adjust the state of each motor to the state that can serve the first ball in the built-in combination as quickly as possible) and then wait to receive the results of the camera visual perception module;

[0076] (4) After the trainer serves, the visual perception module captures a portion of the trainer's hitting trajectory (in this embodiment, the portion of the trajectory is defined as the trajectory segment from the hitting of the ball to the first flight of the ball over the net), inputs the identified segment of the trajectory into the trajectory prediction algorithm module, and outputs the predicted complete hitting trajectory of the trainer (the reason for the need for prediction is that the serving machine needs to obtain the hitting trajectory information in advance so that there is enough time to drive the motor to rotate and serve the ball, so as to simulate the rhythm of the real player returning the ball after the player serves);

[0077] (5) At this time, the serving robot receives the complete trajectory of the trainer's hitting ball sent by visual perception, and determines whether the trajectory is a valid serve (the ball's flight direction is from the trainer to the serving machine, the ball does not touch the court before flying over the net, and the first contact point with the court after flying over the net is within the serving area).

[0078] If it is not a valid serve, continue to wait for receiving the visual perception result; if it is a valid serve, according to the received remaining time T, control the motor to complete the serve of the first ball in the set built-in combination at the end of the remaining time, and serve the set built-in combination in sequence.

[0079] (6) After completing the built-in combination serve, enter the serve preparation state again and wait to receive the visual perception result (the trainer's next serve).

[0080] By repeating this cycle, you can continuously and automatically train the connection after serving.

[0081] 2. Rhythm Synergy Mode

[0082] The rhythm coordination mode is used to automatically adjust the set built-in combination of serving frequencies according to the trainer's return, so that the serving frequency of the ball machine automatically matches the trainer's return frequency, achieving the effect of simulating real-life training (rather than serving the built-in fixed frequency ball regardless of how the trainer returns the ball). The overall process is referenced Figure 2 , the detailed process is as follows:

[0083] (1) First, ensure that the camera visual perception module is visually calibrated before starting the automated intelligent training for the first time or after the camera is moved. The specific method is as described above.

[0084] (2) Then, the application interactive software is used to set the training mode to the rhythm coordination mode, and the built-in combination of the serving machine is set, such as topspin at two points on the left and right sides of the backcourt for 20 consecutive shots, to start the automated intelligent training;

[0085] (3) After the interactive software starts the automated intelligent training, the serving robot and the camera visual perception module will start and initialize accordingly, and then wait for the trainer to return the ball; the details are as follows:

[0086] After receiving the command to start the automated intelligent training, the camera visual perception module first starts the visual perception, turns on the camera video stream and the corresponding recognition algorithm, and then perceives and recognizes the scene in real time; after receiving the command to start the automated intelligent training, the serving robot first starts the serving machine and enables the motors of each joint;

[0087] Then, the serving robot serves the first ball of the set built-in combination according to the set built-in combination, and then prepares to serve the next ball in the set built-in combination (adjusts the state of each motor to the state that can serve the next ball in the set built-in combination as quickly as possible) and waits for the result of receiving the camera visual perception module;

[0088] (4) After the trainee receives the ball, the visual perception module captures a portion of the trainee's hitting trajectory (in this embodiment, the portion of the trajectory is defined as the trajectory segment from hitting the ball to the ball's first flight over the net), inputs the identified segment of the trajectory into the trajectory prediction algorithm module, and outputs the predicted complete trainee's hitting trajectory (the reason for the need for prediction is that the serving machine needs to obtain the hitting trajectory information in advance so that there is enough time to drive the motor to rotate and serve the ball, so as to simulate the rhythm of the real player returning the ball after the player serves);

[0089] (5) At this time, the serving robot receives the complete trajectory of the trainer's hitting ball sent by visual perception, and determines whether the trajectory is a valid ball (the ball's flight direction is from the trainer to the serving machine). If it is not a valid ball, it continues to wait for the visual perception result. If it is a valid ball, it controls the motor according to the received remaining time T to serve the prepared ball at the end of the remaining time (if the remaining time T cannot be estimated due to the trainer's return error, the remaining time is set to the maximum remaining time T_max, for example, 5 seconds);

[0090] (6) Then, it is determined whether all the preset built-in combinations have been served. If so, the rhythm coordination exercise ends. If not, the ball serving machine is made ready to serve the next ball in the preset built-in combination (adjusting the state of each motor to the state in which the next ball in the preset built-in combination can be served as quickly as possible);

[0091] This cycle repeats itself to automatically match the trainee's ball return rhythm until all the balls in the set built-in combination are served.

[0092] 3. Simulation combat mode

[0093] The simulated combat mode starts with serving. It automatically selects the possible return path of the opponent in actual combat according to the trainee's return, and matches the trainee's return frequency, so that the serving frequency and path of the serving machine are close to the real return of the opponent in actual combat, achieving the purpose of simulated combat training.

[0094] Overall process reference Figure 3 , the detailed process is as follows:

[0095] (1) First, ensure that the camera visual perception module is visually calibrated before starting the automated intelligent training for the first time or after the camera is moved. The specific method is as described above.

[0096] (2) Then use the application interactive software to set the training mode to simulate actual combat mode and start automated intelligent training;

[0097] (3) After the interactive software starts the automated intelligent training, the serving robot and the camera visual perception module will start and initialize accordingly, and then wait for the trainer to return the ball; the details are as follows:

[0098] After receiving the command to start the automated intelligent training, the camera visual perception module first starts the visual perception, turns on the camera video stream and the corresponding recognition algorithm, and then perceives and recognizes the scene in real time; after receiving the command to start the automated intelligent training, the serving robot first starts the serving machine and enables the motors of each joint;

[0099] Then, after serving the first ball (simulating the serve of an actual opponent) according to the serving parameters selected by the probability distribution function, wait for the result of receiving the camera visual perception module;

[0100] (4) After the trainee receives the ball, the visual perception module captures a portion of the trainee's hitting trajectory (in this embodiment, the portion of the trajectory is defined as the trajectory segment from hitting the ball to the ball's first flight over the net), inputs the identified segment of the trajectory into the trajectory prediction algorithm module, and outputs the predicted complete trainee's hitting trajectory (the reason for the need for prediction is that the serving machine needs to obtain the hitting trajectory information in advance so that there is enough time to select the serving parameters and drive the motor to rotate and serve the ball, so as to simulate the ball path and rhythm of the real player's return after the player serves);

[0101] (5) At this time, the serving robot receives the complete trajectory of the trainer's hitting ball sent by visual perception, and determines whether the trajectory is a valid return ball (the ball's flight direction is from the trainer to the serving machine, the ball does not touch the court before flying over the net, and the first contact point with the court after flying over the net is within the serving area). If it is not a valid return ball, it continues to wait for receiving the visual perception result;

[0102] If it is a valid return, the serve parameters are selected through the probability distribution function according to the received complete trajectory and the remaining time T (simulating the return of the opponent in actual combat), and the motor is controlled to send the ball out when the remaining time ends;

[0103] (6) If the trainee fails to receive the ball successfully, the serving machine will wait for the maximum allowed time T_max (T_max = 10 seconds in this embodiment) while waiting for the result of the visual perception module. Once this condition is met, it is determined that the trainee has failed to return the ball successfully, and the ball is judged to be over. The next serve and simulated combat training will be continued from the serving parameter selection.

[0104] In this mode, the method of selecting the serving parameters through the probability distribution function (refer to Figure 4 )as follows:

[0105] First, determine whether the parameters to be selected are the parameters of the first serve (corresponding to the simulated opponent's serve in actual combat). If so, randomly generate (without input, directly randomly generate) a set of serve parameters of the serve machine through the serve probability distribution model;

[0106] If not (corresponding to the simulated opponent's return in actual combat), the complete trajectory and the remaining time T received from visual perception are input into the return probability distribution model to obtain a set of serving parameters of the serving machine.

[0107] Among them, the "serve parameters" can be described by ball speed, landing point, rotation direction, rotation speed, net height (arc); the ball speed can also be replaced by the average ball speed, the net height can be replaced by the arc or the serve angle, etc.

[0108] Among them, a method for obtaining the serve probability distribution model and the return probability distribution model is as follows.

[0109] 1) Collect several professional tennis match videos as a dataset;

[0110] 2) All rounds from a certain serve to the nearest next serve are called a "score", and the game video is divided into several video clips according to the "score", and each video clip is a complete score;

[0111] 3) For each video clip, the AI ​​tennis detection model is used to automatically detect tennis balls in the game video, and the flying tennis balls are tracked. Each shot in the score is divided according to the flight direction of the tennis balls, forming a record similar to "athlete A serves -> athlete B receives the serve -> athlete A returns the ball -> athlete B returns the ball ->... until the last shot of the score". At the same time, the start and end time of each shot, the complete trajectory information of this shot and its corresponding ball speed, landing point, rotation direction, rotation speed, past height (arc), and the remaining trajectory flight time information after the tennis ball flies to the net. Among them, the start and end time of each shot is obtained by tracking the flying tennis ball and judging it based on the condition of the reverse flight direction; the complete trajectory information is obtained by dividing the tennis ball detection and tracking results according to the start and end time of each shot; the ball speed, landing point, past height (arc), and the remaining trajectory flight time after the tennis ball flies to the net are obtained by fitting the complete trajectory information; the rotation direction and rotation speed are marked by experienced experts;

[0112] 4) The complete trajectory parameters (ball speed, landing point, rotation direction, rotation speed, net height (arc)) corresponding to the first ball bounce of each video clip are used as the serving data set, and the serving data set is clustered to obtain several groups of "serving parameters" and their corresponding weights (frequency of occurrence in the serving data set). These clustered "serving parameters" and their weights can be regarded as the serving probability distribution model;

[0113] 5) All the adjacent balls in each video clip are taken as input, including the complete trajectory of the previous ball and the remaining trajectory flight time T after the tennis ball flies to the net, and the "serve parameters" of the next ball of the adjacent balls are taken as output to form a return ball data set. A neural network model is established to train the return ball data set, and the trained neural network model is obtained as the return ball probability distribution model.

[0114] In this embodiment, the camera visual perception is used in conjunction with the tennis serving robot to realize the automated intelligent training of tennis training scenes, replacing the manual ball feeding by the coach or sparring partner, thus saving manpower;

[0115] Computer vision and artificial intelligence technologies are used to achieve visual perception of the trainer's hitting trajectory, and the visual perception results are used to automatically trigger or adjust the serving combination of the tennis serving robot to achieve intelligent training.

[0116] The above technical scheme is only an exemplary embodiment of the present invention. For those skilled in the art, it is easy to make various types of improvements or modifications based on the application methods and principles disclosed in the present invention, and it is not limited to the method described in the above specific embodiment of the present invention. Therefore, the method described above is only preferred and does not have a restrictive meaning.

Claims

1. An automated intelligent training method based on a serving robot, comprising the following steps: S1, set the serve training mode; S2, the serving robot serves the first ball and waits for the trainer to return the ball; or the serving robot waits for the trainer to serve the first ball; S3, obtaining a partial trajectory of the trainer's hitting the ball through a visual perception device, and performing subsequent trajectory prediction calculation based on the segment of the trajectory to obtain a complete trajectory of the trainer's hitting the ball and the remaining time; S4, the serving robot determines whether the trajectory of the ball hit by the trainee given by the visual perception device is a valid ball; S5, if it is a valid ball, serve the ball to the training personnel according to the set training mode, wherein: In the first training mode, the serving robot serves the first ball in the set serving combination when the remaining time of the trainee's hitting trajectory is over, and then serves the remaining balls in the set serving combination in sequence according to the set time intervals; In the second training mode, after the serving robot serves the ball, it continuously repeats steps S3-S5, that is, each time it judges that the ball is a valid ball based on the complete hitting trajectory of the trainee output by the visual perception device, it serves the next ball to the trainee according to the preset training mode.

2. The method according to claim 1, characterized in that The training mode includes: The first type of training mode includes: a serving connection mode, in which after the serving robot serves the first ball of the set built-in combination, no matter how the trainer returns the ball, it will serve the balls of the set built-in combination in sequence according to the built-in fixed serving frequency; The second type of training mode includes: rhythm coordination mode and simulated combat mode, among which: In the rhythm coordination mode, after serving each ball, the serving robot will use the visual perception device to obtain the trainer's effective return before serving the next ball, so that the serving frequency matches the trainer's return frequency, and the set built-in combination of balls is served in sequence; In the simulated combat mode, the serving robot automatically selects the possible return paths of the opponents in actual combat according to the trainees' return balls, and matches the trainees' return frequency, so that the serving frequency and ball path of the serving robot simulate the real return balls of the opponents in actual combat.

3. The method according to claim 1, characterized in that: The step S3 specifically includes: S31, using binocular cameras to respectively take pictures of the ball practice scene in real time, and respectively detect the ball targets in the images; respectively track the ball target detection results in real time; and obtain the trajectory of the ball by matching and binocular 3D reconstruction of the binocular tracking results; S32, performing subsequent trajectory prediction calculation based on the trajectory obtained in step S31 to obtain a complete training personnel's hitting trajectory and remaining time T.

4. The method according to claim 1 or 3, characterized in that: The subsequent trajectory prediction and calculation steps specifically include: The x, y, and z coordinates of the input ball trajectory are fitted with the time t respectively, and the complete ball trajectory and the remaining flight time T are directly obtained through the fitting coefficient and the prior landing point z value; Alternatively, a neural network model is built to predict the subsequent trajectory and remaining flight time of the ball.

5. The method according to claim 2, characterized in that: The determination conditions for the valid ball in step S4 include: In the serve connection mode and the simulated actual combat mode, the following conditions must be met at the same time: the ball's flight direction is from the trainer to the serve machine, the ball has no contact with the court before flying over the net, and the first contact point with the court after flying over the net is within the serving area. In the rhythm coordination mode, the effective ball means: the flight direction of the ball is from the trainer to the ball machine.

6. The method according to any one of claims 1 to 5, characterized in that: When the simulated combat training mode is adopted, the serving robot serves the ball each time according to the serving parameters selected by the probability distribution function; the serving parameters include: one or more of the ball speed, landing point, rotation direction, rotation speed, and net height; Methods for selecting serve parameters through probability distribution functions include: First, determine whether the parameters to be selected are the parameters of the first serve. If so, directly and randomly generate a set of serve parameters of the serve machine without input through the serve probability distribution model; If not, the complete trajectory and the remaining time T received from the visual perception device are input into the ball return probability distribution model to obtain a set of serving parameters of the serving machine.

7. The method according to claim 6, characterized in that The method for constructing the serve probability distribution model and the return probability distribution model comprises: 1) Collect several professional ball game videos as a dataset; 2) All rounds from a certain serve to the nearest next serve are called a "score", and the game video is divided into several video clips according to the "score", and each video clip is a complete score; 3) For each video clip, the AI ​​ball detection model is used to automatically detect the ball in the game video and track the flying ball. Each shot in the score is divided according to the flight direction of the ball, forming a record similar to "athlete A serves -> athlete B receives the serve -> athlete A returns the ball -> athlete B returns the ball -> ... until the last shot of the score". At the same time, the start and end time of each shot, the complete trajectory information of this shot and its corresponding ball parameters, and the remaining trajectory flight time information after the ball flies to the net. The ball parameters include: one or more of the ball speed, landing point, rotation direction, rotation speed, and net height; The start and end time of each ball shot are obtained by tracking the flying ball and judging it based on the condition of the reverse direction of flight. The complete trajectory information is obtained by dividing the ball detection and tracking results according to the start and end time of each ball shot. The ball speed, landing point, net height, and the remaining trajectory flight time after the ball flies to the net are calculated by fitting the complete trajectory information. The rotation direction and rotation speed are marked by experienced experts. 4) The complete ball-out parameters corresponding to the first ball-out of each video clip are used as a serving data set, and the serving data set is clustered to obtain several groups of "serving parameters" and their corresponding weights, i.e., the frequencies of occurrence in the serving data set; these clustered "serving parameters" and their weights can be regarded as a serving probability distribution model; 5) Taking the complete trajectory of the previous ball in all adjacent bouncing balls in each video clip and the remaining trajectory flight time T after the ball flies to the net as input and the ball parameters of the next ball in the adjacent bouncing balls as output, a return ball data set is formed, and a neural network model is established to train the return ball data set, and the trained neural network model is obtained as the return ball probability distribution model.

8. The method according to claim 1, characterized in that The following steps are also included: Before starting the automated intelligent training for the first time, or after the visual perception device is moved, the visual perception device is visually calibrated to establish a conversion relationship between the camera coordinate system and the practice field coordinate system.

9. An automated intelligent training system based on a serving robot using the method described in any one of claims 1 to 8, characterized in that: include: A serving robot, a camera-based visual perception module, and an application interaction module, including: An application interaction module is used to send an automated intelligent training start instruction to the serving robot and the visual perception module; and to set a training mode to control the serving robot to serve the trainer; A visual perception module is used to obtain a portion of the trainer's hitting trajectory, and perform subsequent trajectory prediction calculation based on the segment of the trajectory to obtain the trainer's hitting trajectory and remaining time; The serving robot is used to determine whether the trajectory of the trainee's hitting ball given by the visual perception device is a valid ball; and is used to serve the ball to the trainee according to the set training mode, wherein: In the first training mode, the serving robot serves the first ball in the set serving combination when the remaining time of the trainee's hitting trajectory ends, and then serves the remaining balls in the set serving combination in sequence according to the set time intervals; In the second training mode, after each ball is served, the serving robot will judge that the ball is a valid ball based on the complete hitting trajectory of the trainee output by the visual perception device, and then serve the next ball to the trainee according to the preset training mode.

10. The system according to claim 9, characterized in that The visual perception module includes the following submodules: Visual calibration submodule: used to calibrate the visual perception of the camera before starting the automated intelligent training for the first time or after the camera is moved, and to establish a conversion relationship between the camera coordinate system and the practice court coordinate system. This coordinate conversion relationship is used to convert the image coordinates of the ball recognized by the camera into the practice court coordinate system; Real-time recognition submodule: using binocular cameras to simulate binocular vision, binocular cameras respectively take pictures of the ball practice scene in real time, and detect the ball targets in the image respectively; then the ball target detection results are respectively tracked in real time; the tracking results of the binocular cameras are matched and binocular 3D reconstructed to obtain the trajectory of the ball; Trajectory prediction submodule: It is used to perform subsequent trajectory prediction calculations based on the trajectory obtained by the real-time recognition submodule, and output the complete ball trajectory and the remaining time T.

Images