Vision-based Table Tennis Service Occlusion Judgment Method, System and Storage Medium

Abstract

The present invention discloses a method for judging table tennis service occlusion based on vision. The method includes: acquiring video data collected by two cameras; extracting video images from the video data according to a preset frame rate; performing information extraction on each video image to identify the table tennis ball and the upper body of the athlete, obtaining table tennis ball data and upper body data; processing the table tennis ball data and the upper body data to obtain three-dimensional table tennis ball data and three-dimensional upper limb data; judging whether the athlete occludes the service during a set period according to the three-dimensional table tennis ball data and the three-dimensional upper limb data, obtaining a judgment conclusion; the set period is the period between the athlete's ball toss and the racket hitting the ball. The present invention extracts and tracks the table tennis ball data and the upper body data of the video collected by two cameras, and judges whether there is service occlusion accordingly. The real-time performance of obtaining the judgment conclusion is high, the implementation cost is low, and the judgment is accurate, which is suitable for assisting referees in table tennis competitions.

Description

Technical Field

[0001] The present invention relates to the technical field of electronic refereeing in sports, and in particular to a table tennis serve occlusion judgment method, system and storage medium based on vision. Background Art

[0002] According to the Rules of Table Tennis, "a player has the responsibility to ensure that the referee or assistant referee clearly sees the legality of their serve." Existing technology used to determine whether a player's serve is obstructed in table tennis matches primarily relies on Hawkeye technology, known as the Hawkeye Replay Challenge (TTR). This system uses a camera to capture the trajectory of the table tennis ball, collects data, and then uses a computer system to create a 3D animation. The image is manually controlled to calculate various required parameters or playback the data in slow motion.

[0003] The TTR system has the following disadvantages:

[0004] (1) The TTR system determines whether there is an obstruction by repeatedly watching the video from various angles through playback monitoring. However, the camera position on the court does not completely overlap with the receiving side's perspective. There is a possibility of penalty errors in that the receiving side's perspective is not obstructed, but the camera's perspective is obstructed.

[0005] (2) The TTR system has the problem of long video playback and penalty time. The referee needs to repeatedly watch videos from various angles through the playback monitoring to finally determine the penalty result and present it on the large screen on site. This takes a long time and seriously affects the smoothness of the game.

[0006] (3) The TTR system currently only has data on the speed and rotation speed of the winning ball and the number of rounds of the exciting ball, but it cannot display the kinematic data that are more characteristic of table tennis competition, such as the spatial position of the table tennis ball.

[0007] (4) The introduction of the TTR system has significantly increased the costs of referees and funds. Summary of the Invention

[0008] Purpose of the invention: In order to overcome the deficiencies in the prior art, the present invention provides a table tennis serve occlusion judgment method, system and storage medium based on vision, which are timely, low-cost and have detailed data.

[0009] Technical Solution: To achieve the above-mentioned purpose, the present invention provides a table tennis serve occlusion judgment method based on vision, which comprises:

[0010] Get the video data collected by two cameras;

[0011] extracting a video image from the video data according to a preset frame rate;

[0012] Extract information from each of the said video images to identify the table tennis ball therein, and obtain table tennis ball data;

[0013] Extract information from each of the said video images to identify the upper body of the athlete therein, and obtain upper body data; wherein, the upper body data are data corresponding to each joint point of the athlete's upper body;

[0014] Process the table tennis ball data and the upper body data to obtain three-dimensional table tennis ball data and three-dimensional upper limb data; wherein, the three-dimensional upper limb data are three-dimensional data corresponding to each joint point of the athlete's arm;

[0015] Judge whether the athlete blocks the service during a set time period according to the three-dimensional table tennis ball data and the three-dimensional upper limb data, and obtain a judgment conclusion; the set time period is the time period from when the athlete throws the ball to when the racket hits the ball.

[0016] Further, before acquiring the video data collected by the camera, it further includes:

[0017] Perform data initialization;

[0018] Judge whether there is video data imported. If yes, execute the next step; otherwise, re-execute the previous step.

[0019] Further, the method further includes:

[0020] Output the video data and the judgment conclusion to the screen.

[0021] Further, the extracting information from each of the said video images to identify the table tennis ball therein and obtaining table tennis ball data includes:

[0022] Preprocess the video image to obtain a processed image;

[0023] Use an object detection algorithm to extract the table tennis ball area in the processed image, and extract a foreground detection frame;

[0024] Calculate the pixel coordinates of the center of the foreground detection frame, and obtain the pixel coordinates of the table tennis ball in the video image as the table tennis ball data.

[0025] Further, the extracting information from each of the said video images to identify the athlete therein and obtaining upper body data includes:

[0026] Use a human detection network to extract a candidate box for the upper body of the human in the video image;

[0027] Perform key point and pose prediction on the human image within the candidate box for the upper body of the human, and obtain a prediction result;

[0028] Transform the prediction result into the video image to obtain the key point positions corresponding to the video image;

[0029] Exclude redundant poses through NMS to obtain the final key point data as the upper body data.

[0030] Further, the processing of the table tennis data and the upper body data to obtain the table tennis three-dimensional data and the upper limb three-dimensional data includes:

[0031] Obtain the depth difference mapping matrix according to the internal and external parameter matrices of the camera;

[0032] Project the table tennis data and the upper body data into the coordinate system of the camera according to the depth difference mapping matrix;

[0033] According to the pose relationship of the camera relative to the world coordinate system, convert the table tennis data in the coordinate system of the camera and the data corresponding to each joint point of the upper limb in the upper body data into the world coordinate system to obtain the table tennis three-dimensional data and the upper limb three-dimensional data.

[0034] Further, the judgment of whether the athlete blocks the serve during a set period according to the table tennis three-dimensional data and the upper limb three-dimensional data, and the obtained judgment conclusion includes:

[0035] Compare the table tennis three-dimensional data and the upper limb three-dimensional data, and judge whether the athlete blocks the serve during a set period according to the coordinate relationship, or judge whether the athlete blocks the serve during a set period according to the tracking trajectory of the table tennis three-dimensional data.

[0036] Further, the judgment of whether the athlete blocks the serve during a set period according to the coordinate relationship includes:

[0037] Judge whether there is a first set of circumstances at a certain moment to obtain a first judgment result; the first set of circumstances is: there is a point in the upper limb three-dimensional data that coincides with the table tennis three-dimensional data in the X-axis direction, and the coordinate of this point is closer to the center of the table tennis table than the table tennis three-dimensional data;

[0038] If the first judgment conclusion is yes, the obtained judgment conclusion is that the athlete blocks the serve;

[0039] The judgment of whether the athlete blocks the serve during a set period according to the tracking trajectory of the table tennis three-dimensional data includes:

[0040] Track the movement trajectory of the table tennis according to the table tennis three-dimensional data at consecutive moments;

[0041] Based on the movement trajectory of the table tennis ball, determine whether there is a second set condition to obtain a second judgment result; the second set condition is: when the table tennis ball is moving downward, there are several video images without corresponding three-dimensional data of the table tennis ball.

[0042] If the second judgment conclusion is yes, the obtained judgment conclusion is that the athlete blocks the serve.

[0043] A vision-based table tennis serve occlusion judgment system includes:

[0044] Cameras, there are two cameras, which are respectively installed beside two referee positions;

[0045] A display screen, which is used to output information;

[0046] A controller, which is connected to the cameras and the display screen, and is used to implement the above-mentioned vision-based table tennis serve occlusion judgment method.

[0047] A storage medium, on which a computer program is stored, and when the program is executed by a processor, it implements the above-mentioned vision-based table tennis serve occlusion judgment method.

[0048] Beneficial effects: The vision-based table tennis serve occlusion judgment method, system and storage medium of the present invention extract and track the table tennis ball data and upper body data from the videos collected by two cameras, and judge whether there is a serve occlusion based on this. The real-time performance of obtaining the judgment conclusion is high, the implementation cost is low, and the judgment is accurate, which is suitable for assisting referees in table tennis matches. Description of the Drawings

[0049] Att Figure 1 It is a schematic diagram of the composition of a vision-based table tennis serve occlusion judgment system;

[0050] Att Figure 2 It is a schematic flowchart of a vision-based table tennis serve occlusion judgment method. Detailed Embodiments

[0051] The present invention will be further described below with reference to the drawings.

[0052] The vision-based table tennis serve occlusion judgment method of the present invention is based on Att Figure 1The visual-based table tennis serve occlusion judgment system shown in the figure. In the figure, the system includes a camera 1, a display screen 2, and a controller 3. Among them, there are two cameras 1, and the two cameras 1 form a binocular system. The two cameras 1 are respectively installed beside two referee positions so that the viewing angles of the cameras 1 are the same as those of the two referees in a table tennis match. The camera 1 is used to capture the athletes and the table tennis ball on the playing field in real time and obtain the real-time image data of the table tennis ball and the athletes. The display screen 2 is used to output information, and the output information can include video information, the coordinate data of the table tennis ball, the pose data of the athletes, the referee conclusion information, etc. The controller 3 is connected to the camera 1 and the display screen 2 and is used to implement the visual-based table tennis serve occlusion judgment method of the present invention. In the present invention, the controller 3 is a general term for all components involved in control in the system. Specifically in this embodiment, the controller 3 includes a host computer and an image acquisition system. The image acquisition system is used to connect the camera 1 and the host computer to receive the video data from the two cameras 1 in real time. In this embodiment, the image acquisition system is an AD4-CL image acquisition card.

[0053] Based on the above system, the visual-based table tennis serve occlusion judgment method of the present invention includes the following steps S401-S406 (all the following step numbers are not used to limit the execution order of the steps, and at most only represent the preferred execution order. The execution order of some steps can be adjusted as needed or implemented synchronously):

[0054] Step S401, obtain the video data collected by the two cameras.

[0055] In this step, the shooting mode adopted by each camera 1 is 250fps (resolution 2048*1088).

[0056] Step S402, extract the video images in the video data according to the preset frame rate.

[0057] In this step, each frame of video image in the video data can be obtained frame by frame, or a frame of video image can be obtained every set number of frames as needed.

[0058] Step S403, perform information extraction on each video image to identify the table tennis ball therein and obtain the table tennis ball data.

[0059] In this step, the table tennis ball data includes the coordinate data of the table tennis ball in the video image where it is located.

[0060] Step S404, perform information extraction on each video image to identify the upper body of the athlete therein and obtain the upper body data. Among them, the upper body data is the data corresponding to each joint point of the athlete's upper body.

[0061] Step S405: Process the table tennis data and the upper body data to obtain 3D table tennis data and 3D upper limb data; wherein, the 3D upper limb data are the 3D data corresponding to the joint points of the athlete's arm.

[0062] This step is mainly used to convert the upper body data and the table tennis data from the coordinate data in the video image to the coordinate data in the world coordinate system. In this step, the 3D table tennis data and the 3D upper limb data are obtained by integrating the upper body data and the table tennis data in the video images of two cameras at the same moment.

[0063] Step S406: Determine whether the athlete blocks the serve during a set period according to the 3D table tennis data and the 3D upper limb data, and obtain a judgment conclusion; the set period is the period from when the athlete tosses the ball to when the racket hits the ball.

[0064] Preferably, before the above step S401, before obtaining the video data collected by the camera, the following steps S501 - S502 are further included:

[0065] Step S501: Perform data initialization.

[0066] Step S502: Determine whether video data is imported. If yes, execute step S401; otherwise, re - execute step S501.

[0067] Preferably, after the above step S406, the method further includes the following step S601:

[0068] Step S601: Output the video data and the judgment conclusion to the screen.

[0069] Preferably, in the above step S403, the process of extracting information from each video image to identify the table tennis ball and obtaining the table tennis data includes the following steps S701 - S703:

[0070] Step S701: Pre - process the video image to obtain a processed image.

[0071] In this step, the pre - processing includes operations such as converting the image to a grayscale image, Gaussian noise filtering, contrast enhancement, eliminating image noise, and improving the display quality of the image, making the image more suitable for subsequent processing.

[0072] Step S702: Use an object detection algorithm to extract the table tennis ball area in the processed image and extract the foreground detection frame.

[0073] Step S703: Calculate the pixel coordinates of the center of the foreground detection frame, and obtain the pixel coordinates of the table tennis ball in the video image as the table tennis data.

[0074] Preferably, the information extraction of each video image to identify the athletes therein and obtain the upper body data includes the following steps S801 - S804:

[0075] Step S801, use a human detection network to extract the upper body candidate boxes of the human body in the video image;

[0076] In this step, LSTM is used as the human detection network. In other embodiments, other algorithms can also be used as the human detection network.

[0077] Step S802, perform key point and pose prediction on the human body image within the upper body candidate box of the human body to obtain a prediction result;

[0078] In this step, the method of data augmentation is used to train the SPPE stacked hourglass network for key point and pose prediction.

[0079] Step S803, transform the prediction result into the video image to obtain the key point positions corresponding to the video image;

[0080] Step S804, exclude redundant poses through NMS to obtain the final key point data as the upper body data.

[0081] In this step, NMS is used to calculate the similarity of poses, thereby excluding redundant poses, and further excluding the identified redundant key points to reduce the interference of the multi-identified points on the result.

[0082] Preferably, the processing of the table tennis data and the upper body data in the above step S405 to obtain the table tennis three-dimensional data and the upper limb three-dimensional data includes the following steps S901 - S903:

[0083] Step S901, obtain the depth difference mapping matrix according to the internal and external parameter matrices of the camera;

[0084] In this step, the stereoRectify() function is used to obtain the depth difference mapping matrix.

[0085] Step S902, project the table tennis data and the upper body data into the coordinate system of the camera according to the depth difference mapping matrix;

[0086] Step S903, according to the pose relationship of the camera relative to the world coordinate system, convert the table tennis data and the upper body data in the coordinate system of the camera to the world coordinate system to obtain the table tennis three-dimensional data and the upper limb three-dimensional data.

[0087] In this step, the world coordinate system sets the center position of the table tennis table. Before the system starts processing data, it is necessary to fine-tune the coordinate system according to the measured data and manually calculate the rotation matrix to perform rotation correction on the world coordinate system.

[0088] Preferably, in the above step S406, the determination of whether the athlete blocks the serve during the set period according to the three-dimensional data of the table tennis ball and the three-dimensional data of the upper limb, and the obtained judgment conclusion includes the following steps A1:

[0089] Step A1: Compare the three-dimensional data of the table tennis ball with the three-dimensional data of the upper limb, and judge whether the athlete blocks the serve during the set period according to the coordinate relationship, or judge whether the athlete blocks the serve during the set period according to the tracking trajectory of the three-dimensional data of the table tennis ball.

[0090] In this step, when there is a table tennis ball in the video image, it is possible to judge whether the athlete's body blocks the table tennis ball according to the coordinate relationship. When there is no table tennis ball in the video image, it is possible to judge whether the athlete's body blocks the table tennis ball according to the tracking trajectory of the three-dimensional data of the table tennis ball.

[0091] Specifically, the determination of whether the athlete blocks the serve during the set period according to the coordinate relationship in the above step A1 includes the following steps B1 - B2:

[0092] Step B1: Judge whether there is a first set of circumstances at a certain moment to obtain a first judgment result; the first set of circumstances is that there is a point in the three-dimensional data of the upper limb that coincides with the three-dimensional data of the table tennis ball in the X-axis direction, and the coordinates of this point are closer to the center of the table tennis table than the three-dimensional data of the table tennis ball;

[0093] In this step, the X-axis is parallel to the length direction of the table tennis table, that is, the X-axis is perpendicular to the plane where the net is located.

[0094] Step B2: If the first judgment conclusion is yes, the obtained judgment conclusion is that the athlete blocks the serve;

[0095] Through the above steps B1 - B2, the coordinates of the athlete and the table tennis ball can be compared to judge whether there is occlusion.

[0096] Specifically, the determination of whether the athlete blocks the serve during the set period according to the tracking trajectory of the three-dimensional data of the table tennis ball in the above step A1 includes the following steps C1 - C3:

[0097] Step C1: Track the movement trajectory of the table tennis ball according to the three-dimensional data of the table tennis ball at consecutive moments;

[0098] Step C2: Determine whether there is a second preset situation according to the movement trajectory of the table tennis ball to obtain a second judgment result. The second preset situation is that when the table tennis ball is moving downward, there are several video images without corresponding three-dimensional data of the table tennis ball.

[0099] Step C3: If the second judgment conclusion is yes, the obtained judgment conclusion is that the athlete blocks the serve.

[0100] Through the above steps C1 - C3, by combining the movement trajectory of the table tennis ball and whether there is a table tennis ball in the video image, it is determined whether the serve is blocked, and there will be no misjudgment. Because if the table tennis ball is in the ascending stage and there is no corresponding three-dimensional data of the table tennis ball in the video image, it may be that the table tennis ball is out of the camera's field of view. When the table tennis ball is in the descending stage, if there is no corresponding three-dimensional data of the table tennis ball in the video image, it must be blocked.

[0101] The present invention also provides a computer-readable storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, a server, an App application store, etc. A computer program is stored thereon, and when the program is executed by a processor, it implements the method for judging table tennis serve occlusion based on vision of the present invention.

[0102] The method, system and storage medium for judging table tennis serve occlusion based on vision of the present invention extract and track the table tennis data and upper body data from the videos collected by two cameras, and accordingly judge whether the serve is blocked. The real-time performance of obtaining the judgment conclusion is high, the implementation cost is low, and the judgment is accurate, which is suitable for assisting referees in table tennis matches.

[0103] The above are only the preferred embodiments of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present invention.

Claims

1. A vision-based method for judging the occlusion of table tennis serves, characterized in that, The method includes: Obtaining video data collected by two cameras; Extracting video images from the video data according to a preset frame rate; Performing information extraction on each of the video images to identify the table tennis ball therein, and obtaining table tennis ball data; Performing information extraction on each of the video images to identify the upper body of the athlete therein, and obtaining upper body data; wherein, the upper body data is data corresponding to each joint point of the athlete's upper body; Processing the table tennis ball data and the upper body data to obtain three-dimensional table tennis ball data and three-dimensional upper limb data; wherein, the three-dimensional upper limb data is three-dimensional data corresponding to each joint point of the athlete's arm; Judging whether the athlete obstructs the service during a set period according to the three-dimensional table tennis ball data and the three-dimensional upper limb data, and obtaining a judgment conclusion; the set period is the period from when the athlete tosses the ball to when the racket hits the ball; The judging whether the athlete obstructs the service during a set period according to the three-dimensional table tennis ball data and the three-dimensional upper limb data, and obtaining a judgment conclusion includes: Comparing the three-dimensional table tennis ball data and the three-dimensional upper limb data, and judging whether the athlete obstructs the service during the set period according to the coordinate relationship, or judging whether the athlete obstructs the service during the set period according to the tracking trajectory of the three-dimensional table tennis ball data; The judging whether the athlete obstructs the service during the set period according to the coordinate relationship includes: Judging whether there is a first set situation at a certain moment, and obtaining a first judgment result; the first set situation is: there is a point in the three-dimensional upper limb data that coincides with the three-dimensional table tennis ball data in the X-axis direction, and the coordinate of this point is closer to the center of the table tennis table than the three-dimensional table tennis ball data; the X-axis is parallel to the length direction of the table tennis table; If the first judgment conclusion is yes, the obtained judgment conclusion is that the athlete obstructs the service; The judging whether the athlete obstructs the service during the set period according to the tracking trajectory of the three-dimensional table tennis ball data includes: Tracking the movement trajectory of the table tennis ball according to the three-dimensional table tennis ball data at consecutive moments; Judging whether there is a second set situation according to the movement trajectory of the table tennis ball, and obtaining a second judgment result; the second set situation is: when the table tennis ball is moving downward, there are several video images without corresponding three-dimensional table tennis ball data; If the second judgment conclusion is yes, the obtained judgment conclusion is that the athlete obstructs the service.

2. The method for judging table tennis serve occlusion based on vision according to claim 1, characterized in that, Before obtaining the video data collected by the camera, it further includes: Performing data initialization; Judging whether there is video data imported. If yes, execute the next step; otherwise, execute the previous step again.

3. The method for judging table tennis serve occlusion based on vision according to claim 1, wherein The method further includes: Outputting the video data and the judgment conclusion to the screen.

4. The vision-based table tennis serve occlusion determination method according to claim 1, wherein The performing information extraction on each of the video images to identify the table tennis ball therein, and obtaining table tennis ball data includes: Performing preprocessing on the video image to obtain a processed image; Using an object detection algorithm to extract the table tennis ball area in the processed image, and extracting a foreground detection frame; Calculating the pixel coordinates of the center of the foreground detection frame, and obtaining the pixel coordinates of the table tennis ball in the video image as the table tennis ball data.

5. The method for judging table tennis service occlusion based on vision according to claim 1, wherein Performing information extraction on each of the video images to identify the athletes therein, and obtaining upper body data, including: Extracting candidate bounding boxes of the upper body of a human in the video image by using a human detection network; Performing key point and pose prediction on the human image within the candidate bounding box of the upper body of the human to obtain a prediction result; Transforming the prediction result into the video image to obtain the key point positions corresponding to the video image; Excluding redundant poses through NMS to obtain the final key point data as the upper body data.

6. The method for judging table tennis serve occlusion based on vision according to claim 1, characterized in that, Processing the table tennis data and the upper body data to obtain 3D table tennis data and 3D upper limb data, including: Obtaining a depth difference mapping matrix according to the internal and external parameter matrices of the camera; Projecting the table tennis data and the upper body data into the coordinate system of the camera according to the depth difference mapping matrix; According to the pose relationship of the camera relative to the world coordinate system, converting the data corresponding to each joint point of the arm in the table tennis data and the upper body data in the coordinate system of the camera into the world coordinate system to obtain 3D table tennis data and 3D upper limb data.

7. The method for judging table tennis service occlusion based on vision according to claim 1, characterized in that Judging whether an athlete obstructs a serve during a set period according to the coordinate relationship, including: Judging whether there is a first set of circumstances at a certain moment to obtain a first judgment result; the first set of circumstances is that there is a point in the 3D upper limb data that coincides with the 3D table tennis data in the X-axis direction, and the coordinate of this point is closer to the center of the table tennis table than the 3D table tennis data; If the first judgment conclusion is yes, the obtained judgment conclusion is that the athlete obstructs the serve; Judging whether an athlete obstructs a serve during a set period according to the tracking trajectory of the 3D table tennis data, including: Tracking the movement trajectory of the table tennis according to the 3D table tennis data at consecutive moments; Judging whether there is a second set of circumstances according to the movement trajectory of the table tennis to obtain a second judgment result; the second set of circumstances is that when the table tennis is moving downward, there are several video images without corresponding 3D table tennis data; If the second judgment conclusion is yes, the obtained judgment conclusion is that the athlete obstructs the serve.

8. A vision-based ping-pong ball serving occlusion judgment system, characterized in that, It includes: Cameras, there are two cameras, which are respectively installed beside two referee positions; A display screen for outputting information; A controller, which is connected to the cameras and the display screen and is used to implement the vision-based table tennis serve obstruction judgment method according to any one of claims 1-7.

9. A storage medium, characterized in that, A computer program is stored thereon, and when the program is executed by a processor, it implements the vision-based table tennis serve obstruction judgment method according to any one of claims 1-7.

Images