High-speed vision-based three-dimensional trajectory reconstruction of table tennis and hawk-eye referee system

Abstract

This invention discloses a high-speed vision-based 3D trajectory reconstruction and Hawk-Eye referee system for table tennis. The system utilizes an image acquisition module consisting of at least four high-speed cameras distributed around the table, reconstructing the 3D flight trajectory of the table tennis ball through multi-view geometric triangulation. An event detection and penalty module automatically detects key events such as net contact, edge contact, insufficient serve height, serve obstruction, and ball going out of bounds based on the 3D trajectory and the table space model. The Hawk-Eye decision module, combined with a competition rule database, makes automated penalty decisions. The system also includes an adaptive exposure control and multi-feature fusion target detection mechanism to improve detection robustness under complex lighting conditions; and a penalty confidence assessment and dispute event marking mechanism, achieving an intelligent closed loop from trajectory measurement to rule-based penalty in table tennis matches, effectively assisting referees in making rapid and accurate penalty decisions.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a ping-pong ball 3D trajectory reconstruction and eagle-eye referee system based on high-speed vision. Background Technology

[0002] The Hawk-Eye system, a real-time replay system, has been used for refereeing and simulation demonstrations in tennis matches for many years and has been widely adopted in tennis broadcasting, playing a positive role in tennis research and the fairness of matches. This system typically consists of 8 to 10 high-speed cameras, four computers, and a large screen. Multiple high-speed cameras capture the ball's trajectory from different angles, and the computer performs 3D image reconstruction and trajectory prediction. However, the traditional Hawk-Eye system takes about 10 seconds from data acquisition to result demonstration, lacking strong real-time capability. In official matches, it can only serve as an auxiliary referee tool and cannot achieve online real-time rulings. Existing systems mainly focus on trajectory measurement, prediction, and 3D reconstruction, without building dedicated detection and judgment modules for specific rulings in table tennis matches, lacking complete automated ruling capabilities. The smooth, reflective surface of a table tennis ball makes traditional detection methods based on single-color features prone to failure under strong light. Existing technologies lack ruling confidence assessment and automatic dispute marking mechanisms, failing to achieve organic synergy between automated machine rulings and manual review. Summary of the Invention

[0003] The purpose of this invention is to solve the above problems by designing a 3D trajectory reconstruction and Hawk-Eye referee system for table tennis based on high-speed vision.

[0004] To achieve the above objectives, the technical solution of the present invention further includes the following modules in the above-mentioned high-speed vision-based three-dimensional trajectory reconstruction and Hawk-Eye referee system for table tennis: The image acquisition module is used to simultaneously acquire high-speed image sequences from multiple perspectives during the movement of table tennis; The camera synchronization calibration module is used to calibrate the internal and external parameters of each high-speed camera, establish the mapping relationship between the image coordinate system of each camera and the three-dimensional world coordinate system, and obtain the camera calibration parameters. The target detection and tracking module is used to detect and identify the high-speed image sequence in real time, and to use a multi-target tracking algorithm to track the motion of the ping-pong ball between consecutive frames, generating two-dimensional trajectory data from the perspective of each camera. The 3D trajectory reconstruction module is used to reconstruct 3D trajectory data based on 2D trajectory data and camera calibration parameters using a multi-view geometric triangulation method, and to perform spatiotemporal filtering and smoothing on the 3D trajectory data to generate the 3D flight trajectory of the ping-pong ball. The event detection and judgment module is used to detect key events of the ping-pong ball during its flight based on the three-dimensional flight trajectory and the three-dimensional spatial model of the ping-pong table, and obtain the event detection results. The Eagle Eye referee decision module is used to make automated judgment decisions based on event detection results and the table tennis competition rule base, and generate judgment result signals.

[0005] Furthermore, in the aforementioned high-speed vision-based 3D trajectory reconstruction and Hawk-Eye referee system for table tennis, the target detection and tracking module includes the following units: The adaptive exposure control unit is used to dynamically adjust the exposure parameters of each high-speed camera according to the current ambient lighting conditions and the reflective properties of the ping-pong ball surface. The multi-feature fusion detection unit is used to fuse the color features, shape features, and motion features of a ping-pong ball for target detection. The color features are established based on the hue distribution of the ping-pong ball in the HSV color space, the shape features are established based on the roundness and size constraints of the ping-pong ball, and the motion features are established based on the inter-frame difference motion saliency. The detection confidence evaluation unit is used to generate a confidence score for the target detection results in each frame of the image, and to trigger cross-validation of the detection results of adjacent cameras when the confidence score is lower than a preset threshold.

[0006] Furthermore, in the aforementioned table tennis 3D trajectory reconstruction and Hawk-Eye referee system based on high-speed vision, the 3D trajectory reconstruction module includes the following units: The subpixel precision positioning unit is used to perform subpixel-level precision center positioning of the detected ping-pong ball target position; The multi-view feature matching unit is used to establish a corresponding matching relationship between ping-pong ball targets in different camera images at the same time. Triangulation reconstruction unit, used to reconstruct multiple views that have been matched Figure 2 The three-dimensional coordinates of the ping-pong ball are calculated using linear triangulation, based on the 3D coordinates and the corresponding camera projection matrix. The trajectory filtering unit is used to perform Kalman filtering on the reconstructed 3D trajectory sequence to generate the 3D flight trajectory of the ping-pong ball.

[0007] Furthermore, in the aforementioned high-speed vision-based 3D trajectory reconstruction and Hawk-Eye referee system for table tennis, the event detection and penalty module includes the following sub-modules: The net area modeling submodule is used to establish a spatial area model of the net in a three-dimensional world coordinate system based on the camera calibration parameters and the size parameters of the ping-pong table. The spatial area model includes the position of the upper edge line of the net and the height of the net. The trajectory network surface intersection detection submodule is used to detect whether the three-dimensional flight trajectory intersects with the ball net spatial region model, and to record the intersection time, intersection position and intersection velocity vector when an intersection is detected. The net-wipe determination submodule is used to determine whether the cross event is a net-wipe event based on the distance between the cross position and the top edge of the net, the direction of the cross velocity vector, and the continuity of the trajectory before and after the cross moment, and to distinguish between net-wipe and net-wipe loss of points.

[0008] Furthermore, in the aforementioned table tennis ball 3D trajectory reconstruction and Hawk-Eye referee system based on high-speed vision, the event detection and judgment module also includes the following sub-modules: The table surface boundary modeling submodule is used to establish an accurate spatial model of the table tennis table surface and its boundary lines in the three-dimensional world coordinate system based on the camera calibration parameters and the table tennis table size parameters. The spatial model includes at least the positions of the four corner points, the positions of the four side lines, and the table height. The landing point detection submodule is used to detect the intersection of the 3D flight trajectory and the table space model, which serves as the landing point of the ping-pong ball. The edge-grabbing determination submodule is used to determine whether a ball is edge-grabbing based on the spatial distance between the landing point and the table boundary line. When the landing point is located in the table boundary line area and the trajectory has a significant change in direction before and after the landing point, it is determined to be an edge-grabbing event. The edge-grip type module is used to distinguish between upper edge grazing and side edge grazing based on the specific position of the landing point on the edge line and the trajectory incident angle.

[0009] Furthermore, in the aforementioned table tennis ball 3D trajectory reconstruction and Hawk-Eye referee system based on high-speed vision, the Hawk-Eye referee decision module includes the following units: The rule knowledge base unit is used to store the structured representation of the rules of table tennis. The rule knowledge base includes at least the logical expressions of the serving rules, hitting rules, scoring rules, and losing rules. The event rule matching engine is used to match key events detected by the event detection and judgment module with the rules in the rule knowledge base in real time, and trigger the corresponding judgment logic. The penalty confidence calculation unit is used to generate a confidence score for each penalty decision. The confidence score is calculated based on the number of detected event evidences, the consistency of the evidence, and the confidence of each detection step. The dispute event marking unit is used to mark the penalty decision with a confidence score lower than a preset threshold as a dispute event and prompt the referee to conduct a manual review.

[0010] Furthermore, in the aforementioned 3D trajectory reconstruction and eagle-eye referee system for table tennis based on high-speed vision, the frame rate of the high-speed camera in the image acquisition module is no less than 200 frames per second, and each high-speed camera is equipped with a global shutter sensor to eliminate motion blur generated by the high-speed moving table tennis ball during image acquisition.

[0011] Furthermore, in the method for realizing the 3D trajectory reconstruction of a table tennis ball and the Hawk-Eye referee system based on high-speed vision, the method includes the following steps: High-speed image sequences were simultaneously acquired from multiple perspectives during the table tennis game. The intrinsic and extrinsic parameters of each high-speed camera are calibrated, and the mapping relationship between the image coordinate system of each camera and the three-dimensional world coordinate system is established to obtain the camera calibration parameters; The high-speed image sequence is detected and identified in real time, and a multi-target tracking algorithm is used to track the motion of the ping-pong ball between consecutive frames to generate two-dimensional trajectory data from the perspective of each camera. Based on the two-dimensional trajectory data and camera calibration parameters, the three-dimensional trajectory data is reconstructed using the multi-view geometric triangulation method, and the three-dimensional trajectory data is then subjected to spatiotemporal filtering and smoothing to generate the three-dimensional flight trajectory of the ping-pong ball. Based on the three-dimensional flight trajectory and the three-dimensional spatial model of the ping-pong table, key events of the ping-pong ball during flight are detected, and event detection results are obtained. Based on the event detection results, and combined with the table tennis competition rule database, automated penalty decisions are made, and penalty result signals are generated.

[0012] Furthermore, in the method for realizing the 3D trajectory reconstruction of a table tennis ball and the Hawk-Eye referee system based on high-speed vision, the method includes the following steps: Based on the current ambient lighting conditions and the reflective properties of the ping-pong ball surface, the exposure parameters of each high-speed camera are dynamically adjusted. Target detection is performed by fusing the color, shape, and motion features of a ping-pong ball. The color features are established based on the hue distribution of the ping-pong ball in the HSV color space, the shape features are established based on the roundness and size constraints of the ping-pong ball, and the motion features are established based on the motion saliency of inter-frame difference. A confidence score is generated for the target detection results in each frame of the image, and cross-validation of the detection results from adjacent cameras is triggered when the confidence score is lower than a preset threshold.

[0013] Furthermore, in the method for realizing the 3D trajectory reconstruction of a table tennis ball and the Hawk-Eye referee system based on high-speed vision, the method includes the following steps: The detected ping-pong ball target position is centered with sub-pixel accuracy; Establish a corresponding matching relationship between ping-pong ball targets in different camera images at the same time; Based on the matched multiviews Figure 2 The three-dimensional coordinates of the ping-pong ball are calculated using linear triangulation, based on the 3D coordinates and the corresponding camera projection matrix. The reconstructed 3D trajectory sequence is processed by Kalman filtering to generate the 3D flight trajectory of the ping-pong ball.

[0014] Its beneficial effects lie in the fact that a dedicated table tennis event detection and judgment module has been constructed. Specific detection sub-modules have been designed for unique table tennis judgment scenarios such as net grazing, edge grazing, serve toss height, and serve obstruction, achieving a qualitative leap from trajectory measurement to rule-based judgment and filling the gap in existing technologies lacking automated judgment capabilities. A multi-feature fusion target detection and adaptive exposure control scheme is adopted, integrating color, shape, and motion features for detection. Adaptive exposure control, detection confidence assessment, and cross-camera cross-validation mechanisms are introduced, significantly improving the robustness of table tennis detection under complex lighting conditions and effectively overcoming the detection difficulties caused by the smooth and reflective surface of the table tennis ball. Sub-pixel-level center positioning and multi-view epipolar geometric constraint matching, combined with bundled adjustment optimization, achieve higher accuracy in 3D trajectory reconstruction than existing technologies, providing technical support for high-precision edge grazing judgment. The introduction of judgment confidence calculation and automatic dispute marking mechanisms intelligently identifies disputed judgments requiring manual review, achieving organic synergy between automatic machine judgment and manual review, effectively solving the potential risk of misjudgment in fully automated judgment. Through end-to-end latency monitoring and dynamic degradation strategies, the system ensures that end-to-end latency is controlled within 500 milliseconds, truly achieving real-time online judging during the match. This overcomes the shortcomings of traditional Hawk-Eye systems, which take approximately 10 seconds and can only be used for post-match processing. Based on 3D trajectory reconstruction, the system further estimates the spin parameters of the table tennis ball and incorporates this spin information into judging logic such as service violation detection, expanding the technical dimensions of the table tennis Hawk-Eye system and providing technical support for more comprehensive match judging. Attached Figure Description

[0015] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention.

[0016] Figure 1 This is a schematic diagram of the first embodiment of the table tennis ball three-dimensional trajectory reconstruction and eagle-eye referee system based on high-speed vision in this invention. Figure 2 This is a schematic diagram of the second embodiment of the table tennis ball three-dimensional trajectory reconstruction and eagle-eye referee system based on high-speed vision in this invention. Figure 3 This is a schematic diagram of the first embodiment of the table tennis ball three-dimensional trajectory reconstruction and Hawk-Eye referee system based on high-speed vision in this invention. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0018] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms "one," "an," and "this" used herein may also include the plural forms. It should be further understood that the terminology used in this specification includes the presence of features, integers, steps, operations, elements, and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0019] The present invention will now be described in detail with reference to the accompanying drawings. Figure 1 As shown, a 3D trajectory reconstruction and Hawk-Eye referee system for table tennis based on high-speed vision is presented. This 3D trajectory reconstruction and Hawk-Eye referee system includes the following modules: The image acquisition module is used to simultaneously acquire high-speed image sequences from multiple perspectives during the movement of table tennis; Specifically, in this embodiment, the image acquisition module consists of six high-speed cameras, arranged on both sides of the long and short sides of the ping-pong table, with one or two cameras on each side. The height of the cameras from the table surface varies from 2.5 meters to 4 meters, and the optical axis angle between adjacent cameras is controlled between 30 degrees and 60 degrees to ensure that the field of view of each camera can overlap and cover the entire effective area of ​​the ping-pong ball's movement. Each high-speed camera uses a global shutter CMOS sensor, with a frame rate set to 200 frames / second to 500 frames / second and a resolution set to 1280×1024 pixels or higher. The exposure time is adaptively adjusted according to the ambient lighting conditions. The camera synchronization and calibration module generates multiple synchronization trigger signals with a frequency of 200Hz to 500Hz through a synchronization controller, which are connected to the external trigger interface of each camera via control lines to ensure that each camera acquires image 1 at the same time. The camera calibration adopts Zhang Zhengyou's chessboard calibration method, which collects multiple sets of chessboard images with different postures. The intrinsic parameter matrix, distortion coefficient and extrinsic parameters, rotation matrix and translation vector of each camera are solved by optimization algorithm to establish the accurate mapping relationship between the image coordinate system of each camera and the three-dimensional world coordinate system.

[0020] The camera synchronization calibration module is used to calibrate the internal and external parameters of each high-speed camera, establish the mapping relationship between the image coordinate system of each camera and the three-dimensional world coordinate system, and obtain the camera calibration parameters. Specifically, in this embodiment, the real-time target detection and tracking module performs real-time detection and recognition of ping-pong balls in each frame of the acquired image. Because the surface of a ping-pong ball is smooth, it is prone to specular reflection under strong light conditions, resulting in an irregular brightness distribution of the target in the image. Traditional detection methods based on a single color threshold are prone to failure. Therefore, this embodiment employs the following multi-feature fusion detection strategy. The adaptive exposure control unit monitors the grayscale distribution of the ping-pong ball area in the image in real time. When it detects an oversaturated highlight area on the ping-pong ball surface, it dynamically reduces the camera's exposure time or adjusts the aperture to suppress highlight reflection. When the ambient light is insufficient, it appropriately increases the exposure time or increases the gain to ensure that the ping-pong ball target has sufficient contrast.

[0021] The multi-feature fusion detection unit first segments the image in the HSV color space, extracting candidate regions based on the standard orange hue range of a ping-pong ball, with the H channel set to 5 to 25 degrees. Then, it performs shape analysis on these candidate regions, calculating the roundness, the ratio of the square of the perimeter to the area, the size, and the pixel diameter. Regions with a roundness close to 1 and a size matching the projection size of the ping-pong ball in the image are retained. Finally, the inter-frame difference method is used to calculate motion saliency, eliminating false positives in static backgrounds. The detection results of the three features are fused using a weighted voting method to output the final ping-pong ball target position.

[0022] The detection confidence evaluation unit generates a confidence score between 0 and 1 based on indicators such as the color purity, circularity deviation, motion saliency, and contrast between the target and the background of the target area. When the detection confidence of a certain camera viewpoint is lower than 0.6, cross-validation of the detection results from adjacent cameras is triggered—using epipolar geometric constraints, the target positions detected by adjacent cameras are projected onto the image plane of the current camera to verify whether the current detection result is consistent with the detection results of adjacent viewpoints.

[0023] In terms of target tracking between consecutive frames, this embodiment uses a Kalman filter to predict and correlate the motion of the ping-pong ball on the image plane. For each detected target position in a frame, the Mahalanobis distance between it and the predicted position of the Kalman filter is calculated. If the distance is less than a preset threshold, it is determined to be the same target, and the filter state is updated; if the distance is greater than the threshold, a new trajectory hypothesis is initiated, and false trajectories are eliminated through a multi-frame confirmation mechanism.

[0024] The target detection and tracking module is used to detect and recognize high-speed image sequences in real time, and to use a multi-target tracking algorithm to track the motion of the ping-pong ball between consecutive frames, generating two-dimensional trajectory data from the perspective of each camera. Specifically, in this embodiment, real-time detection and recognition of the ping-pong ball target are performed on each frame of the high-speed image sequence. A multi-target tracking algorithm is used to correlate and track the movement of the ping-pong ball between consecutive frames, generating two-dimensional trajectory data from the perspectives of each camera. This module further includes an adaptive exposure control unit, a multi-feature fusion detection unit, and a detection confidence evaluation unit. The adaptive exposure control unit dynamically adjusts the exposure parameters of each high-speed camera according to the current ambient lighting conditions and the reflective characteristics of the ping-pong ball surface to suppress the interference of high-gloss reflection from the ping-pong ball surface on target detection. The multi-feature fusion detection unit fuses the color features of the ping-pong ball, establishes shape features based on the hue distribution of the ping-pong ball in the HSV color space, establishes motion features based on the circularity and size constraints of the ping-pong ball, and establishes target detection based on the saliency of motion difference between frames. The detection confidence evaluation unit generates a confidence score for the target detection results in each frame and triggers cross-validation of the detection results of adjacent cameras when the confidence score is lower than a preset threshold.

[0025] The 3D trajectory reconstruction module receives the associated 2D trajectory data from each camera's viewpoint and reconstructs the 3D spatial position of the ping-pong ball using a multi-view geometric triangulation method.

[0026] First, the subpixel precision localization unit performs subpixel-level center localization on the detected ping-pong ball target. Specifically, using the detected target area as the center, the grayscale distribution of the target area is extracted, and the subpixel precision center coordinates are calculated using the grayscale centroid method: for each pixel within the target area, its grayscale value is used as a weight, and the weighted average of all pixel coordinates is calculated to obtain the subpixel center position. For slight motion blur that may occur due to high-speed movement, an ellipse fitting method is used to fit the target contour, with the center of the ellipse as the precise position of the target.

[0027] Secondly, the multi-view feature matching unit establishes a corresponding matching relationship between ping-pong ball targets in different camera images at the same time. Since each camera has established an epipolar geometric constraint relationship through calibration, for a target point in one camera image, its corresponding point in another camera image must lie on the corresponding epipolar line. Combining the consistency of the ping-pong ball target's appearance, color, size, etc., across different viewpoints, the cross-viewpoint matching relationship can be uniquely determined.

[0028] Subsequently, the triangulation reconstruction unit is based on the matched multi-view... Figure 2The 3D coordinates of the ping-pong ball are calculated using linear triangulation, based on the 2D coordinates and the corresponding camera projection matrix. For any two matching 2D points, a system of linear equations about the 3D points can be established, and the 3D coordinates can be obtained by solving the least squares method. When there are more than three matching points from different perspectives, a bundled adjustment optimization method is used to jointly optimize the observation data from all perspectives, minimizing reprojection errors and obtaining more accurate 3D coordinates.

[0029] Finally, the trajectory filtering unit performs Kalman filtering or particle filtering on the reconstructed 3D trajectory sequence. Due to image detection noise, calibration errors, and triangulation errors, the reconstructed 3D trajectory may exhibit high-frequency jitter. The Kalman filter uses a kinematic model of a ping-pong ball, either a uniform velocity or uniform acceleration model, to smooth the trajectory, suppress observation noise, and simultaneously predict the ping-pong ball's position at future moments. In cases of occlusion or detection loss, the filter's predictions can temporarily replace the observed values, maintaining the continuity of the trajectory.

[0030] The 3D trajectory reconstruction module is used to reconstruct 3D trajectory data based on 2D trajectory data and camera calibration parameters using a multi-view geometric triangulation method, and to perform spatiotemporal filtering and smoothing on the 3D trajectory data to generate the 3D flight trajectory of the ping-pong ball. Specifically, in this embodiment, based on the two-dimensional trajectory data from each camera's perspective and the camera calibration parameters, a multi-view geometric triangulation method is used to reconstruct the spatial position sequence of the ping-pong ball in the three-dimensional world coordinate system. The reconstructed three-dimensional trajectory is then subjected to spatiotemporal filtering and smoothing to generate the ping-pong ball's three-dimensional flight trajectory. This module further includes a sub-pixel precision positioning unit, a multi-view feature matching unit, a triangulation reconstruction unit, and a trajectory filtering unit. The sub-pixel precision positioning unit is used to perform sub-pixel precision center positioning of the detected ping-pong ball target position, based on the gray-level distribution characteristics of the ping-pong ball target, using the gray-level centroid method or ellipse fitting method. The multi-view feature matching unit is used to establish a corresponding matching relationship between ping-pong ball targets between different camera images at the same time, with matching based on epipolar geometric constraints and the appearance consistency constraints of the ping-pong ball target. The triangulation reconstruction unit is used to perform spatiotemporal filtering and smoothing processing on the already matched multi-view... Figure 2 The 3D coordinates of the ping-pong ball and the corresponding camera projection matrix are calculated using linear triangulation or bundled adjustment optimization methods. The trajectory filtering unit performs Kalman filtering or particle filtering on the reconstructed 3D trajectory sequence to suppress observation noise and reconstruction errors.

[0031] The event detection and judgment module is used to detect key events of the ping-pong ball during its flight based on the three-dimensional flight trajectory and the three-dimensional spatial model of the ping-pong table, and obtain the event detection results. Specifically, in this embodiment, based on the three-dimensional flight trajectory and the three-dimensional spatial model of the ping-pong table, key events during the ping-pong ball's flight are automatically detected. These key events include at least: net-grabbing events, edge-grabbing events, insufficient ball toss height events, ball-obstruction events, and ball-out-of-bounds events. According to the relevant rules of the International Table Tennis Federation, these events cover the main controversial scenarios that athletes can challenge during the match.

[0032] The net-touch event detection submodule includes a net area modeling unit, a trajectory-net-surface intersection detection unit, and a net-touch determination unit. The net area modeling unit establishes a spatial model of the net in a three-dimensional world coordinate system based on camera calibration parameters and table tennis table dimensions. This spatial model includes the position of the net's upper edge and the net's height. The trajectory-net-surface intersection detection unit detects whether the three-dimensional flight trajectory intersects with the net's spatial area model, and records the intersection time, position, and velocity vector when an intersection is detected. The net-touch determination unit determines whether the intersection event is a net-touch event based on the distance between the intersection position and the upper edge of the net, the direction of the velocity vector, and the continuity of the trajectory before and after the intersection. It distinguishes between a net-touch event where the ball lands on the opponent's side of the table after touching the net (a point is lost), and a net-touch event where the ball does not land on the opponent's side of the table.

[0033] The edge-grabbing event detection submodule includes a table boundary modeling unit, a landing point detection unit, an edge-grabbing judgment unit, and an edge-grabbing type differentiation unit. The table boundary modeling unit establishes a precise spatial model of the ping-pong table surface and its boundary lines in a 3D world coordinate system based on camera calibration parameters and the ping-pong table size parameters. This spatial model includes at least the positions of the four corner points, the four sidelines, and the table height. The landing point detection unit detects the intersection of the 3D flight trajectory and the table spatial model, which serves as the landing point of the ping-pong ball. The edge-grabbing judgment unit determines whether a ball grazes the edge based on the spatial distance between the landing point and the table boundary line. When the landing point is located within the table boundary line area and the trajectory exhibits a significant abrupt change in direction before and after the landing point, it is determined to be an edge-grabbing event. The edge-grabbing type differentiation unit distinguishes between top-edge edge-grabbing and side-edge-grabbing based on the specific position of the landing point on the sideline and the trajectory's incident angle. The Eagle Eye referee decision module is used to make automated judgment decisions based on event detection results and the table tennis competition rule base, and generate judgment result signals.

[0034] Specifically, in this embodiment, the system automatically makes rulings based on the detection results of the event detection and ruling module, combined with the table tennis competition rule base, and generates a ruling result signal. This module includes a rule knowledge base, an event rule matching engine, a ruling confidence calculation unit, and a disputed event marking unit. The rule knowledge base stores the structured representation of table tennis competition rules, including at least the logical expressions of serving rules, hitting rules, scoring rules, and losing rules. The event rule matching engine matches key events detected by the event detection and ruling module with rules in the rule knowledge base in real time, triggering corresponding ruling logic. The ruling confidence calculation unit generates a confidence score for each ruling decision, calculated based on the number of detected event evidences, the consistency of the evidence, and the confidence of each detection step. The disputed event marking unit marks ruling decisions with confidence scores below a preset threshold as disputed events and prompts the referee for manual review via a human-computer interaction module.

[0035] The 3D visualization and playback module is used to visualize and render the 3D flight trajectory and key events in a 3D virtual scene in real time, and to display the trajectory and events for a specified time period in 3D playback in response to external playback commands. This module includes a multi-view synchronous playback unit, a trajectory animation generation unit, and a penalty result overlay display unit. The multi-view synchronous playback unit simultaneously displays the original images from multiple camera perspectives and the corresponding 3D virtual scene during playback, and simultaneously labels the position of the ping-pong ball and key event markers in each perspective image. The trajectory animation generation unit generates a smooth 3D trajectory animation based on the 3D flight trajectory, and uses color gradients to indicate the speed and / or height changes of the ping-pong ball in the trajectory animation. The penalty result overlay display unit overlays penalty result information onto the 3D virtual scene and camera images. The penalty result information includes at least the penalty type, the event on which the penalty was based, and the penalty timestamp.

[0036] The human-computer interaction module is used to receive the referee's penalty review instructions and replay control instructions, and to confirm or correct the penalty result signal.

[0037] Furthermore, the system of this invention also includes a table tennis ball spin parameter estimation module, used to estimate the spin speed and spin axis direction of the table tennis ball based on the three-dimensional trajectory sequence output by the three-dimensional trajectory reconstruction module and the aerodynamic model of the table tennis ball during flight. The event detection and penalty module is also used to determine whether there is an illegal spin operation during the serve, and whether the spin state of the table tennis ball after the hit conforms to the requirements of the competition rules, based on the spin parameters.

[0038] This system also includes a real-time performance assurance unit, which monitors the latency of the entire processing pipeline, including image acquisition, object detection, 3D reconstruction, and event detection, ensuring that the end-to-end latency from image acquisition to the output of the judgment result does not exceed 500 milliseconds. When a processing latency exceeding a preset threshold is detected, the real-time performance assurance unit automatically reduces the level of detail in the 3D visualization rendering and / or reduces the detection accuracy of non-critical events to ensure the real-time output of the judgment decision.

[0039] Its beneficial effects lie in the fact that a dedicated table tennis event detection and judgment module has been constructed. Specific detection sub-modules have been designed for unique table tennis judgment scenarios such as net grazing, edge grazing, serve toss height, and serve obstruction, achieving a qualitative leap from trajectory measurement to rule-based judgment and filling the gap in existing technologies lacking automated judgment capabilities. A multi-feature fusion target detection and adaptive exposure control scheme is adopted, integrating color, shape, and motion features for detection. Adaptive exposure control, detection confidence assessment, and cross-camera cross-validation mechanisms are introduced, significantly improving the robustness of table tennis detection under complex lighting conditions and effectively overcoming the detection difficulties caused by the smooth and reflective surface of the table tennis ball. Sub-pixel-level center positioning and multi-view epipolar geometric constraint matching, combined with bundled adjustment optimization, achieve higher accuracy in 3D trajectory reconstruction than existing technologies, providing technical support for high-precision edge grazing judgment. The introduction of judgment confidence calculation and automatic dispute marking mechanisms intelligently identifies disputed judgments requiring manual review, achieving organic synergy between automatic machine judgment and manual review, effectively solving the potential risk of misjudgment in fully automated judgment. Through end-to-end latency monitoring and dynamic degradation strategies, the system ensures that end-to-end latency is controlled within 500 milliseconds, truly achieving real-time online judging during the match. This overcomes the shortcomings of traditional Hawk-Eye systems, which take approximately 10 seconds and can only be used for post-match processing. Based on 3D trajectory reconstruction, the system further estimates the spin parameters of the table tennis ball and incorporates this spin information into judging logic such as service violation detection, expanding the technical dimensions of the table tennis Hawk-Eye system and providing technical support for more comprehensive match judging.

[0040] Please see Figure 2 In the high-speed vision-based 3D trajectory reconstruction and Hawk-Eye referee system for table tennis, the target detection and tracking module includes the following units: The adaptive exposure control unit is used to dynamically adjust the exposure parameters of each high-speed camera according to the current ambient lighting conditions and the reflective properties of the ping-pong ball surface. The multi-feature fusion detection unit is used to fuse the color features, shape features and motion features of the ping-pong ball for target detection. The color features are established based on the hue distribution of the ping-pong ball in the HSV color space, the shape features are established based on the roundness constraints and size constraints of the ping-pong ball, and the motion features are established based on the motion saliency of the inter-frame difference. The detection confidence evaluation unit is used to generate a confidence score for the target detection results in each frame of the image, and to trigger cross-validation of the detection results of adjacent cameras when the confidence score is lower than a preset threshold.

[0041] Please see Figure 3 In the high-speed vision-based 3D trajectory reconstruction and Hawk-Eye referee system for table tennis, the event detection and penalty module includes the following sub-modules: The table surface boundary modeling submodule is used to establish an accurate spatial model of the table tennis table surface and its boundary lines in the three-dimensional world coordinate system based on the camera calibration parameters and the table tennis table size parameters. The spatial model includes at least the positions of the four corner points, the positions of the four side lines, and the table height. The landing point detection submodule is used to detect the intersection of the 3D flight trajectory and the table space model, which serves as the landing point of the ping-pong ball. The edge-grabbing determination submodule is used to determine whether a ball is edge-grabbing based on the spatial distance between the landing point and the table boundary line. When the landing point is located in the table boundary line area and the trajectory has a significant change in direction before and after the landing point, it is determined to be an edge-grabbing event. The edge-grip type module is used to distinguish between upper edge grazing and side edge grazing based on the specific position of the landing point on the edge line and the trajectory incident angle.

[0042] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended technical solutions and equivalents.

Claims

1. A 3D trajectory reconstruction and Hawk-Eye referee system for table tennis based on high-speed vision, characterized in that, The three-dimensional trajectory reconstruction and Eagle Eye referee system includes the following modules: The image acquisition module is used to simultaneously acquire high-speed image sequences of the table tennis motion from multiple perspectives; The camera synchronization calibration module is used to calibrate the internal and external parameters of each high-speed camera, establish the mapping relationship between the image coordinate system of each camera and the three-dimensional world coordinate system, and obtain the camera calibration parameters. The target detection and tracking module is used to detect and identify the high-speed image sequence in real time, and to use a multi-target tracking algorithm to track the motion of the ping-pong ball between consecutive frames, generating two-dimensional trajectory data from the perspective of each camera. The 3D trajectory reconstruction module is used to reconstruct 3D trajectory data based on 2D trajectory data and camera calibration parameters using a multi-view geometric triangulation method, and to perform spatiotemporal filtering and smoothing on the 3D trajectory data to generate the 3D flight trajectory of the ping-pong ball. The event detection and judgment module is used to detect key events of the ping-pong ball during its flight based on the three-dimensional flight trajectory and the three-dimensional spatial model of the ping-pong table, and obtain the event detection results. The Eagle Eye referee decision module is used to make automated judgment decisions based on event detection results and the table tennis competition rule base, and generate judgment result signals.

2. The table tennis ball 3D trajectory reconstruction and Hawk-Eye referee system based on high-speed vision as described in claim 1, characterized in that, The target detection and tracking module includes the following units: The adaptive exposure control unit is used to dynamically adjust the exposure parameters of each high-speed camera according to the current ambient lighting conditions and the reflective properties of the ping-pong ball surface. The multi-feature fusion detection unit is used to fuse the color features, shape features, and motion features of a ping-pong ball for target detection. The color features are established based on the hue distribution of the ping-pong ball in the HSV color space, the shape features are established based on the roundness and size constraints of the ping-pong ball, and the motion features are established based on the inter-frame difference motion saliency. The detection confidence evaluation unit is used to generate a confidence score for the target detection results in each frame of the image, and to trigger cross-validation of the detection results of adjacent cameras when the confidence score is lower than a preset threshold.

3. The table tennis ball 3D trajectory reconstruction and Hawk-Eye referee system based on high-speed vision as described in claim 1, characterized in that, The three-dimensional trajectory reconstruction module includes the following units: The subpixel precision positioning unit is used to perform subpixel-level precision center positioning of the detected ping-pong ball target position; The multi-view feature matching unit is used to establish a corresponding matching relationship between ping-pong ball targets in different camera images at the same time. The triangulation reconstruction unit is used to calculate the three-dimensional spatial coordinates of the ping-pong ball using a linear triangulation method based on the matched two-dimensional coordinates of the multi-view view and the corresponding camera projection matrix. The trajectory filtering unit is used to perform Kalman filtering on the reconstructed 3D trajectory sequence to generate the 3D flight trajectory of the ping-pong ball.

4. The table tennis ball 3D trajectory reconstruction and Hawk-Eye referee system based on high-speed vision as described in claim 1, characterized in that, The event detection and judgment module includes the following sub-modules: The net area modeling submodule is used to establish a spatial area model of the net in a three-dimensional world coordinate system based on the camera calibration parameters and the size parameters of the ping-pong table. The spatial area model includes the position of the upper edge line of the net and the height of the net. The trajectory network surface intersection detection submodule is used to detect whether the three-dimensional flight trajectory intersects with the ball net spatial region model, and to record the intersection time, intersection position and intersection velocity vector when an intersection is detected. The net-wipe determination submodule is used to determine whether the cross event is a net-wipe event based on the distance between the cross position and the top edge of the net, the direction of the cross velocity vector, and the continuity of the trajectory before and after the cross moment, and to distinguish between net-wipe and net-wipe loss of points.

5. The table tennis ball 3D trajectory reconstruction and Hawk-Eye referee system based on high-speed vision as described in claim 1, characterized in that, The event detection and judgment module also includes the following sub-modules: The table surface boundary modeling submodule is used to establish an accurate spatial model of the table tennis table surface and its boundary lines in the three-dimensional world coordinate system based on the camera calibration parameters and the table tennis table size parameters. The spatial model includes at least the positions of the four corner points, the positions of the four side lines, and the table height. The landing point detection submodule is used to detect the intersection of the 3D flight trajectory and the table space model, which serves as the landing point of the ping-pong ball. The edge-grabbing determination submodule is used to determine whether a ball is edge-grabbing based on the spatial distance between the landing point and the table boundary line. When the landing point is located in the table boundary line area and the trajectory has a significant change in direction before and after the landing point, it is determined to be an edge-grabbing event. The edge-grip type module is used to distinguish between upper edge grazing and side edge grazing based on the specific position of the landing point on the edge line and the trajectory incident angle.

6. The table tennis ball 3D trajectory reconstruction and Hawk-Eye referee system based on high-speed vision as described in claim 1, characterized in that, The Eagle Eye referee decision-making module includes the following units: The rule knowledge base unit is used to store the structured representation of the rules of table tennis. The rule knowledge base includes at least the logical expressions of the serving rules, hitting rules, scoring rules, and losing rules. The event rule matching engine is used to match key events detected by the event detection and judgment module with the rules in the rule knowledge base in real time, and trigger the corresponding judgment logic. The penalty confidence calculation unit is used to generate a confidence score for each penalty decision. The confidence score is calculated based on the number of detected event evidences, the consistency of the evidence, and the confidence of each detection step. The dispute event marking unit is used to mark the penalty decision with a confidence score lower than a preset threshold as a dispute event and prompt the referee to conduct a manual review.

7. The table tennis ball 3D trajectory reconstruction and Hawk-Eye referee system based on high-speed vision as described in claim 1, characterized in that, The high-speed camera in the image acquisition module has a frame rate of no less than 200 frames per second, and each high-speed camera is equipped with a global shutter sensor to eliminate motion blur caused by the high-speed moving ping-pong ball during image acquisition.

8. The method for implementing the high-speed vision-based three-dimensional trajectory reconstruction and Hawk-Eye referee system for table tennis as described in claim 1, characterized in that, The method includes the following steps: High-speed image sequences were simultaneously acquired from multiple perspectives during the table tennis game. The intrinsic and extrinsic parameters of each high-speed camera are calibrated, and the mapping relationship between the image coordinate system of each camera and the three-dimensional world coordinate system is established to obtain the camera calibration parameters; The high-speed image sequence is detected and identified in real time, and a multi-target tracking algorithm is used to track the motion of the ping-pong ball between consecutive frames to generate two-dimensional trajectory data from the perspective of each camera. Based on the two-dimensional trajectory data and camera calibration parameters, the three-dimensional trajectory data is reconstructed using the multi-view geometric triangulation method, and the three-dimensional trajectory data is then subjected to spatiotemporal filtering and smoothing to generate the three-dimensional flight trajectory of the ping-pong ball. Based on the three-dimensional flight trajectory and the three-dimensional spatial model of the ping-pong table, key events of the ping-pong ball during flight are detected, and event detection results are obtained. Based on the event detection results, and combined with the table tennis competition rule database, automated penalty decisions are made, and penalty result signals are generated.

9. The method for implementing the high-speed vision-based three-dimensional trajectory reconstruction and Hawk-Eye referee system for table tennis as described in claim 8, characterized in that, The method includes the following steps: Based on the current ambient lighting conditions and the reflective properties of the ping-pong ball surface, the exposure parameters of each high-speed camera are dynamically adjusted. Target detection is performed by fusing the color, shape, and motion features of a ping-pong ball. The color features are established based on the hue distribution of the ping-pong ball in the HSV color space, the shape features are established based on the roundness and size constraints of the ping-pong ball, and the motion features are established based on the motion saliency of inter-frame difference. A confidence score is generated for the target detection results in each frame of the image, and cross-validation of the detection results from adjacent cameras is triggered when the confidence score is lower than a preset threshold.

10. A method for implementing the high-speed vision-based three-dimensional trajectory reconstruction and Hawk-Eye referee system for table tennis as described in claim 8, characterized in that, The method includes the following steps: The detected ping-pong ball target position is centered with sub-pixel accuracy; Establish a corresponding matching relationship between ping-pong ball targets in different camera images at the same time; Based on the matched multi-view 2D coordinates and the corresponding camera projection matrix, the 3D spatial coordinates of the ping-pong ball are calculated using the linear triangulation method. The reconstructed 3D trajectory sequence is processed by Kalman filtering to generate the 3D flight trajectory of the ping-pong ball.

Images