Three-dimensional tennis track real-time reconstruction method and system based on multiple cameras

Abstract

The embodiment of the invention discloses a three-dimensional tennis track real-time reconstruction method and system based on multiple cameras, internal and external parameters and calibration results of the multiple cameras are utilized, respective two-dimensional pixel coordinates are converted into three-dimensional rays, and nodes are positioned in combination with a minimum distance algorithm. According to the embodiment of the invention, the real-time reconstruction of the three-dimensional track of the tennis ball is realized by virtue of the synchronous multi-camera two-dimensional image through the steps of camera accurate calibration, two-dimensional to three-dimensional ray conversion, intersection point calculation and screening, data output and the like. Compared with the prior art, the reconstruction precision is improved through accurate calibration and ray intersection screening; by means of multi-camera data synchronization and simple calculation process, data delay is reduced, and real-time performance is prominent; by means of multi-camera data redundancy and a placeholder mechanism, the robustness in a complex environment is enhanced; the output three-dimensional point data can directly support the functions of ball speed calculation and the like, and the expansibility is good.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, specifically to a method and system for real-time reconstruction of a three-dimensional tennis ball trajectory based on multiple cameras. Background Technology

[0002] In tennis matches and related training environments, accurately and in real-time determining the landing point of the ball is crucial. Traditional landing point determination relies on human line judges, which suffers from high subjectivity, susceptibility to errors, and difficulty in accurately capturing the ball during high-speed rallies. With technological advancements, vision-based automated judging systems are gradually being applied.

[0003] In existing technologies, monocular vision-based tennis ball landing point localization methods are a relatively economical solution. These systems typically use a fixed camera to monitor the entire tennis court. The workflow includes camera calibration, usually using the corner points of the court lines as feature points to solve for the camera's intrinsic and extrinsic parameters. In real-time processing, the system identifies the tennis ball's position in consecutive video frames through image acquisition, preprocessing (such as grayscale conversion, denoising, and edge enhancement), and moving target detection (such as frame differencing), and extracts its two-dimensional pixel coordinates. By analyzing the ball's trajectory and trends (e.g., abrupt changes in speed or extreme points in the vertical direction), the system determines the keyframes for the tennis ball's landing. Finally, combining the detected court boundary information (e.g., through Hough transform), the two-dimensional pixel coordinates of the landing point in the image are mapped to the actual three-dimensional coordinate range of the court, and compared with a predefined boundary area to output the ruling.

[0004] However, this single-camera-based 2D vision system has significant limitations. The most fundamental problem is that it can only provide the two-dimensional coordinates of the tennis ball on the image plane. The system needs to rely on a projection model and camera calibration parameters to estimate the tennis ball's position in three-dimensional space through back-projection calculations. This two-dimensional to three-dimensional calculation process inevitably introduces large positional errors, especially in the depth direction (perpendicular to the imaging plane), where accuracy is difficult to guarantee. This error directly affects the accuracy of the landing point judgment and may lead to incorrect judgments.

[0005] Furthermore, existing systems also face challenges in terms of real-time performance. Tennis is a fast-paced sport with intense competition, requiring extremely low latency in the judging system. In existing technologies, whether it's the back-projection calculation of a monocular system or other possible 3D reconstruction methods, inefficient algorithms or imperfect synchronization mechanisms can lead to a significant increase in data processing latency. This latency makes it difficult for the system to keep up with the real-time dynamic changes of the match, failing to meet the immediate judging requirements of high-speed competitive environments and limiting its application value in official matches or high-standard training. Summary of the Invention

[0006] To address this issue, this invention provides a method and system for real-time reconstruction of a three-dimensional tennis ball trajectory based on multiple cameras, thereby solving the technical problem that existing technologies struggle to accurately determine the actual position of the tennis ball in space, resulting in significant positional errors.

[0007] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0008] According to a first aspect of the present invention, a method for real-time reconstruction of a three-dimensional tennis ball trajectory based on a multi-channel camera is provided. The method is applied to a multi-channel camera, which consists of multiple synchronized cameras deployed on a tennis court, for real-time acquisition of a video stream containing a tennis ball, comprising:

[0009] Each camera is calibrated to obtain its intrinsic parameter matrix, distortion coefficients, and extrinsic parameter matrix relative to the world coordinate system.

[0010] Real-time detection of the two-dimensional pixel coordinates of the tennis ball in video frames from each camera;

[0011] Based on the pinhole camera model, the two-dimensional pixel coordinates of each camera are converted into three-dimensional spatial rays originating from the optical center of the camera using calibration parameters;

[0012] Select at least two spatial rays from different cameras and calculate the distance between their closest point pairs;

[0013] If the distance between the nearest point pairs is less than a preset threshold, their intersection point is used as the 3D reconstruction coordinates of the tennis ball; if no valid intersection point exists, an invalid data flag is output.

[0014] Outputs a three-dimensional coordinate sequence in real time.

[0015] Furthermore, each camera is calibrated to obtain its intrinsic parameter matrix, distortion coefficients, and extrinsic parameter matrix relative to the world coordinate system, including:

[0016] Set up physical markers with known three-dimensional coordinates on the site;

[0017] Images containing marker points are captured using each camera;

[0018] Based on the correspondence between the image coordinates and physical coordinates of the marked points, the intrinsic parameter matrix, distortion coefficients, and extrinsic parameter matrix are solved.

[0019] Furthermore, based on the pinhole camera model, the two-dimensional pixel coordinates of each camera are converted into a three-dimensional spatial ray emanating from the optical center of that camera using calibration parameters, including:

[0020] Two-dimensional pixel coordinates are corrected using intrinsic parameter matrices and distortion coefficients;

[0021] The corrected coordinates are converted into normalized direction vectors in the camera coordinate system;

[0022] The direction vector is transformed to the world coordinate system by using the rotation and translation components of the extrinsic parameter matrix.

[0023] Furthermore, at least two spatial rays from different cameras are selected, and the distance between their nearest point pairs is calculated, including:

[0024] The nearest point pair is calculated by minimizing the objective function:

[0025] P1=C1+λ1·d1

[0026] P2=C2+λ2·d2

[0027] Where C1 and C2 are the world coordinates of the camera's optical center, λ1 and λ2 are scalar parameters on two direction vectors, representing the distance extended along the direction vector from the optical center, and d1 and d2 are the ray direction vectors.

[0028] Furthermore, the preset threshold is 50 mm, and when the distance between the nearest point pairs is less than the preset threshold, it is determined to be a valid intersection point.

[0029] Furthermore, invalid data is identified as a specific combination of values ​​[-9999, -9999, -9999].

[0030] Furthermore, the method for selecting rays is as follows: for each frame of data, all cameras are traversed and the ray intersection points are calculated by combining them in pairs.

[0031] According to a second aspect of the present invention, a real-time reconstruction system for a three-dimensional tennis ball trajectory based on multiple cameras is provided, the system comprising:

[0032] The camera calibration module is used to calibrate each camera and obtain its intrinsic parameter matrix, distortion coefficients, and extrinsic parameter matrix relative to the world coordinate system.

[0033] The coordinate detection module is used to detect the two-dimensional pixel coordinates of the tennis ball in video frames from each camera in real time.

[0034] The ray generation module is used to convert the two-dimensional pixel coordinates of each camera into a three-dimensional spatial ray emanating from the optical center of the camera, based on the pinhole camera model and using calibration parameters.

[0035] The point-to-point calculation module is used to select at least two spatial rays from different cameras and calculate the distance between their nearest point pairs.

[0036] The 3D reconstruction module is used to take the intersection of the nearest point pair as the 3D reconstruction coordinates of the tennis ball if the distance between them is less than a preset threshold; if no valid intersection exists, it outputs an invalid data flag.

[0037] The output module is used to output three-dimensional coordinate sequences in real time.

[0038] The embodiments of the present invention have the following advantages:

[0039] This invention utilizes synchronized multi-camera 2D images, employing steps such as precise camera calibration, 2D-to-3D ray conversion, intersection point calculation and filtering, and data output to achieve real-time reconstruction of the 3D trajectory of a tennis ball. Compared to existing technologies, it improves reconstruction accuracy through precise calibration and ray intersection filtering; reduces data latency and enhances real-time performance by utilizing synchronized data from multiple cameras and a simplified calculation process; strengthens robustness in complex environments through multi-camera data redundancy and placeholder mechanisms; and provides excellent scalability by directly supporting functions such as ball speed calculation. Attached Figure Description

[0040] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely exemplary, and those skilled in the art can derive other embodiments based on the provided drawings without creative effort.

[0041] The structures, proportions, sizes, etc. illustrated in this specification are only for the purpose of assisting those skilled in the art in understanding and reading the content disclosed herein, and are not intended to limit the conditions under which the present invention can be implemented. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportions, or adjustments to the size, without affecting the effects and objectives that the present invention can produce, should still fall within the scope of the technical content disclosed in the present invention.

[0042] Figure 1 A flowchart of existing technology;

[0043] Figure 2 A schematic diagram of the logical structure of a real-time reconstruction system for a three-dimensional tennis ball trajectory based on multiple cameras, provided in an embodiment of the present invention;

[0044] Figure 3 This is a flowchart illustrating a method for real-time reconstruction of a three-dimensional tennis ball trajectory based on multiple cameras, provided in an embodiment of the present invention. Detailed Implementation

[0045] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0046] refer to Figure 1 The paper, "Research on Tennis Landing Point Localization Based on Monocular Vision," proposes a method for tennis landing point localization based on monocular vision, intended to replace or supplement expensive professional landing point tracking systems that are difficult to deploy in low-cost environments. Its core technologies include:

[0047] Camera calibration: Using the corner points of the field lines as calibration feature points, the camera's intrinsic and extrinsic parameters and the reflection parameters are solved; the calibration feature points are optimized based on the reprojection error.

[0048] Landing point tracking: By using a background model and tracking algorithm, the foreground target is identified, and the landing frame is determined based on the channel direction change of adjacent frames.

[0049] Landing point positioning: After mitigating the slight reflection of the landing point in the image, the projection model is applied, and the camera parameters are flipped to flip the landing point coordinates to the three-dimensional world coordinate system.

[0050] This system monitors a tennis court using a fixed monocular camera. First, the image acquisition module continuously acquires video frames. The image preprocessing module performs grayscale conversion, noise reduction, and edge enhancement on the images. Then, the tennis ball detection module uses frame differencing to detect the position of the moving object (tennis ball) and extracts the ball's two-dimensional pixel coordinates within the image frames using centroid localization. Finally, the trajectory extraction module performs linear fitting on the ball's pixel positions across consecutive frames to establish a two-dimensional trajectory.

[0051] To determine whether the tennis ball has landed and is within bounds, the motion analysis module extracts the ball's motion trend and identifies possible landing frames based on a sharp drop in velocity or extreme points in the y-direction. Next, the court line detection module detects the edges in the image based on Hough transform and maps the landing pixel coordinates to the actual court coordinate range.

[0052] Finally, the landing point determination module compares the mapping result with the predefined field area and outputs the penalty result.

[0053] This technology is primarily applicable to standard tennis court environments, using monocular vision to identify and determine the landing point of the tennis ball. It relies on stable image acquisition and processing algorithms, achieving a certain degree of automation in landing point detection. However, it can only obtain the planar coordinates of the tennis ball, then performs inverse-image integration and obtains the three-dimensional coordinates of the landing point through a projection model. This makes it difficult to accurately determine the actual position of the tennis ball in space, resulting in significant positional errors and severely impacting the system's judgment accuracy. This invention, through precise synchronization of data from multiple cameras, combined with rigorous camera calibration and ray intersection algorithms, achieves real-time, high-precision three-dimensional coordinate reconstruction, significantly reducing coordinate errors and improving the overall judgment accuracy of the system.

[0054] When processing real-time video data, the system's real-time performance is insufficient due to the inefficiency of synchronization algorithms or the complexity of 3D reconstruction algorithms, resulting in significant delays that cannot meet the judging requirements of high-speed dynamic environments in tennis matches.

[0055] To address the technical problem of accurately determining the actual position of a tennis ball in space, resulting in significant positional errors, this invention designs an efficient shared memory mechanism and an optimized multi-process architecture. Through real-time data processing, fast frame synchronization, and an efficient 3D reconstruction algorithm, it achieves real-time and accurate tennis ball position tracking and judgment, significantly improving the system's real-time response performance.

[0056] refer to Figure 2 This invention discloses a real-time reconstruction system for three-dimensional tennis ball trajectory based on multiple cameras. The system includes: a camera calibration module 1; a coordinate detection module 2; a ray generation module 3; a point pair calculation module 4; a three-dimensional reconstruction module 5; and an output module 6.

[0057] Corresponding to the aforementioned real-time reconstruction system for 3D tennis ball trajectories based on multi-cameras, this invention also discloses a real-time reconstruction method for 3D tennis ball trajectories based on multi-cameras. The following details a method for real-time reconstruction of 3D tennis ball trajectories based on multi-cameras, as disclosed in this invention, in conjunction with the aforementioned real-time reconstruction system for 3D tennis ball trajectories based on multi-cameras.

[0058] refer to Figure 3This invention discloses a method for real-time reconstruction of a 3D tennis ball trajectory based on multiple cameras. The method is applied to multiple synchronous cameras deployed on a tennis court to acquire video streams containing tennis balls in real time. The method includes: calibrating each camera to obtain its intrinsic parameter matrix, distortion coefficient, and extrinsic parameter matrix relative to the world coordinate system; detecting the 2D pixel coordinates of the tennis ball in each camera's video frame in real time; converting the 2D pixel coordinates of each camera into 3D spatial rays emanating from the optical center of the camera using calibration parameters based on a pinhole camera model; selecting at least two spatial rays from different cameras and calculating the distance between their nearest point pairs; if the distance between the nearest point pairs is less than a preset threshold, using their intersection as the 3D reconstructed coordinates of the tennis ball; if no valid intersection exists, outputting an invalid data identifier; and outputting a 3D coordinate sequence in real time.

[0059] Synchronized 2D image data from multiple cameras is used to accurately reconstruct the real-time 3D trajectory of a tennis ball in space using a ray-mapping method. Camera parameters (including intrinsic matrix and distortion coefficients) are pre-calibrated for each camera by acquiring 2D markers on the court and known 3D markers for precise camera calibration. The synchronized 2D pixel coordinates from the multiple cameras are acquired in real time, and based on a pinhole camera model and camera calibration parameters (intrinsic matrix, distortion coefficients, extrinsic matrix), the 2D pixel coordinates are transformed into rays in 3D space.

[0060] Furthermore, each camera is calibrated to obtain its intrinsic parameter matrix, distortion coefficients, and extrinsic parameter matrix relative to the world coordinate system. This includes: setting physical markers with known three-dimensional coordinates on the site; acquiring images containing the markers through each camera; and solving for the intrinsic parameter matrix, distortion coefficients, and extrinsic parameter matrix based on the correspondence between the image coordinates and physical coordinates of the markers.

[0061] Furthermore, based on the pinhole camera model, the two-dimensional pixel coordinates of each camera are converted into three-dimensional spatial rays originating from the optical center of the camera using calibration parameters. This includes: correcting the two-dimensional pixel coordinates using intrinsic parameter matrices and distortion coefficients; converting the corrected coordinates into normalized direction vectors in the camera coordinate system; and converting the direction vectors to the world coordinate system using rotation and translation components of the extrinsic parameter matrix.

[0062] By mapping the two-dimensional pixel coordinates to a normalized direction vector in the camera coordinate system using the camera intrinsic parameters, and then combining the camera extrinsic parameters (rotation matrix and translation vector), the direction vector is transformed to the world coordinate system, forming a three-dimensional ray that starts from the camera optical center and extends along the direction vector.

[0063] Furthermore, at least two spatial rays from different cameras are selected, and the distance between their nearest point pairs is calculated, including: the calculation of the nearest point pairs uses a minimization objective function:

[0064] P1=C1+λ1·d1

[0065] P2=C2+λ2·d2

[0066] Where C1 and C2 are the world coordinates of the camera's optical center, λ1 and λ2 are scalar parameters on two direction vectors, representing the distance extended along the direction vector from the optical center, and d1 and d2 are the ray direction vectors.

[0067] Two cameras construct rays: P1 = C1 + λ1·d1, P2 = C2 + λ2·d2. The closest point pair is obtained by minimizing ||P1 - P2||. Valid intersection points are selected using a distance threshold (e.g., less than 50 mm). If multiple candidate intersection points exist, the intersection point with the smallest ray distance is selected as the 3D reconstruction point for the current frame. If no valid intersection point exists, placeholders (e.g., [-9999,-9999,-9999]) are used to indicate that there is no valid reconstruction data for this frame. Finally, the valid 3D point data is output in real-time to subsequent processing modules for use in functions such as ball speed calculation, landing point detection, and real-time penalty judgment.

[0068] Furthermore, the preset threshold is 50 mm, and when the distance between the nearest point pairs is less than the preset threshold, it is determined to be a valid intersection point.

[0069] Furthermore, invalid data is identified as a specific combination of values ​​[-9999, -9999, -9999].

[0070] Furthermore, the method for selecting rays is as follows: for each frame of data, all cameras are traversed and the ray intersection points are calculated by combining them in pairs.

[0071] The embodiments of the present invention employ the ray intersection method, and can also achieve real-time reconstruction of the three-dimensional trajectory of a tennis ball through reconstruction by a projection system, finding a surface system, or a reconstruction scheme based on light field gaps.

[0072] This invention utilizes the intrinsic and extrinsic parameters and calibration results of multiple cameras to convert their respective two-dimensional pixel coordinates into three-dimensional rays, and then uses a minimum distance algorithm to locate nodes. By employing synchronized two-dimensional images from multiple cameras, and through steps such as precise camera calibration, two-dimensional to three-dimensional ray conversion, intersection point calculation and filtering, and data output, real-time reconstruction of the three-dimensional trajectory of a tennis ball is achieved. Compared to existing technologies, it improves reconstruction accuracy through precise calibration and ray intersection filtering; reduces data latency and enhances real-time performance by utilizing synchronized data from multiple cameras and a simplified calculation process; strengthens robustness in complex environments through multi-camera data redundancy and placeholder mechanisms; and the output three-dimensional point data can directly support functions such as ball speed calculation, demonstrating excellent scalability.

[0073] Although the present invention has been described in detail above with general descriptions and specific embodiments, modifications or improvements can be made to it, which will be obvious to those skilled in the art. Therefore, all such modifications or improvements made without departing from the spirit of the present invention fall within the scope of protection claimed by the present invention.

Claims

1. A method for real-time reconstruction of a three-dimensional tennis ball trajectory based on multiple cameras, characterized in that, The method is applied to multiple cameras, which are multiple synchronized cameras deployed on a tennis court, for real-time acquisition of video streams containing tennis balls, including: Each camera is calibrated to obtain its intrinsic parameter matrix, distortion coefficients, and extrinsic parameter matrix relative to the world coordinate system. Real-time detection of the two-dimensional pixel coordinates of the tennis ball in video frames from each camera; Based on the pinhole camera model, the two-dimensional pixel coordinates of each camera are converted into three-dimensional spatial rays originating from the optical center of the camera using calibration parameters; Select at least two spatial rays from different cameras and calculate the distance between their closest point pairs; If the distance between the nearest point pairs is less than a preset threshold, their intersection point is used as the 3D reconstruction coordinates of the tennis ball; if no valid intersection point exists, an invalid data flag is output. Outputs a three-dimensional coordinate sequence in real time.

2. The method for real-time reconstruction of a three-dimensional tennis ball trajectory based on multiple cameras as described in claim 1, characterized in that, For each camera, calibrate it to obtain its intrinsic parameter matrix, distortion coefficients, and extrinsic parameter matrix relative to the world coordinate system, including: Set up physical markers with known three-dimensional coordinates on the site; Images containing marker points are captured using each camera; Based on the correspondence between the image coordinates and physical coordinates of the marked points, the intrinsic parameter matrix, distortion coefficients, and extrinsic parameter matrix are solved.

3. The method for real-time reconstruction of a three-dimensional tennis ball trajectory based on multiple cameras as described in claim 2, characterized in that, Based on the pinhole camera model, calibration parameters are used to convert the two-dimensional pixel coordinates of each camera into a three-dimensional spatial ray emanating from the optical center of that camera, including: Two-dimensional pixel coordinates are corrected using intrinsic parameter matrices and distortion coefficients; The corrected coordinates are converted into normalized direction vectors in the camera coordinate system; The direction vector is transformed to the world coordinate system by using the rotation and translation components of the extrinsic parameter matrix.

4. The method for real-time reconstruction of a three-dimensional tennis ball trajectory based on multiple cameras as described in claim 3, characterized in that, Select at least two spatial rays from different cameras and calculate the distance between their nearest point pairs, including: The nearest point pair is calculated by minimizing the objective function: P1=C1+λ1·d1 P2=C2+λ2·d2 Where C1 and C2 are the world coordinates of the camera's optical center, λ1 and λ2 are scalar parameters on two direction vectors, representing the distance extended along the direction vector from the optical center, and d1 and d2 are the ray direction vectors.

5. The method for real-time reconstruction of a three-dimensional tennis ball trajectory based on multiple cameras as described in claim 4, characterized in that, The preset threshold is 50 mm. When the distance between the nearest points is less than the preset threshold, it is determined to be a valid intersection point.

6. The method for real-time reconstruction of a three-dimensional tennis ball trajectory based on multiple cameras as described in claim 5, characterized in that, Invalid data is identified by a specific combination of values ​​[-9999, -9999, -9999].

7. The method for real-time reconstruction of a three-dimensional tennis ball trajectory based on multiple cameras as described in claim 6, characterized in that, The method for selecting rays is as follows: for each frame of data, traverse all cameras and calculate the ray intersection points by combining them in pairs.

8. A real-time reconstruction system for a three-dimensional tennis ball trajectory based on multiple cameras, characterized in that, The system includes: The camera calibration module is used to calibrate each camera and obtain its intrinsic parameter matrix, distortion coefficients, and extrinsic parameter matrix relative to the world coordinate system. The coordinate detection module is used to detect the two-dimensional pixel coordinates of the tennis ball in video frames from each camera in real time. The ray generation module is used to convert the two-dimensional pixel coordinates of each camera into a three-dimensional spatial ray emanating from the optical center of the camera, based on the pinhole camera model and using calibration parameters. The point-to-point calculation module is used to select at least two spatial rays from different cameras and calculate the distance between their nearest point pairs. The 3D reconstruction module is used to take the intersection of the nearest point pair as the 3D reconstruction coordinates of the tennis ball if the distance between them is less than a preset threshold; if no valid intersection exists, it outputs an invalid data flag. The output module is used to output three-dimensional coordinate sequences in real time.

Images