Method and system for automatically judging tennis ball out-of-bound

Abstract

The invention discloses a method and system for automatically judging tennis ball out-of-bound, and the method comprises the steps: completing multi-camera lightweight calibration based on a tennis court inherent mark, and building a multi-view collaborative three-dimensional coordinate system; a tennis ball in a video is positioned through a fusion detection algorithm of a frame difference method, an optical flow method and color features, and a real-time three-dimensional coordinate sequence of the tennis ball is obtained by combining multi-view triangulation; when the tennis ball is shielded, a simplified aerodynamic model is called to be matched with a pre-training track library, a fracture track is complemented, and three-dimensional coordinates of a landing point are predicted; and superposing the real-time track and the predicted landing point to a site realistic image constructed based on a Gaussian mixture model, and automatically judging whether the tennis ball is out of bound or not by comparing the space range of the side line / end line of the tennis court through three-dimensional coordinates. The system comprises a multi-camera module, a lightweight calibration module, a fusion detection positioning module, an anti-shielding track prediction module, a real scene fusion module and an out-of-bound judgment module.

Description

Technical Field

[0001] This application relates to the field of intelligent sports refereeing technology, specifically to a method and system for automatically determining whether a tennis ball is out of bounds. Background Technology

[0002] In tennis, "out-of-bounds calls" are a core element affecting the fairness and flow of the game. The core requirement is to accurately identify the spatial relationship between the ball's landing point and the court's sidelines and end lines, and to deliver the call within a reasonable timeframe. As tennis becomes more popular, the demands for calls in different scenarios are becoming increasingly diverse. However, current technological solutions still have multiple pain points and cannot fully meet the industry's needs.

[0003] From the perspective of accuracy and real-time performance in judging, the ball speed in tennis can reach 20-40 m / s, and a deviation in position at the moment of landing (such as within 1 cm) can change the judgment result, requiring the system to have the ability of "high-speed capture + precise positioning". At the same time, the pace of the game requires that the output delay of the judgment result be controlled within 1 second to avoid affecting the progress of the game, which places strict requirements on the real-time performance of data collection, processing and analysis.

[0004] From the perspective of scenario applicability, professional tennis tournaments have extremely high requirements for the accuracy of the rulings (the error must be ≤3cm). However, existing equipment that meets this accuracy (such as multi-high-speed camera collaborative systems) has obvious limitations: the hardware cost is high, with a single set of equipment often costing hundreds of thousands of yuan, and professional personnel are required for installation and debugging. It can only be applied to a few high-end scenarios such as international tournaments and professional leagues. In contrast, in popular scenarios such as amateur leagues, campus competitions, and individual training, it is difficult to deploy such high-end systems due to budget and venue conditions, resulting in the reliance on manual rulings.

[0005] From the perspective of the limitations of human judgment, traditional human judgment relies on the referee's visual observation and experience. However, when tennis is in high-speed motion, factors such as visual persistence at the moment of landing, obstruction by the athlete's body, and interference from the court background (such as the audience seats and light reflections) can easily lead to misjudgments by the referee. Moreover, human judgment cannot trace the trajectory, and when controversial judgments occur, there is a lack of objective basis, which affects the fairness of the game and the experience of the participants.

[0006] From the perspective of the shortcomings of low-cost technical solutions, some ruling schemes for mass-market scenarios (such as simple systems based on ordinary cameras) have key technical bottlenecks: First, insufficient positioning accuracy. Ordinary cameras have low frame rates (usually ≤60fps), making it difficult to capture the continuous trajectory of the tennis ball's high-speed movement, which easily leads to "trajectory breaks" and deviations in landing point prediction. Second, weak anti-interference ability. When faced with athlete obstruction or complex backgrounds (such as worn court lines or shadows), they cannot effectively distinguish the tennis ball from other moving objects, resulting in positioning failure. Third, cumbersome calibration operations. Some multi-view schemes require manual placement of calibration objects (such as standard-length markers) and multiple adjustments, which takes a long time (often exceeding 30 minutes). Ordinary users cannot complete this independently, limiting the promotion and application of the scheme. Summary of the Invention

[0007] In view of the shortcomings of the existing technology, the purpose of this invention is to provide a method and system for automatically determining whether a tennis ball is out of bounds, so as to solve the problems mentioned in the background art.

[0008] According to one aspect of this application, a method for automatically determining whether a tennis ball is out of bounds includes the following steps:

[0009] S1: Perform lightweight calibration of multiple cameras based on the inherent markings of the tennis court and establish a multi-view collaborative three-dimensional coordinate system;

[0010] S11: Establish a real space coordinate system T with the center of the tennis court as the origin. The positive x-axis of the real space coordinate system T is directed towards the end line on one side of the tennis court, the positive y-axis is directed towards the sideline on one side of the tennis court, and the positive z-axis is directed towards the sky. Measure the real three-dimensional coordinates of the inherent markings of the tennis court in the real space coordinate system T. The inherent markings of the tennis court include the intersection of four sidelines and the endpoints of the center lines of two service areas.

[0011] S12: Control N cameras (N≥2) to synchronously capture tennis court images, call a deep learning model to extract the pixel coordinates of the inherent markers in each image, calculate the transformation matrix between each camera and coordinate system T through perspective transformation, and form a multi-view collaborative three-dimensional coordinate system equation set.

[0012] S2: Real-time acquisition of tennis match video data, location of the tennis ball and acquisition of three-dimensional coordinate sequence through fusion detection algorithm;

[0013] S21: Construct a tennis court background image using a Gaussian mixture model, and perform global frame difference and local frame difference on the current video frame to obtain the initial moving object mask image;

[0014] S22: Combine optical flow method to track the vector trajectory of moving objects in the mask image, filter out candidate regions that meet the characteristics of tennis movement, and further filter them by tennis ball color threshold to obtain tennis ball candidate regions.

[0015] S23: Based on the multi-view collaborative 3D coordinate system established in step S1, perform triangulation on the candidate tennis ball area detected by each camera to calculate the real-time 3D coordinates of the tennis ball in the real space coordinate system T, forming a coordinate sequence {(x1,y1,z1,t1),(x2,y2,z2,t2),...,(x n ,y n ,z n ,t n )};

[0016] S3: Detect the occlusion status of the tennis ball. If occlusion exists, complete the trajectory and predict the landing point through the anti-occlusion trajectory prediction module.

[0017] S31: If no three-dimensional coordinates of the tennis ball are detected for 3 consecutive frames, it is determined to be an occlusion state, and the coordinate sequence of the 5 frames before the occlusion is extracted as the historical trajectory.

[0018] S32: Call the simplified aerodynamic model, input the tennis ball's inherent parameters and environmental parameters, and fit the motion velocity vector and acceleration vector by combining the historical trajectory;

[0019] S33: Match the pre-trained trajectory library to correct the output of the simplified aerodynamic model and obtain the completed trajectory and the three-dimensional coordinates (x0, y0, z0) of the landing point during the occlusion period;

[0020] S4: Integrates real-world images and trajectory information to automatically determine whether the tennis ball is out of bounds;

[0021] S41: The real-time coordinate sequence obtained in step S2 and the completed trajectory and landing point coordinates obtained in step S3 are superimposed on the tennis court background image constructed in step S21 to form a real-scene dynamic video and static image containing the trajectory of tennis.

[0022] S42: Extract the three-dimensional coordinate range of the tennis court sidelines / endlines in the real space coordinate system T, and compare the landing point coordinates (x0, y0, 0) with the range: if (x0, y0) exceeds the range, it is determined to be out of bounds; otherwise, it is determined to be in bounds, and a ruling result is generated.

[0023] Preferably, the N cameras mentioned in step S1 are installed 3-5m above the midpoint of the sidelines on both sides of the tennis court, with the shooting angle covering the entire tennis court, and the overlapping area of ​​the shooting range of adjacent cameras is not less than 30% of the area of ​​the tennis court.

[0024] Preferably, the optical flow method in step S22 uses the Lucas-Kanade algorithm, which calculates the grayscale changes of pixels in a 3×3 neighborhood to obtain the instantaneous velocity vector of the moving object, and filters out regions where the velocity vector direction changes continuously and the amplitude conforms to the characteristics of tennis motion.

[0025] Preferably, the simplified aerodynamic model described in step S32 ignores the Magnus force and only considers gravity and air resistance, and its equation set is as follows:

[0026]

[0027] in, For tennis ball speed ( ), ρ is the drag coefficient, and g is the acceleration due to gravity.

[0028] Preferably, the pre-trained trajectory library in step S33 is obtained by collecting more than 1,000 sets of tennis motion data in different scenarios and training with an LSTM neural network, and the trajectory template matching accuracy is ≥95%.

[0029] Preferably, the frame rate of the real-scene dynamic video in step S41 is consistent with the frame rate of the camera acquisition, and the real-scene static image is marked with the coordinates of the landing point and the distance between the edge / end line.

[0030] Preferably, a system for automatically determining whether a tennis ball is out of bounds includes:

[0031] Multi-camera module: includes N (N≥2) high-definition cameras, installed on both sides of the tennis court, for synchronously collecting video data of tennis matches;

[0032] Lightweight calibration module: used to complete camera calibration based on the inherent markings of the tennis court and establish a multi-view collaborative three-dimensional coordinate system. It includes a marking extraction unit and a matrix calculation unit. The marking extraction unit is used to extract the pixel coordinates of the inherent markings in the image, and the matrix calculation unit is used to calculate the perspective transformation matrix and the coordinate system equations.

[0033] Fusion detection and localization module: used to locate tennis balls in videos and obtain three-dimensional coordinate sequences, including background modeling unit, mask generation unit, candidate filtering unit and three-dimensional calculation unit. The background modeling unit constructs a background image through Gaussian mixture model. The mask generation unit is used to perform global / local frame difference. The candidate filtering unit is used to filter by optical flow method and color feature. The three-dimensional calculation unit is used for multi-view triangulation.

[0034] Anti-occlusion trajectory prediction module: used to complete the trajectory and predict the landing point when occlusion occurs, including occlusion judgment unit, model calculation unit and trajectory matching unit;

[0035] The real-scene fusion module is used to overlay the trajectory and landing point onto the real-scene image, including the real-scene generation unit and the overlay rendering unit;

[0036] Out-of-bounds judgment module: used to compare the coordinates of the landing point with the edge / end line and generate the judgment result, including a coordinate extraction unit and a result judgment unit;

[0037] The multi-camera module is connected to the lightweight calibration module and the fusion detection and positioning module respectively, outputting an empty field image frame to the lightweight calibration module and a real-time video stream to the fusion detection and positioning module;

[0038] The lightweight calibration module is connected to the fusion detection and positioning module and the anti-occlusion trajectory prediction module, and outputs the multi-view collaborative three-dimensional coordinate system equations and real space coordinate system parameters;

[0039] The fusion detection and localization module is bidirectionally connected to the anti-occlusion trajectory prediction module, outputting a real-time three-dimensional coordinate sequence and an occlusion trigger signal to it, and receiving the completed trajectory point sequence fed back by it.

[0040] The fusion detection and positioning module and the anti-occlusion trajectory prediction module are respectively connected to the real scene fusion module, and output real-time coordinate sequence, complete trajectory and predicted landing point coordinates;

[0041] The real-scene fusion module is connected to the out-of-bounds judgment module and outputs a real-scene dynamic frame with trajectory and a static summary image.

[0042] The anti-occlusion trajectory prediction module and the fusion detection and positioning module are respectively connected to the out-of-bounds judgment module, and output the landing point coordinates;

[0043] The boundary judgment module is connected to the display module and outputs a dynamic video stream and a static image with the judgment result.

[0044] Preferably, the multi-camera module uses a regular high-definition camera.

[0045] Preferably, it also includes a display module for displaying real-time dynamic video, static images, and judgment results.

[0046] Preferably, the calibration process of the lightweight calibration module does not require manual placement of calibration materials and can be completed with just one acquisition of tennis court footage.

[0047] The advantages of this application compared to existing technologies are:

[0048] 1. By using the inherent markings of the tennis court to replace manual calibration materials, the calibration time is reduced from 30 minutes to 5 minutes, the operation complexity is significantly reduced, and the problem of cumbersome calibration of existing multi-view schemes is solved;

[0049] 2. By combining "global frame difference (anti-background interference) + local frame difference (anti-static interference) + optical flow method (anti-high-speed motion interference) + color features (anti-similar object interference)", a tennis ball positioning accuracy of ≤1cm is achieved under ordinary camera conditions, which is better than monocular vision solutions;

[0050] 3. A hybrid approach combining a simplified aerodynamic model and a pre-trained trajectory library is proposed, which avoids the high computational cost of complex models and solves the problem of trajectory breakage under occlusion, filling the technical gap of existing solutions in occlusion scenarios;

[0051] 4. It uses a regular high-definition camera, eliminating the need for a high-speed camera or binocular equipment, thus reducing hardware costs; at the same time, it achieves real-time judgment through lightweight computing, meeting the needs of mass competitions. Attached Figure Description

[0052] Figure 1 This is a flowchart of a method for automatically determining whether a tennis ball is out of bounds according to an embodiment of this application.

[0053] Figure 2 This is a schematic diagram of a system structure for automatically determining whether a tennis ball is out of bounds, according to an embodiment of this application. Detailed Implementation

[0054] To make the content of this application easier to understand, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0055] like Figure 1 As shown, a method for automatically determining whether a tennis ball is out of bounds includes the following steps:

[0056] Step S1: Perform lightweight calibration of multiple cameras based on the inherent markings of the tennis court, and establish a multi-view collaborative 3D coordinate system; further including:

[0057] Step S11: Coordinate system definition and inherent marker selection; the core of this step is to establish a mapping reference between the real physical space and the camera pixel space. In specific implementation:

[0058] The coordinate system T is established with the geometric center of the tennis court as the origin (0,0,0). The x-axis extends along the end lines (pointing towards one end line is the positive direction), the y-axis extends along the sidelines (pointing towards one sideline is the positive direction), and the z-axis is perpendicular to the ground and pointing upwards (ground height z=0). This coordinate system strictly follows the standard court dimensions of the International Tennis Federation (ITF): the distance between the end lines is 23.77m (x∈[-11.89m,11.89m]), and the distance between the sidelines is 12.8m (y∈[-6.4m,6.4m]).

[0059] For the selection and measurement of inherent markers, six permanent marker points on the tennis court were chosen as calibration references, including: the four intersections of the sidelines, namely A(-11.89,-6.4,0), B(11.89,-6.4,0), C(11.89,6.4,0), and D(-11.89,6.4,0), corresponding to the four corners of the court; and the two endpoints of the center line of the service court, E(-5.485,0,0) and F(5.485,0,0). The center line of the service court is 10.97m long, and its midpoint coincides with the origin of the coordinate system. These marker points do not move during the match and have high contrast (white lines against the green / brown court), facilitating image recognition.

[0060] Step S12: Camera Deployment and Image Acquisition; Use N ≥ 2 ordinary high-definition cameras (resolution ≥ 1920×1080, frame rate ≥ 60fps), no high-speed camera equipment is required. In actual deployment, it is preferable to use 2 cameras, installed 3-5m directly above the midpoint of the sidelines on both sides of the tennis court, with an optimal height of 4m. The lenses should face the center of the court, ensuring that each camera covers the entire court, and that the overlapping area of ​​the two cameras' shooting ranges is ≥ 30%, avoiding blind spots. Through hardware triggering or software timestamp alignment, ensure that multiple cameras acquire images at the same time, with a time error ≤ 1ms, avoiding positioning deviations caused by acquisition time differences.

[0061] The acquired images are processed by a deep learning model pre-trained on a tennis court marker dataset, which can automatically identify the pixel coordinates of six inherent markers. For example, the pixel coordinates of marker A in camera 1 are (u1A, v1A), the pixel coordinates of marker A in camera 2 are (u2A, v2A), and so on, to obtain the pixel coordinates of all markers.

[0062] For each camera, substitute the six marked real 3D coordinates (x, y, 0) and image pixel coordinates (u, v) into the perspective transformation formula:

[0063]

[0064] The 3×3 transformation matrix M is solved using the least squares method to achieve the mapping from real coordinates to pixel coordinates. The matrices of multiple cameras together constitute a multi-view collaborative 3D coordinate system, providing a foundation for subsequent 3D positioning of tennis balls.

[0065] No manual placement of calibration plates or markers is required; calibration can be completed with just one image acquisition, taking ≤5 minutes.

[0066] Step S2: Real-time acquisition of tennis match video data, location of the tennis ball using a fusion detection algorithm, and acquisition of a 3D coordinate sequence; further including:

[0067] Step S21: Background Modeling and Mask Generation; Gaussian Mixture Model (GMM) background modeling: After system startup, the first 100 frames of the ball-free scene are captured, and the GMM model is trained for each pixel. The model parameters are dynamically updated over time to adapt to slow changes in light. After the background image is generated, stationary objects (such as the ball net and the referee's chair) can be effectively filtered out.

[0068] Moving objects are extracted using the double frame difference method, specifically:

[0069] Global frame difference is calculated by performing pixel difference operations between the current frame and the background image to obtain a global difference image. The formula is as follows:

[0070]

[0071] Among them I t For the current frame, B is the background image, and T is the background image. global =30 (grayscale threshold, which can be adjusted adaptively) The result is a binary image (1 represents the motion area, 0 represents the background).

[0072] Local frame difference: The pixel difference between the current frame and the previous frame is calculated to obtain a local difference image.

[0073]

[0074] Where T local =20, used to highlight objects in momentary motion (such as tennis balls).

[0075] Integrating with computation, for D global and D local Perform a logical AND operation to obtain the initial moving object mask image, while filtering out static interference in the global frame difference and noise in the local frame difference.

[0076] Step S22: Tennis candidate area screening; specifically:

[0077] Motion feature selection using optical flow method: The Lucas-Kanade algorithm is used to calculate the optical flow vector of each connected region in the mask image.

[0078] 1) Calculate the gray-level gradient matrix G and temporal gradient vector b of the 3×3 neighborhood for each pixel within the region;

[0079] 2) Solve for the optical flow vector (u,v) using the least squares method: ;

[0080] 3) Filtering criteria: velocity amplitude Converted to actual speed, combined with the camera's focal length, it covers all tennis ball motion states within the range of 0-40m / s, with the angle between adjacent frame velocity vectors ≤17°, a smoothness threshold of 0.3, and avoids non-spherical objects such as birds.

[0081] Color feature filtering is used, as tennis balls are typically bright yellow. Thresholds are set in the RGB color space: R∈[240,255], G∈[200,230], B∈[0,50]. The average RGB values ​​are extracted from the candidate regions after optical flow filtering, and regions that meet the threshold range are retained, ultimately yielding a unique tennis ball candidate region. If multiple regions exist, they are excluded based on historical trajectory correlation.

[0082] Step S23: Multi-view triangulation, for the coordinates of the tennis ball pixels detected by each camera. Combined with the transformation matrix M obtained in step S1 i By inversely tracing back its possible rays in three-dimensional space, and calculating (least square fitting) the spatial intersections of multiple rays, the three-dimensional coordinates of the tennis ball in coordinate system T are obtained. ;

[0083] Coordinate sequence generation: Recording the three-dimensional coordinates of consecutive frames in timestamp order. The sampling interval of the sequence is consistent with the camera frame rate to ensure the continuity of the trajectory.

[0084] Step S3: Detect the tennis ball's occlusion status. If occlusion exists, complete the trajectory and predict the landing point using the anti-occlusion trajectory prediction module; further including:

[0085] Step S31: Occlusion detection and historical trajectory extraction; if no tennis ball candidate region is detected for 3 consecutive frames, and the tennis ball coordinates in the previous frame satisfy z < 2m (the possible height range of occlusion), then it is determined to be an occlusion state; the coordinate sequence of the 5 frames before occlusion is extracted as input, and the motion trend before occlusion is obtained by quadratic polynomial fitting:

[0086]

[0087] Differentiation yields the velocity vector before occlusion. and acceleration vector .

[0088] Step S32: Simplify aerodynamic model calculations; in tennis, the Magnus force (generated by spin) has a relatively small impact on the landing point (error ≤ 2cm), so it is ignored to reduce the computational load; the main considerations are gravity and air resistance:

[0089] gravity: (Direction along the -z axis);

[0090] Air resistance: ,in (Windward area), C D =0.5 (drag coefficient, experimentally determined). It is a unit vector of velocity;

[0091] Solving the system of equations of motion:

[0092]

[0093] in For tennis ball speed ( ), Here, m is the drag coefficient, R is the weight of the tennis ball, and R is the diameter of the tennis ball. , The Runge-Kutta method was used for numerical solution with a time step of 0.01 s to calculate the trajectory during the occlusion period.

[0094] Step 33: Trajectory database matching and landing point prediction; specifically:

[0095] A pre-trained trajectory library was constructed by collecting over 1000 sets of tennis motion data from different scenarios (serve, forehand, backhand, and volley). Each set contains a complete trajectory (from the moment the ball hits to the moment it lands). Trajectory templates were obtained through training with an LSTM neural network. The templates include feature parameters such as velocity, angle, and spin, with a matching accuracy of ≥95%.

[0096] Trajectory correction compares the simplified model-calculated trajectory with the most similar template in the trajectory library, dynamically adjusting the drag coefficient C. D (Correction range ±0.1), reducing errors caused by model simplification.

[0097] Landing point prediction, when z(t) = 0 and v z (t)≤0 (at the moment of landing), record (x0,y0,0) at this moment as the predicted landing point, with an error ≤3cm.

[0098] Step S4: Integrate real-world image and trajectory information to automatically determine whether the tennis ball is out of bounds; further includes:

[0099] Step S41: Real-scene fusion and visualization; specifically, dynamic video generation, rendering real-time trajectory points and completed trajectory points as red lines in time stamp order, marking the landing point prediction box (5cm×5cm) in yellow, and overlaying it onto the background image to form a dynamic video, with the frame rate consistent with the camera (60fps) to ensure smooth display; static image generation, extracting the background image at the moment of landing, marking the complete trajectory (red), landing point (yellow cross), and edge / end line (white), and displaying the distance between the landing point and the nearest edge / end line (accurate to ±1cm), such as "landing point is +0.8cm from the right edge line (within the boundary)".

[0100] Step S42: Out-of-bounds judgment logic; specifically, the court boundary coordinates are defined. Based on coordinate system T, the spatial range of the sidelines / end lines is extracted. The end line is x∈[-11.89m, 11.89m], and exceeding this range indicates the end line is out of bounds; the sideline is y∈[-6.4m, 6.4m], and exceeding this range indicates the sideline is out of bounds; special areas such as the service area can be limited by additional coordinate ranges, such as the service area x∈[-11.89m, -6.4m]. [4m]∪[6.4m,11.89m]; Penalty rules: compare the landing point (x0,y0,0) with the above range. If x0<-11.89m or x0>11.89m, the end line is out of bounds; if y0<-6.4m or y0>11.89m, the end line is out of bounds; otherwise, it is within bounds; Result output: the penalty result (text + icon) is superimposed on the dynamic video and static image, and output through the display module with a delay of ≤0.5s.

[0101] Combination Figure 2 A system for automatically determining whether a tennis ball is out of bounds includes:

[0102] The multi-camera module consists of N (N≥2) high-definition cameras, each with a resolution ≥1920×1080, a frame rate ≥60fps, and a lens focal length of 8-12mm. It supports IP network transmission or USB data interface. The cameras are installed 3-5m directly above the midpoint of the sidelines on both sides of the tennis court, with the lenses facing the center of the court, ensuring that each camera covers the entire court, and the overlapping area of ​​the shooting range of adjacent cameras is ≥30% (eliminating blind spots). During the system startup phase, the module responds to the calibration command and synchronously acquires one frame of an empty court image without the ball (including complete court markings). During the match, it continuously acquires real-time video streams at a frame rate ≥60fps, with each frame accompanied by a precise timestamp (synchronization error ≤1ms), providing raw data for subsequent processing.

[0103] The lightweight calibration module includes a marker extraction unit equipped with a pre-trained YOLOv8 deep learning model to identify the pixel coordinates of inherent markers on the tennis court (intersections of the four sidelines and endpoints of the two service court midlines) in the image; a matrix calculation unit, based on the principle of perspective transformation, fits the real 3D coordinates of the markers with the pixel coordinates to solve the transformation matrix between the camera and the real spatial coordinate system; this module establishes a 3D coordinate system T with the center of the tennis court as the origin (x-axis along the end line, y-axis along the sideline, z-axis perpendicular to the ground); calculates and outputs the transformation matrices M1, M2...Mn of each camera, forming a multi-view collaborative 3D coordinate system equation set, providing a spatial reference for the 3D positioning of the tennis ball.

[0104] The fusion detection and localization module includes a background modeling unit that uses a Gaussian mixture model (GMM) to generate a dynamically updated court background image through training on the previous 100 frames of ball-free images; a mask generation unit that performs "global frame difference (current frame - background image)" and "local frame difference (current frame - previous frame)" on the current frame and generates a moving object mask through AND operation; a candidate filtering unit that combines the Lucas-Kanade optical flow method (filtering areas with speeds of 0-40m / s and trajectory smoothness ≤0.3) and color thresholds (RGB: 240-255, 200-230, 0-50) to locate a unique tennis ball candidate area; and a 3D calculation unit that, based on multi-view triangulation, converts the tennis ball pixel coordinates detected by each camera into 3D coordinates (x, y, z) in coordinate system T. This module outputs a real-time tennis ball 3D coordinate sequence {(x1, y1, z1, t1), ...} to achieve continuous tracking of the tennis ball's trajectory.

[0105] The anti-occlusion trajectory prediction module includes an occlusion determination unit, which triggers an occlusion handling mechanism based on the logic that no tennis ball coordinates are detected for three consecutive frames; a model calculation unit, which calculates the motion trajectory based on a simplified aerodynamic model (considering gravity and air resistance, ignoring Magnus force), taking into account tennis ball parameters (mass 57g, radius 3.3cm) and environmental parameters (air density 1.225kg / m³); and a trajectory matching unit, which calls a pre-trained trajectory library (containing 1000+ sets of service, smash, and other scene templates) to correct model calculation errors and improve prediction accuracy. This module completes the trajectory in occluded scenarios and outputs the coordinate sequence during occlusion and the predicted landing point value (x0, y0, 0).

[0106] The real-scene fusion module includes a real-scene generation unit that performs noise reduction and enhancement processing on the GMM background image, preserving the clarity of key markings such as field edges and end lines; and an overlay rendering unit that overlays the real-time trajectory (red solid line), the completed trajectory (orange dashed line), and the landing point (yellow cross box) onto the real-scene image according to the timestamp, generating dynamic video frames and static summary images; this module outputs a real-scene dynamic video (60fps) with trajectory annotations and static images (including the distance annotation of the landing point from the edge line), realizing trajectory visualization.

[0107] The out-of-bounds judgment module includes a coordinate extraction unit that pre-stores the three-dimensional range of the tennis court sidelines / endlines in coordinate system T (endline x∈[-11.89,11.89]m, sideline y∈[-6.4,6.4]m); and a result judgment unit that compares the landing point coordinates (x0,y0,0) with the boundary range and outputs the result of "in bounds", "endline out of bounds", or "sideline out of bounds", along with the specific distance exceeded (accurate to ±1cm). This module judges the out-of-bounds status based on three-dimensional coordinate logic and provides the judgment result data to the display module.

[0108] The display module uses a 55-inch or larger 4K display screen, supports HDMI or network interface input, and has a response latency of ≤100ms. This module receives the dynamic video stream with the penalty result output by the out-of-bounds judgment module and the static image, and displays them according to the logic of "real-time video first, and static image after landing", to ensure that the penalty result and trajectory visualization are presented synchronously.

[0109] The relationships and interaction mechanisms between the various modules are as follows:

[0110] Multi-camera module → Lightweight calibration module: After receiving the "synchronous acquisition command" from the lightweight calibration module, the multi-camera module captures an open field image at the same time and transmits it to the marker extraction unit; if there is no such command, the camera is in standby mode to ensure the time consistency of the calibration data.

[0111] The output radiation of the lightweight calibration module: The matrix calculation unit synchronously sends the transformation matrix and coordinate system parameters to the fusion detection and localization module (for 3D coordinate transformation) and the anti-occlusion trajectory prediction module (for constraining the physical boundary of the trajectory). Both rely on this reference data to complete the core calculation.

[0112] Fusion detection and localization module → Anti-occlusion trajectory prediction module:

[0113] Under normal circumstances, the fusion detection and localization module continuously outputs real-time coordinate sequences to the anti-occlusion module as the basis for occlusion determination;

[0114] When no tennis ball is detected for 3 consecutive frames, an "occlusion signal" is triggered along with the historical coordinates of the previous 5 frames, driving the anti-occlusion module to switch from standby to working state.

[0115] Anti-occlusion trajectory prediction module → Fusion detection and localization module:

[0116] During the occlusion period, the anti-occlusion module feeds back the complete trajectory points to the fusion module to update the global trajectory library;

[0117] After the occlusion is removed, the deviation between the predicted landing point and the actual detection point is output to assist the fusion module in correcting the positioning parameters.

[0118] The fusion detection and positioning module sends the real-time coordinate sequence to the overlay rendering unit of the real-scene fusion module to generate the red trajectory line;

[0119] The anti-occlusion trajectory prediction module sends the completed trajectory and landing point coordinates to the overlay rendering unit to generate orange dashed lines and yellow markers;

[0120] The optimized background image of the real-scene generation unit serves as the base layer, which is then integrated with the aforementioned trajectory elements to form visual content.

[0121] Real-scene fusion module → Out-of-bounds judgment module: Sends the static summary image to the result judgment unit as visual evidence of the judgment result;

[0122] Core Processing Layer → Out-of-Bounds Judgment Module: The actual landing point output by the fusion module (when there is no occlusion) or the predicted landing point output by the anti-occlusion module (when there is occlusion) serves as the core data input for judgment;

[0123] Out-of-bounds judgment module → Display module: Sends the dynamic video stream and static image with the judgment result (such as "out of bounds 0.5cm") to the display module, ensuring that the output delay is ≤0.5s to meet the real-time requirements of the game.

[0124] Startup phase: After the system is powered on, the lightweight calibration module automatically triggers the multi-camera module to acquire open field images. After completing the calibration, it sends a "start signal" to the fusion detection and positioning module to start the real-time processing flow.

[0125] Landing penalty trigger: When the fusion module detects z=0 (actual landing) or the anti-occlusion module calculates z=0 and v_z≤0 (predicted landing), the out-of-bounds judgment module is immediately triggered to perform boundary comparison and output the penalty result first;

[0126] Anomaly handling mechanism: If data from a certain camera is interrupted, the fusion detection and positioning module automatically switches to single-view positioning mode (accuracy is temporarily reduced but function is maintained) and sends a "device abnormality" prompt to the display module.

[0127] The above embodiments are only used to illustrate the technical solutions of the embodiments of this application, and are not intended to limit them. Although the embodiments of this application have been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features, without departing from the spirit and scope defined by the claims of this application.

Claims

1. A method for automatically determining whether a tennis ball is out of bounds, characterized in that, Includes the following steps: S1: Perform lightweight calibration of multiple cameras based on the inherent markings of the tennis court and establish a multi-view collaborative three-dimensional coordinate system; S11: Establish a real space coordinate system T with the center of the tennis court as the origin. The positive x-axis of the real space coordinate system T is directed towards the end line on one side of the tennis court, the positive y-axis is directed towards the sideline on one side of the tennis court, and the positive z-axis is directed towards the sky. Measure the real three-dimensional coordinates of the inherent markings of the tennis court in the real space coordinate system T. The inherent markings of the tennis court include the intersection of four sidelines and the endpoints of the center lines of two service areas. S12: Control N cameras (N≥2) to synchronously capture tennis court images, call a deep learning model to extract the pixel coordinates of the inherent markers in each image, calculate the transformation matrix between each camera and coordinate system T through perspective transformation, and form a multi-view collaborative three-dimensional coordinate system equation set. S2: Real-time acquisition of tennis match video data, location of the tennis ball and acquisition of three-dimensional coordinate sequence through fusion detection algorithm; S21: Construct a tennis court background image using a Gaussian mixture model, and perform global frame difference and local frame difference on the current video frame to obtain the initial moving object mask image; S22: Combine optical flow method to track the vector trajectory of moving objects in the mask image, filter out candidate regions that meet the characteristics of tennis movement, and further filter them by tennis ball color threshold to obtain tennis ball candidate regions. S23: Based on the multi-view collaborative 3D coordinate system established in step S1, perform triangulation on the tennis ball candidate areas detected by each camera to calculate the real-time 3D coordinates of the tennis ball in the real space coordinate system T, forming a coordinate sequence {(x1,y1,z1,t1),(x2,y2,z2,t2),...,(x n ,y n ,z n ,t n )}; S3: Detect the occlusion status of the tennis ball. If occlusion exists, complete the trajectory and predict the landing point through the anti-occlusion trajectory prediction module. S31: If no three-dimensional coordinates of the tennis ball are detected for 3 consecutive frames, it is determined to be an occlusion state, and the coordinate sequence of the 5 frames before the occlusion is extracted as the historical trajectory. S32: Call the simplified aerodynamic model, input the tennis ball's inherent parameters and environmental parameters, and fit the motion velocity vector and acceleration vector by combining the historical trajectory; S33: Match the pre-trained trajectory library to correct the output of the simplified aerodynamic model and obtain the completed trajectory and the three-dimensional coordinates (x0, y0, z0) of the landing point during the occlusion period; S4: Integrates real-world images and trajectory information to automatically determine whether the tennis ball is out of bounds; S41: The real-time coordinate sequence obtained in step S2 and the completed trajectory and landing point coordinates obtained in step S3 are superimposed on the tennis court background image constructed in step S21 to form a real-scene dynamic video and static image containing the tennis ball trajectory. S42: Extract the three-dimensional coordinate range of the tennis court sidelines / endlines in the real space coordinate system T, and compare the landing point coordinates (x0, y0, 0) with the range: if (x0, y0) exceeds the range, it is determined to be out of bounds; otherwise, it is determined to be in bounds, and a ruling result is generated.

2. The method for automatically determining whether a tennis ball is out of bounds according to claim 1, characterized in that, The N cameras mentioned in step S1 are respectively installed 3-5m above the midpoint of the sidelines on both sides of the tennis court, with the shooting angle covering the entire tennis court, and the overlapping area of ​​the shooting range of adjacent cameras is not less than 30% of the area of ​​the tennis court.

3. The method for automatically determining whether a tennis ball is out of bounds according to claim 1, characterized in that, The optical flow method described in step S22 uses the Lucas-Kanade algorithm, which calculates the grayscale changes of pixels in a 3×3 neighborhood to obtain the instantaneous velocity vector of the moving object, and filters out regions where the velocity vector direction changes continuously and the amplitude conforms to the characteristics of tennis motion.

4. The method for automatically determining whether a tennis ball is out of bounds according to claim 1, characterized in that, The simplified aerodynamic model described in step S32 neglects the Magnus force and only considers gravity and air resistance. Its equations are as follows: ； in, For tennis ball speed ( ), ρ is the drag coefficient, and g is the acceleration due to gravity.

5. The method for automatically determining whether a tennis ball is out of bounds according to claim 1, characterized in that, The pre-trained trajectory library mentioned in step S33 is obtained by collecting more than 1,000 sets of tennis motion data in different scenarios and training with an LSTM neural network, with a trajectory template matching accuracy of ≥95%.

6. The method for automatically determining whether a tennis ball is out of bounds according to claim 1, characterized in that, The frame rate of the real-scene dynamic video in step S41 is consistent with the frame rate of the camera acquisition, and the real-scene static image is marked with the coordinates of the landing point and the distance to the edge / end line.

7. A system for automatically determining whether a tennis ball is out of bounds, characterized in that, include: Multi-camera module: includes N (N≥2) high-definition cameras, installed on both sides of the tennis court, for synchronously collecting video data of tennis matches; Lightweight calibration module: used to complete camera calibration based on the inherent markings of the tennis court and establish a multi-view collaborative three-dimensional coordinate system. It includes a marking extraction unit and a matrix calculation unit. The marking extraction unit is used to extract the pixel coordinates of the inherent markings in the image, and the matrix calculation unit is used to calculate the perspective transformation matrix and the coordinate system equations. Fusion detection and localization module: used to locate tennis balls in videos and obtain three-dimensional coordinate sequences, including background modeling unit, mask generation unit, candidate filtering unit and three-dimensional calculation unit. The background modeling unit constructs a background image through Gaussian mixture model. The mask generation unit is used to perform global / local frame difference. The candidate filtering unit is used to filter by optical flow method and color feature. The three-dimensional calculation unit is used for multi-view triangulation. Anti-occlusion trajectory prediction module: used to complete the trajectory and predict the landing point when occlusion occurs, including occlusion judgment unit, model calculation unit and trajectory matching unit; The real-scene fusion module is used to overlay the trajectory and landing point onto the real-scene image, including the real-scene generation unit and the overlay rendering unit; Out-of-bounds judgment module: used to compare the coordinates of the landing point with the edge / end line and generate the judgment result, including a coordinate extraction unit and a result judgment unit; The multi-camera module is connected to the lightweight calibration module and the fusion detection and positioning module respectively, outputting an empty field image frame to the lightweight calibration module and a real-time video stream to the fusion detection and positioning module; The lightweight calibration module is connected to the fusion detection and positioning module and the anti-occlusion trajectory prediction module, and outputs the multi-view collaborative three-dimensional coordinate system equations and real space coordinate system parameters; The fusion detection and localization module is bidirectionally connected to the anti-occlusion trajectory prediction module, outputting a real-time three-dimensional coordinate sequence and an occlusion trigger signal to it, and receiving the completed trajectory point sequence fed back by it. The fusion detection and positioning module and the anti-occlusion trajectory prediction module are respectively connected to the real scene fusion module, and output real-time coordinate sequence, complete trajectory and predicted landing point coordinates; The real-scene fusion module is connected to the out-of-bounds judgment module and outputs a real-scene dynamic frame with trajectory and a static summary image. The anti-occlusion trajectory prediction module and the fusion detection and positioning module are respectively connected to the out-of-bounds judgment module, and output the landing point coordinates; The boundary judgment module is connected to the display module and outputs a dynamic video stream and a static image with the judgment result.

8. The system for automatically determining whether a tennis ball is out of bounds according to claim 7, characterized in that, The multi-camera module uses a standard high-definition camera.

9. The system for automatically determining whether a tennis ball is out of bounds according to claim 7, characterized in that, It also includes a display module for real-time display of live dynamic videos, still images, and judgment results.

10. A system for automatically determining whether a tennis ball is out of bounds according to claim 7, characterized in that, The calibration process of the lightweight calibration module does not require manual placement of calibration materials and can be completed with just one capture of the tennis court image.

Images