A tennis highlight clip generation method and device based on multi-dimensional competition semantics and space-time features
By constructing a multidimensional tensor evaluation model that combines tennis trajectory and player posture characteristics, the problem of the inability to identify highly entertaining confrontations in existing technologies has been solved, achieving high-quality automatic editing and special effects rendering.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHANGHAI FUTURE MIND CO LTD
- Filing Date
- 2026-05-11
- Publication Date
- 2026-06-05
AI Technical Summary
Existing automatic editing systems for tennis matches are unable to identify and edit out truly entertaining and challenging matches. They are prone to misjudging unforced errors as highlights and lack awareness of players' micro-kinematic characteristics, resulting in low-quality video highlights.
A multidimensional tensor evaluation model is constructed, which combines visual camera recognition of tennis ball trajectory, player running distance, number of rallies and skeletal point posture. The model calculates the excitement of the match through multidimensional features, eliminates boring segments and accurately captures high-quality moments of confrontation, and supports automatic editing and special effects rendering.
It enhances the viewing experience and professionalism of video highlights, accurately identifies long-distance runs and clutch saves, filters out false highlights, and generates video clips that are close to the level of human professionals.
Smart Images
Figure CN122160601A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, and in particular to a method for generating tennis highlights based on multidimensional event semantics and spatiotemporal features. Background Technology
[0002] With the rapid development of sports broadcasting technology and computer vision, the demand for automatic video summarization and highlight extraction technologies for long-duration sports events is increasing. Automated editing can greatly reduce the cost of manual editing and improve the efficiency of distributing event highlights.
[0003] In tennis matches, real-time tracking of tennis balls and recording of match videos are already possible using Hawk-Eye systems or visual cameras deployed around the court. The key challenge in the field of intelligent sports video processing is how to automatically extract truly exciting shots from these long videos containing massive amounts of redundant information (such as players preparing, wiping sweat, and retrieving the ball).
[0004] Current technical solutions mainly fall into two categories:
[0005] The rule-based and score-event-driven solution relies on the referee scoring system. When key score changes such as "Ace", "Break Point", and "Match Point" occur, the system automatically extracts a fixed-length video clip before and after the point as a highlight.
[0006] Schemes based on multimodal shallow feature-driven approaches (such as acoustic and optical flow) can extract audio energy peaks in a video (such as loud cheers from the audience or a player's loud shout when hitting the ball) or trigger clips through simple, dramatic changes in the visuals.
[0007] Existing automatic editing systems are like scorekeepers who only look at the result, not the process. For example, if player A makes a double fault on their serve and player B scores an immediate point, the system will edit that in because it sees the "point" or hears the cheers from player B's fans. However, this kind of shot is extremely boring for the audience. Conversely, if A and B go back and forth on the baseline for over 30 rallies, during which player B even makes an extreme diving save, although B might ultimately lose the point (without triggering the score edit), the entire process is incredibly exciting. Current technology cannot understand "running distance," "number of rallies," and "player posture," therefore it cannot identify and edit these truly spectacular "miracle shots."
[0008] It is evident that the main drawbacks of existing technologies are:
[0009] The evaluation dimensions are too simplistic and lack process semantics: it relies solely on score changes or the outcome of the match, ignoring the most entertaining aspect of tennis: the "confrontational process" (such as the number of rallies and the area covered by the players' running).
[0010] There is a risk of misjudging "false highlights": it is easy to mistake unforced errors (such as simple net shots or out-of-bounds shots) or double faults, which are extremely unwatchable moments, as highlights.
[0011] Lack of perception of human micro-kinematic characteristics: Existing visual tracking focuses more on the "ball" and ignores the "person", failing to recognize the player's extreme extension, diving, sliding and other high-difficulty body postures, thus losing key elements of visual impact. Summary of the Invention
[0012] To address the aforementioned shortcomings, this disclosure provides a method and apparatus for generating tennis highlight clips based on multidimensional match semantics and spatiotemporal features. By combining the recognition of tennis ball trajectories using a visual camera, it simultaneously extracts features such as the running distance of both players, the number of rallies, the extreme save posture calculated based on skeletal points, and unforced errors, constructing a multidimensional tensor to comprehensively calculate the "excitement level" of the match. This eliminates boring clips, accurately captures truly high-quality moments of competition, and supports subsequent automatic editing and special effects rendering.
[0013] The tennis highlight generation method based on multidimensional event semantics and spatiotemporal features disclosed herein is fundamentally based on the construction of a multidimensional tensor evaluation model. The main steps include:
[0014] S1 acquires the full-scene real-time high-definition video stream and performs basic data analysis and "round" segment cutting;
[0015] S2, for each segmented “round”, perform multi-dimensional spatiotemporal and semantic feature extraction;
[0016] S3, calculates "excitement level" based on the extracted multidimensional tensor;
[0017] S4, the identification and editing of highlight clips.
[0018] Furthermore, step S1 specifically includes:
[0019] The video capture equipment deployed within the tennis court was used to obtain a real-time high-definition video stream of the entire court.
[0020] Using target detection and tracking algorithms, the three-dimensional spatial coordinate sequence of the tennis ball and the 2D and / or 3D bounding boxes and coordinate trajectories of the two players are output in real time.
[0021] Dead ball / live ball cycle segmentation: Based on the continuity of the tennis ball trajectory and the serving action, the video stream is automatically segmented into multiple independent "round" segments, which serve as the basic units for subsequent analysis.
[0022] Furthermore, step S2 specifically includes:
[0023] For each segmented "round," features from multiple dimensions are extracted to construct a tensor, including one or more of the following: temporal adversarial features, spatial consumption features, micro-pose features, and high-level semantic features.
[0024] (1) Temporal confrontation characteristics, i.e., the number of rounds N:
[0025] The total number of racket strokes in a rally is counted based on the number of times the tennis ball's trajectory alternates between the two sides of the net.
[0026] (2) Space consumption characteristics, i.e. running distance D:
[0027] Track the trajectory of the center point of the player's boundary frame and calculate the sum of the total displacement distance of the two players on the field during the round;
[0028] (3) Microscopic posture characteristics, i.e., the score S for the ultimate save posture:
[0029] The player's extreme save posture is identified and recorded as an "extreme save," and a score is awarded for the extremely high posture.
[0030] (4) High-level semantic features, i.e., error penalty factor P:
[0031] By analyzing the landing point of the last shot in a rally and the running status before the shot, if the rally is determined to end due to an unforced error, a penalty factor is assigned.
[0032] Furthermore, in step S2, the features extracted for each cut-out "round" segment also include: acoustic features;
[0033] The acoustic characteristics include: the crispness of the hit and / or the decibel level of the cheers.
[0034] Furthermore, in step S2, the specific method for determining and extracting the player's extreme save posture can be any of the following:
[0035] (1) Method 1
[0036] A human pose estimation model is applied to the player's video area to extract the coordinates of several key skeletal points of the player;
[0037] Calculate the spatial relationships of specific skeletal points, including: calculating the Euclidean distance between the ankle joints to determine whether the body is in a "large stride / split" position; and calculating the angle between the vertical line of the body's center of gravity and the supporting foot to determine whether the body is in a "maximum forward / lateral lean" position.
[0038] When the spatial relationship of a specific skeletal point exceeds a preset threshold, it is recorded as an "extreme save".
[0039] (2) Method Two
[0040] Calculate the aspect ratio of the player's bounding box and the rate of change of its area;
[0041] The player's extreme save posture is determined by the instantaneous widening of the boundary frame or the sharp drop in center of gravity when the player makes a flying or splitting motion.
[0042] Furthermore, the specific methods for steps S3 and S4 include:
[0043] S3, input the features extracted in step S2 into the tensor evaluation algorithm to calculate the overall brilliance score E for each round. The specific calculation method includes:
[0044] Based on the extracted feature dimensions, select the corresponding weight coefficients for weighted fusion calculation;
[0045] When the extracted features are the temporal adversarial features, spatial consumption features, micro-pose features, and high-level semantic features, the weighted fusion function is:
[0046]
[0047] In the formula, w1, w2, w3, and w4 are the weight coefficients for each dimension; f(N) and g(D) are monotonically increasing normalization functions; max(S) A , S B () indicates that the highest possible stance achieved by either player is scored.
[0048] S4. When the overall brilliance score E of a certain round is greater than the preset highlight threshold, the round is judged as a highlight segment.
[0049] Extract the start frame to the end frame of the round and generate independent video clips.
[0050] Furthermore, the specific methods for steps S3 and S4 include:
[0051] Using a deep learning model: Construct a classification or regression model based on a multilayer perceptron (MLP) or support vector machine (SVM), and use a data sequence including the number of rounds, running distance, and posture as the input feature vector. Train the model using historical event data that has been manually labeled with "excitement level". The model directly outputs the excitement evaluation results for each round.
[0052] Extract the start frame to the end frame of the highlight round and generate independent video clips.
[0053] Furthermore, the method also includes the following steps:
[0054] S5, Based on the spatial coordinates recorded in step S2, automatically add special effects during the video rendering stage, including:
[0055] Automatically generate local slow motion at the video frame that triggers the "score with extreme save posture";
[0056] Using the coordinates of a tennis ball's trajectory, visual effects are rendered on the shot with the fastest ball speed.
[0057] A tennis highlight generation device based on multidimensional event semantics and spatiotemporal features, applying the above method, mainly includes:
[0058] The video acquisition and basic data parsing module is used to acquire the full-scene real-time high-definition video stream and perform basic data parsing and segment cutting.
[0059] The multidimensional spatiotemporal and semantic feature extraction module is used to extract multidimensional spatiotemporal and semantic features for each segmented "round" fragment;
[0060] The "Excitement Level" calculation module is used to calculate and evaluate "excitement level" based on multidimensional tensors.
[0061] The automatic editing module is used for threshold determination and editing of highlight clips.
[0062] Furthermore, the device also includes an intelligent special effects rendering module, used to automatically add special effects during the video rendering stage.
[0063] The method for generating exciting clips disclosed herein is mainly accomplished through the following technical means:
[0064] (1) Comprehensive evaluation of the excitement of tennis matches: Instead of relying solely on scoring events, a multi-dimensional evaluation model is constructed by integrating the number of rallies, the running distance of both sides, the skeletal posture characteristics of the human body, and the semantics of unforced errors.
[0065] (2) Quantification of extreme ball-saving posture based on skeletal key points: In the field of automatic tennis editing, human skeletal point extraction technology is introduced to calculate joint distance and center of gravity shift, so as to quantify and identify the tennis player's "extreme ball-saving (such as sliding step, flying body)" action;
[0066] (3) Error weighting / penalty mechanism: Based on semantic recognition of unforced errors, the weighting or points deduction is applied when calculating highlight scores to avoid editing low-quality error rounds into highlights.
[0067] (4) Dynamic rendering driven by the above multi-dimensional features: Based on the detected highlight feature trigger points (such as the moment of extreme posture occurrence, the moment of highest ball speed), slow motion or trajectory trailing effects are automatically superimposed on the corresponding timestamp and spatial coordinates of the original video.
[0068] Compared with the prior art, the beneficial effects of this disclosure are:
[0069] ① Deep understanding of the game: Able to truly understand the game from a kinematic and competitive perspective. A point may be due to a simple mistake by the opponent, but a tug-of-war involving long-distance running, high-frequency rallies, and extreme saves, even without a change in the score, can be accurately captured by this method, greatly enhancing the entertainment value and professionalism of the highlights.
[0070] ② Strong anti-interference ability and high accuracy: The introduction of error penalty factor effectively filters out false highlights such as "double error" and "simple net drop", making the quality of the final generated video clips far higher than that of traditional audio or score triggering systems.
[0071] ③ The visual presentation is more impactful: By combining the three-dimensional coordinates of skeletal points and trajectories, the system can not only "cut" out videos, but also "create" special effects (such as precisely positioned slow-motion playback), reaching a production level close to that of a professional post-production team. Attached Figure Description
[0072] The above and other objects, features and advantages of this disclosure will become more apparent from the more detailed description of exemplary embodiments of this disclosure taken in conjunction with the accompanying drawings, in which the same reference numerals generally represent the same components.
[0073] Figure 1 Here is a flowchart of an exemplary multidimensional sports event semantic and spatiotemporal feature tennis highlight generation system according to this disclosure;
[0074] Figure 2 This is a schematic diagram illustrating the principle of determining the ultimate save posture based on key skeletal points. Detailed Implementation
[0075] Preferred embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While preferred embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure will be thorough and complete, and will fully convey the scope of the present disclosure to those skilled in the art.
[0076] This disclosure provides a method for generating tennis highlight clips based on multidimensional event semantics and spatiotemporal features. It focuses on solving the technical problems of existing automatic tennis event editing systems that rely solely on "scoring results" or "acoustic features (such as audience cheers)" to determine highlights. This often results in the generated highlights ignoring truly entertaining high-difficulty rallies, players' extreme saves, and other process-related high points during the match. It also addresses the issue of easily misclassifying low-quality scoring errors, such as unforced mistakes, as highlight clips. The solution constructs an automated highlight clip editing and rendering scheme that truly "understands" tennis and is based on the extraction of comprehensive multidimensional spatiotemporal and semantic features.
[0077] In one exemplary implementation:
[0078] The overall process of the tennis highlight generation method based on multidimensional event semantics and spatiotemporal features disclosed herein is as follows: Figure 1 As shown, the specific steps are as follows:
[0079] Step 1: Video and Basic Data Analysis
[0080] 1. Utilize visual cameras or Hawk-Eye systems deployed within the tennis courts to acquire real-time high-definition video streams of the entire court.
[0081] 2. Using target detection algorithms (such as the YOLO series) and tracking algorithms, output the three-dimensional spatial coordinate sequence of the tennis ball and the 2D / 3D bounding boxes and coordinate trajectories of the two players in real time.
[0082] 3. Dead / Live Ball Cycle Division: Based on the continuity of the tennis ball trajectory and the serving action, the video stream is automatically divided into multiple independent "rally" segments, which serve as the basic units for subsequent analysis.
[0083] Step 2: Multidimensional Spatiotemporal and Semantic Feature Extraction
[0084] For each segment of "rounds", extract features from the following four dimensions to construct a tensor:
[0085] 1. Timing-based competitive characteristics (number of rounds N):
[0086] The total number of racket strokes in a rally is counted based on the number of times the tennis ball's trajectory alternates between the two sides of the net.
[0087] 2. Space consumption characteristics (running distance D):
[0088] Track the trajectory of the center point of the player's boundary frame and calculate the sum of the total displacement distance of the two players on the field during the round.
[0089] 3. Microscopic posture characteristics (score S for extreme save posture):
[0090] Human pose estimation models (such as OpenPose, HRNet, MMPose, Mediapipe, etc.) are applied to the player's video region to extract the coordinates of several key skeletal points of the player.
[0091] Calculate the spatial relationships of specific skeletal points. For example, calculate the Euclidean distance between the ankle joints to determine if the body is in a "large stride / split" position; calculate the angle between the vertical line of the body's center of gravity and the supporting foot to determine if the body is in a "maximum forward / lateral lean" position.
[0092] When the angle or distance exceeds the preset threshold, an extremely high posture score is assigned and recorded as an "extreme save".
[0093] As shown in Appendix 2, the system demonstrates the geometric judgment logic of the athlete's "extreme ball-saving posture". When the distance between the two ankles d_ankle in the two-dimensional or three-dimensional skeletal coordinates extracted by the system is greater than the set stride threshold, or when the tilt angle θ formed by the calculated vertical line of the body's center of mass and the line connecting the supporting foot is greater than the preset tilt threshold, the system will assign an extremely high "posture brilliance score" in the multidimensional tensor containing spatiotemporal features at that moment.
[0094] 4. High-level semantic features (error penalty factor P):
[0095] By analyzing the landing point (out of bounds or into the net) of the last shot in the rally and the running state before the shot, if the rally is determined to end due to an extremely simple unforced error (such as hitting the ball directly into the net without running), a penalty factor is assigned.
[0096] III. Step 3: Calculation Model of "Excitement" Based on Multidimensional Tensors
[0097] The extracted features are then input into the tensor evaluation algorithm. For simplicity and efficiency, a weighted fusion function can be used to calculate the overall brilliance score E for this round:
[0098]
[0099] Where w1, w2, w3, and w4 are the weight coefficients for each dimension; f(N) and g(D) are monotonically increasing normalization functions; max(S) A , S B() indicates that the highest extreme posture of the players from both sides is scored.
[0100] Step 4: Automatic Editing and Intelligent Effects Rendering
[0101] 1. Threshold Determination and Editing: When the overall excitement score E of a certain round is greater than E... threshold When the preset highlight threshold is reached, the system determines that the rally is a highlight segment. The system extracts the start frame (serve or receive) to the end frame (3 seconds after the ball is dead) of the rally and generates an independent video clip.
[0102] 2. Automatic Effects Rendering: Based on the spatial coordinates recorded in step 2, effects are automatically added during the video rendering stage. For example, at the video frame that triggers the "score with an extreme save posture," a local slow motion is automatically generated; using the tennis ball trajectory coordinates, a "trailing spark" visual effect is rendered on the shot with the fastest ball speed.
[0103] In this embodiment, the "extreme save" feature extraction uses complex skeletal point extraction (such as OpenPose). However, a lighter alternative can be used: directly calculating the aspect ratio of the player's bounding box and its rate of change of area. When a player performs a diving or splitting motion, the bounding box widens instantaneously or the center of gravity drops sharply. This shallow geometric deformation can also be used as an approximate feature input for the extreme pose.
[0104] In this embodiment, the "excitement level" is calculated using a weighted mathematical formula. In fact, a deep learning model can also be used instead: that is, a classification / regression model based on a multilayer perceptron (MLP) or support vector machine (SVM) is constructed, and the data sequences such as the number of rounds, running distance, and posture are used as input feature vectors. The model is trained using historical event data with manually labeled "excitement level", and the model directly outputs the excitement level of the round.
[0105] In this embodiment, four feature dimensions are mainly utilized: temporal adversarial features, spatial consumption features, microscopic posture features, and high-level semantic features. In fact, while retaining the spatiotemporal and semantic features, existing acoustic features (sound of the ball hitting, decibel level of cheers at the scene) can be incorporated and used as the fifth dimension in the multidimensional tensor for joint calculation, further expanding the system's fault tolerance and evaluation comprehensiveness.
[0106] The above technical solutions are merely exemplary embodiments of the present invention. For those skilled in the art, based on the application methods and principles disclosed in the present invention, it is easy to make various types of improvements or modifications, and not limited to the methods described in the specific embodiments of the present invention. Therefore, the methods described above are merely preferred and not restrictive.
Claims
1. A method for generating tennis highlight clips based on multidimensional event semantics and spatiotemporal features, characterized in that, Includes the following steps: S1: Acquire the full real-time high-definition video stream and perform basic data parsing and "round" segment cutting; S2, for each segmented "round", perform multi-dimensional spatiotemporal and semantic feature extraction; S3, calculates "excitement level" based on the extracted multidimensional tensor; S4, the identification and editing of highlight clips.
2. The method according to claim 1, characterized in that, Step S1 specifically includes: Utilize video capture equipment deployed within the tennis courts to obtain real-time high-definition video streams of the entire court; Using target detection and tracking algorithms, the system outputs the three-dimensional spatial coordinate sequence of the tennis ball and the 2D and / or 3D bounding boxes and coordinate trajectories of the two players in real time. Dead ball / live ball cycle segmentation: Based on the continuity of the tennis ball trajectory and the serving action, the video stream is automatically segmented into multiple independent "round" segments, which serve as the basic units for subsequent analysis.
3. The method according to claim 1, characterized in that, Step S2 specifically includes: For each segmented "round," features from multiple dimensions are extracted to construct a tensor, including one or more of the following: temporal adversarial features, spatial consumption features, micro-pose features, and high-level semantic features. (1) Temporal confrontation characteristics, i.e., the number of rounds N: The total number of racket strokes in a rally is counted based on the number of times the tennis ball's trajectory alternates between the two sides of the net. (2) Space consumption characteristics, i.e. running distance D: Track the trajectory of the center point of the player's boundary frame and calculate the sum of the total displacement distance of the two players on the field during the round; (3) Microscopic posture characteristics, i.e., the score S for the ultimate save posture: The system identifies and records a player's extreme save posture as an "extreme save," assigning a high score to the posture. (4) High-level semantic features, i.e., error penalty factor P: By analyzing the landing point of the last shot in a rally and the running state before the shot, if the rally is determined to end due to an unforced error, a penalty factor is assigned.
4. The method according to claim 3, characterized in that, In step S2, the features extracted for each cut-out "round" segment also include: acoustic features; The acoustic characteristics include: the crispness of the hit and / or the decibel level of the cheers.
5. The method according to claim 3, characterized in that, In step S2, the specific method for determining and extracting the player's extreme save posture can be any of the following: (1) Method 1 A human pose estimation model is applied to the player's video area to extract the coordinates of several key skeletal points of the player; Calculate the spatial relationships of specific skeletal points, including: calculating the Euclidean distance between the two ankle joints to determine whether the person is in a "big step / split" state; Calculate the angle between the vertical line of the body's center of gravity and the supporting foot to determine whether the body is in a "maximum forward / lateral tilt" state. When the spatial relationship of a specific skeletal point exceeds a preset threshold, it is recorded as an "extreme save". (2) Method Two Calculate the aspect ratio of the player's bounding box and the rate of change of its area; The player's extreme save posture is determined by the instantaneous widening of the boundary frame or the sharp drop in center of gravity when the player makes a flying or splitting motion.
6. The method according to any one of claims 3-5, characterized in that, The specific methods for steps S3 and S4 include: S3, input the features extracted in step S2 into the tensor evaluation algorithm to calculate the overall brilliance score E for each round. The specific calculation method includes: Based on the extracted feature dimensions, select the corresponding weight coefficients for weighted fusion calculation; When the extracted features are the temporal adversarial features, spatial consumption features, micro-pose features, and high-level semantic features, the weighted fusion function is: In the formula, w1, w2, w3, and w4 are the weight coefficients for each dimension; f(N) and g(D) are monotonically increasing normalization functions; max(S) A , S B () indicates that the highest possible stance achieved by either player is scored. S4. When the overall brilliance score E of a certain round is greater than the preset highlight threshold, the round is judged as a highlight segment. Extract the start frame to the end frame of the round and generate independent video clips.
7. The method according to any one of claims 3-5, characterized in that, The specific methods for steps S3 and S4 include: Using a deep learning model: Construct a classification or regression model based on a multilayer perceptron (MLP) or support vector machine (SVM), and use a data sequence including the number of rounds, running distance, and posture as the input feature vector. Train the model using historical event data that has been manually labeled with "excitement level". The model directly outputs the excitement evaluation results for each round. Extract the start frame to the end frame of the exciting round and generate independent video clips.
8. The method according to claim 1, characterized in that, It also includes the following steps: S5, Based on the spatial coordinates recorded in step S2, automatically add special effects during the video rendering stage, including: Automatically generate local slow motion at the video frame that triggers the "score with extreme save posture"; Using the coordinates of a tennis ball's trajectory, visual effects are rendered on the shot with the fastest ball speed.
9. A tennis highlight generation device based on multidimensional event semantics and spatiotemporal features, applying the method described in any one of claims 1-8, characterized in that, include: The video acquisition and basic data parsing module is used to acquire the full-scene real-time high-definition video stream and perform basic data parsing and segment cutting. The multidimensional spatiotemporal and semantic feature extraction module is used to extract multidimensional spatiotemporal and semantic features for each segmented "round"; The "Excitement Level" calculation module is used to calculate and evaluate "excitement level" based on multidimensional tensors. The automatic editing module is used for threshold determination and editing of highlight clips.
10. The apparatus according to claim 9, characterized in that, Also includes: The intelligent special effects rendering module is used to automatically add special effects during the video rendering stage.