Motion video key clip extraction method based on behavior analysis

Abstract

The invention provides a motion video key clip extraction method based on behavior analysis. The method comprises the following steps: firstly, performing decoding and frame standardization on a match video stream, detecting and positioning a coach target, and obtaining and expanding a bounding box; extracting a candidate skeleton key point set in the extended region, selecting the skeleton with the most key points as the skeleton of the coach, and performing cross-frame tracking; analyzing the skeleton in real time, triggering hand fine analysis and extracting hand key points when a scoring trend appears, comprehensively judging based on geometry, kinematics and time sequence rules, and generating a trigger signal when a specific scoring gesture is detected; and the background thread extracts fragments before and after the triggering moment from the video stream as key fragments, and stores the key fragments in association with metadata such as event types, coach identities and competition states.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and artificial intelligence, and particularly to a method for extracting key segments of sports video based on behavior analysis. BACKGROUND

[0002] With the deep integration of information technology and sports science, the technical and tactical analysis of martial arts, wrestling, boxing and other combat sports is rapidly evolving from traditional manual video watching mode to data-based, intelligent and refined direction. The traditional analysis paradigm relies heavily on experienced coaches, athletes and professional analyst teams, who need to invest a large amount of time to repeatedly watch, pause and play back the competition or training videos for tens of minutes or even hours, and manually mark key events such as scoring moments, tactical changes and foul behaviors. The inherent defects of this mode are obvious and increasingly prominent: first, the human cost is high and the efficiency is extremely low, which cannot meet the growing demand for instant feedback and massive data analysis; second, the analysis results have strong subjectivity, and different analysts have different standards, attention and experience levels for key events, resulting in inconsistent and objective quantitative standards, and even important tactical details may be missed; third, the output is unstructured data, and the manually marked results are mostly discrete time point records or accompanying text descriptions, which are difficult to directly use for subsequent statistical analysis, data mining and model building, and cannot form valuable data assets.

[0003] To address these challenges, both academia and industry are actively exploring the use of computer vision and artificial intelligence technologies to achieve automatic analysis of sports video. The current mainstream research path mostly focuses on identifying and understanding the behavior of athletes, the core of the competition. For example, by deploying pose estimation algorithms (such as OpenPose, HRNet) to track the skeletal dynamics of athletes, and combining temporal action recognition models (such as C3D, I3D, Transformer-based models) to classify their specific actions, such as shooting in football and shooting in basketball. However, when this technology path is applied to martial arts, sanda and Brazilian jiu-jitsu, etc. near combat projects, it encounters huge technical bottlenecks. The core feature of these projects is the long-term, large-area body contact, entanglement and occlusion between athletes, which severely damages the body shape of individual athletes, and the skeletal key points are easily occluded by the opponent's body and cannot be detected, and the action boundaries are blurred, which makes the recognition algorithm solely relying on the athlete's posture accuracy drop sharply, far from meeting the reliability requirements of commercial applications.

[0004] Against this backdrop, existing technologies generally overlook a crucial information source on the field: coaches. In formal competitions like Jiu-Jitsu, coaches are not only issuers of tactical instructions, but their specific gestures towards the referee or technical table—such as raising their hands to indicate a point, an advantage, or a foul—are themselves authoritative key events in the competition. These instructive gestures are often highly correlated with official rulings and are important turning points signifying changes in the game's momentum. Compared to the chaotic and complex grappling movements of the two athletes on the field, the coach's behavior within a designated area, especially their standardized instructive gestures, is highly identifiable. Therefore, building a system that automatically "reads" the coach's behavioral language and uses it as an anchor point to index and segment video footage would open up a new, more efficient, and robust technological path for sports video analysis.

[0005] However, putting this concept into practice faces numerous challenges. While existing general object detection (such as the YOLO series), pose estimation (such as the YOLO-Pose series), and gesture recognition (such as MediaPipe) technologies are mature, how to organically and deeply integrate them for specific application scenarios to form an end-to-end automated process from coach detection, pose understanding, intent judgment to video clip production remains an unresolved technical challenge. Specifically, how to ensure the recall and accuracy of coach detection in complex competition environments, how to accurately identify the coach's skeleton among multiple potential targets, how to accurately capture specific gestures with instructional intent at low computational cost while effectively filtering out everyday, meaningless body movements, and finally, how to seamlessly transform these recognition results into structured video clips with rich contextual information for analysis, are the technical barriers that urgently need to be overcome in this field.

[0006] Chinese patent application CN118015510A discloses a method for extracting and describing key segments of sports videos based on referee gestures. The method includes: extracting frames from a video file or live stream to obtain frame images; detecting referees in the frame images using a custom-trained detection model to obtain referee detection boxes; extracting key points of the human skeleton within the referee detection boxes, and calculating and logically judging the "point-line-angle" relationship between these key points using custom rules; if the relationship between the key points satisfies the description of the referee's rule actions, focusing on the referee's hand area to obtain finer-grained video description information and performing logical judgment; and outputting and describing the key video segments after determining that the conditions for extracting and describing the key segments are met. The drawback of this method is that it does not clearly define a mechanism for accurately and uniquely identifying referees in multi-person scenarios. If multiple non-referee individuals with similar postures appear in the frame, the system may confuse or lose targets, leading to the interruption of the subsequent processing chain. At the same time, this method relies on the macroscopic skeletal relationship of "point--line--corner" as the only prerequisite for triggering hand area analysis. This hard threshold judgment may be too rigid for the diverse and non-standard hand-raising postures of referees, and may easily lead to missed detections. Summary of the Invention

[0007] To address the aforementioned technical issues, this invention proposes a method for extracting key segments from sports videos based on behavioral analysis. By performing hierarchical intelligent analysis of coaches—a key information source—from coarse to fine and conditionally triggered, this method achieves accurate and efficient capture and video content production of scoring instructions that are crucial in a match. This significantly improves the automation, efficiency, and objectivity of match review, tactical analysis, and training evaluation.

[0008] The purpose of this invention is to provide a method for extracting key segments from sports videos based on behavior analysis, including acquiring video stream data of sports event scenes, and further including the following steps:

[0009] Step 1: Perform preprocessing on the video stream data, including decoding and frame normalization;

[0010] Step 2: The first target detection model is used to process each frame of the video stream to detect and locate the coach target in the image and obtain its initial bounding box information;

[0011] Step 3: Expand the initial bounding box by a preset ratio to generate an expanded bounding box that fully encompasses the coach and his / her range of motion.

[0012] Step 4: Within the extended bounding box area, apply the pose estimation model to extract the set of candidate body skeleton key points; if a single target is detected, the set is directly used as the full-body skeleton key points of the coach target; if there are at least two candidate sets, the set with the most valid key points is selected as the full-body skeleton key points of the coach target by comparing the number of valid key points in each set, and cross-frame identity tracking is performed.

[0013] Step 5: Analyze the key points of the coach's entire skeleton in real time. When it is determined that the vertical coordinate of the wrist joint is higher than the vertical coordinate of the shoulder joint, it is determined that a scoring trend has occurred and the fine hand analysis process is triggered.

[0014] Step 6: After the hand detail analysis process is triggered, the hand key point model is called to process the hand area of ​​the coach target and extract the detailed skeletal joints of the hand.

[0015] Step 7: Based on preset geometric rules, kinematic angle rules and temporal logic, comprehensively analyze the position, posture, angle between the arm and shoulder and duration of the state of the fine skeletal joints of the hand to determine whether the coach's target has made a specific gesture that conforms to the scoring instructions.

[0016] Step 8: If it is determined that the coach's target makes the specific gesture, a trigger signal for a key event is generated, and a video segment containing a preset time length before and after the trigger moment is extracted from the video stream data through an independent background processing thread, based on the trigger signal, as the key segment;

[0017] Step 9: Associate and store the key fragment with metadata containing the trigger time, event type, coach identity, and current event status.

[0018] Preferably, step 1 includes sending the video stream data into a video preprocessing unit, decoding the video stream into a sequence of image frames, and performing size normalization and color space correction on each image frame.

[0019] In any of the above schemes, it is preferred that the first target detection model is a YOLOv5 model, which is trained in a supervised manner using a dataset containing at least 20,000 labeled images of coaches under different scenes, poses, and lighting conditions, so as to achieve generalization ability and high robustness in detecting coach targets.

[0020] In any of the above schemes, the preferred embodiment is that the bounding box expansion process includes enlarging the width and height of the initial bounding box by a preset coefficient greater than 1, while keeping the center of the bounding box unchanged, and ensuring that the expanded bounding box does not exceed the original boundary of the image.

[0021] In any of the above schemes, the preferred option is that the pose estimation model is the YOLOv8-Pose model, which is used to obtain 17 human skeleton key points in the COCO format.

[0022] In any of the above schemes, it is preferred that the hand key point model is the MediaPipe Hands model, which is used to achieve high-precision positioning of twenty-one hand three-dimensional joints.

[0023] In any of the above schemes, the preferred embodiment is that the kinematic angle rule includes: calculating the arm span angle formed by the shoulder joint, elbow joint and wrist joint, and the angle between the vector formed by the shoulder joint and elbow joint and the vertical line of the body trunk, and determining whether these angles fall within the preset effective command posture angle range.

[0024] In any of the above schemes, the preferred method for judging the timing logic is to establish a state machine, and when a pose that conforms to the geometric rules and kinematic angle rules is detected to appear continuously for more than a preset frame number threshold, it is determined to be a valid scoring instruction gesture.

[0025] In any of the above solutions, step 7 preferably includes the following sub-steps:

[0026] Step 71: Analyze the internal geometry of the hand, calculate the angle between the joints of the fingers to determine whether the index finger is straight, and calculate the distance from the fingertips of the other fingers to the palm to determine whether the hand is clenched.

[0027] Step 72: Calculate the arm span angle formed by the three points of shoulder, elbow and wrist, and the angle with the main body, and determine whether the above angle is within the preset angle range;

[0028] Step 73: When a pose satisfies all the geometric and kinematic rules and is maintained stably for more than a preset frame threshold, it is recognized as a valid scoring instruction gesture.

[0029] In any of the above schemes, it is preferred that the independent background processing thread maintains a video frame loop buffer of fixed duration. When the key event trigger signal is received, all video frames in the buffer are merged with video frames of a preset duration that are collected after the trigger time, and encoded into the final key segment video file.

[0030] In any of the above schemes, it is preferred that the metadata is a structured string, the format of which includes at least one field among the following: live room ID, match round, red team name, blue team name, scoring team, and scoring timestamp, and is stored in a CSV file or database record.

[0031] This invention proposes a method for extracting key segments from sports videos based on behavior analysis. By conducting in-depth and multi-level intelligent analysis of the coach's behavior, it achieves automated and accurate capture of scoring instructions that are crucial in the game. Using these as core events, it intelligently generates key video segments containing rich contextual information and structured metadata, thereby completely revolutionizing the workflow of traditional sports analysis and significantly improving its efficiency, objectivity, and data value. Attached Figure Description

[0032] Figure 1 This is a flowchart of a preferred embodiment of a motion video key segment extraction method based on behavior analysis according to the present invention.

[0033] Figure 2 This is a schematic diagram of another preferred embodiment of a motion video key segment extraction method based on behavior analysis according to the present invention. Detailed Implementation

[0034] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0035] Example 1

[0036] like Figure 1 As shown, a method for extracting key segments from sports videos based on behavior analysis is described. Step 100 involves obtaining video stream data of a sports event scene.

[0037] Step 110 is executed to preprocess the video stream data by decoding and frame normalization. The video stream data is sent to the video preprocessing unit to decode the video stream into a sequence of images frame by frame, and to normalize the size and correct the color space of each image frame.

[0038] Step 120 involves using a first target detection model to process each frame of the video stream to detect and locate the coach target in the image and obtain its initial bounding box information. The first target detection model is a YOLOv5 model, which is trained in a supervised manner using a dataset containing at least 20,000 labeled images of coaches under different scenes, poses, and lighting conditions to achieve generalization ability and high robustness in detecting coach targets.

[0039] Step 130 involves expanding the initial bounding box by a preset ratio to generate an expanded bounding box that fully encompasses the coach and his / her range of motion. The expansion process includes enlarging the width and height of the initial bounding box by a preset coefficient greater than 1, while keeping the center of the bounding box unchanged and ensuring that the expanded bounding box does not exceed the original boundary of the image.

[0040] In step 140, within the extended bounding box area, a pose estimation model is applied to extract a set of candidate body skeleton key points. If a single target is detected, the set is directly used as the full-body skeleton key points of the coach target. If there are at least two candidate sets, the set with the most valid key points is selected as the full-body skeleton key points of the coach target by comparing the number of valid key points in each set, and cross-frame identity tracking is performed. The pose estimation model is the YOLOv8-Pose model, which is used to obtain 17 human skeleton key points in the COCO format.

[0041] Step 150 is executed, and the key points of the coach's skeletal structure are analyzed in real time. When it is determined that the vertical coordinate of the wrist joint is higher than that of the shoulder joint, a scoring trend is identified, and the fine hand analysis process is triggered.

[0042] Step 160 is executed. After the hand fine analysis process is triggered, the hand key point model is called to process the hand area of ​​the coach target and extract the fine skeletal joints of the hand. The hand key point model is the MediaPipeHands model, which is used to achieve high-precision positioning of twenty-one hand three-dimensional joints.

[0043] Step 170 involves comprehensively analyzing the position, posture, angle between the arm and shoulder, and duration of the state of the fine skeletal joints of the hand based on preset geometric rules, kinematic angle rules, and temporal logic. It determines whether the coach's target has made a specific gesture that conforms to the scoring instruction. The kinematic angle rules include: calculating the arm span angle formed by the shoulder joint, elbow joint, and wrist joint, and the angle between the vector formed by the shoulder joint and elbow joint and the vertical line of the body trunk, and determining whether these angles fall within the preset effective instruction posture angle range.

[0044] The timing logic judgment method is as follows: a state machine is established, and when a posture that conforms to the geometric rules and kinematic angle rules is detected to appear continuously for more than a preset frame number threshold, it is determined to be a valid scoring instruction gesture.

[0045] Step 170 includes the following sub-steps:

[0046] Perform step 171, analyze the internal geometry of the hand, determine whether the index finger is straight by calculating the angle between the joints of the fingers, and determine whether the hand is clenched by calculating the distance from the fingertips of the other fingers to the palm.

[0047] Perform step 172 to calculate the arm span angle formed by the three points of shoulder, elbow and wrist, and the angle with the main body, and determine whether the above angle is within the preset angle range;

[0048] Execute step 173. When a pose satisfies all the geometric and kinematic rules and is maintained stably for more than a preset frame threshold, it is confirmed as a valid scoring instruction gesture.

[0049] In step 180, if it is determined that the coach's target makes the specific gesture, a trigger signal for a key event is generated. Then, through an independent background processing thread, a video segment containing a preset time length before and after the trigger moment is extracted from the video stream data based on the trigger signal as the key segment. The independent background processing thread maintains a fixed-length video frame loop buffer. When the key event trigger signal is received, all video frames in the buffer are merged with video frames of the preset time length that are collected after the trigger moment and encoded into the final key segment video file.

[0050] Step 190 involves associating and storing the key fragment with metadata containing information such as trigger time, event type, coach identity, and current match status. The metadata is a structured string whose format includes at least one field among the following: live stream ID, match round, red team name, blue team name, scoring team, and scoring timestamp, and is stored in a CSV file or database record.

[0051] Example 2

[0052] This invention discloses a method, system, and computer storage medium for extracting key segments from sports videos based on behavior analysis. The method and system aim to solve the core technical problems of traditional sports video analysis, especially in close combat sports such as Jiu-Jitsu, in the automatic identification, analysis, and extraction of key instruction behaviors of coaches and related video segments, which are characterized by low efficiency, heavy reliance on manual labor, strong subjectivity, and insufficient objectivity. The method proposed in this invention has a rigorous and unique technical process, specifically including the following steps: First, continuous video stream data containing sports competition scenes such as Jiu-Jitsu is acquired through video capture equipment and preprocessed according to standardization. Next, a YOLOv5 target detection model, trained on over 20,000 labeled samples, is used to perform robust real-time detection and localization of the coach target in the video frames, and the detected bounding boxes are expanded to provide sufficient contextual information for subsequent analysis. Then, within the expanded region, the YOLOv8-Pose pose estimation algorithm is invoked, and based on the principle of maximizing the number of skeletal points, the key points of the coach's entire skeleton are accurately identified from multiple possible skeletons, and continuous tracking across frames is established. Finally, the system monitors the coach's pose in real time. When it is determined that the hand coordinates are higher than the shoulder coordinates, forming a "hand-raising tendency," this behavior will serve as a trigger condition to activate a high-precision hand key point detection model (such as MediaPipe). The system performs detailed tracking and analysis of the coach's hands. Furthermore, by establishing a composite rule model integrating the spatial geometry of multiple hand joints, specific angle thresholds of the arm and shoulder, and the duration of the movement, it accurately determines specific scoring instruction gestures made by the coach. When the system determines a valid scoring instruction gesture, it uses it as a key event trigger signal. Finally, an independent video processing thread, based on this trigger signal, automatically extracts video segments containing specific time windows before and after the event from a continuously maintained video frame cache, and encapsulates and saves them together with metadata containing real-time event status (such as match ID, round, score, etc.) to form structured key segment data. This invention, through hierarchical intelligent analysis of the coach—a key information source—from coarse to fine, conditionally triggered, achieves accurate and efficient capture and video content production of decisive scoring instruction moments in the game, greatly improving the automation, efficiency, and objectivity of match review, tactical analysis, and training evaluation.

[0053] A method for extracting key segments from motion videos based on behavior analysis includes the following steps:

[0054] S1) Acquire video stream data of the sports event scene, and perform preprocessing such as decoding and frame normalization on the video stream data.

[0055] S2) The first target detection model is used to process each frame of the video stream to detect and locate the coach target in the image and obtain its initial bounding box information.

[0056] In step S2), the first target detection model is a YOLOv5 model. This model is trained in a supervised manner using a dataset containing at least 20,000 labeled images of coaches under different scenes, poses, and lighting conditions to achieve generalization ability and high robustness in detecting coach targets.

[0057] S3) Expand the initial bounding box by a preset ratio to generate an expanded bounding box that fully includes the coach and his / her range of motion.

[0058] In step S3), the bounding box expansion process specifically involves enlarging the width and height of the initial bounding box by a preset coefficient greater than 1 (e.g., 1.15), while keeping the center of the bounding box unchanged and ensuring that the expanded bounding box does not exceed the original boundary of the image.

[0059] S4) Within the extended bounding box region, apply a pose estimation model to extract one or more candidate skeleton key point sets; by comparing the number of valid key points in each candidate set, select the set with the most key points as the coach's full-body skeleton key points, and perform cross-frame identity tracking.

[0060] In step S4), the pose estimation model is the YOLOv8-Pose model, which is used to obtain 17 human skeleton key points in the COCO format.

[0061] S5) Analyze the key points of the coach's entire skeletal structure in real time. When it is determined that the vertical coordinate of the wrist joint is higher than that of the shoulder joint, it is determined that a scoring trend has occurred and the fine hand analysis process is triggered.

[0062] S6) After the scoring trend is triggered, the hand key point model is called to process the coach's hand area and extract the fine skeletal joints of the hand.

[0063] In step S6), the hand key point model is the MediaPipe Hands model, which is used to achieve high-precision positioning of twenty-one hand three-dimensional joints.

[0064] S7) Based on preset geometric rules, kinematic angle rules and temporal logic, comprehensively analyze the position and posture of the fine skeletal joints of the hand, the angle between the arm and the shoulder, and the duration of the state to determine whether the coach has made a specific gesture that conforms to the scoring instructions.

[0065] The kinematic angle rules in step S7) specifically include: calculating the arm span angle formed by the shoulder joint, elbow joint and wrist joint, and the angle between the vector formed by the shoulder joint and elbow joint and the vertical line of the body trunk, and determining whether these angles fall within the preset effective command posture angle range.

[0066] The timing logic judgment in step S7) specifically involves: establishing a state machine, which determines a valid scoring instruction gesture only when a pose conforming to the geometric rules and kinematic angle rules appears continuously for more than a preset frame threshold, in order to filter out instantaneous or unconscious accidental actions.

[0067] S8) If it is determined that the coach makes the specific gesture, a key event trigger signal is generated, and a video segment containing a preset time length before and after the trigger moment is extracted from the video stream data through an independent background processing thread, based on the trigger signal, as the key segment.

[0068] In step S8), the independent background processing thread maintains a fixed-duration video frame loop buffer. When the key event trigger signal is received, all video frames in the buffer are merged with video frames of a preset duration that are collected after the trigger time, and encoded into the final key segment video file.

[0069] S9) Associate and store the key fragment with metadata containing the trigger time, event type, coach identity and current event status information.

[0070] In step S9), the metadata is a structured string, the format of which includes fields such as live room ID, match round, red team name, blue team name, scoring team, and scoring timestamp, and is stored in a CSV file or database record.

[0071] A motion video key segment extraction system based on behavior analysis that implements any of the methods described above includes:

[0072] The video acquisition and preprocessing unit is used to acquire and process the raw video stream;

[0073] The coach detection and tracking unit embeds a first target detection model trained on a large number of samples and a bounding box processing module, which is used to robustly lock and continuously track the coach target;

[0074] The posture and gesture analysis unit embeds a posture estimation model and a hand key point model, and integrates a coach confirmation module based on maximizing the number of skeletal points, an analysis trigger module based on hand raising trends, and a composite rule engine that integrates geometry, kinematics, and temporality to determine the occurrence of specific scoring instruction gestures.

[0075] The key segment generation and storage unit runs on an independent computing thread and is used to perform asynchronous video segment extraction, encapsulation, and structured metadata storage operations based on the trigger signals output by the posture and gesture analysis unit.

[0076] Example 3

[0077] This invention addresses the limitations of existing sports video analysis techniques, particularly in grappling sports like Jiu-Jitsu, which rely heavily on athlete identification and fail to adequately utilize coaches as a key information source. It proposes a novel, efficient, behavior-based method, system, and related technical products for extracting key segments from sports videos. This invention aims to automate and accurately capture crucial scoring instructions during matches through in-depth, multi-layered intelligent analysis of coach behavior. Using these instructions as core events, it intelligently generates key video segments containing rich contextual information and structured metadata, thereby revolutionizing the traditional workflow of sports analysis and significantly improving its efficiency, objectivity, and data value.

[0078] The technical solution of this invention is a logically rigorous, progressive method for extracting key segments from motion videos based on behavior analysis. It transforms raw, linear video stream data into a highly structured, easily searchable, and analyzable collection of key event videos through a series of sophisticated computer vision and artificial intelligence processing steps.

[0079] The first stage of the technical process of this invention begins with the data acquisition and preparation phase. The system acquires real-time match video streams by connecting to high-definition video capture devices deployed at key locations in the competition venue, or obtains raw video data by reading previously stored match recordings locally. This data is first sent to a video preprocessing unit, which uses mature video processing libraries such as OpenCV to decode the video stream into a sequence of image frames. Each frame is then normalized (uniformly scaled to 1920x1080 pixels) and its color space corrected (converted from the camera's default BGR format to the RGB format required by the model), ensuring that subsequent processing modules receive input with a uniform format and stable quality.

[0080] The second stage involves its core, multi-stage intelligent analysis process. The first stage is robust coach detection. The system employs a deeply customized first-object detection model, specifically the YOLOv5 model. The uniqueness of this model lies in its use of non-standard pre-trained weights. Instead, it undergoes supervised learning and fine-tuning through a specially constructed, massive dataset containing over 20,000 images of coaches captured under various conditions (events, venues, lighting, and shooting angles). This dataset covers a wide range of coach postures, including standing, squatting, shouting, instructing, and making various gestures, thus endowing the model with strong generalization ability and high-precision recognition capabilities even in complex background interference. For each input frame, the YOLOv5 model can quickly output the initial bounding box position of the coach target. To address potential minor positioning errors in consecutive frames or incomplete bounding box cropping due to pose changes, this invention specifically designs a "box expansion processing" step. The system enlarges the width and height of the detected initial bounding box by a preset scaling factor (e.g., 1.15) to generate an expanded bounding box. This step is crucial, as it ensures that subsequent pose analysis has the full range of motion of the coach's limbs, preventing analysis failures caused by arms or hands being cropped out of the bounding box.

[0081] After obtaining an expanded bounding box containing sufficient context, the method enters the second stage: accurate coach skeleton recognition and identity verification. Within the image region defined by the expanded bounding box, the system invokes a high-performance pose estimation model; in this embodiment, YOLOv8-Pose is used. YOLOv8-Pose can detect all targets matching human features within this region and output their corresponding 17 COCO-format skeletal keypoints. Here, this invention addresses a key ambiguity issue: the expanded bounding box may contain not only the coach but also parts of the referees, staff, or athletes' bodies. To address this, this invention proposes a heuristic filtering strategy based on information completeness. The system iterates through all candidate skeletons output by YOLOv8-Pose within this region and calculates the number of keypoints successfully detected (i.e., with a confidence level above a certain threshold) in each skeleton. Finally, the system identifies the skeleton with the most valid keypoints as belonging to the target coach. This strategy is based on the reasonable assumption that, in most cases, the coach, as the primary detection target, has the most complete body within the bounding box.

[0082] The third stage is condition-triggered fine-grained hand analysis. To maximize accuracy while minimizing computational resource consumption, this invention does not perform time-consuming hand detail analysis for every frame. Instead, it designs an efficient "intent prediction" mechanism. The system monitors the locked coach's skeletal data in real time, particularly the relative positions of the wrist and shoulder joints. When the system detects that the Y-coordinate value of either wrist joint (in the image coordinate system, a smaller Y-value indicates a higher position) is less than the Y-coordinate value of the shoulder joint on the same side, it determines that the coach has a "hand-raising tendency." This simple yet effective geometric judgment is considered a potential instruction signal and serves as a trigger to activate the next layer of analysis.

[0083] Only after the "hand-raising trend" is triggered does the method enter the fourth stage: high-precision gesture command determination. At this point, the system calls a specialized hand keypoint detection model, such as Google's MediaPipe Hands, and limits its analysis scope to a small area around the wrist point determined by YOLOv8-Pose. The MediaPipe Hands model can accurately resolve 21 three-dimensional fine joints of the human hand. After acquiring these joint point data, this invention does not rely on a black-box classification model, but instead uses a transparent and interpretable composite rule engine built based on the principles of human kinematics and geometry. The engine's judgment logic is multi-dimensional: First, it analyzes the internal geometry of the hand, calculating the angles between the finger joints to determine if the index finger is straight, and calculating the distance from the fingertips to the palm to determine if it's clenched. Second, it introduces more macroscopic kinematic constraints, namely, user-supplemented key conditions, calculating the arm span angle formed by the shoulder, elbow, and wrist, and the angle between the upper arm (shoulder-elbow vector) and the body's main trunk. These angles must fall within a predefined reasonable range to ensure that this is an intentional and standardized command action, rather than a random hand raise. Finally, to combat accidental and transient motion interference, the engine also includes a temporal consistency filter. A gesture must be consistently maintained for more than a preset frame threshold (e.g., 15 consecutive frames) while satisfying all geometric and kinematic rules to be ultimately recognized as a valid scoring command gesture.

[0084] When the composite rule engine determines that a valid scoring instruction gesture has occurred, the core task of the analysis process is completed, and the system immediately generates a key event trigger signal. This signal is a rich data packet containing a precise timestamp, event type ("coach scoring instruction"), and current match status information (live room ID, match round, real-time scores of both sides, etc.).

[0085] Finally, the execution of the method enters the final stage: the generation and structured storage of key segments. To avoid the real-time analysis process being blocked by time-consuming file read and write operations, this invention specifically designs an asynchronous processing architecture. A separate background video processing thread is dedicated to this task. During system operation, this thread continuously maintains a first-in-first-out (FIFO) circular video frame buffer queue, which always stores video frames from the most recent period (e.g., 10 seconds). When this thread receives a key event trigger signal from the main analysis thread, it immediately performs an atomic operation: merging all existing video frames in the buffer queue (the context before the event) with video frames acquired for a short period after the trigger (e.g., 2 seconds) (the reaction after the event), and then calling a video encoder (e.g., H.264) to encode this complete frame sequence into a separate MP4 video file. At the same time, based on the information in the trigger signal, this thread generates a line of structured string data, whose format strictly follows a preset specification, and appends this string to a CSV file or inserts it into a database. Ultimately, the system produces a series of video clips that are centered on the coach's key instructions, are of appropriate length, and contain complete information, along with precisely corresponding structured data records that can be queried and analyzed at any time.

[0086] The beneficial effects of this invention are reflected in several aspects: First, by focusing on coaches as a high signal-to-noise ratio information source and employing an innovative hierarchical trigger-based analysis strategy, it achieves high-precision and high-efficiency automated identification of key events in complex combat sports such as Jiu-Jitsu, freeing analysts from tedious manual labor. Second, based on a clear and quantifiable rule engine, it completely eliminates the subjective bias of manual analysis, ensuring the objectivity, consistency, and reproducibility of the analysis results. Third, it creates a completely new form of data asset, intelligently reconstructing massive amounts of unstructured long videos into a series of short video databases centered on "events" and accompanied by rich metadata tags, providing an unprecedented high-quality data foundation for conducting in-depth technical and tactical data mining, quantitative evaluation of athlete performance, and even automated generation of event highlights. Fourth, this method and system can serve as a powerful real-time review tool, providing coaches and athletes with rapid and accurate decision review, intuitively reproducing the complete scene of each key scoring or tactical instruction moment, thereby greatly assisting daily training and improving on-site competitive performance.

[0087] Example 4

[0088] Figure 2 As shown, the complete execution flow of a motion video key segment extraction method based on behavior analysis provided in this embodiment of the invention can be meticulously decomposed into the following series of interrelated and logically progressive steps:

[0089] Step 1: Accessing and Standardizing Video Data Sources

[0090] The starting point of this method is the acquisition of video data. In a typical Jiu-Jitsu match live broadcast or post-match analysis scenario, this system is designed to flexibly access multiple video sources. This can be real-time video streams transmitted from cameras. Simultaneously, the system is also fully compatible with offline processing of pre-recorded match video files (such as MP4, MOV, AVI, etc.).

[0091] Regardless of the video source, the raw video data is first imported into the system's "Video Acquisition and Preprocessing Unit." The core function of this unit is to use industry-standard open-source multimedia processing frameworks, such as OpenCV's VideoCapture module, to decode the input video, restoring it from compressed encoding formats (such as H.264) into a series of continuous, uncompressed digital image frames. To ensure consistency and efficiency in subsequent algorithm processing, each frame undergoes a series of standardized preprocessing operations. The first is size normalization: regardless of the original video resolution, all frames are uniformly resizing to a fixed standard resolution, such as 1920x1080 pixels. This operation not only fixes the input size for subsequent deep learning models, facilitating batch processing, but also balances image detail with computational load. The second is color space conversion: since OpenCV reads images in BGR (blue, green, red) channel order by default, while most deep learning models (such as those trained in PyTorch or TensorFlow) typically require RGB (red, green, blue) format input, a channel order conversion is necessary. The output at this stage is a sequence of highly uniform image frames flowing at a fixed frame rate (e.g., 30 frames per second), providing a stable and reliable data foundation for all subsequent analysis modules.

[0092] Step 2: Coach Detection and Bounding Box Optimization Based on Deeply Customized Model

[0093] The preprocessed image frame sequence is fed frame by frame into the core of the "Coach Detection and Tracking Unit"—a deeply customized YOLOv5 object detection model. This YOLOv5 model is a key component of the invention; its superior performance does not stem from using publicly available general weights, but rather from an extremely time-consuming and labor-intensive, yet crucial, model training process. Our team collected over 20,000 high-quality coach image samples from hundreds of different Jiu-Jitsu and MMA events, covering various venue environments from small local competitions to international tournaments. The samples exhibit extremely high diversity, including different lighting conditions (bright, dim, top lighting, side lighting), different coach attire (spring / summer clothing, autumn / winter clothing), different body postures (standing to direct, squatting to observe, bending over to observe), and various complex background interferences (moving crowds, flickering screens, complex venue markings). All these images underwent meticulous manual annotation, with bounding boxes precisely outlining the coach's position.

[0094] Leveraging this large and diverse private dataset, we performed transfer learning and extensive fine-tuning on the YOLOv5 model. The resulting coach.pt weight file is able to quickly and reliably identify coach targets in real-world match videos with extremely high accuracy (mAP) and recall, even in highly challenging environments.

[0095] For each input frame, the model outputs the bounding box coordinates (x, y, w, h) of one or more candidate targets, along with their corresponding confidence scores. The system sets a high confidence threshold (e.g., 0.75), and a detection is considered valid only if its confidence score is higher than this threshold.

[0096] However, even high-performance detection models can exhibit "clipping" in their output bounding boxes due to extreme changes in the target's pose (such as a fully extended arm). This means the bounding box might just cover the torso, but the outstretched arm or hand remains outside the box. This is fatal for subsequent pose analysis that relies on complete limb information. To address this issue, this invention introduces a crucial "bounding box expansion" logic. Upon receiving the initial bounding box from YOLOv5, the system doesn't use it directly but programmatically enlarges it: the width *w* and height *h* of the initial bounding box are multiplied by a preset expansion factor (e.g., 1.15, i.e., a 15% enlargement), while keeping the center point of the bounding box unchanged. This operation generates a new, larger "expanded bounding box," which, at the cost of a small amount of background purity, ensures with a very high probability that all parts of the coach's body are completely included when making large limb movements, providing a solid foundation for subsequent accurate analysis.

[0097] Step 3: Accurate estimation of coach posture based on the principle of maximizing bone points

[0098] After obtaining the extended bounding box, the system crops the image region defined by the box from the original frame and uses it as input to the first submodule of the "Pose and Gesture Analysis Unit"—the full-body pose estimator. In this embodiment, we selected the performance-balanced YOLOv8-Pose model. This model is designed to detect all visible human bodies within a given image region and output a skeletal structure for each human body containing 17 keypoints (conforming to the COCO dataset definition, including head, neck, shoulders, elbows, wrists, hips, knees, and ankles).

[0099] At this point, the present invention needs to address a common ambiguity problem in real-world scenarios: due to the expanded bounding box, the area within the box may include background figures besides the target coach, such as passing referees, assistant coaches, or even half of an athlete's body. This can lead to YOLOv8-Pose outputting multiple incomplete skeletons within the area. Randomly selecting one could result in choosing the wrong target. To address this, the present invention innovatively proposes a heuristic filtering strategy based on "maximizing information completeness." Specifically, the system first collects all candidate skeletons output by YOLOv8-Pose within the area. Then, for each candidate skeleton, the system iterates through all 17 predefined keypoints and counts the number of "valid keypoints" with a confidence score higher than a certain threshold (e.g., 0.5). Finally, the system determines the skeleton with the most valid keypoints as belonging to the target coach being tracked. The underlying logic of this strategy is that the coach, as the core of detection and tracking, usually has the most complete body parts within the bounding box, and therefore should have the most detectable skeleton points. This simple yet robust rule greatly improves the ability to uniquely and correctly identify the coach's skeleton in crowded scenarios.

[0100] Step 4: Conditional Hand Analysis Based on "Hand Raising Trend"

[0101] To achieve high-precision gesture recognition while keeping computational costs within an acceptable range, enabling the system to run in real time, this invention designs a cascaded analysis architecture that proceeds from coarse to fine and is triggered by conditions. Instead of running a computationally expensive fine-grained gesture recognition model in every frame, the system first performs a low-computation "intent prediction."

[0102] This predictive mechanism is defined as "hand-raising tendency" detection. In the "Posture and Gesture Analysis Unit," the system monitors the skeletal data of the coach in real time. Specifically, it continuously compares the Y-coordinates of the coach's left and right wrist joints (left_wrist, right_wrist) with the Y-coordinates of the corresponding shoulder joints (left_shoulder, right_shoulder). In a standard image coordinate system, a smaller Y-value indicates a higher position. Therefore, when the system detects that wrist_y < shoulder_y, it means the coach has made a gesture of raising their hand above shoulder height. In the context of sports competition, this gesture is highly likely an intentional signal indicating a readiness to issue a command. Therefore, the appearance of this "hand-raising tendency" is used as a key trigger signal to activate the next stage of fine-grained analysis. This design effectively filters out the vast majority of coaches who are in a natural standing, sitting, or with their arms hanging down—states without any intention to give a command—only allocating computational resources to the moment when a key gesture is most likely to occur.

[0103] Step 5: Precise Judgment of Scoring Instruction Gestures Based on a Composite Rule Engine

[0104] The high-precision hand keypoint detector, the second submodule of the "Pose and Gesture Analysis Unit," is activated only when the "hand-raising trend" is triggered. In this embodiment, we use Google's MediaPipe Hands model. The system defines a compact, dynamic region of interest (ROI) around the raised wrist joint coordinates provided by YOLOv8-Pose and feeds this small image patch containing only the hand into MediaPipe Hands. This model can infer 21 keypoints of the human hand in 3D space in real time with extremely high accuracy and efficiency, including the wrist, palm, and four joints for each finger.

[0105] After obtaining the coordinate sequence of these 21 precise hand joint points, the system enters the core command determination stage. This invention does not employ an end-to-end "black box" deep learning classifier, but instead constructs a completely transparent, interpretable, and flexibly configurable "composite rule engine." The engine's determination logic is multi-dimensional and multi-layered, ensuring extremely high discrimination accuracy.

[0106] Internal geometric rules of the hand: determining the basic shape of a gesture. For example, the geometric rules of a "pointer" gesture (scoring one point) can be quantified and defined as follows:

[0107] Index finger extension: Calculate the four points of the index finger: the metacarpophalangeal joint (MCP), proximal interphalangeal joint (PIP), distal interphalangeal joint (DIP), and fingertip (TIP), ensuring they are approximately collinear. This can be achieved by calculating the angles between the vectors (PIP - MCP) and (DIP - PIP), and between (DIP - PIP) and (TIP - DIP), both of which are close to 180 degrees (e.g., greater than 170 degrees).

[0108] Clenching the remaining fingers: Calculate the Euclidean distance from the fingertips of the middle, ring, and little fingers to their corresponding metacarpophalangeal joints. If this distance is less than a preset threshold, it is determined to be a clenched and bent state.

[0109] Overall body kinematics rules: These are key conditions supplemented by the user to exclude informal or weak hand raises. The rule engine combines the full-body skeletal information provided by YOLOv8-Pose to perform constraint checks.

[0110] Arm-shoulder angle: Calculate the angle between the vector formed by the three points of shoulder, elbow and wrist (elbow-shoulder) and the vertical line of the torso (which can be approximated by the line connecting the midpoint of the neck and hip). This angle must be within a preset range that represents a “powerful lift” (e.g., between 30 and 145 degrees).

[0111] Arm angle: Calculate the angle between the upper arm vector (elbow - shoulder) and the forearm vector (wrist - elbow), i.e., the degree of elbow flexion. This angle must also meet constraints within a specific range to conform to the shape of standard instruction gestures. These angle thresholds are all customizable parameters and can be fine-tuned according to different events or referee habits.

[0112] Temporal Consistency Rules: To filter out fleeting or unintentionally formed poses that satisfy the above rules, this invention introduces a state machine and a temporal filtering mechanism. When a pose in a frame simultaneously satisfies all the above geometric and kinematic rules, the system state switches from "no gesture" to "candidate gesture," and a frame counter is started. In the following consecutive frames, if the pose is maintained, the counter continues to accumulate. Only when the accumulated counter value exceeds a preset frame threshold (15 frames, equivalent to 0.5 seconds in a 30fps video) will the system finally confirm the state as a "valid command gesture," and at this moment, send out a key event trigger signal. If the pose is interrupted during this period, the counter and state machine will be immediately reset. This mechanism greatly enhances the system's ability to resist interference from noisy actions.

[0113] Step Six: Asynchronous Key Fragment Generation and Structured Data Encapsulation

[0114] Once the composite rule engine determines that a valid scoring instruction gesture has occurred and generates a key event trigger signal, the main analysis process concludes. To ensure that the real-time performance of the main process (usually running on the main thread) is not affected by time-consuming operations such as disk I / O, this invention employs an asynchronous processing architecture.

[0115] A separate "key segment generation and storage unit" runs in one or more background worker threads. The core component of this unit is a first-in-first-out (FIFO) circular buffer queue (collections.deque), which continuously stores video frames processed by the main process into the queue during system operation and automatically discards the oldest frames, thus always keeping the most recent fixed-length (e.g., 10 seconds) video content in memory.

[0116] When a worker thread receives a critical event trigger signal from the main thread's event queue, it will perform the following series of automated operations:

[0117] Video clip extraction: It first retrieves all currently stored video frames in the buffer queue (corresponding to 300 frames in the 10 seconds before the event). Then, it continues to capture a short period of video (2 seconds, or 60 frames) after the event is triggered from the live video stream. These two frame sequences are seamlessly stitched together to form a 13-second video clip that fully records the cause and effect of the event.

[0118] Video Encoding and Saving: The worker thread calls a video encoding library (such as OpenCV's VideoWriter) to compress and save the stitched frame sequence into a single MP4 video file using the efficient H.264 encoding format. The filename can be automatically generated based on the event ID and timestamp to ensure uniqueness.

[0119] Structured metadata generation and storage: This is another major highlight of this invention. The worker thread not only saves the video but also generates corresponding structured metadata. It parses the rich information carried in the trigger signals and calls a function similar to `build_data_string` to format this information into a single CSV (comma-separated value) string. This string is strictly arranged according to a predefined format.

[0120] "live_room_id_123", "Round 2", "Red Team", "Player A", "Player B", "Coach's Score", "2 points", "2025-06-01 16:30:45"

[0121] This data line contains the live stream ID, match round, scoring team color, names of players from both the red and blue teams, a detailed description of the scoring event, and a timestamp accurate to milliseconds. Then, a worker thread appends this string to a specified CSV log file. This method ensures that all key events are recorded in a single spreadsheet file that is easily read and analyzed using Excel, Python Pandas, or database tools.

[0122] Through the close collaboration of the above six steps, this invention successfully and intelligently transforms a lengthy, unstructured sports match video into a new data asset consisting of a series of high-value, concise key video clips with rich queryable tags and their corresponding structured database. This brings revolutionary efficiency improvements and data dimension expansion to the field of sports tactical analysis.

[0123] To better understand this invention, specific embodiments have been described in detail above, but these are not intended to limit the invention. Any simple modifications made to the above embodiments based on the technical essence of this invention still fall within the scope of this invention. Each embodiment in this specification focuses on its differences from other embodiments; similar or identical parts between embodiments can be referred to mutually. For system embodiments, since they basically correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

Claims

1. A method for extracting key segments from sports videos based on behavior analysis, comprising acquiring video stream data of a sports event scene, and further comprising the following steps: Step 1: Perform preprocessing on the video stream data, including decoding and frame normalization; Step 2: The first target detection model is used to process each frame of the video stream to detect and locate the coach target in the image and obtain its initial bounding box information; Step 3: Expand the initial bounding box by a preset ratio to generate an expanded bounding box that fully encompasses the coach and his / her range of motion. Step 4: Within the extended bounding box region, apply a pose estimation model to extract the set of key points of the candidate body skeleton; If a single target is detected, the set is directly used as the full-body skeletal key points of the coach's target; If there are at least two candidate sets, the set with the most valid key points is selected as the full-body skeletal key points of the coach target by comparing the number of valid key points in each set, and cross-frame identity tracking is performed. Step 5: Analyze the key points of the coach's skeleton in real time. When it is determined that the vertical coordinate of the wrist joint is higher than that of the shoulder joint, it is determined that a scoring trend has occurred and the fine hand analysis process is triggered. Step 6: After the hand detail analysis process is triggered, the hand key point model is called to process the hand area of ​​the coach target and extract the detailed skeletal joints of the hand. Step 7: Based on preset geometric rules, kinematic angle rules and temporal logic, comprehensively analyze the position, posture, angle between the arm and shoulder and duration of the state of the fine skeletal joints of the hand to determine whether the coach's target has made a specific gesture that conforms to the scoring instructions. Step 8: If it is determined that the coach's target makes the specific gesture, a trigger signal for a key event is generated, and a video segment containing a preset time length before and after the trigger moment is extracted from the video stream data through an independent background processing thread, based on the trigger signal, as the key segment; Step 9: Associate and store the key fragment with metadata containing the trigger time, event type, coach identity, and current event status.

2. The method for extracting key segments from motion videos based on behavior analysis as described in claim 1, characterized in that, Step 1 includes sending the video stream data into a video preprocessing unit, decoding the video stream into a sequence of images frame by frame, and performing size normalization and color space correction on each frame.

3. The method for extracting key segments from motion videos based on behavior analysis as described in claim 2, characterized in that, The first target detection model is a YOLOv5 model, which is trained in a supervised manner using a dataset containing at least 20,000 labeled images of coaches under different scenes, poses, and lighting conditions to achieve generalization ability and high robustness in detecting coach targets.

4. The method for extracting key segments from motion videos based on behavior analysis as described in claim 3, characterized in that, The bounding box expansion process includes enlarging the width and height of the initial bounding box by a preset coefficient greater than 1, while keeping the center of the bounding box unchanged and ensuring that the expanded bounding box does not exceed the original boundary of the image.

5. The method for extracting key segments from motion videos based on behavior analysis as described in claim 4, characterized in that, The pose estimation model is the YOLOv8-Pose model, used to obtain 17 human skeleton key points conforming to the COCO format.

6. The method for extracting key segments from motion videos based on behavior analysis as described in claim 5, characterized in that, The hand key point model is the MediaPipe Hands model, which is used to achieve high-precision positioning of twenty-one three-dimensional hand joints.

7. The method for extracting key segments from motion videos based on behavior analysis as described in claim 6, characterized in that, The kinematic angle rules include: calculating the arm span angle formed by the shoulder joint, elbow joint and wrist joint, and the angle between the vector formed by the shoulder joint and elbow joint and the vertical line of the body trunk, and determining whether these angles fall within the preset effective command posture angle range.

8. The method for extracting key segments from motion videos based on behavior analysis as described in claim 7, characterized in that, The timing logic judgment method is as follows: a state machine is established, and when a posture that conforms to the geometric rules and kinematic angle rules is detected to appear continuously for more than a preset frame number threshold, it is determined to be a valid scoring instruction gesture.

9. The method for extracting key segments from motion videos based on behavior analysis as described in claim 8, characterized in that, Step 7 includes the following sub-steps: Step 71: Analyze the internal geometry of the hand, calculate the angle between the joints of the fingers to determine whether the index finger is straight, and calculate the distance from the fingertips of the other fingers to the palm to determine whether the hand is clenched. Step 72: Calculate the arm span angle formed by the three points of shoulder, elbow and wrist, and the angle with the main body, and determine whether the above angle is within the preset angle range; Step 73: When a pose satisfies all the geometric and kinematic rules and is maintained stably for more than a preset frame threshold, it is recognized as a valid scoring instruction gesture.

10. The method for extracting key segments from motion videos based on behavior analysis as described in claim 9, characterized in that, The independent background processing thread maintains a fixed-duration video frame loop buffer. When the key event trigger signal is received, all video frames in the buffer are merged with video frames collected after the trigger time for a preset duration, and encoded into the final key segment video file.

Images