Basketball event automatic labeling method based on visual multi-modal fusion and rule constraint, electronic device and storage medium

Abstract

This application discloses an automatic annotation method, electronic device, and storage medium for basketball game events based on visual multimodal fusion and rule constraints. The method includes: extracting visual structured features and a scoreboard text time sequence from a basketball game video stream; detecting event anchor points based on field transitions in the text sequence, defining segments to be annotated, and generating a candidate event type list; inputting the visual features within the segments into a multimodal large-scale model, performing semantic understanding under the constraints of the candidate list to obtain a preliminary annotation sequence; using a game state machine combined with a basketball rule knowledge base to perform logical verification on the sequence, correcting or marking conflicting events to output the final annotation sequence; and the state machine outputting constraint instructions to dynamically adjust the visual extraction parameters of subsequent frames. This invention overcomes the high false alarm rate of single visual detection and achieves highly robust fully automatic annotation.

Description

Technical Field

[0001] This application relates to the fields of computer vision and sports data analysis technology, and in particular to an automatic annotation method, electronic device and storage medium for basketball game events based on visual multimodal fusion and rule constraints. Background Technology

[0002] Video data analysis of basketball games plays a crucial role in sports broadcasting, tactical review, and sports big data statistics. With the development of computer vision technology, technical solutions for automatically generating game status and statistical data based on game videos are gradually becoming a focus of industry attention.

[0003] However, traditional sports data statistics mainly rely on manual review of video footage round by round, which is not only time-consuming and labor-intensive, but also fails to meet the real-time data output requirements of live sports broadcasts. To improve data statistics efficiency, the industry has gradually introduced artificial intelligence solutions based on pure vision-based motion recognition. These solutions typically use object detection technology to track players and the basketball, attempting to automatically extract game events by recognizing specific actions such as shooting and passing. Although vision-based automated detection can alleviate the burden on humans to some extent, its application in high-intensity basketball game videos still faces significant technical limitations.

[0004] Existing visual motion recognition solutions typically determine motion based solely on local pixel changes within the frame. When players are densely obscured in the game footage or the broadcast camera frequently switches, the system is highly susceptible to losing track of its target, leading to numerous misjudgments of crucial events such as ball possession changes or invalid shots. Furthermore, existing video analytics architectures often process image feature detection and game rule determination as completely independent processes. The system relies on the visual results of a single segment to output events, resulting in significant discrepancies between the output statistics and the actual objective results of the game.

[0005] In summary, overcoming the problem of visual detection misjudgment in complex game scenes and ensuring that automatically generated event data strictly conforms to the actual logic of the game are the technical challenges that urgently need to be solved in this field. Summary of the Invention

[0006] This application proposes an automatic annotation method for basketball game events based on visual multimodal fusion and rule constraints, which can overcome the problem of visual detection misjudgment under complex game scenes and ensure that the automatically generated event data strictly conforms to the real logic of the game.

[0007] According to one embodiment of this application, an automatic annotation method for basketball game events based on visual multimodal fusion and rule constraints is proposed, including: Visual structured feature sequences and scoreboard text time sequences are extracted from the video stream of a basketball game. The visual structured features include on-court player features and basketball trajectory features. Based on the specified field jump detection event anchor point in the scoreboard text time sequence, the corresponding video segment range to be labeled is divided according to the type of the event anchor point, and a candidate event type list corresponding to the type of the event anchor point is generated; The visual structured feature sequence within the range of the video segment to be labeled is input into a multimodal large model, and semantic understanding is performed under the constraints of the candidate event type list to obtain a preliminary event labeling sequence. By utilizing the game state machine that maintains the game's logical state, and combining it with a pre-set basketball rule knowledge base, a logical consistency check is performed on the preliminary event labeling sequence. Based on the visual structured feature sequence, conflicting events detected are corrected or marked to obtain the final event labeling sequence. The competition state machine generates constraint instructions for the extraction process of the visual structured feature sequence based on the current competition logic state, and feeds the constraint instructions back to the extraction process to dynamically adjust the extraction parameters used for subsequent video frames.

[0008] According to one embodiment of this application, an electronic device is provided, the device including a memory and a processor, the memory being used to store computer instructions executable on the processor, and the processor being used to implement the method as described in any of the preceding claims when executing the computer instructions.

[0009] According to one embodiment of this application, a computer-readable storage medium is provided that stores a computer program thereon, which, when executed by a processor, implements the method as described in any of the preceding claims.

[0010] This application proposes an automatic event annotation method for basketball games based on visual multimodal fusion and rule constraints. By detecting event anchor points based on specified field transitions in the scoreboard text time sequence and dividing the segments to be annotated, it achieves precise localization of key event time periods, avoiding invalid searches of the entire video. The method inputs visual structured feature sequences into a large multimodal model and performs semantic understanding under the constraints of a candidate event type list, overcoming the ambiguity inherent in conventional single-action detection in complex scenes and improving the accuracy of initial event recognition. Furthermore, a game state machine containing a basketball rule knowledge base is used to perform logical consistency checks and correct conflicts in the initial event annotation sequence, ensuring that the final output event sequence strictly conforms to the actual game logic and reducing false alarms that violate common sense. In particular, the game state machine in this scheme can generate constraint instructions based on the current game logic state and feed these instructions back to the visual extraction process to dynamically adjust parameters. Through the interaction between vision and logic, the perceptual stability of the system in dynamic and complex scenes is improved.

[0011] Other features and advantages of the technical solution proposed in this application are described below. Attached Figure Description

[0012] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this specification and, together with the description, serve to explain the principles of this specification.

[0013] Figure 1 This is a flowchart of an automatic annotation method for basketball game events based on visual multimodal fusion and rule constraints, provided in an embodiment of this application; Figure 2 This is a schematic diagram of the architecture of an automatic annotation system for basketball event events provided in an exemplary embodiment of this application; Figure 3 This is a schematic diagram illustrating the process of maintaining and allocating player identifiers according to an exemplary embodiment of this application; Figure 4 This is a schematic diagram of event anchor point detection and the division of the video segment to be labeled provided in an exemplary embodiment of this application; Figure 5 This is a schematic diagram of the competition state machine rule verification provided in an exemplary embodiment of this application; Figure 6 This is a schematic diagram of the hardware structure of an electronic device provided in at least one embodiment of this application. Detailed Implementation

[0014] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0015] This application addresses the shortcomings of relying solely on visual recognition, which can easily misjudge invalid shots and ball possession changes as valid events, and the lack of rule verification leading to discrepancies between the annotations and the actual game. It uses deterministic jumps in the scoreboard text as event anchors to constrain the semantic understanding range of a multimodal large model. Then, the game state machine, combined with a basketball rule knowledge base, performs logical consistency verification on the annotation results and feeds back the game state to the visual feature extraction process. As a result, it is possible to obtain a sequence of event annotations consistent with the game rules based solely on the video stream.

[0016] Please refer to Figure 1 , Figure 1 This document presents a flowchart of an automatic event annotation method for basketball games based on visual multimodal fusion and rule constraints, as provided in an embodiment of this application. This method can be applied to the automatic annotation of game events in recorded or live video streams of basketball games. The video stream is read and processed frame by frame by an electronic device, eliminating the need for additional sensors or manual review of each play-by-play (PBP) session. The method includes the following steps S1 to S4.

[0017] Step S1: Extract visual structured feature sequences and scoreboard text time sequences from the video stream of the basketball game. The visual structured features include on-court player features and basketball trajectory features.

[0018] The aforementioned features can be extracted in parallel from the original video stream by the visual perception layer, and data alignment of each feature type is performed using a unified timestamp. The visual structured feature sequence consists of visual structured features corresponding to each video frame arranged in temporal order, where each visual structured feature may include on-court player features and basketball trajectory features for that frame. On-court player features can be obtained through player and basketball detection, maintenance and allocation of player cross-frame identifiers, and extraction of player skeleton key points, and may include player tracking features and player skeleton key points, as well as the recognition results of player action categories (e.g., fast break). The extraction of basketball trajectory features and the temporal sequence of scoreboard text can be extracted independently.

[0019] For player detection, video frames can be extracted from the video stream at a fixed frame rate, such as 2 to 5 frames per second. The extracted images are then normalized in size and color space. Size normalization maintains the image aspect ratio and fills the edges with black borders to the standard input size of the detection network. Color space normalization reduces the impact of white balance differences between different cameras. A deep learning object detection network can then be used to detect players and basketballs in each frame. To accommodate objects that appear larger in the foreground and smaller in the background, the detection network can employ a Feature Pyramid Network (FPN) structure to detect simultaneously at multiple scales. Since basketballs occupy a relatively small portion of the frame, a small object detection branch can be used to improve the sensitivity for detecting objects with a diameter less than 30 pixels. The detection network outputs bounding box coordinates and confidence scores for players. The confidence score characterizes the likelihood that the object within the detection box is a player or a basketball. The detection network can also assign temporary labels for cross-frame association to each detected object. The temporary identifier is an identifier assigned frame by frame during the detection phase. It can be inherited or newly created in subsequent cross-frame matching and can serve as a player identifier that uniquely identifies the same player in time sequence.

[0020] After obtaining the detection results for each player, players can be divided into teams based on their jersey color scheme. For example, the dominant color within the bounding box can be extracted, and K-means clustering can be used to divide players into home and away teams. The number of clusters can be K=2. Then, the offensive and defensive teams can be confirmed by combining the opening jump ball position or the direction of attack. Post-processing can then be performed on the detection results. For example, non-maximum suppression (NMS) can be used to eliminate duplicate detection boxes for the same target, and low-quality detection results can be filtered according to a confidence threshold. In some examples, the confidence threshold can be between 0.6 and 0.7. Detection boxes in adjacent frames can also be smoothed to reduce jitter. After the above detection and post-processing, each frame can output a record, the content of which includes a frame timestamp, target type (player or basketball), bounding box coordinates (x, y, w, h), confidence score, team identifier, and temporary identifier.

[0021] Based on the detection, detection boxes of the same player in adjacent frames can be associated to maintain the temporal consistency of their identifiers. In some implementations, the maintenance and allocation of cross-frame identifiers can be achieved as follows: extract the positional features, appearance features, and topological features of each player in the current frame, where the topological features of each player represent the relative distance relationship between that player and other players in the current frame; for players in the previous frame and players in the current frame, calculate the prediction deviation distance of the positional features, the appearance distance between the appearance features, and the topological distance between the topological features, and calculate the weighted sum of the prediction deviation distance, the appearance distance, and the topological distance to construct a cross-frame cost matrix; solve the cross-frame cost matrix based on the minimum cost assignment algorithm, and when the cost of a matching pair does not exceed a preset matching threshold, inherit the player identifier of the previous frame to the corresponding player in the current frame; otherwise, determine that the matching has failed; when a shot change is detected in the video stream, extract the topological features of each player in the new frame after the switch, increase the weight of the topological distance in constructing the cross-frame cost matrix, and restore the player identifiers before the shot change.

[0022] The aforementioned positional features can be the center pixel coordinates of the player detection box, while the appearance features can be feature vectors extracted by the embedding model that characterize the player's visual appearance, such as a 512-dimensional feature vector. In this application, topological features refer to the relative distance vectors between each player and the other players on the field, used to characterize the tactical positioning relationships between players. Unlike appearance information such as jersey color and face, relative distance relationships exhibit good stability before and after camera zooming and panning.

[0023] When constructing the cross-frame cost matrix, the prediction bias distance can be the distance between the predicted and measured values ​​of the player's position. Position prediction can be obtained by extrapolating the instantaneous velocity vector of the previous frame. For example, the player's center coordinates in the previous frame can be extrapolated to the prior position in the current frame along the velocity vector direction, and then the Mahalanobis distance between the prior position and the measured position of the player in the current frame can be calculated as the prediction bias distance. The appearance distance can be the cosine distance between the appearance feature vectors of the two frames. The topological distance can be the Euclidean distance between the topological feature vectors of the two frames. For the i-th known player in the previous frame and the j-th player in the current frame, the corresponding elements in the cost matrix can be represented as a weighted sum of the three, i.e., C[i,j]=w1×Dmotion+w2×Dappear+w3×Dtopo, where Dmotion, Dappear, and Dtopo are the prediction bias distance, appearance distance, and topological distance, respectively, and w1, w2, and w3 are their respective weights. The three can take values ​​in [0,1] and satisfy w1+w2+w3=1.

[0024] The minimum cost assignment algorithm can employ either the Hungarian algorithm or a graph matching algorithm. The minimum cost assignment algorithm solves for the cost matrix to find the matching allocation that minimizes the total cost, outputting matching pairs between player identifiers from the previous frame and detection boxes in the current frame. When the cost of a matching pair does not exceed a preset matching threshold, the player in the current frame inherits the player identifier from the previous frame; matching pairs whose cost exceeds the threshold are considered failed matches. In some examples, the matching threshold can be 0.6. For failed matches, unmatched players in the current frame can be assigned new player identifiers, which can be generated in a globally incrementing manner; unmatched player identifiers from the previous frame can be temporarily stored in a sleep pool for a preset number of frames, such as 30 frames, for later recovery, or marked for review.

[0025] After the identifier inheritance is completed, the basketball rules can be used to verify the allocation results. For example, based on the rule that no more than 5 players can be on the court at the same time for a single team, the number of player identifiers corresponding to each team in the current frame can be checked; when the number of player identifiers for a certain team is abnormal, it can be determined that there is an error in the identifier allocation, and the relevant detection boxes can be re-matched to correct it.

[0026] Because the pixel positions of the same player are no longer continuous before and after the camera switch, motion prediction based on pixel position loses its reliability. However, the relative distance between players remains stable before and after camera zooming and panning. Therefore, this implementation adopts a process of increasing the weight of topological distance when switching cameras, so as to make it the main basis for restoring player identification.

[0027] By incorporating position, appearance, and relative distance topology information into the cost matrix, compared to cross-frame association based on a single feature, the labeling can remain stable when players have similar appearances or experience short-term occlusion. When camera switching causes appearance information such as jersey color and face to become invalid, the player labeling can be recovered by using relative positioning relationships, which helps maintain the temporal continuity of the labeled data.

[0028] The weights of each distance in the weighted sum can be dynamically adjusted according to the scene in which the current frame is located. In some implementations, this dynamic adjustment includes: extracting the global screen feature changes between adjacent frames, the number of player detections, the confidence of player detection boxes, the intersection over union (IoU) ratio between player detection boxes, the inter-frame displacement velocity of players, and / or the decreasing state of the remaining time field in the scoreboard text time sequence as identification criteria to identify the scene type of the current frame. The scene type includes normal continuous shot scenes, player occlusion scenes, high-speed offensive and defensive transition scenes, shot switching transition scenes, and stationary standing scenes; and dynamically adjusting the weights of the prediction deviation distance, the appearance distance, and the topological distance according to the scene type.

[0029] The scene type of the current frame can be determined based on some or all of the above identification criteria, and different weight values ​​can be configured for different scenes. When a certain feature is unavailable in the current scene type, for example, if the confidence of the feature is lower than the preset confidence threshold, or if the image information on which the feature is based has become invalid in the current scene type, the corresponding weight can be temporarily adjusted to zero.

[0030] According to this implementation method, by dynamically adjusting the weight of each distance according to the scene type, various features can play a greater role in their more reliable scenes, reducing the label jump caused by the failure of a certain type of feature in a specific scene.

[0031] The weights for each scenario can be preset empirical values, or they can be adaptively updated based on the statistical characteristics of the match footage. In some implementations, the method further includes: calculating the player identifier jump rate for each scenario type; when the player identifier jump rate for any scenario type exceeds a preset threshold, adaptively adjusting the weight corresponding to that scenario type.

[0032] In this embodiment, the weight of the corresponding scene is adaptively adjusted according to the player's icon jump rate, so that the weight setting changes with the statistical characteristics of the game screen, further reducing the icon jump in the same scene.

[0033] During the annotation process, after the successful and unsuccessful matching results for each frame are determined, the player identifier jump rate for that frame can be statistically analyzed. In addition to the player identifier jump rate, the matching success rate and rule verification anomaly rate for each scene type can also be recorded. The adaptive adjustment of weights can be performed offline using historical annotation data or manual review results, or it can be updated online during the annotation process.

[0034] On-field player features can also include key points of the player skeleton, which can be used for motion understanding and visual annotation. Individual player images can be cropped from the original image based on the player detection box and its identifier, appropriately expanded (e.g., expanded by 10% to 15%) to include the complete limb, and then scaled to the input size of the skeleton extraction network (e.g., 256×192). A top-down strategy is used to independently locate key points for each player, output the two-dimensional pixel coordinates and visibility confidence of each key point, and bind the corresponding player identifier.

[0035] In some implementations, when extracting the key points of the player's skeleton, key points related to basketball movements can be given a higher detection priority than other key points to improve the localization accuracy of key points related to basketball movements. Key points related to basketball movements mainly include arm key points and leg key points, as shown in the following examples: High priority (key arm points): shoulders, elbows, and wrists – used to identify shooting, passing, and stealing actions. High priority (key leg points): hips, knees, and ankles – used to identify running, jumping, and defensive postures. Standard priority: Head key points (nose, eyes, ears) - used to assist in posture judgment.

[0036] After extracting the skeleton key points, their coordinates can be post-processed and corrected. In some embodiments, the method further includes at least one of the following correction operations for the skeleton key point coordinates: calculating the current length ratio between adjacent limbs; when the current length ratio exceeds a preset reasonable ratio range, scaling the coordinates of the corresponding key points according to the historical frame average ratio of the corresponding limb; calculating the joint vector angle; when the joint vector angle exceeds a preset reasonable angle range, truncating the joint vector angle to the boundary value and remapping the joint coordinates, or reducing the confidence of the corresponding key point; calculating the coordinate difference between left and right symmetrical key points; when the coordinate difference exceeds a preset symmetry threshold, averaging the coordinates of the corresponding key points, or marking the corresponding key points as having abnormal postures.

[0037] Since the length proportions of human limbs are relatively fixed in the short term and should not change significantly between adjacent frames, recognizing basketball movements based on the above-corrected data can effectively improve recognition quality. During correction, the corresponding value of the current frame is first calculated and compared with a preset range; values ​​exceeding the range are corrected accordingly. In one example, the preset ranges and processing methods for each correction operation are shown in the table below: The above three correction operations can be used individually or in combination.

[0038] In some embodiments, the method further includes adaptive temporal filtering of the skeleton keypoint coordinates: calculating the inter-frame displacement velocity of the keypoints and determining their motion state based on the confidence level of the keypoints; when the motion state is a stationary state, using a moving average filter with a first window length; when the motion state is a preset slow state, using a moving average filter with a second window length less than the first window length; when the motion state is a preset high-speed state, using a Kalman filter; when the confidence level is lower than a preset keypoint confidence threshold, using linear interpolation to complete the process and marking the corresponding keypoints as interpolated states. Those skilled in the art can determine the velocity ranges corresponding to the slow and high-speed states based on experience.

[0039] For example, when the displacement speed is less than 5 pixels / frame and the confidence level is higher than 0.8, it is determined to be a stationary state, and a moving average filter with a window of 5 frames can be used; when the displacement speed is between 5 and 15 pixels / frame, it is determined to be a slow state, and a moving average filter with a window of 3 frames can be used; when the displacement speed is not less than 15 pixels / frame, it is determined to be a high-speed state, and a Kalman filter can be used to preserve the sharpness of the motion; when the confidence level is lower than the preset key point confidence threshold (e.g., 0.3), linear interpolation can be used to complete the data and mark it as an interpolation state.

[0040] By applying anatomical constraint correction and adaptive temporal filtering to the coordinates of key points in the skeleton, the jitter and offset of key points under occlusion or rapid movement can be reduced, thereby improving the skeleton annotation accuracy of actions such as shooting and passing.

[0041] The processed keypoint coordinates can be rendered back to the original video frames. For example, different colors can be used to distinguish teams (blue for the home team and red for the away team), and line segments can be used to connect the relevant keypoints to form a skeleton structure. The corresponding player labels can be marked next to the skeleton. The rendered skeleton-enhanced video can be used as input enhancement for subsequent multimodal large models, or it can be used for manual review interfaces.

[0042] Basketball trajectory features can be obtained through continuous detection and trajectory smoothing of the target basketball. The pixel coordinates and confidence level of the basketball's center can be detected in each frame, and basic filtering (e.g., median filtering) can be applied to the detection results of consecutive frames to eliminate false detections in single frames. Basketball trajectory features can include smoothed trajectories. The identified historical position sequence of the basketball can be stored in a sliding window of a preset length, and the smoothed trajectory can be generated using interpolation or moving average algorithms. When basketball detection is lost, extrapolation prediction is performed based on the last known speed and direction of movement, and the smoothed trajectory is maintained within a preset tolerance frame number. The preset length of the sliding window can be, for example, 30 frames. When generating the smoothed trajectory, spline interpolation can be used, for example, to eliminate detection jitter and obtain a continuous trajectory curve. When the basketball is obstructed by a player, causing detection loss, linear extrapolation prediction can be performed based on the last known speed and direction of movement, and the smoothed trajectory is maintained within a preset tolerance frame number (e.g., 15 frames), until the detection is restored and the trajectory is corrected. By using sliding window smoothing and loss extrapolation, a continuous basketball trajectory can be maintained when jitter or short-term loss is detected, providing a basis for the spatiotemporal alignment of events.

[0043] While generating smooth motion trajectories, the system can also determine the association state between the basketball and the players. The Euclidean distance between the basketball's center and each player's bounding box can be calculated in real time to identify the nearest player and determine the association state based on that distance. In one example, the determination criteria, rendering colors, and semantic meanings of each association state are shown in the table below: The aforementioned associated states can be rendered along with the smooth motion trajectory to the video frame. For example, the current position of the basketball can be marked with a bright dot, and the movement path over a recent period (e.g., 2 seconds) can be displayed as a semi-transparent trajectory line, with the color of the trajectory line gradually changing over time. The basketball and the nearest player can also be connected by line segments of corresponding colors according to their associated states; for example, a solid red line for possession, a dashed yellow line for controllable possession, and a dotted blue line for free possession. The associated player's identifier and distance value can be labeled next to the basketball. Changes in the basketball-player associated states can serve as one of the criteria for determining keyframes.

[0044] The time sequence of scoreboard text can be obtained through character recognition in the scoreboard area. A combination of static localization and dynamic detection can be used to determine the scoreboard area. For example, the user can annotate the scoreboard area in several frames and create a Region of Interest (ROI) mask. Subsequent processing is then performed only on this region to reduce computational load. This region can be grayscaled, Gaussian denoising performed, and contrast enhanced. Then, the Otsu adaptive thresholding algorithm is used for binarization. Connectivity analysis is used to extract the character bounding rectangles, and noise regions are filtered based on geometric features such as aspect ratio, area, and position. Finally, an OCR model can be used to recognize characters in each segmented region. In one example, considering the characteristics of a basketball game scoreboard, seven recognition fields can be defined, as shown in the table below: Step S2: Based on the specified field jump detection event anchor point in the scoreboard text time sequence, divide the corresponding video segment range to be labeled according to the type of the event anchor point, and generate a candidate event type list corresponding to the type of the event anchor point.

[0045] Basketball games involve numerous deterministic events corresponding to changes in scoreboard fields, such as score changes, offensive timing resetting, and quarter switching. The designated fields refer to pre-agreed fields in the scoreboard text time sequence used to trigger event anchors, and may include the score field (including home and away team scores), offensive timing field, and quarter field as shown in the field examples above. Scoreboard field changes are highly deterministic hard signals and can serve as event anchors for coarse-grained localization of video segments requiring annotation. When detecting field changes, a time-series sliding window of a certain duration (e.g., 5 seconds) can be maintained to monitor changes in the values ​​of each scoreboard field. To avoid false detections in a single frame, a method of confirming only when multiple consecutive frames show consistent results can be used; for example, a change is confirmed only after three consecutive frames show consistent results. For each detected event anchor, the anchor type, trigger timestamp, segment start frame, segment end frame, and a list of candidate event types can be output.

[0046] In some implementations, detecting event anchors based on specified field jumps in the scoreboard text time sequence, dividing the corresponding range of video segments to be labeled, and generating a candidate event type list may include: determining events where the value of the score field in the scoreboard text time sequence increases as score change anchors, with candidate event types corresponding to the score change anchors including field goals made and free throws made; determining events where the offensive timer field in the scoreboard text time sequence is reset from a decreasing state to a full offensive timer duration while the score field remains unchanged as offensive timer full reset anchors, with candidate event types corresponding to the offensive timer full reset anchors including fouls, out of bounds, violations, defensive rebounds, and changes in possession, and classifying field goals made and offensive timers as candidate event types. Rebounds are excluded from the candidate event type list; events where the offensive timing field is reset from a decreasing state to the duration of a second offensive possession are identified as offensive timing half-reset anchors, and the candidate event types corresponding to the offensive timing half-reset anchors include missed shots and offensive rebounds, while defensive rebounds are excluded from the candidate event type list; and / or events where the quarter field changes in the scoreboard text time sequence are identified as quarter switching anchors, and the candidate event types corresponding to the quarter switching anchors include quarter breaks and opening jump balls; and based on the trigger time of each of the event anchors, the range of the video segments to be labeled is divided according to the preset forward segment time window and backward segment time window corresponding to the type of the event anchor.

[0047] The complete offensive possession timer refers to the initial offensive possession timer value corresponding to a new complete offensive possession, such as 24 seconds; the secondary offensive possession timer refers to the initial offensive possession timer value corresponding to when the offensive team regains possession after an offensive rebound or other situation, such as 14 seconds. These values ​​can be adjusted depending on the game format. In game formats using other offensive possession timer rules, the complete offensive possession timer and the secondary offensive possession timer can be other values ​​specified by the corresponding game format. The determination of the event anchor point still depends on whether the offensive possession timer field has been reset from a decreasing state to the corresponding duration corresponding to that game format.

[0048] According to this implementation method, four types of anchor points can be determined based on the jump patterns of different fields on the scoreboard, and the candidate event types also differ accordingly. An increase in the score field value indicates that a score may have occurred in this round, so the score change anchor point corresponds to a successful shot or free throw. A change in the offensive timer field from decreasing to the full offensive timer duration while the score field remains unchanged indicates that an offensive play has ended without the offensive team scoring, and possession may have changed. Therefore, a full offensive timer reset anchor point corresponds to fouls, out-of-bounds errors, violations, defensive rebounds, and possession changes, excluding successful shots and offensive rebounds that signify a score or possession retained by the offensive team from the candidates. A change in the offensive timer field from decreasing to the second offensive timer duration indicates that the offensive team regained control of the ball after a missed shot, and possession did not change. Therefore, a half offensive timer reset anchor point corresponds to missed shots and offensive rebounds, excluding defensive rebounds that signify possession changes from the candidates. A change in the quarter field indicates that the game has entered a quarter or a new quarter, so the quarter switching anchor point corresponds to the quarter break and the opening tip-off.

[0049] When defining the range of video segments to be labeled, the trigger time of each event anchor point can be used as a benchmark, and segments can be extracted forward and backward according to the time window corresponding to the anchor point type. The candidate event type list can also be further refined based on the specific numerical changes in the scoreboard fields, for example, determining the specific type of the score based on the increment of the score field.

[0050] By using the deterministic jumps of the scoreboard field as event anchors, and generating a list of candidate event types according to the anchor type, and excluding events that do not conform to the logic of the anchor, the candidate range can be narrowed before semantic understanding is performed on the multimodal large model. This reduces the situation where pure visual recognition misjudges invalid actions as valid events in occlusion and camera switching scenarios.

[0051] For score change anchor points, the increase in the score field value on the scoreboard may be caused by replay footage, scoreboard recognition delay, or false detection. This can be further confirmed by combining the soft signal of the basketball trajectory. In some implementations, the method further includes establishing a spatiotemporal alignment window between the basketball trajectory features and the scoreboard text time sequence. After determining the event of an increase in the score field value as the score change anchor point, when the basketball trajectory features represent the basketball passing through the basket area within the spatiotemporal alignment window, and the increase in the score field value triggering the score change anchor point occurs within a preset score verification time window after this representation, the score change anchor point is confirmed as a valid scoring event, and the type of the valid scoring event is determined from the candidate event types corresponding to the score change anchor point.

[0052] The aforementioned spatiotemporal alignment window is used to align the basketball trajectory characteristics (a soft signal) with the scoreboard text sequence (a hard signal) in time, thus mutually verifying each other. The spatiotemporal alignment window can be extended forward and backward based on the moment of the scoreboard field transition, and its duration should be greater than the inherent delay of the scoreboard display and less than the minimum interval between two adjacent events; for example, it can be 3 seconds before to 1 second after the moment. The preset score verification time window can be, for example, 0.3 to 2 seconds. After confirming a valid score, the specific type of the valid score event can be determined from the corresponding candidate event types by combining the release position and basketball trajectory; for example, distinguishing between a two-pointer and a three-pointer based on whether the release position is outside the three-point line. Confirming a valid score by pairing the soft signal of the basketball crossing the rim with the hard signal of the scoreboard field transition in time can eliminate misjudgments caused by abnormal scoreboard transitions or scores not from the current possession.

[0053] Step S3: Input the visual structured feature sequence within the range of the video segment to be labeled into the multimodal large model, perform semantic understanding under the constraints of the candidate event type list, and obtain the preliminary event labeling sequence output by the multimodal large model.

[0054] After defining the range of video segments to be labeled, the visual structured feature sequence within the segment can serve as the basis for semantic understanding. The multimodal large model can be a general large model with visual language understanding capabilities, or a dedicated multimodal model trained based on basketball game data.

[0055] In some embodiments, the method further includes constructing event evidence data before inputting the visual structured feature sequence within the range of the video segment to be labeled into the multimodal large model: extracting basic frames at a preset frame rate, increasing the frame extraction density near the triggering time of the event anchor point, and retaining keyframes corresponding to at least one of the following: basketball leaving the hand, basketball approaching the basket, changes in the score field or offensive timing field in the scoreboard text time sequence, and changes in the relationship between the player and the basketball, to obtain a keyframe image sequence; encapsulating the keyframe image sequence and the visual structured feature sequence into the event evidence data, which is then used as input to the multimodal large model.

[0056] The frame extraction method used in this embodiment combines fixed frame extraction and event-triggered frame extraction. The preset frame rate can be, for example, 2 to 5 frames per second; near the trigger time of the event anchor point, since the correlation between image changes and event determination is high during this period, the frame extraction density can be increased. Simultaneously, the aforementioned keyframe types can be retained, where the change in the relationship between players and the basketball can be determined based on the change in distance between the basketball center and the boundary frames of each player. The number of keyframes retained for each video segment to be labeled can be, for example, 8 to 32 frames.

[0057] Event evidence data can be encapsulated from keyframe image sequences and visual structured feature sequences. The visual structured feature sequences can include player identification, position, speed, and team affiliation; player skeletal key points and their confidence levels; the basketball's position, speed, and its association with the player; and the score, remaining time in the quarter, and offensive countdown from the scoreboard text time sequence. When the multimodal large model supports video input, enhanced video clips overlaid with player identification, skeletal key points, basketball trajectory, and scoreboard recognition results can also be directly input.

[0058] The aforementioned player tracking features, player skeletal key points, and basketball trajectory features can be input into the multimodal large model simultaneously, or only a portion of them can be input. When a certain type of feature is unavailable due to limitations in image conditions, such as when a player is largely occluded resulting in missing skeletal key points, the large model can perform semantic understanding based solely on the remaining features and the candidate event type list to determine the event type, but its confidence level can be correspondingly reduced. This type confidence level is used to characterize the degree of certainty the multimodal large model has regarding its event type determination.

[0059] When inputting event evidence data into a multimodal large model, a list of candidate event types corresponding to the segment should also be input to constrain the model's output range. This requires the model to select event types only from the candidate event types and not output excluded event types. The input can be organized into cue words, which may include task description, game status, visual evidence, candidate event range, and output requirements. The game status can provide the current score, current quarter, and changes in the offensive timer. Visual evidence can provide the basketball trajectory, player actions, and scoreboard changes. The candidate event range can provide the candidate event types and excluded event types corresponding to the anchor point. The output requirements can limit the model to select event types from the candidate event types and provide the identifiers of the players involved, the precise frame number within the segment, and the confidence level. An example of a cue word is as follows: [Task] Mark Basketball Game Events [Match Status] Current Score: Home Team 85 - Away Team 92; Current Quarter: Q3; Offensive Timer: Reset from 05 seconds to 24 seconds [Visual Evidence] Basketball trajectory: flew from (520,340) to (600,200), but did not cross the net; Player action: player marked 1023 raised his arm to catch the basketball; Scoreboard changes: offensive countdown 05→24, score unchanged. [Candidate Events Scope] Possible events: fouls, out-of-bounds errors, violations, defensive rebounds, changes of possession; Excluded events: field goals made, offensive rebounds

Output Requirements

[0060] By performing semantic understanding under the constraints of a list of candidate event types, the multimodal large model can avoid making judgments among all event types. Its output can be limited to the range that matches the jump pattern of the scoreboard, thereby reducing the possibility of confusion between actions with similar visual appearances, such as judging the action of grabbing the backboard as a shot.

[0061] Step S4: Using the game state machine that maintains the game's logical state, combined with a preset basketball rule knowledge base, perform a logical consistency check on the preliminary event labeling sequence. Based on the visual structured feature sequence, correct or mark the conflicting events detected, and obtain the final event labeling sequence.

[0062] The preliminary event labeling sequence output by the multimodal large model is the semantic understanding result of each segment of the image. Its judgment is mainly based on the image itself, which may not be consistent with the event sequence and causal relationship constrained by the basketball rules. This application uses a game state machine that maintains the logical state of the game, combined with a pre-set basketball rule knowledge base, to perform logical consistency verification on each item of the preliminary event labeling sequence.

[0063] The game state machine maintains the context of the game up to the current moment, including the current game logic state, ball possession direction, the type of the previous event, and the order of free throws. The basketball rule knowledge base stores rules unrelated to specific games. When checking individual event annotations, the rules in the knowledge base can be used. When verifying the sequence and causal relationships between events, such as whether ball possession changes after a score, whether the order of a shot attempt and a rebound is reversed, or whether the number of free throws after a shooting foul matches the ball possession, the context maintained by the game state machine can be used as a reference. In this case, the state machine provides the current game logic state. The verification results are then fed back to the game state machine to update the context. For example, if an event is verified and corrected from an offensive rebound to a defensive rebound, triggering a ball possession change, the ball possession direction maintained by the game state machine will be flipped. This flip will then serve as the basis for verifying subsequent events.

[0064] The game's logical states can include normal offensive and defensive states with a live ball, live ball shooting in flight, score verification states, visually static states with a dead ball, free throw states, and camera transition states. The transitions between these six game logical states can be determined by combining changes in the scoreboard text sequence, basketball trajectory characteristics, and on-court player characteristics, with changes in scoreboard fields being the primary criterion.

[0065] The basketball rule knowledge base can predefine several rules for verification, such as the six rules shown in the table below: The game state machine can apply the above rules to verify the event labels in the preliminary event labeling sequence one by one according to the current game logic state, and determine the event labels that do not conform to the rules as conflict events.

[0066] For the conflict events detected, corrections or markings can be made according to the type of conflict.

[0067] In some implementations, logical consistency checks are performed on the initial event labeling sequence, and conflicting events are corrected or marked, including at least one of the following: when an event is labeled as a successful shot, but the score field in the corresponding scoreboard text time sequence does not increase, it is determined to be a conflicting event, and the visual structured feature sequence is backtracked to check whether there is a prior violation in the offensive timing field of the scoreboard text time sequence; when an event is labeled as an offensive rebound, but the offensive timing field in the corresponding scoreboard text time sequence is reset to the full offensive timing duration instead of the secondary offensive timing duration, it is determined to be a conflicting event, and the conflicting event is corrected to a defensive rebound and a change of possession is triggered; when multiple consecutive events labeled as free throws correspond to non-shooting fouls, it is determined to be a conflicting event, and the conflicting event is marked for manual review; the type confidence of events that pass the logical consistency check is increased, the type confidence of conflicting events that can be automatically corrected is decreased, and conflicting events that cannot be automatically corrected are marked for manual review and the type confidence of the event is decreased.

[0068] The three types of conflicts listed in this implementation correspond to the scoring logic, offensive timing logic, and foul / free throw logic in the table above. Specifically, tracing back the visual structured feature sequence to check if a violation occurred before the offensive timing field, we can check if the offensive timing field had reached zero before the basketball left the hand: if it had, the shot occurred after the violation, the shot was not successful, and the event can be corrected to an offensive timing violation; if it had not reached zero, the conflict may stem from scoreboard recognition anomalies or the basketball not entering the basket, which can be further determined by considering whether the basketball's trajectory crossed the basket area. The foul type corresponding to multiple consecutive free throw events can be provided by the multimodal large model in its initial event annotation output based on visual evidence, for example, distinguishing between shooting fouls and non-shooting fouls based on whether the fouled player was in a shooting motion at the time of the foul.

[0069] In addition to the three types of conflicts mentioned above, conflicts caused by missed events can also be detected based on event sequence logic. For example, if the same player holds the ball for a period of time exceeding the full offensive timer without any passing or shooting events during that time, this situation does not conform to the violation judgment of the offensive timer and there is a risk of violation. The visual structured feature sequence within that period can be traced back to check whether any passing or shooting events were missed.

[0070] After verification, the confidence level of each event can be dynamically adjusted: events that pass the logical consistency check have their judgment consistent with the rules, and their confidence level can be increased, for example, by 10%; events with conflicts that can be corrected have their original judgment incorrect, and their confidence level can be reduced while being corrected, for example, by 20%; events with conflicts that cannot be automatically corrected are marked for manual review and their confidence level is reduced, for example, by 50%. The events that have undergone the above verification, correction, or marking are arranged chronologically to form the final event annotation sequence. Each event in this sequence can include the event type, the identifiers of the players involved, a timestamp, type confidence level, and event description, and can also include verification results, such as the rules passed, the conflicted rules, the corrective actions taken, and the review mark. The final event annotation sequence can be output in a preset data format as standardized round-by-round data.

[0071] By introducing rule-level logical consistency checks on top of the semantic understanding results of a multimodal large model, event annotations that do not conform to basketball rules can be identified and corrected. This overcomes the defect of annotation results that do not match the actual game due to the separation of detection and rules. Marking events that cannot be automatically corrected and lowering their confidence also allows for focusing manual review efforts on a small number of questionable events. Taking the semi-reset anchor point of offensive timing as an example, its candidate event types can include missed shots and offensive rebounds. Both types of events can occur when the offensive team grabs an offensive rebound after a missed shot. If the multimodal large model marks this segment as a successful shot, this marking does not conform to the form of the offensive timing field returning to the second offensive timing duration: after a successful shot, possession changes, and the offensive timing should return to the complete offensive timing duration. Therefore, it can be detected as a conflicting event in the logical consistency check, and thus corrected or marked.

[0072] Based on the above steps S1 to S4, the competition state machine generates constraint instructions for the extraction process of the visual structured feature sequence according to the current competition logic state, and feeds the constraint instructions back to the extraction process to dynamically adjust the extraction parameters used for subsequent video frames.

[0073] In actual gameplay, the current game logic state itself limits the content that may and may not appear in subsequent shots. Therefore, in addition to receiving the detection results reflected by the visual structured feature sequence, the game state machine can also feed back its maintained game logic state to the extraction process. The constraint instructions may include at least one of the following: a region of interest mask, a detection category filtering flag, a detection confidence threshold, and a player identifier lock flag, wherein the detection category filtering flag is used to indicate that the detection of one or more types of targets is blocked during the extraction process.

[0074] In some implementations, the constraint instructions are fed back to the extraction process of the visual structured feature sequence to dynamically adjust the extraction parameters for subsequent video frames. This includes: when the game state machine is in a free throw state, adjusting the extraction parameters through the constraint instructions to instruct the detection of fast break categories in the player features on the court to be masked during the extraction process, and increasing the attention weight of the preset free throw line area; and / or, when the game state machine is in a camera switching transition state, adjusting the extraction parameters through the constraint instructions to instruct the player identifiers of each player before the switch to be locked during the extraction process, and pausing cross-frame identifier updates based on pixel positions.

[0075] By feeding back the game logic state into the extraction process of the visual structured feature sequence, the extraction parameters of subsequent video frames are adjusted according to the progress of the game. This allows for the exclusion of detection categories that do not match the specific game stage, and the concentration of detection resources on the screen areas related to event determination. This improves the consistency between the feature extraction results and the current game context, and also reduces player identification jumps during shot transitions. As a result, it provides a more reliable visual structured feature sequence for subsequent processing, significantly improving the final annotation quality.

[0076] Please refer to Figure 2 , Figure 2 This is a schematic diagram of the architecture of an automatic basketball event annotation system provided as an exemplary embodiment of this application. As shown in the figure, the system can be divided into a visual perception layer, an event anchor detection layer, a multimodal annotation layer, and a rule constraint engine layer from bottom to top. Each layer can be deployed on the same electronic device, or it can be deployed on different electronic devices and communicate with each other via a network.

[0077] The visual perception layer comprises four modules: player detection and label maintenance, player skeleton keypoint extraction, basketball trajectory tracking, and scoreboard text recognition. These four modules extract corresponding features in parallel from the basketball game video stream and align them using a unified timestamp. The outputs of the first three modules constitute a visual structured feature sequence, while the output of the scoreboard text recognition module constitutes a temporal sequence of scoreboard text.

[0078] like Figure 2 As shown, the features output by the player detection and identification maintenance module and the player skeleton key point extraction module, after being combined with the features output by the basketball trajectory tracking module, flow to the multimodal annotation layer and are encapsulated as event evidence data, serving as the basis for semantic understanding. The scoreboard text time sequence output by the scoreboard text recognition module then flows to the event anchor detection layer. The reason for this design is that field transitions in the scoreboard text time sequence are hard signals, whose values ​​are given by the game timing system and can serve as a deterministic basis for whether an event has occurred; player features and basketball trajectory are soft signals, which represent the content of the screen and are suitable for determining the specific type of event, but not suitable for determining whether an event has occurred alone. Therefore, the detection of event anchors is based on the scoreboard text time sequence.

[0079] The basketball trajectory features output by the basketball trajectory tracking module are also horizontally connected to the event anchor detection layer, and paired with the jump of the score field within the spatiotemporal alignment window to confirm the valid score, that is, to further verify the score change anchor points that have been triggered by hard signals.

[0080] The event anchor detection layer detects transitions in specified fields, including score changes, full-time resets of attack timers, half-time resets of attack timers, and segment transitions. Based on the anchor type, it outputs the range of video segments to be labeled and a list of candidate event types. The layer's output is divided into two paths: the segment range is routed down to event evidence data, limiting the extraction range of keyframes; the candidate event type list is directly input into the multimodal large model, constraining its semantic understanding output range. These two outputs act on the input content and output range of the large model, respectively, jointly constraining the semantic understanding.

[0081] In the multimodal annotation layer, event evidence data is input into the multimodal big model. The big model performs semantic understanding under the constraints of the candidate event type list and outputs a preliminary event annotation sequence.

[0082] The rule constraint engine layer includes a basketball rule knowledge base, a game state machine, and a logic consistency verification and conflict event correction or marking module. The game state machine provides the current game logic state to the verification module. The verification module, in conjunction with the basketball rule knowledge base, verifies the preliminary event label sequence one by one and sends the verification results back to the game state machine to update its game logic state, forming a state update loop within the layer. The verified, corrected, or marked events are arranged chronologically as the final event label sequence output by this layer.

[0083] like Figure 2 As shown, in addition to providing the game logic state to the verification module of this layer, the game state machine also generates constraint instructions based on the current game logic state. These constraint instructions may include at least one of the following: a region of interest mask, a detection category filtering flag, a detection confidence threshold, and a player identifier lock flag. These instructions are then fed back to the visual perception layer to dynamically adjust the extraction parameters used for subsequent video frames.

[0084] Please refer to Figure 3 , Figure 3 This is a schematic diagram illustrating the process of maintaining and allocating player identifiers, provided as an exemplary embodiment of this application.

[0085] like Figure 3As shown, the maintenance and allocation of player identifiers are performed cyclically on a frame-by-frame basis. The objects processed fall into two categories: one is the known players from the previous frame, whose state information includes position status represented by pixel coordinates, motion status represented by velocity vectors, a 512-dimensional appearance feature vector, and topological features representing relative distance relationships; the other is the players in the current frame, which are newly detected players and have not yet established a correspondence with the previous frame, possessing only position status, appearance features, and topological features, but no motion status. Among the aforementioned state information, the position status and motion status of the known players from the previous frame are used together to infer their prior position in the current frame. The distance between this prior position and the measured position of the player in the current frame is the prediction bias distance; the distances between the appearance features and topological features of the known players from the previous frame and the players in the current frame are the appearance distance and topological distance, respectively.

[0086] According to this application, the scene type of the current frame is identified before constructing the cost matrix. This is because the weights w1, w2, and w3 of the three distance classes in the cost matrix depend on the scene of the current frame.

[0087] In this example, the judgment conditions and corresponding weight values ​​for each scene type are as follows: In normal continuous shot scenes, the global feature changes between adjacent frames are small, the number of detected players is stable, and the average confidence of the detection boxes is high. The motion continuity and appearance features are both reliable, so w1=0.40, w2=0.35, and w3=0.25 can be set. In player occlusion scenes, the intersection-union ratio of two or more detection boxes exceeds the preset threshold or players are clustered in local areas. The appearance features are easily affected by occlusion, so the appearance weight can be reduced and the topology weight increased. For example, w1=0 can be set. 0.30, w2=0.20, w3=0.50; In high-speed offensive and defensive transition scenarios, the inter-frame displacement speed of multiple players exceeds the preset speed threshold, and the direction of the basketball trajectory is consistent with the direction of the player group's movement. Motion prediction has certain reference value, and w1=0.45, w2=0.25, w3=0.30 can be set. In static standing scenarios, the remaining time field of the quarter stops decreasing, and the overall speed of the players is lower than the preset threshold. The appearance features and standing topology are relatively stable, and w1=0.20, w2=0.45, w3=0.35 can be set. Based on experience, the preferred value range for w1 is 0.05 to 0.70, the preferred value range for w2 is 0.10 to 0.60, and the preferred value range for w3 is 0.10 to 0.75.

[0088] Figure 3The processing of camera transition scenes is shown separately. Other scenes usually only adjust the weights according to the scene type, while camera transition scenes also consider the topological features of each player in the new frame after the switch, and reconstruct the relative distance relationship between players in the new frame based on this. At the same time, motion prediction based on pixel coordinates is basically ineffective in this scene, so the weight of motion distance can be greatly reduced and the weight of topological distance can be increased, for example, by taking w1=0.10, w2=0.25, and w3=0.65.

[0089] Once the weights are determined, the position of the current frame can be predicted using the velocity vector of the previous frame, and an N×M cost matrix C can be constructed according to C[i,j]=w1×Dmotion+w2×Dappear+w3×Dtopo, where N is the number of players in the previous frame and M is the number of players in the current frame. Then, the Hungarian algorithm can be used to solve for the minimum total cost allocation on the cost matrix C, outputting the matching pairs between player identifiers and detection boxes.

[0090] like Figure 3 As shown, matches with a matching cost of no more than 0.6 are considered successful, and the current frame's detection box inherits the original player identifier. Matches with a matching cost exceeding 0.6 are considered unsuccessful. In this case, unmatched detection boxes are assigned new player identifiers, while the original unmatched player identifiers are entered into the sleep pool and retained for 30 frames for later recovery. After processing, the process proceeds to the next frame.

[0091] Figure 3 The dashed lines in the diagram indicate the adaptive adjustment of weights. Player identifier changes typically occur when a match fails and a new identifier is assigned. If statistics are only collected when a match succeeds, change data cannot be captured. Therefore, after both the successful and failed match results for a frame are determined, the change rate is calculated based on all match results for that frame. This process runs in parallel with the main process executed frame by frame, continuously calculating the match success rate and player identifier change rate for each scene type. When the change rate for a certain scene type exceeds a preset threshold, the weight corresponding to that scene type is adaptively adjusted, and the adjustment result applies to the weight settings for subsequent similar scenes.

[0092] Please refer to Figure 4 , Figure 4 This is a schematic diagram illustrating event anchor point detection and the division of the video segment range to be labeled, provided as an exemplary embodiment of this application.

[0093] Figure 4 The four timelines correspond to the score change anchor point, the offensive timer full reset anchor point, the offensive timer half reset anchor point, and the quarter switch anchor point, respectively. The shaded area on each timeline represents the range of the video segment to be labeled corresponding to that type of anchor point, with its left end being the starting point of the forward segment time window and its right end being the ending point of the backward segment time window.

[0094] like Figure 4 As shown, the trigger conditions for the four types of anchor points are all changes in a specified field in the scoreboard text time sequence: an increase in the score field corresponds to a score change anchor point; a reset of the offensive timer field from a decreasing state to 24 seconds (i.e., the full offensive timer duration) while the score field remains unchanged corresponds to a full offensive timer reset anchor point; a reset of the offensive timer field from a decreasing state to 14 seconds (i.e., the second offensive timer duration) corresponds to a half offensive timer reset anchor point; and a change in the quarter field corresponds to a quarter switching anchor point. The latter two types of anchor points both involve resetting the offensive timer field, the difference being the value after the reset. A reset to the full offensive timer duration indicates the end of an offensive possession and a change in possession, while a reset to the second offensive timer duration indicates that the offensive team regains control of the basketball and possession has not changed. Whether possession changes determines the different candidate event types for the two types. A full reset anchor point also requires the condition that the score field remains unchanged, because the offensive timer also resets to the full offensive timer duration after a successful shot, so the difference lies in whether the score field increases.

[0095] Figure 4 The document also shows the candidate and excluded event types for each type of anchor point. For the offensive timer fully reset anchor point, the candidate event types are foul, out of bounds, violation, defensive rebound, and change of possession, while excluding field goals and offensive rebounds. For the offensive timer half-reset anchor point, the candidate event types are missed field goals and offensive rebounds, while excluding defensive rebounds.

[0096] For score change anchor points, the candidate event types are field goal and free throw, which can be further refined based on the increment of the score field. When the score field value increases by 2, the score can only come from a two-point field goal, so the candidate event type can be determined as a two-point field goal, excluding three-point field goals and free throws. When the score field value increases by 3, the score may come from a three-point field goal or from the accumulation of three consecutive free throws. This can be distinguished by whether the remaining time in the quarter has stopped decreasing; if it has stopped decreasing, it is judged as a free throw; otherwise, it is judged as a three-point field goal. When the score field value increases by 1, the candidate event type can be determined as a free throw, excluding field goals. Figure 4 The text also shows the pairing of the soft signal of basketball passing the basket and the hard signal of the scoring field in the spatiotemporal alignment window at this type of anchor point. Only when the two are successfully paired in sequence can the anchor point be confirmed as a valid scoring event.

[0097] Figure 4The image shows a set of preferred values ​​for various time windows. These time windows can also be selected within a certain range: the forward time window for the score change anchor point can be 2 to 8 seconds, and the backward time window can be 1 to 4 seconds; the forward time window for the offensive timer full reset anchor point can be 4 to 12 seconds, and the backward time window can be 1 to 5 seconds; the forward time window for the offensive timer half reset anchor point can be 3 to 8 seconds, and the backward time window can be 1 to 4 seconds; the forward time window for the quarter change anchor point can be 2 to 10 seconds, and the backward time window can be 3 to 15 seconds.

[0098] Please refer to Figure 5 , Figure 5 This is a schematic diagram of a competition state machine rule verification provided as an exemplary embodiment of this application. Figure 5 The left side shows the various logical states and state transitions maintained by the competition state machine, while the right side shows the process of performing logical consistency checks and adjusting confidence levels on the initial event label sequence.

[0099] like Figure 5 As shown on the left side, the game state machine maintains six types of game logic states. The transition conditions between states are mainly based on the changes of each field in the scoreboard text time sequence, and the characteristics of basketball movement trajectory and on-court players are used as auxiliary judgments.

[0100] When the remaining time of the quarter decreases continuously, the offensive timer field decreases continuously, and the score field remains unchanged, the normal live-ball offensive and defensive state can be maintained if ball possession and player movement remain continuous. When the basketball trajectory indicates that the basketball has left the player's hand and is flying towards the basket, the player can transition from the normal live-ball offensive and defensive state to the live-ball shooting flight state.

[0101] When the ball is in flight during a live shot, if the score field increases and the basketball trajectory indicates that the ball has passed through the rim area, the game can enter the score verification state. If the score field remains unchanged but the offensive timing field returns to the second offensive timing duration, and the offensive team regains control of the ball, the game can return to the normal live-ball offensive and defensive state, and the candidate event type will be set to offensive rebound. If the score field remains unchanged but the offensive timing field returns to the full offensive timing duration, the game can return to the normal live-ball offensive and defensive state, and the candidate event type will be set to defensive rebound or change of possession.

[0102] When the remaining time of the quarter stops and the score field remains unchanged, if the overall player movement decreases, a ball-handling preparation appears, or the referee's screen appears, the game can enter a dead-ball visual stillness state. While in the score verification state, if the score field increases by 1 point and the remaining time of the quarter stops, or if the free-throw line position and free-throw trajectory are detected, the game can enter a free-throw state. While in the score verification state, if the score field increases by 2 or 3 points and the offensive timing field returns to the full offensive timing duration, the score can be confirmed by combining the release position, basketball trajectory, and player identifier, outputting a two-point shot made or three-point shot made event, and returning to the live-ball normal offensive and defensive state. While in the dead-ball visual stillness state or free-throw state, if the remaining time of the quarter resumes decreasing and the offensive timing field begins to decrease continuously, the game can return to the live-ball normal offensive and defensive state.

[0103] Furthermore, during normal offensive and defensive play with the live ball, if the offensive timing field reaches or is close to zero, the remaining time in the quarter stops, and the score field remains unchanged, while the basketball trajectory indicates that the shot was not released or the basketball did not touch the rim, the game can enter a dead-ball visual stillness state, and the candidate event type can be pointed to an offensive timing violation. During score verification, if the score field changes are unstable, or if the score field increases but the changes in the offensive timing field and the remaining time in the quarter do not conform to basketball rules, the visual structured feature sequence within that period can be traced back, and the state can be transitioned to a dead-ball visual stillness state or the corresponding event can be marked as pending verification.

[0104] In any state, if the quarter number field changes or the remaining time in the quarter reaches zero, and the screen displays inter-quarter footage or a player leaving the court, the game enters a dead-ball visual stillness state. In any state, if the scoreboard area is lost, the global screen features change abruptly, the positions of the court lines or basket change discontinuously, or the number and spatial distribution of player detection frames change abruptly, the game enters a camera transition state.

[0105] In the transition state of camera switching Figure 5 The dashed arrow indicates that this state is not caused by the game process itself, but by the discontinuous changes in the screen. Therefore, other game logic states can enter this state. Figure 5 The entry condition for this state is the loss of the scoreboard area or a sudden change in the global screen features. The exit condition is that the scoreboard recognition and player detection numbers stabilize and the rule verification passes. During this state, the game logic state is frozen rather than rewritten, so its recovery condition does not involve the decrementing of scoreboard fields. In this state, the game state machine can also temporarily suspend the output of new event labels and retain the last stable scoreboard recognition results, game logic state, and ball possession direction before the switch. Output will resume after the state recovers to avoid misinterpreting screen changes during the switch as events.

[0106] Figure 5The right side illustrates the logical consistency verification process. After the initial event label sequence is input, each event is verified using rules R1 to R6 from the basketball rule knowledge base. During verification, the current game logic state is obtained from the game state machine on the left. The verification results are processed in two stages: first, a rule conflict is checked; events without conflicts pass the verification with a 10% increase in confidence. For conflicting events, it is then determined whether they can be automatically corrected. If they can be automatically corrected, they are corrected based on the visual structured feature sequence. For example, an event labeled as an offensive rebound but with the offensive timer reset to 24 seconds is corrected to a defensive rebound, triggering a change of possession, and its confidence is reduced by 20%. Events that cannot be automatically corrected are marked for manual review. For example, multiple consecutive free throws correspond to non-shooting fouls, and their confidence is reduced by 50%. The results from the three branches are combined and output in chronological order as the final event label sequence.

[0107] like Figure 5 As shown, the verification results are also sent back to the game state machine to update its current game logic state. For example, if an event is automatically corrected from an offensive rebound to a defensive rebound after verification, and a change of possession is triggered, the possession direction maintained by the game state machine will be updated accordingly. This update will then affect its verification of subsequent events.

[0108] Figure 5 The dashed arrows below indicate the flow of constraint instructions generated by the competition state machine. These constraint instructions are fed back to the extraction process of the visual structured feature sequence, used to dynamically adjust the extraction parameters for subsequent video frames. Figure 5 Two scenarios are shown in the figure.

[0109] During free throws, the game is paused, and players from both sides line up on either side of the free throw line. Fast breaks are not visible in the frame, and the content relevant to event determination is concentrated in the free throw line area. According to this application, the game state machine can use a detection category filtering flag in the constraint instructions to block detection of fast break categories among the player features during the extraction process, preventing this category from participating in the determination in subsequent video frames. Simultaneously, it increases the attention weight of the preset free throw line area, for example, by limiting the detection range with a attention area mask for the free throw line area, or by lowering the detection confidence threshold of targets within this area, allowing for more thorough extraction of players, the basketball, and key skeletal features within the area. The preset free throw line area can be determined based on the court line recognition results.

[0110] When in a transitional shot, the image content changes discontinuously, and the pixel positions of the same player before and after the switch are no longer continuous. If cross-frame identifier updates are continued based on pixel positions, players in the switched image may be mistakenly associated with other players from the previous frame, causing player identifier jumps. According to this application, the game state machine can use constraint instructions to lock the player identifiers of each player before the switch during the extraction process. This means retaining the player identifiers, team identifiers, positional status, appearance features, and topological features from the previous frame, and pausing cross-frame identifier updates based on pixel positions. Once the switched image stabilizes—for example, when the scoreboard recognition results and player detection counts stabilize for several consecutive frames—the correspondence between players before and after the switch is reconstructed by increasing the topological distance weight as described above. Player identifiers are then restored, and rule verification is performed using team identifiers, the number of players on the field, and the direction of possession. After verification, the game state machine can exit the transitional shot state and restore the corresponding game logic state based on the current scoreboard recognition results.

[0111] In the two scenarios described above, the former utilizes the certainty of the footage during the free throw phase, while the latter utilizes the fact that pixel positions are no longer continuous during camera transitions. Both are based on the current logical state of the game.

[0112] The following describes the implementation of this application in a typical application scenario, based on the processing of a complete round.

[0113] The scenario depicts a broadcast video stream of a basketball game. The game is in its third quarter, with the scoreboard showing the home team at 85 points and the away team at 92 points. The method described in this application can process this video stream in real time, outputting event annotations for each round without requiring manual monitoring.

[0114] At the start of this round, the visual perception layer extracts video frames at 2 to 5 frames per second, normalizes the size and color space of each frame, and then feeds them into the detection network. The detection network locates the players and the basketball on the court in each frame, outputs their respective bounding boxes and confidence scores, and classifies the players into home and away teams based on their jersey color scheme. For each detected player, the visual perception layer maintains their positional state, motion state, 512-dimensional appearance features, and relative distance vectors to other players on the court. It also constructs a cross-frame cost matrix frame by frame and solves it using the Hungarian algorithm to ensure that the player identifiers of each player are consistent in time. In this round, the away team players are assigned player identifiers of 1023 and 1045.

[0115] In this round, the broadcast footage switched from a wide shot to a close-up of the half-court during the attack's initiation phase. This shot transition caused discontinuous changes in the pixel positions of the players, rendering pixel-position-based motion prediction ineffective. At this point, the state machine, based on the abrupt change in global screen features, determined to enter a shot transition state. It locked the player identifiers of each player before the transition using constraint instructions and paused cross-frame identifier updates based on pixel position. Once the screen stabilized, the visual perception layer extracted the topological features of each player in the new screen after the transition, increasing the weight of topological distance to 0.65 and decreasing the weight of motion distance to 0.10 to reconstruct the correspondence between players before and after the transition. Player 1023's player identifier was thus restored after the shot transition without any identifier jump.

[0116] Towards the end of the possession, the basketball travels from (520, 340) to (600, 200). The basketball trajectory tracking module maintains and smooths this trajectory using a 30-frame sliding window, indicating that the basketball did not cross the basket area. Simultaneously, the scoreboard text recognition module continuously outputs the scoreboard text time sequence, where the offense timing field progresses from 05 seconds to 24 seconds, while the score field remains unchanged.

[0117] The event anchor detection layer confirms the field change after three consecutive frames of consistent recognition results, classifying it as an offensive timer reset anchor point. Based on the type of this anchor point, the layer uses its trigger time as a benchmark to divide the video segment to be annotated into 8 seconds before and 3 seconds after the trigger time, and generates a list of candidate event types corresponding to this anchor point. The candidate event types include fouls, out of bounds, violations, defensive rebounds, and changes in possession, while excluding shots made and offensive rebounds. Since the score field did not increase, the score change anchor point was not triggered in this play, and the trajectory representation of the basketball not crossing the rim also corroborates the unchanged score field.

[0118] Next, the multimodal annotation layer constructs the event evidence data corresponding to this segment. During frame extraction, the frame extraction density is increased near the anchor point trigger time, and keyframes corresponding to the basketball leaving the hand, the basketball approaching the basket, changes in the offensive timing field, and changes in the relationship between the player and the basketball are retained, resulting in a keyframe image sequence of this segment with 8 to 32 frames. The keyframe image sequence and the visual structured feature sequence within this segment are encapsulated together as event evidence data. This event evidence data, along with a list of candidate event types, is input into the multimodal large model. The input game state is: current score, home team 85 points, away team 92 points, current quarter Q3, offensive timing reset from 05 seconds to 24 seconds; the input visual evidence is: basketball trajectory from (520,340) to (600,200) without crossing the net, player 1023 raising his arm to catch the basketball, and the scoreboard offensive timing reset from 05 seconds to 24 seconds with no change in score; the input candidate event range is the candidate event types and excluded event types mentioned above.

[0119] The multimodal large model performs semantic understanding under the constraints of the aforementioned candidate event type list, outputting the event label for this segment: the event type is defensive rebound, the players involved are identified as 1023 and 1045, the timestamp is 00:05:24:120, the type confidence is 0.87, and the event description is "player 1023 grabs the rebound and possession changes." The action of player 1023 raising their arm to grab the basketball, under pure visual recognition, is similar to the action features of shooting and offensive rebounding. Without constraints, it would easily be identified as an offensive rebound. However, according to this application, by resetting the offensive timing field to the full offensive timing duration—a hard signal—it indicates that possession has changed, and the offensive rebound has been excluded from the candidates. Therefore, under the candidate event type list, this action can only fall under defensive rebound.

[0120] The rule constraint engine layer performs a logic consistency check on the event annotation. The game state machine provides the current game logic state, and the check process applies each rule in the basketball rule knowledge base: this event is not a shot made, so the ball possession logic after a score, as defined by R1, does not apply; the number of player identifiers for each team on the court does not exceed 5, consistent with R3; the basketball first leaves the hand, does not go into the basket, and is then caught by a player, the sequence of events consistent with R5. This event does not conflict with any rules and does not require correction. Its confidence level increases from 0.87 to 0.92, and the review is marked as automatically passed. Thus, this event is written into the final event annotation sequence, which includes the event type "defensive rebound," the player identifiers 1023 and 1045 involved, the timestamp 00:05:24:120, the type confidence level of 0.92, and the verification results of the passed rules R3 and R5, as well as the conflict rules being "empty" and the correction operation being "none."

[0121] Simultaneously, the game state machine updates its maintained game logic state based on the verification result: In this round, when the score field remains unchanged and the offensive timing field returns to the complete offensive timing duration, the state machine reverts from the live-ball shooting flight state to the normal live-ball offensive and defensive state, and updates the ball possession direction to the away team. This game logic state will then serve as the basis for verification in the next round, generating constraint instructions for the extraction process of the aforementioned visual structured feature sequence. If a foul occurs in a subsequent round and the game transitions to a free-throw state, the state machine can use constraint instructions to shield the detection of fast-break categories among the player features on the court and increase the attention weight of the preset free-throw line area, making the extraction process of the visual perception layer compatible with the screen content at this stage.

[0122] There are two potential pitfalls in the above process. First, camera switching can render player appearance features indistinguishable. The proposed solution addresses this by restoring player identification based on the consistency of relative distances between players before and after the camera switch, rather than relying on jersey color or facial features. This ensures the event is correctly attributed to player 1023. Second, the action of grabbing a rebound is difficult to distinguish from an offensive rebound under purely visual recognition. The proposed solution excludes offensive rebounds from the candidate list based on the reset pattern of the scoreboard's offensive timing field, and then uses rule verification to confirm that the possession change matches, ensuring the annotation result aligns with the actual progress of the game. Compared to solutions relying solely on on-screen action recognition, the annotation results of this application are consistent with the game progress reflected on the scoreboard, reducing the likelihood of incorrectly identifying invalid actions or possession changes as valid events.

[0123] This application integrates the deterministic signals of the scoreboard, the semantic information of the image, and the logical constraints of basketball rules into the same processing procedure, and allows the established game context to influence the extraction of visual perception. Event labeling no longer depends on the recognition results of a single image, but is simultaneously verified by deterministic signals and game rules, overcoming the shortcomings of labeling that does not match the actual game due to the separation of detection and rules. High-quality, round-by-round data that can be used for post-game analysis can be obtained based on the video stream.

[0124] Figure 6 An electronic device provided in at least one embodiment of this application includes a memory and a processor. The memory is used to store computer instructions that can be executed on the processor. The processor is used to implement the automatic annotation method for basketball game events based on visual multimodal fusion and rule constraints as described in any embodiment or implementation of this application when executing the computer instructions.

[0125] At least one embodiment of this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the automatic annotation method for basketball event based on visual multimodal fusion and rule constraints as described in any embodiment or implementation of this application.

[0126] Those skilled in the art will understand that the embodiments of this application can be provided as methods, systems, electronic devices, or computer program products, and can be implemented in a completely hardware, completely software, or a combination of hardware and software. The embodiments in this specification are described in a progressive manner, and identical or similar parts can be referred to mutually.

[0127] The processing flow and logical sequence depicted in the accompanying drawings are not necessarily required to be executed in the specific order or sequence shown. Where there is no conflict, the operation steps may be executed in a different order than that shown in the embodiments, or parallel processing may be used to achieve the desired result. Furthermore, the separation of the various system modules and components described above should not be construed as requiring such separation in all embodiments; the described program components and systems can typically be integrated into a single software or hardware product.

[0128] The above description is merely a preferred embodiment of one or more embodiments of this specification and is not intended to limit the scope of one or more embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments of this specification should be included within the scope of protection of one or more embodiments of this specification.

Claims

1. An automatic annotation method for basketball game events based on visual multimodal fusion and rule constraints, characterized in that, include: Visual structured feature sequences and scoreboard text time sequences are extracted from the video stream of a basketball game. The visual structured features include on-court player features and basketball trajectory features. Based on the specified field jump detection event anchor point in the scoreboard text time sequence, the corresponding video segment range to be labeled is divided according to the type of the event anchor point, and a candidate event type list corresponding to the type of the event anchor point is generated; The visual structured feature sequence within the range of the video segment to be labeled is input into a multimodal large model, and semantic understanding is performed under the constraints of the candidate event type list to obtain a preliminary event labeling sequence. By utilizing the game state machine that maintains the game's logical state, and combining it with a pre-set basketball rule knowledge base, a logical consistency check is performed on the preliminary event labeling sequence. Based on the visual structured feature sequence, conflicting events detected are corrected or marked to obtain the final event labeling sequence. The competition state machine generates constraint instructions for the extraction process of the visual structured feature sequence based on the current competition logic state, and feeds the constraint instructions back to the extraction process to dynamically adjust the extraction parameters used for subsequent video frames.

2. The method according to claim 1, characterized in that, Extracting on-field player features from the visual structured feature sequence, including maintaining and assigning player identifiers in the following manner: Extract the positional features, appearance features, and topological features of each player in the current frame. The topological features of each player represent the relative distance relationship between that player and other players in the current frame. For players in the previous frame and players in the current frame, calculate the prediction deviation distance of the position features, the appearance distance between the appearance features, and the topological distance between the topological features, and calculate the weighted sum of the prediction deviation distance, the appearance distance, and the topological distance to construct a cross-frame cost matrix; The cross-frame cost matrix is ​​solved based on the minimum cost assignment algorithm. When the cost of the matching pair does not exceed the preset matching threshold, the player identifier of the previous frame is inherited to the player corresponding to the current frame; otherwise, the matching is judged as a failure. When a shot change is detected in the video stream, the topological features of each player in the new frame after the change are extracted, and the weight of the topological distance in constructing the cross-frame cost matrix is ​​increased to restore the player identification before the shot change.

3. The method according to claim 2, characterized in that, The method further includes adjusting the weights of the prediction bias distance, the appearance distance, and the topology distance when calculating the weighted sum: The global screen feature changes between adjacent frames, the number of players detected, the confidence of player detection boxes, the intersection-over-union ratio between player detection boxes, the inter-frame displacement velocity of players, and / or the decreasing state of the remaining time field in the current quarter in the scoreboard text time sequence are extracted as recognition criteria to identify the scene type of the current frame. The scene types include normal continuous shot scenes, player occlusion scenes, high-speed offensive and defensive transition scenes, shot switching transition scenes, and stationary standing scenes. The weights of the prediction deviation distance, the appearance distance, and the topology distance are dynamically adjusted according to the scene type.

4. The method according to claim 3, characterized in that, The method further includes: Statistics on player icon jump rate under various scenario types; When the player identifier jump rate in any scene type exceeds a preset threshold, the weight corresponding to that scene type is adaptively adjusted.

5. The method according to claim 1, characterized in that, The on-field player features include key points of the player skeleton, and the method further includes at least one of the following correction operations for the coordinates of the key points of the skeleton: Calculate the current length ratio between adjacent limb segments. When the current length ratio exceeds the preset reasonable ratio range, scale and correct the coordinates of the corresponding key points according to the average ratio of the historical frames of the corresponding limb segments. Calculate the joint vector angle. When the joint vector angle exceeds the preset reasonable angle range, truncate the joint vector angle to the boundary value and remap the joint coordinates, or reduce the confidence of the corresponding key point. Calculate the coordinate difference between key points that are symmetrical to the left and right. When the coordinate difference exceeds a preset symmetry threshold, the coordinates of the corresponding key points are corrected by averaging or the corresponding key points are marked as having abnormal posture.

6. The method according to claim 1, characterized in that, The on-field player features include key points of the player skeleton, and the method further includes adaptive temporal filtering of the coordinates of the key points of the skeleton: Calculate the inter-frame displacement velocity of key points and determine the motion state of key points based on their confidence level; When the motion state is stationary, a moving average filter with a first window length is used; when the motion state is a preset slow state, a moving average filter with a second window length less than the first window length is used; when the motion state is a preset high-speed state, a Kalman filter is used; when the confidence level of a key point is lower than a preset key point confidence threshold, linear interpolation is used to complete the key point and the corresponding key point is marked as an interpolation state.

7. The method according to claim 1, characterized in that, The on-field player characteristics include key points of the player's skeletal structure, and the method further includes: When extracting key points of the player's skeleton, key points related to basketball movements are given a higher detection priority than other key points in order to improve the positioning accuracy of key points related to basketball movements.

8. The method according to claim 1, characterized in that, Based on the specified field transition event anchor points in the scoreboard text time sequence, the corresponding range of video segments to be labeled is divided and a candidate event type list is generated, including: An event in which the value of the score field in the scoreboard text sequence increases is identified as a score change anchor point. The candidate event types corresponding to the score change anchor point include field goals made and free throws made. An event in which the offensive timer field in the scoreboard text sequence is reset from a decreasing state to a full offensive timer duration while the score field remains unchanged is identified as an offensive timer full reset anchor point. The candidate event types corresponding to the offensive timer full reset anchor point include fouls, out of bounds, violations, defensive rebounds, and changes in possession. Field goals made and offensive rebounds are excluded from the candidate event type list. An event in which the offensive timer field is reset from a decreasing state to a second offensive timer duration is identified as an offensive timer half reset anchor point. The candidate event types corresponding to the offensive timer half reset anchor point include missed field goals and offensive rebounds. Defensive rebounds are excluded from the candidate event type list. And / or an event in which the quarter field in the scoreboard text sequence changes is identified as a quarter switching anchor point. The candidate event types corresponding to the quarter switching anchor point include quarter breaks and opening tip-offs. Based on the trigger time of each event anchor point, the range of the video segments to be labeled is divided according to the preset forward segment time window and backward segment time window corresponding to the type of event anchor point.

9. The method according to claim 8, characterized in that, The method further includes: Establish a spatiotemporal alignment window between the basketball trajectory features and the scoreboard text time sequence; After determining the event of increasing the value of the score field as the score change anchor point, when the basketball trajectory feature represents the basketball passing through the basket area within the spatiotemporal alignment window, and the increase in the value of the score field that triggered the score change anchor point occurs within a preset score verification time window after this representation, the score change anchor point is confirmed as a valid score event, and the type of the valid score event is determined from the candidate event types corresponding to the score change anchor point.

10. The method according to claim 1, characterized in that, The method further includes constructing event evidence data before inputting the visual structured feature sequence within the range of the video segment to be labeled into the multimodal large model: The basic frames are extracted according to the preset frame rate. The frame extraction density is increased near the triggering time of the event anchor point. The key frames corresponding to at least one of the following are retained: the basketball leaves the hand, the basketball approaches the basket, the change of the score field or the offensive timing field in the scoreboard text time sequence, and the change of the relationship between the player and the basketball, so as to obtain a key frame image sequence. The keyframe image sequence and the visual structured feature sequence are encapsulated into event evidence data, which is then used as input to the multimodal large model.

11. The method according to claim 1, characterized in that, Performing a logical consistency check on the preliminary event-labeled sequence, and correcting or marking conflicting events detected during the check, includes at least one of the following: When an event is marked as a successful shot, but the score field in the corresponding scoreboard text time sequence does not increase, it is determined to be a conflict event. The visual structured feature sequence is then traced back to check whether there is a prior violation in the offensive timing field of the scoreboard text time sequence. When an event is marked as an offensive rebound, but the offensive timing field in the corresponding scoreboard text timing sequence is reset to the full offensive timing duration instead of the secondary offensive timing duration, it is determined to be a conflict event. The conflict event is corrected to a defensive rebound and a change of possession is triggered. When multiple consecutive events marked as free throws correspond to non-shooting fouls, they are identified as conflict events and marked as pending manual review. Increase the type confidence of events that pass the logical consistency check, decrease the type confidence of events that have conflicts but can be automatically corrected, and mark events that have conflicts and cannot be automatically corrected as requiring manual review and decrease the type confidence of those events.

12. The method according to claim 1, characterized in that, The game logic states include normal offensive and defensive states with a live ball, live ball shooting flight states, score verification states, visually static states with a dead ball, free throw states, and camera transition states; the constraint instructions are fed back to the extraction process of the visual structured feature sequence to dynamically adjust the extraction parameters used for subsequent video frames, including: When the game state machine is in the free throw state, the extraction parameters are adjusted by the constraint instruction to indicate that the detection of fast break category in the characteristics of the players on the court is blocked during the extraction process, and the attention weight of the preset free throw line area is increased. And / or when the game state machine is in a camera switching transition state, the extraction parameters are adjusted by the constraint instructions to indicate that the player identifiers of each player before the switch are locked during the extraction process, and the cross-frame identifier update based on pixel position is paused.

13. An electronic device, characterized in that, The device includes a memory and a processor, the memory being used to store computer instructions executable on the processor, and the processor being used to implement the method of any one of claims 1 to 12 when executing the computer instructions.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it implements the method described in any one of claims 1 to 12.

Images