A method for motion localization in sports videos based on behavioral feature recognition

By using successor state anchors and an improved VideoMamba model in sports videos, combined with the Murty algorithm, the accuracy problem of motion localization methods in sports videos under camera movement and subject occlusion was solved, achieving accurate and stable localization of motion boundaries and improving the ability to recognize motion regions in complex scenes.

CN123071901APending Publication Date: 2026-11-10TIANJIN UNIV OF TECH & EDUCATION (TEACHER DEV CENT OF CHINA VOCATIONAL TRAINING & GUIDANCE)
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611302664.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-26
Publication Date
2026-11-10

AI Technical Summary

Technical Problem

Existing motion localization methods for sports videos suffer from decreased localization accuracy when the camera moves, the subject is occluded, or there are continuous motion transitions. Furthermore, features from adjacent motions are mixed into the current motion, leading to incorrect connection of motion boundaries.

Method used

The method combines successor state anchors with an improved VideoMamba model. Anchors are determined by the cross-frame behavioral relationship between the moving subject and the object of action, forming a common initial hidden state. This state is then continued along both the reverse and forward time sequences. Action boundaries are locked at the points where the behavioral relationship is continuously interrupted. The Murty algorithm is used to generate candidate continuation schemes to restore the local state, thereby reducing interference from adjacent actions and motion outside the boundary.

Benefits of technology

It improves the accuracy and stability of action boundary localization, enhances the ability to continuously recognize action ranges in complex sports scenarios, and reduces the impact of camera movement and subject occlusion on action boundary judgment.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123071901A_ABST
    Figure CN123071901A_ABST
Patent Text Reader

Abstract

This invention discloses a sports video action localization method based on behavioral feature recognition, belonging to the field of computer vision technology, and includes the following steps: S1, acquiring sports video and correcting frame timing, extracting motion regions; S2, tracking the moving subject and the object of action to form a sports behavior relationship sequence; S3, detecting the moment when the behavior relationship transitions to the successor state and determining the successor state anchor point; S4, integrating video tags into an improved VideoMamba model, forming a common initial hidden state with anchor points, updating along a bidirectional temporal sequence, and locking the action boundary at the point of relationship interruption; S5, retrieving candidate continuity relationships and calculating the joint association cost; S6, using the Murty algorithm to generate a state writing recovery scheme; S7, recovering local continuity to form candidate action intervals; S8, determining the final action interval based on the continuity of state transitions. This invention can improve the accuracy of action boundary localization and the ability to distinguish continuous actions.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method for locating sports video motions based on behavioral feature recognition. Background Technology

[0002] Existing motion localization methods for sports videos typically rely on manual annotation, object tracking, or deep learning-based temporal recognition models to detect moving subjects in the video and determine the start and end times of the motion based on feature changes in consecutive video frames. While these methods can extract motion segments in typical sports scenarios, their accuracy tends to decrease when there is camera movement, subject occlusion, or continuous motion transitions.

[0003] With the development of computer vision technology, action localization methods based on video temporal features are gradually being applied to competition analysis and training evaluation. However, existing methods mostly focus on the characteristics of the moving subject itself, and do not make sufficient use of the behavioral relationships between the moving subject and objects such as balls and equipment. When the target is lost for a short time or the trajectory intersects, the action relationship is easily interrupted.

[0004] In addition, existing temporal models typically transmit features continuously within a complete video sequence. Irrelevant motion outside the action interval can easily participate in the calculation, causing features of adjacent actions to be mixed into the current action, resulting in problems such as action boundaries being advanced, delayed, or incorrectly connected.

[0005] Therefore, how to provide a sports video motion localization method based on behavioral feature recognition is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] One objective of this invention is to propose a sports video action localization method based on behavioral feature recognition. This invention employs a combination of successor state anchor points and an improved VideoMamba model. Anchor points are determined by the cross-frame behavioral relationship between the moving subject and the object of action, and a common initial hidden state is formed using these anchor points. This state is then continued along both the reverse and forward time sequences, locking the action boundary at the point where the behavioral relationship is continuously interrupted. Based on this, the Murty algorithm is used to generate candidate continuation schemes and restore the local state writing, thereby correcting and filtering the action interval. This reduces interference from adjacent actions and movements outside the boundary, and has the advantages of accurate action boundary localization, strong ability to distinguish continuous actions, and good adaptability to complex sports scenes.

[0007] A sports video motion localization method based on behavioral feature recognition according to an embodiment of the present invention includes the following steps:

[0008] S1. Acquire sports video, correct frame timing and extract motion regions to form sports video timing data;

[0009] S2. Track the moving subjects and objects in the time-series data of sports videos, and form a sequence of sports behavior relationships based on cross-frame behavioral characteristics;

[0010] S3. Detect the moment when the sequence of sports behavior relationships transitions from a stable action state to a subsequent state, and determine the anchor point of the subsequent state.

[0011] S4. Convert sports video time series data into video tags and connect them to the improved VideoMamba model. At the time corresponding to the subsequent state anchor point, a common initial hidden state is formed and the hidden state is updated along the reverse and forward time order. At the point where the behavior relationship is continuously interrupted, the last continuous hidden state is locked and the writing of video tags outside the boundary and bidirectional merging are blocked to form the initial action interval and boundary locking state.

[0012] S5. Retrieve candidate relationships that have geometric continuity between the behavior relationships outside and inside the initial action interval, forming a set of candidate continuity relationships with joint association costs;

[0013] S6. Use the Murty algorithm to process the candidate succession relationship set, generate a joint succession scheme set according to the joint association cost, and convert it into a candidate state to write into the recovery scheme set.

[0014] S7. Write the corresponding video marker to the hidden state at the recovery position according to the candidate state writing recovery scheme one by one, and recover the local continuation from the hidden state of the boundary locked state to form a candidate action interval set containing the initial action interval.

[0015] S8. Determine the final action range based on the continuity of the action state transitioning to the subsequent state according to the behavioral relationship within the candidate action range, thus forming the sports video action localization result.

[0016] Optionally, S1 specifically includes:

[0017] S11. Read the timestamps of video frames in the sports video, rearrange the video frames according to the timestamps, and form a time-corrected frame sequence.

[0018] S12. Estimate the global motion transformation between adjacent video frames in the temporal correction frame sequence, register adjacent video frames and extract the inter-frame change region after registration to form a motion region sequence.

[0019] S13. Associate the motion region sequence with the time-corrected frame sequence according to the video frame order to form sports video time-series data.

[0020] Optionally, S2 specifically includes:

[0021] S21. Perform target detection on the motion region in the sports video time series data, distinguish between the moving subject and the object of action, extract geometric features, and form motion target detection results;

[0022] S22. Based on the appearance consistency and motion continuity in the moving target detection results, perform target association, calculate geometric changes according to the associated trajectory, and form cross-frame behavior features;

[0023] S23. Calculate the relative geometric changes between the moving subject and the object of action based on the cross-frame behavior characteristics, and arrange the relative geometric changes according to the time sequence of video frames to form a sports behavior relationship sequence.

[0024] Optionally, S3 specifically includes:

[0025] S31. Compare the behavioral relationships of adjacent video frames along the sports behavior relationship sequence to locate the transition moment from the action state to the subsequent state, and form a state transition moment sequence.

[0026] S32. Verify the behavioral relationship in chronological order starting from the transition time in the state transition time sequence. When the behavioral relationship returns to the action state, proceed to the next transition time. When the behavioral relationship maintains the successor state in consecutive video frames, determine the corresponding transition time as the successor state anchor point.

[0027] Optionally, the improved VideoMamba model includes a video tokenization layer, a location embedding layer, a VideoMamba encoding layer, and a normalization and prediction layer connected in sequence.

[0028] The video tagging layer uses three-dimensional convolution to extract local video features along the temporal and spatial dimensions of sports video time-series data, and unfolds them into a video tagging sequence according to the frame time sequence.

[0029] The location embedding layer superimposes location codes corresponding to frame temporal and spatial locations onto the video tag sequence, and determines the video tag corresponding to the time of the subsequent state anchor point based on the frame temporal location codes.

[0030] The VideoMamba coding layer performs selective state space calculation according to the spatial arrangement of video markers within the anchor point time. It uses the end hidden state as the common initial hidden state, updates the hidden state along the reverse and forward time order, retains the last continuous hidden state when the sports behavior relationship is continuously interrupted, stops updating the hidden state of the corresponding direction of video markers outside the boundary, and blocks video markers outside the boundary from participating in the bidirectional state merging within the locked interval, forming a boundary locked state.

[0031] The normalization and prediction layer normalizes the hidden states within the locked interval, determines the action boundary based on the state continuation stop position in reverse and forward order, forms the initial action interval, and retains the boundary locked state.

[0032] Optionally, the formation of the boundary locking state specifically includes:

[0033] Within the VideoMamba coding layer, selective state space updates are performed according to the video tag space order corresponding to the time of the successor state anchor point, setting the terminal hidden state corresponding to the last video tag as the initial hidden state of reverse-time continuation and forward-time continuation, respectively.

[0034] Reverse chronological continuation reads video markers and updates the reverse chronological hidden state moment by moment from the subsequent state anchor point; forward chronological continuation reads video markers and updates the forward chronological hidden state moment by moment from the subsequent state anchor point.

[0035] When the sports behavior relationship remains continuous, the current hidden state is passed to the next moment for continued updating. When the sports behavior relationship is not recovered in consecutive video frames, the last consecutive hidden state before the interruption is retained and the reading of the video marker outside the boundary in the corresponding direction is stopped.

[0036] The reverse-time hidden states from the reverse-time stop position to the subsequent state anchor point are arranged in reverse frame time order and merged with the forward-time hidden states from the subsequent state anchor point to the forward-time stop position according to frame time order. The video marker outside the boundary is blocked from being written into the corresponding direction hidden state, forming a boundary locked state.

[0037] Optionally, S5 specifically includes:

[0038] S51. Starting from the two sides of the initial action interval, extract the sports behavior relationship sequence along the corresponding time direction to form a continuous relationship within the boundary and a relationship to be continued outside the boundary.

[0039] S52. Compare the geometric motion changes of the relationship to be continued outside the boundary and the continuous relationship inside the boundary frame by frame, and filter out candidate relationships that maintain motion continuity.

[0040] S53. Calculate the geometric deviations between the moving subject and the object of action in the candidate succession relationship and the corresponding moving target within the boundary. Accumulate the geometric deviations of the same candidate succession relationship along the succession time sequence to form a joint association cost, thus forming a set of candidate succession relationships with joint association costs.

[0041] Optionally, S6 specifically includes:

[0042] S61. Using the combination of the moving subject and the object of action within the initial action interval as rows and the combination of the moving subject and the object of action in the candidate succession relationship as columns, write the joint association cost between the combinations into the corresponding matrix elements to form a joint association cost matrix.

[0043] S62. The Murty algorithm is used to find the minimum cost allocation of the joint association cost matrix. The preceding associations are fixed in sequence and the current association is excluded to construct the sub-allocation problem, forming a set of joint succession schemes that increase in total cost.

[0044] S63. Determine the start time of the candidate connection relationship corresponding to the joint connection scheme as the state write recovery position, and determine the corresponding video marker as the recovery write object to form a set of candidate state write recovery schemes.

[0045] Optionally, S7 specifically includes:

[0046] S71. When the recovery position is outside the start boundary of the initial action interval, the reverse timing boundary lock is selected to be hidden; when it is outside the end boundary, the forward timing boundary lock is selected to be hidden, and the write blocking of the video marker not selected in the joint continuation scheme is maintained.

[0047] S72. Take the selected boundary-locked hidden state as the initial state, and continue to process the video tags corresponding to the candidate continuation relationship along the corresponding time direction from the self-recovery position. When the behavior relationship is continuous, write the updated hidden state at each moment.

[0048] S73. When the candidate continuation relationship ends or the behavior relationship is interrupted again, the state continuation is stopped. The corresponding action boundary is corrected at the last continuous moment. If no continuous continuation is formed, the initial action interval is retained to form a candidate action interval set.

[0049] Optionally, S8 specifically includes:

[0050] S81. Using the subsequent state anchor point as the boundary, verify whether the behavioral relationship within each candidate action interval is continuously maintained along the reverse time sequence, and verify whether the behavioral relationship is continuously maintained along the forward time sequence.

[0051] S82. Retain candidate action intervals that are continuously verified in reverse order to the action start boundary and continuously verified in forward order to the action end boundary, and exclude candidate action intervals that are interrupted in either direction.

[0052] S83. Select the interval with the lowest corresponding joint association cost from the retained candidate action intervals as the final action interval. If no candidate action interval passes the verification, retain the initial action interval to form the sports video action localization result.

[0053] The beneficial effects of this invention are:

[0054] This invention constructs a cross-frame sports behavior relationship sequence by correcting the timing of sports video frames, extracting motion regions, and tracking the moving subject and the object of action. It can identify the stable transition from the action state to the subsequent state from the relative geometric changes between the moving subject and the object of action, and use the subsequent state anchor point as the action positioning benchmark, thereby reducing the impact of camera movement, subject occlusion, and continuous action switching on the judgment of action boundaries.

[0055] This invention improves the state continuation method of the VideoMamba model by updating the common initial hidden state formed at the anchor point along both the reverse and forward time sequences. The last continuous hidden state is retained at the point where the behavior relationship is continuously interrupted. At the same time, the writing of video markers outside the boundary and their participation in bidirectional state merging are blocked, thereby reducing the interference of adjacent actions and irrelevant motion features on the current action interval and improving the positioning accuracy and stability of the action start boundary and end boundary.

[0056] Furthermore, this invention employs the Murty algorithm to jointly allocate candidate succession relationships between the moving subject and the object of action, and recovers local state succession from the boundary-locked hidden state. This can correct the interruption of action relationships caused by short-term target loss, trajectory intersection, or object of action handover, thereby improving the continuous recognition capability of action intervals and the reliability of positioning results in complex sports scenarios. Attached Figure Description

[0057] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0058] Figure 1 This is an overall flowchart of a sports video motion localization method based on behavioral feature recognition proposed in this invention;

[0059] Figure 2 This is a schematic diagram of the improved VideoMamba model in a sports video action localization method based on behavioral feature recognition proposed in this invention.

[0060] Figure 3 This is a schematic diagram illustrating the bidirectional boundary locking and local state recovery of a sports video action localization method based on behavioral feature recognition proposed in this invention. Detailed Implementation

[0061] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0062] refer to Figures 1 to 3 A method for motion localization in sports videos based on behavioral feature recognition includes the following steps:

[0063] S1. Acquire sports video, correct frame timing and extract motion regions to form sports video timing data;

[0064] S2. Track the moving subjects and objects in the time-series data of sports videos, and form a sequence of sports behavior relationships based on cross-frame behavioral characteristics;

[0065] S3. Detect the moment when the sequence of sports behavior relationships transitions from a stable action state to a subsequent state, and determine the anchor point of the subsequent state.

[0066] S4. Convert sports video time series data into video tags and connect them to the improved VideoMamba model. At the time corresponding to the subsequent state anchor point, a common initial hidden state is formed and the hidden state is updated along the reverse and forward time order. At the point where the behavior relationship is continuously interrupted, the last continuous hidden state is locked and the writing of video tags outside the boundary and bidirectional merging are blocked to form the initial action interval and boundary locking state.

[0067] S5. Retrieve candidate relationships that have geometric continuity between the behavior relationships outside and inside the initial action interval, forming a set of candidate continuity relationships with joint association costs;

[0068] S6. Use the Murty algorithm to process the candidate succession relationship set, generate a joint succession scheme set according to the joint association cost, and convert it into a candidate state to write into the recovery scheme set.

[0069] S7. Write the corresponding video marker to the hidden state at the recovery position according to the candidate state writing recovery scheme one by one, and recover the local continuation from the hidden state of the boundary locked state to form a candidate action interval set containing the initial action interval.

[0070] S8. Determine the final action range based on the continuity of the action state transitioning to the subsequent state according to the behavioral relationship within the candidate action range, thus forming the sports video action localization result.

[0071] In this embodiment, S1 specifically includes:

[0072] Sports videos are read frame by frame by a video decoder. The video frame timestamp is taken from the display timestamp in the video encapsulation data. After reading all valid timestamps, the median of the positive difference between adjacent valid timestamps is calculated and used as the frame interval. When a timestamp is missing, it is filled in with adjacent valid timestamps and the frame interval. When timestamps are repeated, the video frames are retained and arranged according to the decoding order. When timestamps are out of order, they are first rearranged in ascending order of timestamps and then the video frames within the same timestamp are processed. When all timestamps are missing, the timestamp is generated by multiplying the video frame number by the reciprocal of the video frame rate. When the video frame rate cannot be read, the processing of the current video segment is stopped. This forms a timing correction frame sequence.

[0073] After adjacent video frames in the temporal correction frame sequence are converted into grayscale images, ORB feature points are extracted and bidirectional matching is performed using Hamming distance. Then, RANSAC is used to remove mismatches and obtain the homography matrix from the current video frame to the previous video frame. When RANSAC returns a valid homography matrix and the proportion of inliers reaches the valid lower limit determined by the validation set, the homography matrix is ​​saved as the global motion transform. If the solution fails, the previous valid global motion transform is used. If the first group of adjacent video frames cannot obtain a valid global motion transform, the current video frame is marked as an invalid frame. After registering adjacent video frames using the global motion transform, the absolute difference of grayscale is calculated pixel by pixel. The mean and standard deviation of the absolute difference of grayscale are used to form the change judgment value. Then, median filtering, closing operation and small connected region deletion are performed to form the inter-frame change region.

[0074] Inter-frame variation regions are bound to a time-series correction frame sequence via correction timestamps and video frame numbers. Each sequence position stores the corrected video frame, corresponding timestamp, and motion region mask. Invalid frames are not written into the sequence, and adjacent valid frames retain the original time interval. After frame-by-frame binding is completed, sports video time-series data is formed, and the motion region in the sports video time-series data is used as the image range for detecting the moving subject and the object of action.

[0075] In this embodiment, S2 specifically includes:

[0076] In sports video time series data, the motion region is used as the target detection range. In basketball videos, the main moving subject is the athlete, and the target is the basketball. After appropriately expanding the motion region, it is input into the trained target detection network. The target detection network outputs the target category, rectangular region, detection confidence, and appearance features. The lower limit of detection confidence is the value corresponding to the highest combined result of detection accuracy and recall in the validation set. Results below the lower limit of detection confidence are deleted, and non-maximum suppression is performed on overlapping results of the same category. The center position and width and height of the rectangular region are normalized to the video frame size to form the moving target detection result.

[0077] Targets of the same category are associated across frames using cosine similarity of appearance features and Kalman prediction position deviation. The two results are weighted to form the association cost. The weight and upper limit of the association cost are the parameter combination that corresponds to the lowest sum of the number of switching of the identity of the moving subject and the number of misconnection of the trajectory of the target in the validation set. The Hungarian algorithm is used to find the matching result with the lowest total cost. Targets that are successfully matched are written into the same associated trajectory. Unmatched detection results establish new trajectories. Unmatched trajectories wait for the target to reappear within the trajectory retention period determined by the validation set. When the reappearing target meets the judgment of appearance consistency and prediction position deviation, the original trajectory number is restored. If no match is found after the trajectory retention period, the corresponding trajectory is terminated. The motion speed is then calculated by the change of adjacent positions and the difference of timestamps, and combined with the displacement direction and regional scale changes to form cross-frame behavior features.

[0078] The sports behavior relationship is calculated based on the relative geometric changes between the moving subject and the object of action within the same video frame. The difference between the x-coordinate of the object of action's center and the x-coordinate of the moving subject's center is divided by the width of the moving subject's region, and the difference between the y-coordinates of their centers is divided by the height of the moving subject's region. The ratio of the distance between their centers to the length of the video frame's diagonal and the difference between the velocity of the object of action and the velocity of the moving subject are also calculated. When the center of the object of action is within 20% of the width and height of the region extending outward from the moving subject's region, or when the distance between their centers is no greater than 15% of the length of the video frame's diagonal, the corresponding subject and object combination is retained. The relative position, relative distance, and relative velocity are connected sequentially according to the video frame. During the short-term loss of the target, the trajectory prediction position is used to maintain the relative geometric calculation. If the target is not recovered by the end of the retention period, the current combination is terminated. This forms a sports behavior relationship sequence, which serves as the input for action state transition detection.

[0079] In this embodiment, S3 specifically includes:

[0080] In sports behavior relationship sequences, the action state and subsequent state are determined by the relative geometric change between the trajectory of the same moving subject and the trajectory of the same object. In this implementation, basketball shooting action is used as the implementation object. The action state indicates that the basketball is continuously within the athlete's control range, the center of the basketball is located within the range of 20% outward from the athlete's detection area, and the center distance between the basketball and the athlete does not increase continuously. The subsequent state indicates that the basketball leaves the control range and enters independent movement. The distance increment is obtained by calculating the current center distance minus the center distance of the previous frame frame by frame. The relative velocity is obtained by calculating the magnitude of the basketball velocity vector minus the athlete's velocity vector. When the distance increment is continuously positive and the relative velocity is greater than 0.4 times the diagonal length of the video frame per second, the behavior relationship is determined to enter the subsequent state.

[0081] The state transition time sequence is generated sequentially along the frames within the same combination of moving subject and object. The behavior relationship categories of adjacent video frames are compared frame by frame. When the previous video frame belongs to the action state and the current video frame belongs to the successor state, the timestamp of the current video frame is recorded as the transition time. When the transition from the action state to the successor state occurs again within the same stable verification window, the transition time with the earliest time sequence is retained. After completing the comparison of all video frames, a state transition time sequence arranged in chronological order is formed.

[0082] The state transition time is verified starting from the first item in the sequence. The lower limit of the stable holding time of the subsequent state is determined by the labeled verification set. The duration with the lowest sum of the false anchor rate and the missing anchor rate is taken and converted into the number of stable holding frames according to the current sports video frame rate. The number of stable holding frames is read continuously from the video frame corresponding to the transition time. When all the subsequent states are maintained, the transition time is determined as the anchor point of the subsequent state. When the behavior relationship returns to the action state during the verification period, the current transition time is deleted and the process moves to the next transition time. If the target is lost for a short time, the predicted position is used to continue the calculation within the trajectory retention period determined above. If the target is not recovered after the trajectory retention period, the verification of the current transition time is terminated.

[0083] When there are multiple stable transition moments in the state transition time sequence, the transition moment with the earliest time sequence and the continuous trajectory number of the moving subject and the trajectory number of the object being acted upon is taken as the subsequent state anchor point, and the corresponding frame number, timestamp, trajectory number of the moving subject, and trajectory number of the object being acted upon are written into the anchor point record; when all transition moments fail the stability maintenance verification, the motion localization of the current video segment is stopped and the process is transferred to manual review. The resulting subsequent state anchor point is established to correspond with the same frame video marker in the sports video time series data, serving as the starting position for the bidirectional state continuation of the improved VideoMamba model.

[0084] In this embodiment, the improved VideoMamba model includes a video tokenization layer, a location embedding layer, a VideoMamba encoding layer, and a normalization and prediction layer connected in sequence:

[0085] The original VideoMamba model uses 3D convolution to convert continuous video frames into video tags. After adding trainable positional codes and classification tags to the video tag sequence, 24 VideoMamba coding blocks perform spatiotemporal forward and backward scans on the complete sequence. Finally, the classification tags are normalized and the video classification result is output through a single-layer perceptron. The complete sequence scan will write the motion before and after the action interval into the bidirectional hidden state. When directly used for sports action boundary localization, it is easily affected by the short-term loss of adjacent actions and the target object.

[0086] The improved VideoMamba model consists of a video tokenization layer, a location embedding layer, a VideoMamba encoding layer, and a normalization and prediction layer connected in sequence. The model input is sports video time series data containing subsequent state anchors. The duration of the input video segment is determined based on the duration distribution of the labeled action intervals in the training set and converted into the number of input frames according to the current sports video frame rate. When the number of frames on one side of the video boundary is insufficient, the nearest valid frame is copied to fill in the gap, and the state writing of the corresponding video tag of the filled frame is turned off.

[0087] The video tagging layer consists of three-dimensional convolutional units and tag unrolling units. The three-dimensional convolutional units use the kernel size, stride, and number of output channels corresponding to the original VideoMamba model and pre-trained parameters to extract local video features along the temporal and spatial dimensions. The tag unrolling units first arrange the local video features in the order of time slices, and then unroll them into a video tag sequence in the spatial order from top to bottom and from left to right within each time slice, and save the temporal index and spatial index for each video tag.

[0088] The location embedding layer consists of a location vector table and an anchor point index mapping unit. Each video tag is added to the corresponding location vector channel by channel to form a video tag with location encoding. The frame number of the subsequent state anchor point is subtracted from the starting frame number of the input video segment to obtain the anchor point frame offset. Then, it is converted into the anchor point time index according to the step size of the video tagging layer in the time dimension. All video tags in the time slice corresponding to the anchor point time index constitute the anchor point time video tag.

[0089] When the position vector table of the original VideoMamba model is inconsistent with the time length of the current video tag sequence, the spatial position vector arrangement remains unchanged, and linear interpolation is performed on the position vectors along the time dimension to make the number of position vectors consistent with the number of current time slices; the original classification tags are retained in the original classification output path, and the formation of hidden states at the anchor end, reverse time sequence continuation, forward time sequence continuation, and action boundary determination all use video tags as the processing object.

[0090] The VideoMamba coding layer consists of consecutive coding blocks with the same number of layers as the original pre-trained model. Each coding block includes a layer normalization unit, an input projection unit, a one-dimensional depthwise convolution unit, a selective state space unit, a gating unit, an output projection unit, and a residual connection. After layer normalization and input projection, the video tags are divided into content branches and gating branches. The content branches are processed by one-dimensional depthwise convolution and SiLU and then enter the selective state space unit. The selective state space unit generates the current hidden state based on the current content and the previous hidden state. The gating branch adjusts the output of the current state. The output projection restores the gating state output to the input dimension of the coding block and adds it to the input of the coding block to form the input of the next coding block.

[0091] The improved VideoMamba encoding layer first processes all spatial video tags corresponding to the anchor time index. It then copies the last hidden state corresponding to the last spatial video tag to a common initial hidden state for both reverse-time continuation and forward-time continuation. Reverse-time continuation processes video tags from the time slice before the anchor point toward the start of the video, while forward-time continuation processes video tags from the time slice after the anchor point toward the end of the video. The hidden states for both directions are saved separately, and the relationship continuity results of the same time slice are received. When the relationship remains continuous, the current video tag is allowed to write to the hidden state of the corresponding direction. When the relationship is continuously interrupted, the last continuous hidden state of the corresponding direction is fixed, and the writing of video tags outside the stop position is disabled.

[0092] The boundary lock state consists of the reverse-time boundary lock hidden state, the forward-time boundary lock hidden state, the stop positions in two directions, the hidden states of each time slice within the lock interval, and a state update mask of the same length as the video marker sequence. The reverse-time boundary lock hidden state and the forward-time boundary lock hidden state are respectively taken from the last consecutive hidden states saved when stopping in the corresponding direction. Each position in the state update mask records whether the corresponding video marker is allowed to be written. The normalization and prediction layer performs layer normalization on the hidden states within the lock interval, converts the two stop positions into action start frames and action end frames, forms the initial action interval, and retains the boundary lock state.

[0093] The improved VideoMamba model is fine-tuned using labeled sports video segments. Training samples are labeled with the action state, successor state, and successor state anchor point corresponding to the moving subject and the object of action. The video tokenization layer, position embedding layer, and VideoMamba encoding layer load the pre-trained parameters of the original VideoMamba model. The normalization and prediction layers output the action state or successor state based on the normalized hidden state of each time slice. During training, cross-entropy is used to calculate the difference between the time slice state output and the labeled state, and AdamW is used to update the model parameters. If the validation loss does not decrease after multiple consecutive rounds, the model parameters with the lowest validation loss are saved and training ends. Anchor point index mapping, bidirectional state recording, state update mask control, and stopping position determination are executed according to the aforementioned processing logic and do not participate in gradient updates.

[0094] Compared to the original VideoMamba model, the improvements focus on the state starting point of selective state space computation, bidirectional propagation topology, and the writable range of hidden states. The original model uses both ends of the video sequence as the scanning starting point and aggregates information within the complete sequence. The improved model uses subsequent state anchors to form a unified state benchmark, reconstructing the complete bidirectional scan into a state continuation that unfolds independently from the same anchor point to both sides of time, and directly converting the continuous results of behavioral relationships into control variables for the writable range of hidden states. This change allows the action start boundary and action end boundary to share the same semantic benchmark, while allowing the hidden states in each direction to stop expanding at the closed position of the action relationship. Movement outside the boundary cannot continue to rewrite the state inside the locked interval. From the perspective of state propagation within the model, it weakens the feature crosstalk caused by adjacent actions, camera movements, and short-term trajectory interruptions, transforming the VideoMamba coding layer from a complete video feature aggregator into a bidirectional boundary locator capable of generating action intervals around the state transitions of the subject and the object of action.

[0095] In this embodiment, the formation of the boundary locking state specifically includes:

[0096] The processing of boundary locking states is located in the selective state space units of each coding block within the VideoMamba coding layer. Each coding block establishes a reverse-time state record and a forward-time state record. A single state record stores the current time index, the last consecutive time index, the current hidden state, the state update mask, and the consecutive interruption count. After the current time slice is processed, the hidden state corresponding to the last spatial video marker is written into the state record, which serves as the previous hidden state to be read when the adjacent time slice continues to be calculated.

[0097] The video markers within the time slice corresponding to the subsequent state anchor point enter the selective state space unit in a fixed spatial order from top to bottom and from left to right. The current video marker is processed by input projection, one-dimensional depth convolution, and SiLU to generate the current input content. The selective state space unit uses the current input content to control the retention ratio of the previous hidden state and generates the current input write amount. Then, the retained previous hidden state is superimposed with the current input write amount to form the current hidden state. The current hidden state is gated, output projected, and residual connected to form the encoding result of the current spatial position. Each video marker within the time slice reads the hidden state updated by the previous spatial position in sequence. After the last spatial video marker is processed, the anchor point end hidden state is obtained. The anchor point end hidden state is copied into two copies with the same value and dimension and written into the reverse-time state record and the forward-time state record respectively, as the common initial hidden state in the two time directions.

[0098] The reverse-time state continuation starts from the time slice before the anchor point. Each time, the time index is decremented by 1 and all spatial video markers in the corresponding time slice are read. Within each time slice, the spatial order continues to be processed from the upper left corner to the lower right corner. The first spatial video marker reads the current hidden state in the reverse-time state record, and the hidden state generated by the last spatial video marker is used as the end hidden state of the reverse-time time slice of this time slice. When the sports behavior relationship maintains the action state and the trajectory number of the moving subject and the trajectory number of the object are both consecutive with the corresponding anchor point, the end hidden state of the reverse-time time slice is written into the state record, the current time index is written as the last consecutive time index, and then the time index is decremented by 1 to enter the previous time slice.

[0099] The sequential state continuation starts from the time slice after the anchor point. Each time, the time index is incremented by 1 and all spatial video markers in the corresponding time slice are read. The spatial order within each time slice is the same as that of the anchor point time slice. The first spatial video marker reads the current hidden state in the sequential state record, and the hidden state generated by the last spatial video marker is used as the end hidden state of the sequential time slice of this time slice. When the sports behavior relationship maintains the successor state and the trajectory number of the moving subject and the trajectory number of the object are both consecutive with the corresponding combination of the anchor point, the end hidden state of the sequential time slice is written into the state record, the current time index is written as the last consecutive time index, and then the time index is incremented by 1 to enter the next time slice. The reverse sequential state record and the sequential state record are updated separately. The hidden state generated on one side is not written into the state record on the other side.

[0100] The continuous relationship results are mapped to the model time slice through the video frame number and the time step of the video tokenization layer. When the trajectory number of the moving subject, the trajectory number of the object, and the behavior relationship category of all valid video frames in the same time slice are consistent, the current time slice is recorded as continuous. When there are invalid video frames in the time slice, the behavior relationship corresponding to the valid video frames is used. When the trajectory number of the moving subject or the trajectory number of the object changes, the current time slice is recorded as interrupted. The action state is verified in reverse time sequence and the subsequent state is verified in forward time sequence. During the short-term occlusion of the target, the behavior relationship is continued to be calculated using the trajectory prediction position.

[0101] When a sports behavior relationship is interrupted for the first time in any direction, the last consecutive hidden state before the interruption is copied to the lock candidate cache, the hidden state generated within the interruption time slice is written to the temporary state cache, and the number of consecutive interruption time slices is accumulated. The upper limit of consecutive interruption time slices is determined by the annotation verification set. The number of erroneous truncations caused by short-term occlusion and the number of erroneous continuations caused by misconnection of adjacent actions are counted for different number of time slices. The number of time slices corresponding to the lowest sum of the two counts is taken as the upper limit of consecutive interruption time slices. When the sports behavior relationship is restored before reaching the upper limit of consecutive interruption time slices and the trajectory number of the moving subject and the trajectory number of the object remain continuous, the hidden state in the temporary state cache is written to the corresponding direction state record in chronological order, and the continuation is continued with the last hidden state in the temporary state cache.

[0102] When the continuous interruption of sports behavior reaches the upper limit of the continuous interruption time slice, the hidden state in the temporary state buffer is deleted, the last continuous hidden state saved in the lock candidate buffer is restored to the current hidden state in the corresponding direction, the last continuous time index before the interruption is determined as the state continuation stop position, and the state update mask corresponding to the video marker outside the stop position is written to 0; when the state update mask is 1, the video marker continues to update the hidden state in the corresponding direction and enters the encoding output in the lock interval; when the state update mask is 0, the corresponding video marker stops writing hidden states and stops participating in the bidirectional state merging in the lock interval.

[0103] The first coding block determines the reverse-time stop position, the forward-time stop position, and the state update masks in both directions based on the relationship continuity result, and passes the stop position and state update mask to the remaining coding blocks; the remaining coding blocks perform bidirectional state continuation with the common initial hidden state formed by the anchor time slice of their respective coding blocks. The video tag encoding result output by the previous coding block is used as the input of the next coding block. All coding blocks share the same boundary range and update their own hidden states respectively. Video tags outside the boundary are continuously kept write-blocked in deep coding.

[0104] The bidirectional state merging process first reads the time-slice hidden states stored between the reverse-time stop position and the anchor point. Since the reverse-time states are generated in descending order of time index, the storage order needs to be reversed to the normal frame time sequence. Then, the forward-time-slice hidden states between the anchor point and the forward-time stop position are read. When the anchor point hidden state appears at the end of the reverse-time sequence and the beginning of the forward-time sequence at the same time, only one copy of the reverse-time sequence is retained. The forward-time sequence is connected starting from the time slice after the anchor point. The connected hidden state sequence, the reverse-time boundary-locked hidden state, the forward-time boundary-locked hidden state, the two stop positions and the corresponding state update mask together form the boundary-locked state, and enter the normalization and prediction layer to determine the initial action range.

[0105] If the video continues in reverse chronological order and remains in motion until the start time slice of the sports video, the start time slice is taken as the stop position for reverse chronological order. If the video continues in forward chronological order and remains in subsequent state until the end time slice of the sports video, the end time slice is taken as the stop position for forward chronological order. If the source video frame of the time slice where the subsequent state anchor point is located is marked as an invalid frame or a complete time slice video tag is not generated, the initialization of the current anchor point is terminated and the next stable transition time in the state transition time sequence is read. If no valid anchor point video tag can be generated for any of the stable transition times, the current video segment is transferred to manual review.

[0106] Compared to the original VideoMamba, which establishes independent forward and reverse state chains from both ends of a complete video tag sequence, this implementation unifies the state source of selective state space computation to the subsequent state anchor point and changes the action boundary from the temporal position output by the prediction layer to the state writing stop position inside the coding layer. The two temporal directions start from the same end hidden state but maintain independent state records, solving the boundary reference deviation caused by the different semantics at the start of scanning on both sides. The state update mask further transforms the continuous range of sports behavior relationships into the effective writing range of state space units, so that video tags outside the boundary lose the propagation path to change the locked hidden state. The action start boundary and action end boundary change from the same action state to closing on both sides. The VideoMamba coding layer changes from a bidirectional feature convergence process covering the complete sequence to a bidirectional boundary localization process that establishes a finite state domain around the same subsequent state anchor point, reducing the interference of adjacent actions, irrelevant movements and short-term occlusion on the action interval from the internal state propagation level.

[0107] In this embodiment, S5 specifically includes:

[0108] The start and end boundaries of the initial action interval serve as the starting points for retrieving candidate continuation relationships. The sports behavior relationship sequence is extracted along the reverse time sequence from the start boundary and along the forward time sequence from the end boundary. The reading time of continuous relationships within the boundary and the retrieval time of continuation relationships outside the boundary are determined based on the duration distribution of correct continuation relationships in the labeled training set and converted into the corresponding number of frames according to the current sports video frame rate. If there are insufficient video frames between the video start point or video end point and the corresponding boundary, the actual existing video frames are read.

[0109] Least square straight line fitting is performed on the trajectory of the moving subject and the trajectory of the object in the continuous relationship within the boundary. The position change direction obtained by fitting is used as the basis for extrapolation of the position outside the boundary. At the same time, the scale change is formed according to the width and height changes of adjacent detection areas. Thus, the predicted trajectory of the moving subject, the predicted trajectory of the object, and the predicted relative displacement between the two are generated.

[0110] In the boundary-bound continuation relationship, each moving subject and the object of action are compared frame by frame according to the corresponding time direction. The normalized distance between the center of the moving subject and the center of the predicted subject is taken as the subject position deviation, the normalized distance between the center of the object of action and the center of the predicted object is taken as the object position deviation, and the normalized vector difference between the candidate relative displacement and the predicted relative displacement is taken as the relationship deviation. The upper limit of the judgment of the three types of deviations is taken as the 95th percentile value of the deviation corresponding to the correct continuation sample in the labeled training set. The number of candidate continuation stable frames is determined by the labeled validation set. The number of consecutive frames corresponding to the lowest sum of the number of misconnections and the number of missed connections is taken. When the candidate combination continuously reaches the candidate continuation stable frame number and satisfies the judgment of the three types of deviations, it is determined as a candidate continuation relationship.

[0111] The candidate succession relationship accumulates the subject position change deviation, subject scale change deviation, object position change deviation, and object scale change deviation along the succession time sequence, and then forms the subject cumulative deviation and object cumulative deviation respectively. The weights of the subject cumulative deviation and object cumulative deviation are both in the range of 0 to 1 and the sum of the two is 1. The weight combination with the lowest sum of the number of switching of the identity of the moving subject and the number of misconnection of the trajectory of the acting object in the verification set is taken as the weight for calculating the joint association cost, thus forming the joint association cost corresponding to the candidate succession relationship.

[0112] The candidate succession relationship set with joint association cost is established on the start boundary side and the end boundary side respectively. Each candidate succession relationship is associated with the corresponding time direction, the trajectory of the moving subject, the trajectory of the object of action, the succession time range and the joint association cost. After being sorted in order of joint association cost from low to high, it is passed into the joint succession scheme generation process. When no candidate succession relationship is formed on one side, the initial action boundary of the corresponding side is retained and the candidate relationship on the other side is processed. When no candidate succession relationship is formed on both sides, the initial action interval is retained and the current candidate succession processing is ended.

[0113] In this embodiment, S6 specifically includes:

[0114] The candidate continuation relationship set with joint association cost summarizes the candidate continuation relationships on the start boundary side and the end boundary side. The combination of the moving subject and the object that need to be continued at the start boundary of the initial action interval is treated as the row of the reverse time sequence matrix, and the combination of the moving subject and the object that need to be continued at the end boundary is treated as the row of the forward time sequence matrix. The combination of the moving subject and the object in the candidate continuation relationship is treated as the column of the matrix and the corresponding time direction is recorded. When the same moving subject and the object appear on both sides at the same time, they are still treated as a combination within two intervals with different directions.

[0115] Each row of the joint association cost matrix corresponds to a combination of a moving subject and an object within an interval with a time direction, and each column corresponds to a candidate combination of a moving subject and an object. The joint association cost between two combinations is written at the intersection of the matrix rows and columns. When the moving subject category, object category, or time direction is inconsistent, an unallocable cost is written. The unallocable cost is the maximum value of the effective matrix elements plus 1, and then multiplied by the sum of the number of rows and columns of the matrix.

[0116] When the number of candidate combinations is greater than the number of combinations within the interval, the remaining candidate combinations are allowed to remain unassigned. When the number of candidate combinations is less than the number of combinations within the interval, virtual candidate combinations are added until the number of columns is the same as the number of rows. The cost of the virtual candidate combinations is the non-assignable cost. When the assignment result includes virtual candidate combinations, the corresponding direction maintains the initial action boundary, and the remaining valid associations continue to enter the joint continuation scheme.

[0117] The Murty algorithm first calls the Hungarian allocation to find the lowest total cost allocation in the joint association cost matrix and writes the lowest total cost allocation into a priority queue. Then, it fixes the previous associations according to the matrix row order and excludes the current associations to construct the sub-allocation problem. The sub-allocation results are written into the priority queue in order of total cost from low to high. After each allocation of the head of the queue, the algorithm continues to split. For allocations with the same set of association pairs, only the one with the lower total cost is retained. The upper limit of the number of joint succession schemes is the minimum number of schemes when the coverage of correct succession schemes in the validation set reaches 95%. The algorithm ends when the upper limit of the number of joint succession schemes is reached or the priority queue is empty.

[0118] In each joint continuation scheme, candidate continuation relationships are read from the initial action interval boundary outwards. The first frame that is adjacent to the boundary and continuously reaches the candidate continuation stable frame number and satisfies the judgment of subject position deviation, object position deviation, and relationship deviation is determined as the state writing recovery position. The recovery position corresponds to the video frame number minus the starting frame number of the model input video segment, then divided by the time step of the video tokenization layer and the integer quotient is used to obtain the recovery time index. The moving subject detection area and the action object detection area are mapped to the spatial grid of the video tokenization layer. The video tokens whose grid center falls into any detection area are written into the recovery object set. When the detection area does not cover the grid center, the video token where the center of the detection area is located is used.

[0119] The candidate state writing recovery scheme saves the recovery direction, state writing recovery position, recovery time index, video tag writing sequence, and joint association cost. When the joint continuation scheme selects candidate relationships on both the start and end boundary sides, it forms a bidirectional recovery scheme. When only one side of the valid candidate relationship is selected, it forms a unidirectional recovery scheme. The output of the Murty algorithm is limited to the candidate state writing recovery scheme. The hidden state update is completed by the local state continuation recovery process.

[0120] Compared to the original Murty algorithm, which only generates multiple allocation results sorted by total cost for the cost matrix, this implementation modifies the allocation object, cost meaning, and output destination of the algorithm to be geared towards action boundary recovery. It combines the moving subject with time direction and the target object as a joint allocation object, writes the geometric deviations of the two types of targets accumulated along the sequential time into the same joint association cost, and further converts each allocation result into a candidate state writing recovery scheme that includes recovery direction, state writing recovery position, and video tag writing sequence. This allows the multiple association results generated by the Murty algorithm to be applied to the reverse-sequence boundary locking hidden state and the forward-sequence boundary locking hidden state respectively. The improved VideoMamba model verifies the actual correction effect of each allocation result on the action boundary through local state continuation, thereby transforming the original multi-solution allocation process into a multi-path recovery entry point for boundary locking state.

[0121] In this embodiment, S7 specifically includes:

[0122] After the candidate state write recovery scheme enters the VideoMamba encoding layer, it first reads the reverse-time boundary lock hidden state and the forward-time boundary lock hidden state saved in the boundary lock state. When the scheme includes the reverse-time recovery direction, it copies the reverse-time boundary lock hidden state; when it includes the forward-time recovery direction, it copies the forward-time boundary lock hidden state. Two recovery copies are established for bidirectional recovery schemes. Each candidate state write recovery scheme starts independently from the original saved value in the boundary lock state. The hidden state generated by the current scheme is only written to the recovery copy of this scheme. The next scheme rereads the original boundary lock state.

[0123] Video marker write recovery is performed in the state update mask of the corresponding direction. First, the direction mask saved by the boundary lock state is copied. Then, the video marker write sequence in the candidate state write recovery scheme is read. The positions of the recovery position and the positions of the video marker write sequence in the recovery direction are changed from 0 to 1. The start boundary side opens for writing from the recovery position in the direction of decreasing time index, and the end boundary side opens for writing from the recovery position in the direction of increasing time index. The video markers of the moving subject and the target of action in the same time slice are all kept with a mask value of 1. The other spatial video markers and the video markers of the moving subject and the target of action that are not selected in the joint continuation scheme are kept with a mask value of 0. This forms a local write range that only covers the current candidate continuation path.

[0124] Local state continuation is completed within the selective state space unit of the VideoMamba coding layer. Each recovery time slice traverses all spatial video markers in the current time slice in spatial order from top left to bottom right. Video markers with a state update mask of 0 retain their current hidden state and do not generate any writes. Video markers with a state update mask of 1 use the previously disclosed input projection, one-dimensional depthwise convolution, selective state update, gating, and residual path to generate their current hidden state. After a single time slice is processed, the hidden state corresponding to the last valid write video marker is taken as the hidden state at the end of the time slice and passed to the next time slice in the corresponding time direction.

[0125] The reverse-temporal local continuation process progresses from the starting boundary side to an earlier time slice. After each time slice's hidden state update, the trajectory of the moving subject, the trajectory of the object being acted upon, and the relative geometric changes within the same time slice are verified. If the trajectory number of the moving subject remains continuous, the subject's identity is confirmed to be continuous. When the trajectory number is re-established due to occlusion, the appearance consistency judgment in the cross-frame association stage is used, and the upper limit of the subject's position deviation determined in the candidate continuation relationship screening stage is used. The continuity of the object's trajectory is determined by the upper limit of the object's position deviation, and the continuity of the relative geometric changes is determined by the upper limit of the relationship deviation. When all three continuity judgments are satisfied and the behavioral relationship remains in the action state, the hidden state at the end of the time slice is saved and the previous time slice is processed.

[0126] The sequential local continuation progresses from the end boundary side of the recovery position to the later time slice. The identity of the moving subject, the trajectory of the object of action, and the relative geometric changes follow the judgment method of the reverse sequential local continuation. When all three consecutive judgments are satisfied and the behavioral relationship maintains the subsequent state, the hidden state at the end of the time slice is saved and the next time slice is processed. The reverse sequential recovery copy and the sequential recovery copy are updated separately. The hidden state on one side is not written to the recovery copy on the other side.

[0127] When a candidate continuation relationship is interrupted for the first time, the last consecutive hidden state and time index before the interruption are written to a temporary lock location, the hidden state generated by the interruption time slice is written to a temporary cache, and the number of consecutive interruption time slices is accumulated. When the sports behavior relationship is restored before reaching the upper limit of consecutive interruption time slices and the continuous judgment of the moving subject and the object of action is satisfied again, the hidden state in the temporary cache is written to the recovery copy in chronological order and the continuation continues. When the consecutive interruption reaches the upper limit of consecutive interruption time slices, the temporary cache is deleted, the last consecutive hidden state before the interruption is restored to the current direction end hidden state, and the local continuation of the current candidate scheme is stopped. When the candidate continuation relationship reaches the end of the record, the state is updated with the last consecutive time slice at the end.

[0128] After the reverse-temporal local continuation stops, the start boundary time index of the initial action interval is replaced with the last continuous time index. After the forward-temporal local continuation stops, the end boundary time index of the initial action interval is replaced with the last continuous time index. The time index is multiplied by the time step of the video tokenization layer and added to the start frame number of the input video segment to obtain the corrected boundary frame. When the current candidate scheme completes continuous continuation on only one side, only the boundary on the corresponding side is corrected. When both sides complete continuous continuation, the start boundary and the end boundary are corrected respectively. If the local continuation does not cross the initial boundary or a continuous hidden state is not formed after the first open write, the initial action interval is retained.

[0129] Each candidate state that forms a continuous local connection is written into the recovery scheme to generate a candidate action interval. The candidate action interval retains the corresponding joint association cost and enters the action state and subsequent state continuity verification. The initial action interval is always retained as the basic candidate interval. When no effective local connection is formed on either side, only the initial action interval is output, thus forming a candidate action interval set that includes the initial action interval and the effective correction interval.

[0130] In this embodiment, S8 specifically includes:

[0131] The continuity verification of the candidate action interval is based on the time boundary of the subsequent state anchor point. The action state is verified in reverse time from the start boundary to the subsequent state anchor point, and the subsequent state is verified in forward time from the subsequent state anchor point to the end boundary. The identity of the moving subject, the trajectory of the object of action, and the relative geometric changes are judged using the local state continuation stage. The short-term interruption recovery adopts the upper limit of the continuous interruption time slice determined in the boundary locking state formation stage.

[0132] When the reverse-time verification continuously reaches the action start boundary and the forward-time verification continuously reaches the action end boundary, the current candidate action interval is retained. When a continuous interruption occurs in any direction before reaching the corresponding boundary, the current candidate action interval is deleted. The initial action interval is retained as the result when no candidate action interval passes the verification.

[0133] When multiple candidate action intervals pass the verification, the interval with the lowest joint association cost is selected. When the joint association costs are the same, the interval with the smaller total frame difference between the start boundary and the end boundary relative to the initial action interval is selected. If they are still the same, the interval with the earlier output order of the Murty algorithm is selected. When no candidate action interval passes the verification, the initial action interval is output. The start frame, end frame and corresponding timestamp of the final action interval form the action localization result of the sports video.

[0134] Example 1:

[0135] To verify the feasibility of this invention in practice, it was applied to an indoor basketball game video analysis system to automatically locate shooting actions in the game video. The video captures scenes that occur consecutively, including camera panning, cross-blocking by athletes, dribbling breakthroughs, passing, shooting, and rebounding. Traditional action localization methods mainly determine action boundaries based on changes in athlete posture or temporal characteristics within the entire video range. When the basketball is briefly obscured by an athlete's body or a defender, dribbling, passing, and shooting are easily linked as the same action, or the shooting action is interrupted due to the reappearance of the basketball, resulting in delayed start boundaries, advanced end boundaries, and erroneous merging of adjacent actions.

[0136] The input video resolution is 1920×1080 pixels, and the frame rate is 30 frames per second. First, the frame order is corrected according to the video frame timestamps. Global motion transformation is estimated and image registration is completed through feature matching of adjacent frames. Motion regions are extracted from the registration results. A trained object detection network is used to identify the athlete and the basketball. The cross-frame target trajectory is associated with appearance features and motion continuity. A sequence of sports behavior relationships is formed based on the relative position and motion changes of the athlete and the basketball. When the basketball transitions from a stable tethered state to a free-flying state, the subsequent state anchor point is determined. The sports video is then transformed using 3D convolution. Instead of video tagging, the VideoMamba model is improved to form a common initial hidden state at the anchor point. The start boundary of the shooting preparation action is located in reverse time and the end boundary of the follow-through action is located in forward time. When the behavior relationship is continuously interrupted, the writing of video tags outside the boundary is stopped. When the trajectory of the basketball is interrupted due to occlusion, the candidate succession relationship between the athlete and the basketball is searched outside the initial action interval. The Murty algorithm is used to generate a joint succession scheme. The hidden state is locked from the corresponding boundary to restore the local state succession. After the continuity of the action state and the successor state is verified, the final shooting action interval is output.

[0137] To verify the feasibility of this invention in the aforementioned scenario, 1500 game videos containing shooting actions and adjacent interfering actions were selected as samples. These were divided into 1000 training videos, 200 verification videos, and 300 test videos. The start boundary, subsequent state anchor point, and end boundary of 428 shooting actions in the test videos were manually labeled frame by frame. The traditional temporal convolutional action localization method, the original VideoMamba method, and the method of this invention were compared. The action detection rate, average temporal intersection-over-union ratio, start boundary error, end boundary error, and adjacent action mismerging rate were statistically analyzed. The statistical results are shown in Table 1 below.

[0138] Table 1 Comparison of Boundary Positioning Effects in Basketball Shooting Motion

[0139] Evaluation indicators Traditional temporal convolution action localization method Original VideoMamba method Method of the present invention Action detection rate / % 88.6 92.3 96.5 Average Time Intersection and Union Ratio 0.72 0.79 0.87 Start boundary average error / frame 7.8 5.6 3.1 End boundary average error / frame 9.6 6.9 3.6 Adjacent action mis-merging rate / % 10.7 6.8 2.8

[0140] As shown in Table 1, the method of this invention outperforms both the traditional temporal convolutional action localization method and the original VideoMamba method in five metrics: action detection rate, average temporal intersection-over-union ratio (MTU), average error at the start boundary, average error at the end boundary, and mis-merging rate of adjacent actions. The traditional temporal convolutional action localization method has an action detection rate of 88.6%, an MTU of 0.72, average errors at the two boundaries of 7.8 frames and 9.6 frames respectively, and a mis-merging rate of 10.7% for adjacent actions, indicating that conventional temporal feature extraction is easily affected by camera movement. The original VideoMamba method improves the action detection rate to 92.3%, the MTU to 0.79, reduces the average errors at the two boundaries to 5.6 frames and 6.9 frames respectively, and reduces the mis-merging rate of adjacent actions to 6.8%, but the latent states transmitted within the sequence may mix in information from outside the interval. The method of this invention achieves an action detection rate of 96.5%, an average temporal intersection-over-union ratio of 0.87, and reduces the average errors of the two boundaries to 3.1 frames and 3.6 frames, respectively, while reducing the false merging rate of adjacent actions to 2.8%. The action detection rate and average temporal intersection-over-union ratio reflect the proportion of action localization and the degree of overlap between intervals, respectively, while the average boundary error and the false merging rate of adjacent actions reflect the boundary offset and the incorrect connection of actions, respectively. This demonstrates that the present invention can simultaneously address action detection, boundary determination, and continuous action separation.

[0141] This embodiment performs frame timing correction, video frame registration, and motion region extraction on sports videos. It then detects and correlates athletes and basketballs across frames, constructing a sequence of sports behavior relationships based on their relative geometric changes. The moment the basketball transitions from a sturdy, held state to a released, flying state is determined as the subsequent state anchor point. The improved VideoMamba model forms a common initial hidden state at the anchor point moment, updating the hidden state along both counter-clockwise and clockwise sequences. When the behavior relationship is continuously interrupted, the last consecutive hidden state is retained, while video marker writing outside the boundary and participation in bidirectional state merging are stopped, reducing interference from adjacent actions. For relationship breaks caused by short-term basketball occlusion, athlete crossing, and trajectory interruption, the Murty algorithm generates candidate continuation schemes based on the joint association cost of the athlete-basketball combination. It then restores local continuation by locking the hidden state at the corresponding boundary and determines the final action interval based on the continuity of state transitions. This processing method can reduce the boundary offset of shooting actions, decrease erroneous connections between consecutive actions, and improve positioning stability under complex motion conditions.

[0142] The above are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for motion localization in sports videos based on behavioral feature recognition, characterized in that, Includes the following steps: S1. Acquire sports video, correct frame timing and extract motion regions to form sports video timing data; S2. Track the moving subjects and objects in the time-series data of sports videos, and form a sequence of sports behavior relationships based on cross-frame behavioral characteristics; S3. Detect the moment when the sequence of sports behavior relationships transitions from a stable action state to a subsequent state, and determine the anchor point of the subsequent state. S4. Convert sports video time series data into video tags and connect them to the improved VideoMamba model. At the time corresponding to the subsequent state anchor point, a common initial hidden state is formed and the hidden state is updated along the reverse and forward time order. At the point where the behavior relationship is continuously interrupted, the last continuous hidden state is locked and the writing of video tags outside the boundary and bidirectional merging are blocked to form the initial action interval and boundary locking state. S5. Retrieve candidate relationships that have geometric continuity between the behavior relationships outside and inside the initial action interval, forming a set of candidate continuity relationships with joint association costs; S6. Use the Murty algorithm to process the candidate succession relationship set, generate a joint succession scheme set according to the joint association cost, and convert it into a candidate state to write into the recovery scheme set. S7. Write the corresponding video marker to the hidden state at the recovery position according to the candidate state writing recovery scheme one by one, and recover the local continuation from the hidden state of the boundary locked state to form a candidate action interval set containing the initial action interval. S8. Determine the final action range based on the continuity of the action state transitioning to the subsequent state according to the behavioral relationship within the candidate action range, thus forming the sports video action localization result.

2. The sports video motion localization method based on behavioral feature recognition according to claim 1, characterized in that, S1 specifically includes: S11. Read the timestamps of video frames in the sports video, rearrange the video frames according to the timestamps, and form a time-corrected frame sequence. S12. Estimate the global motion transformation between adjacent video frames in the temporal correction frame sequence, register adjacent video frames and extract the inter-frame change region after registration to form a motion region sequence. S13. Associate the motion region sequence with the timing correction frame sequence according to the video frame order to form sports video timing data.

3. The sports video motion localization method based on behavioral feature recognition according to claim 1, characterized in that, S2 specifically includes: S21. Perform target detection on the motion region in the sports video time series data, distinguish between the moving subject and the object of action, extract geometric features, and form motion target detection results; S22. Based on the appearance consistency and motion continuity in the moving target detection results, perform target association, calculate geometric changes according to the associated trajectory, and form cross-frame behavior features; S23. Calculate the relative geometric changes between the moving subject and the object of action based on the cross-frame behavior characteristics, and arrange the relative geometric changes according to the time sequence of video frames to form a sports behavior relationship sequence.

4. The sports video motion localization method based on behavioral feature recognition according to claim 1, characterized in that, S3 specifically includes: S31. Compare the behavioral relationships of adjacent video frames along the sports behavior relationship sequence to locate the transition moment from the action state to the subsequent state, and form a state transition moment sequence. S32. Verify the behavioral relationship in chronological order starting from the transition time in the state transition time sequence. When the behavioral relationship returns to the action state, proceed to the next transition time. When the behavioral relationship maintains the successor state in consecutive video frames, determine the corresponding transition time as the successor state anchor point.

5. The sports video motion localization method based on behavioral feature recognition according to claim 1, characterized in that, The improved VideoMamba model includes a video tokenization layer, a location embedding layer, a VideoMamba coding layer, and a normalization and prediction layer connected in sequence. The video tagging layer uses three-dimensional convolution to extract local video features along the temporal and spatial dimensions of sports video time-series data, and unfolds them into a video tagging sequence according to the frame time sequence. The location embedding layer superimposes location codes corresponding to frame temporal and spatial locations onto the video tag sequence, and determines the video tag corresponding to the time of the subsequent state anchor point based on the frame temporal location codes. The VideoMamba coding layer performs selective state space calculation according to the spatial arrangement of video markers within the anchor point time. It uses the end hidden state as the common initial hidden state, updates the hidden state along the reverse and forward time order, retains the last continuous hidden state when the sports behavior relationship is continuously interrupted, stops updating the hidden state of the corresponding direction of video markers outside the boundary, and blocks video markers outside the boundary from participating in the bidirectional state merging within the locked interval, forming a boundary locked state. The normalization and prediction layer normalizes the hidden states within the locked interval, determines the action boundary based on the state continuation stop position in reverse and forward order, forms the initial action interval, and retains the boundary locked state.

6. The sports video motion localization method based on behavioral feature recognition according to claim 5, characterized in that, The formation of the boundary locking state specifically includes: Within the VideoMamba coding layer, selective state space updates are performed according to the video tag space order corresponding to the time of the successor state anchor point, setting the terminal hidden state corresponding to the last video tag as the initial hidden state of reverse-time continuation and forward-time continuation, respectively. Reverse chronological continuation reads video markers and updates the reverse chronological hidden state moment by moment from the subsequent state anchor point; forward chronological continuation reads video markers and updates the forward chronological hidden state moment by moment from the subsequent state anchor point. When the sports behavior relationship remains continuous, the current hidden state is passed to the next moment for continued updating. When the sports behavior relationship is not recovered in consecutive video frames, the last consecutive hidden state before the interruption is retained and the reading of the video marker outside the boundary in the corresponding direction is stopped. The reverse-time hidden states from the reverse-time stop position to the subsequent state anchor point are arranged in reverse frame time order and merged with the forward-time hidden states from the subsequent state anchor point to the forward-time stop position according to frame time order. The video marker outside the boundary is blocked from being written into the corresponding direction hidden state, forming a boundary locked state.

7. The sports video motion localization method based on behavioral feature recognition according to claim 1, characterized in that, S5 specifically includes: S51. Starting from the two sides of the initial action interval, extract the sports behavior relationship sequence along the corresponding time direction to form a continuous relationship within the boundary and a relationship to be continued outside the boundary. S52. Compare the geometric motion changes of the relationship to be continued outside the boundary and the continuous relationship inside the boundary frame by frame, and filter out candidate relationships that maintain motion continuity. S53. Calculate the geometric deviations between the moving subject and the object of action in the candidate succession relationship and the corresponding moving target within the boundary. Accumulate the geometric deviations of the same candidate succession relationship along the succession time sequence to form a joint association cost, thus forming a set of candidate succession relationships with joint association costs.

8. The sports video motion localization method based on behavioral feature recognition according to claim 1, characterized in that, S6 specifically includes: S61. Using the combination of the moving subject and the object of action within the initial action interval as rows and the combination of the moving subject and the object of action in the candidate succession relationship as columns, write the joint association cost between the combinations into the corresponding matrix elements to form a joint association cost matrix. S62. The Murty algorithm is used to find the minimum cost allocation of the joint association cost matrix. The preceding associations are fixed in sequence and the current association is excluded to construct the sub-allocation problem, forming a set of joint succession schemes that increase in total cost. S63. Determine the start time of the candidate connection relationship corresponding to the joint connection scheme as the state write recovery position, and determine the corresponding video marker as the recovery write object to form a set of candidate state write recovery schemes.

9. The sports video motion localization method based on behavioral feature recognition according to claim 1, characterized in that, Specifically, S7 includes: S71. When the recovery position is outside the start boundary of the initial action interval, the reverse timing boundary lock is selected as the hidden state; when it is outside the end boundary, the forward timing boundary lock is selected as the hidden state, and the write blocking of the video marker not selected in the joint continuation scheme is maintained. S72. Take the selected boundary-locked hidden state as the initial state, and continue to process the video tags corresponding to the candidate continuation relationship along the corresponding time direction from the self-recovery position. When the behavior relationship is continuous, write the updated hidden state at each time step. S73. When the candidate continuation relationship ends or the behavior relationship is interrupted again, the state continuation is stopped. The corresponding action boundary is corrected at the last continuous moment. If no continuous continuation is formed, the initial action interval is retained to form a candidate action interval set.

10. A method for sports video motion localization based on behavioral feature recognition according to claim 1, characterized in that, S8 specifically includes: S81. Using the subsequent state anchor point as the boundary, verify whether the behavioral relationship within each candidate action interval is continuously maintained along the reverse time sequence, and verify whether the behavioral relationship is continuously maintained along the forward time sequence. S82. Retain candidate action intervals that are continuously verified in reverse order to the action start boundary and continuously verified in forward order to the action end boundary, and exclude candidate action intervals that are interrupted in either direction. S83. Select the interval with the lowest corresponding joint association cost from the retained candidate action intervals as the final action interval. If no candidate action interval passes the verification, retain the initial action interval to form the sports video action localization result.