End-to-end time sequence action positioning method and system fusing boundary focusing and relative boundary attention

Abstract

The invention belongs to the technical field of video action positioning, and particularly discloses an end-to-end time sequence action positioning method and system fusing boundary focusing and relative boundary attention, and the method comprises the following steps: collecting unclipped video data, inputting the data into a feature encoder, and extracting fragment-level video sequence features; setting an encoder module, introducing a boundary focusing module BFM, predicting and sequencing a boundary perception score for each segment-level video sequence feature, and selectively encoding the video sequence feature with the boundary perception score greater than a threshold value; setting a decoder module, introducing a relative boundary attention module RBAM, and iteratively refining an action interval of a video sequence feature in a multi-layer decoder; and respectively predicting a category label and a time boundary of each candidate action through the classification head and the regression head. By adopting the technical scheme, the model accurately distributes attention to related targets, more positive samples are effectively captured, redundant overlapping is reduced, and the accuracy and efficiency of time sequence action detection in a complex video scene are improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of video action localization, and relates to an end-to-end temporal action localization method and system fusing boundary focus and relative boundary attention. BACKGROUND

[0002] With the explosive growth of online video data, video understanding has become one of the hot directions of computer vision. As an important part of video understanding, temporal action localization (TAL) needs to predict the action semantic label and the accurate time boundary (start and end time) in a long-time uncropped video at the same time.

[0003] Recently, inspired by the Detection Transformer (DETR), the Transformer method has made significant progress in TAL with an end-to-end architecture and efficient temporal modeling. A large number of variants use a two-stage strategy to further improve accuracy: in the first stage, the encoder densely processes the feature map and selects a set of candidate proposals; in the second stage, the decoder makes predictions through more refined cross-attention.

[0004] However, the first stage often produces a large number of inaccurate candidate boxes, which is rooted in the encoding redundancy: the encoder globally models the foreground and background without explicit prior, leading to invalid computation and difficulty in effectively focusing on valuable foreground information. The lack of accuracy of the candidate box further increases the difficulty of the decoder to refine the boundary, resulting in invalid or excessive overlap of the prediction.

[0005] In addition, DETR assigns a unique positive sample to each real box during inference through Hungarian matching to avoid redundant detection, but this also brings the problem of sparse supervision of positive samples: usually only one positive sample corresponds to each real box, and most candidates become negative samples. In complex scenes such as small targets, low contrast or occlusion, this sparse supervision exacerbates the difficulty of training convergence and increases the demand for data and training time. SUMMARY

[0006] The purpose of the present application is to solve the above-mentioned problems existing in the prior art, and to provide an end-to-end temporal action localization method and system fusing boundary focus and relative boundary attention.

[0007] In order to achieve the above-mentioned purpose, the basic scheme of the present application is as follows: an end-to-end temporal action localization method fusing boundary focus and relative boundary attention, comprising the following steps:

[0008] Uncropped video data is collected and input into a feature encoder to extract segment-level video sequence features;

[0009] The encoder module is set, and the boundary focusing module BFM is introduced. A lightweight MLP is used to predict and rank the boundary awareness scores of each segment-level video sequence feature;

[0010] A multi-layer Transformer structure is arranged in the encoder module. Each layer is independently set with a reserved proportion. Only the corresponding proportion of video sequence features ranked at the top of the boundary awareness scores is selectively encoded. The shallow encoder is set with a higher reserved proportion to maintain global information. The reserved proportion of the subsequent deep encoder is gradually decreased layer by layer, so that the deep layer gradually focuses on the action boundary area, and the hierarchical screening and feature refinement from coarse to fine are realized.

[0011] The decoder module is set, and the relative boundary attention module RBAM is introduced. Each decoding query is parameterized as a dynamic anchor box represented by the action center and the time length. The action interval of the video sequence feature is iteratively refined in the multi-layer decoder by using the attention based on the relative position.

[0012] The classification head and the regression head are used to respectively predict the class label and the time boundary of each candidate action output by the last layer of the decoder.

[0013] The working principle and beneficial effects of the basic scheme are as follows: the boundary focusing module (BFM) is introduced in the technical scheme. A lightweight MLP is used to predict and rank the boundary awareness scores of each video segment (token). The high-score tokens are selectively encoded in the encoder, so as to reduce redundancy and suppress background interference.

[0014] The relative boundary attention module (RBAM) is proposed. Each decoding query is parameterized as a dynamic anchor box represented by the action center and the time length. The action interval is iteratively refined in the multi-layer decoder by using the attention based on the relative position. By explicitly modeling the relative distance and time overlap relationship between queries, the attention is accurately allocated to the related target, more positive samples are effectively captured, redundancy overlap is reduced, and the accuracy and efficiency of the temporal action detection in the complex video scene are improved.

[0015] Further, the boundary focusing module BFM is introduced. The method for predicting and ranking the boundary awareness scores of each segment-level video sequence feature by using a lightweight MLP is as follows:

[0016] The boundary awareness score is defined in the BFM to measure the proximity of the nearest action boundary at each time step:

[0017] ,

[0018] ,

[0019] wherein, ​the distance to the action center the recent action start and end boundary, the time distance to the recent boundary, the total duration of the action instance, the rate of control score decay with distance; at the boundary , therefore , the boundary relevance is the highest; away from the boundary, decaying in square term;

[0020] With the boundary-aware score BAS calculated by the real boundary as supervision, a lightweight MLP is trained to regress the BAS of each video sequence feature, using cross-layer hierarchical screening: the BAS of each layer feature of the encoder module is estimated by a lightweight MLP, and the BAS of the high layer is up-sampled and fused with the low layer feature:

[0021] ,

[0022] wherein, represents the boundary-aware score BAS corresponding to the updated l−1 layer feature guided by the high layer, which integrates the semantic boundary information of the upper layer and the local features of the current layer, so that the model can accurately perceive the action boundary at different levels; is an up-sampling operation, used to enlarge and align the boundary-aware score s l of the high layer to the time scale of the low layer feature, so as to fuse the semantic boundary information from the high layer; is the l−1 layer feature, is the BAS corresponding to the l−1 layer, is a learnable weight, which performs hierarchical screening according to the fused BAS, and only the high-score segment is retained to enter the Transformer layer of the encoder module. The boundary focusing module (BFM) is introduced, which uses a lightweight MLP to predict and sort the boundary-aware score for each video segment (token), and selectively encodes high-score tokens in each layer of the encoder, thereby reducing redundancy and suppressing background interference.

[0023] Further, the relative boundary attention module RBAM is introduced, which parameterizes each decoding query as a dynamic anchor box represented by the action center and the duration as:

[0024]

[0025] each query is uniformly represented as , wherein is the action center, is half the duration, corresponding to the interval ​​; Within the same decoding layer, multiple queries may correspond to adjacent or partially overlapped candidate intervals on the time axis. RBAM injects relative position bias in self-attention, enabling queries to perceive relative temporal and scale relationships. For the layer decoder query: , define the relative relationship vector between the two queries:

[0026] ,

[0027] The first term of the formula represents the center relative distance and is normalized by ; The second term represents the time length ratio difference; Map each component to high dimension through sine-cosine position encoding:

[0028] ,

[0029] where, : the action center of the i-th query, representing the center position of the candidate action on the time axis;

[0030] : the half time length of the i-th query, representing half the duration of the action interval;

[0031] ; is the relative boundary relationship vector between the two queries and , which encodes their relative position and scale difference on the time axis;

[0032] and are the action centers of the i-th and j-th queries, respectively; : the input relative position vector ; s: scaling factor, used to adjust the frequency scale; T: the maximum length of the video time scale; : encoding dimension; k: dimension index (encoding of the first dimension); : the half time length of the i-th and j-th queries, respectively;

[0033] and linearly transformed to get the relationship embedding , add the lower bound to avoid subsequent gradient vanishing, and use it together with for self-attention:

[0034] ,

[0035] where, , , Query, Key and Value matrices of the l-th decoder, respectively; : feature dimension; : relative boundary bias term introduced; Softmax: normalization of attention scores; : updated query feature after incorporating relative position relation;

[0036] combined with cross-attention and feed-forward network:

[0037] ,

[0038] where, is the global feature output by the encoder module, i.e., video feature; : self-attention output from the previous formula; AttnCross (·): cross-attention mechanism for incorporating query feature and video global feature.

[0039] Introducing relative boundary attention module RBAM, each decoding query is parameterized as a dynamic anchor box represented by action center and duration, which is beneficial for subsequent use.

[0040] Further, the action interval of video sequence feature is iteratively refined in multi-layer decoder using relative position-based attention, specifically:

[0041] Multi-layer decoding provides an "inter-layer refinement" mechanism: each layer is based on the further correction of deviation predicted by the previous layer, assuming that the parameter output by the l-th layer is , and the increment is , then before entering the (l+1)-th layer:

[0042] ,

[0043] where, is Sigmoid, is its logit, used to stabilize the learning range and training process.

[0044] By refining the action interval layer by layer through relative timing clues, the action detection accuracy is improved.

[0045] Further, the loss function is set to jointly optimize the three items of classification, boundary regression and boundary focus score: ,

[0046] where, denotes the total loss function; , , : weighting coefficients of the three losses, used to balance the contribution of each part; ​​Boundary focus loss is used to strengthen the model's attention to features at the action boundary; With Supervise the category prediction and time boundary regression respectively, and introduce a boundary focus score for each predicted segment , and train using a Focal Loss variant :

[0047] ,

[0048] Where the positive and negative sample balance coefficient ; Measure the consistency of the predicted boundary score and the true boundary quality , defined as:

[0049] ,

[0050] When , When , s∈[0,1]: the predicted boundary focus score, indicating the degree to which the model considers the segment to be close to the boundary; θ∈ [ 0, 1 ]: the true boundary quality, 1 indicates that the segment is exactly located at the action boundary, and 0 indicates that it is not at the boundary.

[0051] Under the multi-task loss of the DETR style, the classification, boundary regression and boundary focus score are jointly optimized, which significantly improves the positioning accuracy on short duration or boundary fuzzy actions.

[0052] The application also provides an end-to-end temporal action localization system based on the method, comprising a data acquisition unit, a feature encoder, an encoder module and a decoder module.

[0053] The data acquisition unit is used to acquire uncut video data and transmit it to the feature encoder.

[0054] The feature encoder receives uncut video data and extracts video sequence features.

[0055] A boundary focus module BFM is introduced in the encoder module, which uses a lightweight MLP to predict and sort the boundary perception score for each video sequence feature, and sets a multi-layer Transformer structure, each layer selectively encodes the video sequence feature with a boundary perception score greater than a threshold.

[0056] A relative boundary attention module (RBAM) is introduced in the decoder module to parameterize each decoding query as a dynamic anchor box characterized by action center and duration, iteratively refine the action interval of video sequence features in a multi-layer decoder using relative position-based attention, and predict the category label and time boundary of each candidate action through classification and regression heads, respectively.

[0057] The system introduces explicit boundary prior and dynamic query to improve the accuracy and efficiency of temporal action detection in complex video scenes.

[0058] Further, the feature encoder comprises:

[0059] A video slice unit is configured to divide the uncropped video into continuous time segments and sample them in steps at a preset frame rate;

[0060] A spatio-temporal feature extraction network is configured to extract segment-level features from each segment and output a feature vector sequence with a dimension of C; the spatio-temporal feature extraction network is a 3D convolution network pre-trained on a large-scale video dataset;

[0061] A feature normalization unit is configured to normalize and linearly map the segment-level features, unify the channels to the input dimension of the encoder, and optionally superimpose time position encoding;

[0062] An optional multi-scale branch is configured to aggregate segment-level features at different time scales and output pyramid temporal features to improve the representation ability of long and short actions.

[0063] The structure is simple and easy to use.

[0064] Further, the encoder module is a multi-layer Transformer structure and integrates a boundary focus module (BFM), specifically comprising:

[0065] An intra-layer self-attention and feedforward network models the temporal context of the input sequence through residual and normalization;

[0066] A boundary focus branch estimates a boundary awareness score (BAS) for each time step of the layer using a lightweight MLP, and performs upsampling fusion and weight adjustment between layers according to

[0067] A hierarchical screening mechanism performs threshold / Top-k selection on the features of each layer based on the fused BAS, and only sends high-score segments to subsequent self-attention calculation to reduce complexity;

[0068] Optional sparse attention and Dropout are used to further improve computational efficiency and generalization performance;

[0069] ​The encoder output includes filtered temporal features and corresponding multi-layer BAS, which are used for query interaction and boundary refinement during the decoding stage.

[0070] In the encoder module, high-resolution tokens are selectively encoded at each layer to reduce redundancy and suppress background interference.

[0071] Furthermore, the decoder module includes:

[0072] Query initialization unit, set the quantity to Learnable decoding queries, and parameterizing each query as The dynamic anchor frame is represented, where For the motion center, Half the duration;

[0073] The Relative Boundary Attention Module (RBAM) injects a bias based on the relationship between the relative center distance and duration ratio between queries into self-attention, enabling query interaction to perceive the relative time and scale relationship.

[0074] Cross-attention units enable the updated query to align with the encoder's global features. The data is then fused to obtain a context-enhanced representation of the candidate intervals.

[0075] Inter-layer refinement mechanism, for Perform incremental updates layer by layer to stabilize the training range and achieve finer action ranges through multi-layer iterations;

[0076] The prediction head is a lightweight MLP that includes a classification head and a regression head. It outputs the class probability and temporal boundary parameters of candidate actions, respectively, and performs end-to-end training in combination with the aforementioned loss function.

[0077] The decoder module uses a set of dynamic queries, combined with a relative temporal boundary attention mechanism, to explicitly model the relative offset and duration ratio between candidate segments, suppressing redundant / overlapping predictions and improving boundary alignment capabilities. Attached Figure Description

[0078] Figure 1 This is a flowchart illustrating the end-to-end temporal action localization method that integrates boundary focusing and relative boundary attention according to the present invention.

[0079] Figure 2 This is a schematic diagram of the boundary awareness score of the end-to-end temporal action localization method that integrates boundary focusing and relative boundary attention according to the present invention. Detailed Implementation

[0080] Embodiments of the present application are described below in the detailed description and illustrated in the accompanying drawings by which like or similar elements, symbols and / or features are referred to by like or similar labels, are intended to be the same or to be congruent (i.e., to have the same reference number), and are intended to be the same as or to be performing a comparable function. The embodiments described below are examples for purposes of illustration only and are not intended to limit the application.

[0081] In the description of the present application, it needs to be understood that the orientation or positional relationship indicated by the terms "longitudinal", "transverse", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer" and the like is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the present application and simplifying the description, and therefore cannot be understood as indicating or implying that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application.

[0082] In the description of the present application, unless otherwise specified and limited, it needs to be explained that the terms "mounting", "connection", "connection" should be understood broadly, for example, it can be mechanical connection or electrical connection, it can be the communication inside two elements, it can be direct connection or indirect connection through intermediate medium, and the specific meaning of the above terms can be understood by the person skilled in the art according to the specific circumstances.

[0083] Temporal action localization (TAL) aims to identify the action category and its time boundary in untrimmed videos. Existing DETR-based models often face problems such as low quality of candidate proposals, strong background interference, and sparse positive sample supervision during detection.

[0084] The application discloses an end-to-end temporal action localization method fusing boundary focus and relative boundary attention, as shown in Figure 1 The method comprises the following steps:

[0085] Collect untrimmed video data and input into a feature encoder to extract a segment-level video sequence feature (dimension ), specifically:

[0086] Divide the original untrimmed video into a plurality of time segments, each segment containing a plurality of consecutive frames, then use a pre-trained video classification network (I3D) to encode each segment to extract a segment-level feature vector. Each video is therefore represented as a time sequence of features , wherein denotes the number of time segments (i.e., sequence length) after the video is divided, reflecting how many feature points the video is sampled in the time dimension; C denotes the feature dimension of each segment.

[0087] An encoder module is set, and a boundary focus module BFM is introduced to predict and rank the boundary awareness scores of each segment-level video sequence feature (token) using a lightweight MLP;

[0088] A multi-layer Transformer structure is provided in the encoder module, and each layer selectively encodes video sequence features with a boundary awareness score greater than a threshold, so that the model can focus more on segments that are more likely to contain actions; tokens closer to the action boundary (start / end) are given higher weights, and the remaining tokens are aggregated into a global background embedding, thereby preserving key boundary semantics while maintaining overall context.

[0089] A decoder module is set, and a relative boundary attention module RBAM is introduced to convert each decoding query parameter (query) into a dynamic anchor box represented by the action center and the duration, and iteratively refine the action interval of the video sequence feature in the multi-layer decoder using the relative position-based attention; The decoder introduces a set of dynamic queries, and combines the relative position attention mechanism based on the position clues to iteratively refine the predicted action center and duration.

[0090] The classification head and the regression head are used to predict the class label and the time boundary of each candidate action (i.e., output feature) output by the last layer of the decoder, respectively. Both the classification head and the regression head (structures already existing in the DETR detection framework) directly act on the output features of the last layer of the decoder, and use a lightweight multi-layer perceptron MLP to predict, respectively.

[0091] Given an untrimmed video , represented as a sequence of features , where is the segment feature at the th discrete time step. The task is to predict a set of actions , where is the th instance category, is the start and end time.

[0092] In a preferred embodiment of the present invention, a boundary focusing module (BFM) is introduced, and a lightweight MLP is used to predict and rank boundary awareness scores for each segment-level video sequence feature (e.g., ...). Figure 2 The method shown is as follows:

[0093] To guide the model to focus more effectively on action boundaries, a boundary awareness score is defined in BFM. To measure the proximity of each time step to the nearest action boundary:

[0094] ,

[0095] ,

[0096] in, Distance from the center of motion The start and end boundaries of the most recent action, The time distance to the nearest boundary. The total duration of this action instance. To control the rate at which the fraction decays with distance; at the boundary Therefore This indicates the highest correlation at the boundary; when far from the boundary, Decrease by the square term;

[0097] Using the boundary-aware score (BAS) calculated from the real boundaries as supervision, a lightweight MLP is trained to regress and predict the BAS for each video sequence feature. To further improve accuracy and efficiency, a cross-layer hierarchical selection method is adopted: the BAS of each layer of the encoder module features is estimated using the lightweight MLP, and the BAS of the higher layers is upsampled and fused with the features of the lower layers.

[0098] ,

[0099] in, The boundary awareness score (BAS) represents the updated features of layer l-1 after higher-level guidance. It integrates the semantic boundary information of the upper layer with the local features of the current layer, enabling the model to accurately perceive action boundaries at different levels. This is an upsampling operation used to increase the boundary-aware score s of higher layers. l Magnify and align to the time scale of low-level features in order to fuse semantic boundary information from high-level features; For the first Layer features, For the first The BAS corresponding to the layer, For learnable weights, hierarchical screening is performed according to the fused BAS, and only high-score fragments are retained to enter the Transformer layer of the encoder module, which efficiently locates the boundary, reduces background interference and redundant calculation.

[0100] BFM: Select and enhance the time segments highly related to the boundary, efficiently suppress the redundant background interference, and significantly improve the boundary positioning ability.

[0101] Accurate positioning of the start and end time of the action is the core difficulty of TAL. The existing Transformer method (such as TadTR) usually represents the candidate action with a fixed query vector, which is difficult to adaptively adjust the query span with the content, and the attention calculation does not explicitly model the relative time relationship between candidates, which is easy to confuse in adjacent / overlapping actions. Therefore, RBAM is proposed, which emphasizes boundary perception and relative relationship modeling, and is composed of relative position attention and multi-layer progressive refinement.

[0102] In a preferred scheme of the application, the relative boundary attention module RBAM is introduced, and the method of parameterizing each decoding query as a dynamic anchor box represented by the action center and the time length (that is, the two parameters of the decoding query query in the relative boundary attention module (RBAM) are parameterized as:

[0103] Each query is uniformly represented as , wherein is the action center, is the half time length, corresponding to the interval ; In the same decoding layer, the candidate intervals corresponding to multiple queries may be adjacent or partially overlapped on the time axis, and if the self-attention only depends on the content features, it is difficult to explicitly distinguish the relative order, overlap degree or time length difference.

[0104] RBAM injects relative position bias in self-attention, so that the interaction between queries can perceive the relative time and scale relationship, and the query of the first layer decoder: , the relative relationship vector between two queries is defined as:

[0105] ,

[0106] The first term of the formula represents the center relative distance and is normalized by ; The second term represents the time length proportion difference; map each component to high dimension through sine-cosine position encoding:

[0107] ,

[0108] wherein, : the action center of the i-th query, indicating the center position of the candidate action on the time axis;

[0109] : the half duration of the i-th query, indicating the half duration of the action interval;

[0110] ; is the relative boundary relation vector between two queries and , which encodes their relative position and scale difference on the time axis;

[0111] and are the action centers of the i-th and j-th queries, respectively; : the input relative position vector ; s: scaling factor for adjusting the frequency scale; T: the maximum length of the video time scale; : the encoding dimension; k: the dimension index (the encoding of the k-th dimension); are the half durations of the i-th and j-th queries, respectively;

[0112] and the linear transformation relation embedding is added to avoid the subsequent gradient vanishing, which is combined with for self-attention:

[0113] ,

[0114] where, , , : Query, Key and Value matrices of the l-th layer decoder, respectively; : feature dimension; : the introduced relative boundary bias term; Softmax: normalization of attention score; : the updated query feature after fusing the relative position relation;

[0115] and the cross-attention and feedforward network combination update:

[0116] ,

[0117] where, is the global feature output by the encoder module, i.e., the video feature; ​: self-attention output from the previous formula; AttnCross (·) : cross-attention mechanism for fusing query features and video global features. Compared with the traditional inter-query attention , , the above makes the model explicitly perceive the relative relationship between time and scale in the interaction, and more effectively distinguish adjacent or significant action segments.

[0118] In a preferred embodiment of the present application, the relative position-based attention is used to iteratively refine the action interval of the video sequence features in the multi-layer decoder (i.e., the time interval parameterized by the decoding query , + ), specifically:

[0119] Single-layer relative position attention can distinguish intervals, but still has challenges in complex scenarios such as long actions or ambiguous boundaries. Multi-layer decoding provides an "inter-layer refinement" mechanism: each layer is based on the previous layer's prediction to further correct the bias. Let the parameters of the th layer output be , and the increment be , then before entering the th layer:

[0120] ,

[0121] where is Sigmoid, is its logit, used to stabilize the learning range and training process.

[0122] In a preferred embodiment of the present application, a loss function is set, which jointly optimizes classification, boundary regression, and boundary focus score under the multi-task loss of the DETR style: ,

[0123] where represents the total loss function; , , : weighting coefficients of the three losses, used to balance the contributions of each part; boundary focus loss, used to strengthen the model's attention to features at the action boundary; and supervise the category prediction and time boundary regression, respectively. To further improve the boundary accuracy, a boundary focus score is introduced for each predicted segment, and the Focal Loss variant is used to train :

[0124] ,

[0125] wherein the positive-negative sample balance coefficient , α = 0.25, γ = 2 as the default value can provide stable and good convergence effect, so this standard setting is directly adopted; α is a positive-negative sample balance coefficient, controlling the relative weight of positive samples (close to the action boundary) and negative samples (non-boundary segments) in the total loss; γ is a difficult sample adjustment coefficient, controlling the attention degree of the model to difficult samples (prediction error) during training; the predicted boundary score is consistent with the quality of the real boundary, defined as:

[0126] ,

[0127] when (high-quality boundary) ; when , s∈[0,1]: the predicted boundary focus score, indicating the degree to which the model considers the segment close to the boundary; θ∈ [ 0, 1 ]: the real boundary quality, 1 indicates that the segment is exactly located at the action boundary, and 0 indicates that it is not at the boundary.

[0128] minimizing promotes better alignment between the predicted boundary and the real boundary, significantly improving positioning accuracy on short duration or fuzzy boundary actions.

[0129] The method is evaluated on two standard TAL benchmarks: THUMOS14 (20 classes, 200 segments of training uncut videos, 213 segments of testing; trained on the validation set according to the convention and evaluated on the test set), and a larger scale ActivityNet1.3 (200 classes, a total of 19,994 segments: 10,024 of training, 4,926 of validation; trained on the training set and evaluated on the validation set).

[0130] The evaluation index is the average precision mean (mAP): THUMOS14 is averaged at tIoU threshold ; ActivityNet1.3 is averaged at , as shown in Table 1. The tIoU of the predicted and real segments exceeding the threshold and the same class are considered as correct detection.

[0131] ​Following most TAL models, the input is offline extracted video features. THUMOS14 uses pre-trained two-stream I3D on Kinetics, extracting features every 8 frames; ActivityNet1.3 uses two-stream TSN, extracting features at 5 FPS. Video features are linearly interpolated to a fixed length of 100 as usual. Optimization employs AdamW with warm-up and cosine annealing: THUMOS14 is trained for 30 epochs, reducing the learning rate after 25 epochs; ActivityNet1.3 is trained for 15 epochs, reducing the learning rate after 12 epochs; the initial learning rate , the decay coefficient is 0.1.

[0132] Table 1. Comparison results of different methods on THUMOS14 and ActivityNet-1.3

[0133]

[0134] In Table 1, Model: model name; Feature: feature extraction method (feature encoder used to generate video segment features); THUMOS14 (mAP@{tIoU}): detection accuracy (mAP, changes with different tIoU thresholds) on the THUMOS14 dataset, the higher the threshold, the more accurate the predicted time interval is required to overlap with the true interval. ActivityNet-1.3 (mAP@{tIoU}): detection accuracy (mAP, changes with different tIoU thresholds) on the ActivityNet-1.3 dataset; tIoU (temporal Intersection over Union): temporal Intersection over Union (a measure of the degree of overlap between the predicted action interval and the true action interval) Avg: average mAP value (average performance at different thresholds).

[0135] On THUMOS14, the method of the present application leads in each tIoU threshold, especially in more stringent thresholds (0.6 / 0.7) with significant improvement. For example, when IoU=0.6, the method reaches 47.7, which is about 2.1 percentage points higher than the strong baseline; when IoU=0.7, it reaches 32.9, which is about 4.3 percentage points higher.

[0136] This shows that by explicitly modeling the boundary and refining the dynamic query, the method of the present application is more accurate in aligning the action boundary, especially effective for short actions and overlapping scenes. On the larger scale of ActivityNet1.3, the overall improvement is relatively moderate, but still shows clear advantages at more stringent thresholds (0.75 / 0.95), reflecting the potential of boundary alignment under complex conditions.

[0137] The application also provides an end-to-end timing action positioning system based on the method.

[0138] The data acquisition unit is used to acquire uncropped video data and transmit it to the feature encoder. The feature encoder receives the uncropped video data and extracts video sequence features.

[0139] A boundary focusing module BFM is introduced in the encoder module, which uses a lightweight MLP to predict and rank the boundary awareness scores of each video sequence feature, and sets a multi-layer Transformer structure, each layer selectively encodes video sequence features with a boundary awareness score greater than a threshold. The boundary focusing module (BFM) learns the boundary awareness score (BAS) of each segment through a lightweight MLP, and selects the segments closer to the action boundary for encoding based on the score, effectively suppressing background noise and reducing computational cost.

[0140] A relative boundary attention module RBAM is introduced in the decoder module, which parameterizes each decoding query as a dynamic anchor box represented by the action center and the duration, iteratively refines the action interval of the video sequence feature in the multi-layer decoder using the relative position-based attention, and predicts the category label and time boundary of each candidate action through the classification head and the regression head respectively. The relative boundary attention module (RBAM) parameterizes the query as "center + duration" in the decoding stage, and introduces the relative boundary attention to refine the boundary layer by layer, improving the sensitivity of positive samples and reducing redundant overlap.

[0141] The system introduces explicit boundary prior and dynamic query to improve the accuracy and efficiency of timing action detection in complex video scenes.

[0142] In a preferred embodiment of the application, the feature encoder comprises:

[0143] The video slicing unit is used to divide the uncropped video into continuous time segments at a preset frame rate and sample them by a step size;

[0144] The spatio-temporal feature extraction network is used to extract segment-level features for each segment and output as a feature vector sequence with dimension C. The spatio-temporal feature extraction network is a 3D convolution network pre-trained on a large-scale video dataset;

[0145] The feature normalization unit is used to normalize and linearly map the segment-level features, unify the channels to the encoder input dimension, and optionally superimpose the time position encoding;

[0146] The optional multi-scale branch is used to converge segment-level features at different time scales and output pyramid timing features, improving the representation ability for long and short actions.

[0147] In a preferred scheme of the present application, the encoder module is a multi-layer Transformer structure integrated with a boundary focusing module BFM, which specifically includes:

[0148] Intra-layer self-attention and feedforward network, which model the temporal context of the input sequence through residual and normalization;

[0149] Boundary focusing branch, which estimates the boundary awareness score BAS for each time step of the layer using a lightweight MLP, and performs up-sampling fusion and weight adjustment between layers according to

[0150] Hierarchical screening mechanism, which performs threshold / Top-k selection on the features of each layer according to the fused BAS, and only sends high-score segments to subsequent self-attention calculation to reduce complexity;

[0151] Optional sparse attention and Dropout, which are used to further improve computational efficiency and generalization performance;

[0152] The encoder output includes the screened temporal features and the corresponding multi-layer BAS, which are used for query interaction and boundary refinement in the decoding stage.

[0153] In a preferred scheme of the present application, the decoder module includes:

[0154] Query initialization unit, which sets the number of learnable decoding queries to and parameterizes each query as a dynamic anchor box represented by , where is the action center, is half the duration;

[0155] Relative boundary attention module (RBAM), which injects a relationship bias based on the relative center distance and duration ratio between queries in self-attention, so that the query interaction perceives the relative time and scale relationship;

[0156] Cross-attention unit, which fuses the updated query and the global features of the encoder to obtain a context-enhanced representation of the candidate interval;

[0157] Inter-layer refinement mechanism, which performs layer-by-layer incremental update on to stabilize the training range and realize multi-layer iterative action interval refinement;

[0158] Prediction head, which includes a lightweight MLP containing a classification head and a regression head, respectively outputs the class probability and time boundary parameters of the candidate action, and combines the aforementioned loss function for end-to-end training.

[0159] ​The boundary focusing module (BFM) explicitly suppresses the background region in the early stage of encoding, and gives higher weights to tokens closer to the action boundary, reducing the redundant encoding of background segments. The relative boundary attention module (RBAM) represents each query as the center and length of the target in the decoding stage, and introduces a relative position bias to iteratively refine the boundary between layers and encourage more accurate positive sample feedback.

[0160] By strengthening boundary recognition and query refinement, the present application effectively alleviates the dual problems of inaccurate first-stage candidates and dominant negative samples in the second stage, achieving faster convergence and more robust detection performance, especially in small / weak target scenarios. A large number of experiments show that the combination of "boundary prior + dynamic query" not only reduces coding redundancy, but also fully releases the potential of the DETR end-to-end detection framework, providing a new perspective for video action localization and general object detection.

[0161] The specific embodiments described herein merely illustrate the spirit of the present application. Those skilled in the art of the present application can make various modifications or supplements to the described specific embodiments or use similar ways to replace them, without deviating from the spirit of the present application or exceeding the scope defined by the appended claims.

Claims

1. An end-to-end temporal action localization method that fuses boundary focus and relative boundary attention, characterized in that, The method comprises the following steps: Collecting uncropped video data and inputting the video data into a feature encoder to extract video sequence features at a segment level; Setting an encoder module and introducing a boundary focus module BFM, using a lightweight MLP to predict and sort boundary awareness scores for each video sequence feature at a segment level; The encoder module is provided with a multi-layer Transformer structure, and each layer selectively encodes video sequence features with a boundary awareness score greater than a threshold value; A decoder module is set, and a relative boundary attention module RBAM is introduced, which parameterizes each decoding query as a dynamic anchor box represented by an action center and a time length, and iteratively refines the action interval of the video sequence feature in the multi-layer decoder using relative position-based attention; The classification head and the regression head are used to predict the category label and the time boundary of each candidate action output by the last layer of the decoder.

2. The end-to-end temporal action localization method that fuses boundary focus and relative boundary attention according to claim 1, wherein, The method of introducing the boundary focus module BFM and using the lightweight MLP to predict and sort the boundary awareness scores for each video sequence feature at a segment level is as follows: The boundary-aware score is defined in BFM to measure the proximity of the most recent action boundary at each time step: ， ， wherein, is the distance to the action center is the most recent action start or end boundary, is the time distance to the most recent boundary, is the total duration of the action instance, is the rate at which the control score decays with distance; at the boundary , so , the boundary correlation is highest; far from the boundary, decays as a square term; Under the supervision of the boundary awareness scores BAS calculated by the real boundaries, a lightweight MLP is trained to regress and predict the BAS of each video sequence feature, and a cross-layer hierarchical screening is adopted: the BAS of each layer of the encoder module is estimated by a lightweight MLP, and the BAS of the higher layer is upsampled and fused with the lower layer feature: ， in, The boundary awareness score (BAS) represents the updated features of layer l-1 after higher-level guidance. It integrates the semantic boundary information of the upper layer with the local features of the current layer, enabling the model to accurately perceive action boundaries at different levels. This is an upsampling operation used to increase the boundary-aware score s of higher layers. l Magnify and align to the time scale of low-level features in order to fuse semantic boundary information from high-level features; For the first Layer features, For the first The BAS corresponding to the layer, As learnable weights, a hierarchical selection is performed based on the fused BAS, retaining only high-scoring segments to enter the Transformer layer of the encoder module.

3. The end-to-end temporal action localization method that fuses boundary focus and relative boundary attention according to claim 1, wherein, The method of introducing the relative boundary attention module RBAM and parameterizing each decoding query as a dynamic anchor box represented by an action center and a time length is as follows: Each query is uniformly represented as where is the action center, is the half duration, and corresponds to the interval Within the same decoding layer, multiple queries correspond to candidate intervals that can be adjacent or partially overlapped in the time axis. RBAM injects relative position bias in self-attention, enabling queries to perceive relative temporal and scale relationships. For the queries of the layer decoder: , define the relative relationship vector between the two queries: ， The first term of the equation represents the center relative distance and is given by normalized; the second term represents the time proportion difference; and each component is mapped to a high dimension through a sine-cosine position encoding: ， wherein, : the action center of the i-th query, representing the center position of the candidate action on the time axis; : half duration of the ith query, representing the duration of half of the action interval; ; are two queries with a relative boundary relation vector between them, which encodes their relative position and scale difference on the time axis; with are the action centers of the i-th and j-th query, respectively; : input relative position vector ; s: scaling factor for adjusting the frequency scale; T: maximum length of the video time scale; : encoding dimension; k: dimension index; are the half-lengths of the i-th and j-th query, respectively; and linearly transformed to get relational embeddings , adding lower bounds to avoid subsequent vanishing gradients, combined with self-attention: ， wherein, , , : represent the query Query, key Key and value Value matrices of the l-th layer decoder, respectively; : feature dimension; : introduced relative boundary bias term; Softmax: normalization on attention scores; : updated query feature after fusing relative position relation; Combined with cross-attention and a feedforward network for updating: ， wherein, global feature output by the encoder module, i.e., video feature; : self-attention output from the previous formula; AttnCross (·): cross-attention mechanism for fusing query feature and video global feature.

4. The end-to-end temporal action localization method that fuses boundary focus and relative boundary attention according to claim 3, wherein, The method of iteratively refining the action interval of the video sequence feature in the multi-layer decoder using relative position-based attention is as follows: Multi-layer decoding provides an "inter-layer refinement" mechanism: each layer further corrects the bias based on the prediction of the previous layer, assuming that the first layer is predicted based on the input image. The parameters output by the layer are with an increment of and the process goes to the next layer layer. ， where, is the Sigmoid, is its logit, used to stabilize the learning range and training process.

5. The end-to-end temporal action localization method that fuses boundary focus and relative boundary attention according to claim 1, wherein, A loss function is set to jointly optimize the three terms of classification, boundary regression, and boundary focusing score: , wherein, represents the total loss function; , , : the weighting coefficients of three losses, used to balance the contribution of each part; boundary-focused loss, used to strengthen the model's attention to features at the action boundary; and respectively supervise the category prediction and the time boundary regression, introducing a boundary-focused score for each predicted segment , and adopting a Focal Loss variant for training : ， Among them, the positive and negative sample balance coefficient ; Measure the prediction boundary score With true boundary quality Consistency is defined as: ， When time ; when time , s e [0, 1]: predicted boundary focus score, indicating how close the model thinks the segment is to the boundary; Q e [0, 1]: true boundary quality, 1 means the segment is exactly at the action boundary, 0 means not at the boundary.

6. An end-to-end time-of-arrival positioning system based on the method of one of claims 1 to 5, characterized in that, The method comprises a data acquisition unit, a feature encoder, an encoder module, and a decoder module. The data acquisition unit is used to collect uncropped video data and transmit the uncropped video data to the feature encoder. The feature encoder receives the uncropped video data and extracts video sequence features. In the encoder module, the boundary focus module BFM is introduced, a lightweight MLP is used to predict and sort the boundary awareness scores for each video sequence feature, and a multi-layer Transformer structure is set, each layer selectively encodes video sequence features with a boundary awareness score greater than a threshold value. In the decoder module, the relative boundary attention module RBAM is introduced, each decoding query is parameterized as a dynamic anchor box represented by an action center and a time length, and relative position-based attention is used to iteratively refine the action interval of the video sequence feature in the multi-layer decoder, and the classification head and the regression head are used to predict the category label and the time boundary of each candidate action.

7. The end-to-end timing action positioning system of claim 6, wherein, The feature encoder comprises: A video slice unit is configured to divide the uncropped video into continuous time segments at a preset frame rate and sample the time segments in steps; A spatiotemporal feature extraction network is configured to extract segment-level features for each segment and output a feature vector sequence with a dimension of C; the spatiotemporal feature extraction network is a 3D convolution network pre-trained on a large-scale video dataset. A feature normalization unit is configured to normalize and linearly map the segment-level features, unify the channels, and optionally superimpose a time position encoding to the encoder input dimension; An optional multi-scale branch is configured to aggregate the segment-level features at different time scales and output pyramid temporal features to improve the representation ability of long and short actions.

8. The end-to-end timing action positioning system of claim 6, wherein, The encoder module is a multi-layer Transformer structure and integrates a boundary focusing module (BFM). Specifically, the encoder module includes: Intra-layer self-attention and feedforward networks are configured to model the temporal context of the input sequence through residual and normalization; Boundary-focused branch, which employs a light-weight MLP to estimate a boundary-aware score BAS for each time step of the layer and performs upsampling fusion and weight adjustment between layers perform upsampling fusion and weight adjustment; A hierarchical screening mechanism is configured to perform threshold / Top-k selection on the features of each layer according to the fused BAS, and only send high-scored segments to subsequent self-attention calculation to reduce complexity; Optional sparse attention and Dropout are configured to further improve the computational efficiency and generalization performance; The encoder output includes screened temporal features and corresponding multi-layer BAS, which are used for query interaction and boundary refinement in the decoding stage.

9. The end-to-end timing action positioning system of claim 6, wherein, The decoder module includes: The query initialization unit sets the number of learnable decoding queries and parameterizes each query as a dynamic anchor box represented by , where is the action center, is half the duration; A relative boundary attention module (RBAM) is configured to inject a bias based on the relative center distance and time length ratio between queries in the self-attention, so that the query interaction can perceive the relative time and scale relationship; across attention units, the updated query is combined with the encoder global features fused, obtaining a context-enhanced representation of the candidate interval; Inter-layer refinement mechanism, to Stabilize the training range by doing incremental update layer by layer, and realize the action interval refinement of multi-layer iteration; A prediction head includes a lightweight MLP containing a classification head and a regression head, which respectively output the category probability and time boundary parameters of the candidate action, and are trained end-to-end in combination with the aforementioned loss function.

Images