Video action detection method based on nonlinear spatiotemporal relative position bias

Abstract

The application designs a video action detection method based on nonlinear space-time relative position bias, and has the characteristics that the method comprises the following steps: acquiring video data; inputting the video data into a video action detection model based on a transformer, calculating the relative coordinates of a cube based on a target point and a reference point by using nonlinear transformation; outputting the nonlinear bias of each dimension, calculating an improved relative position bias B, and outputting key space-time features by using an attention mechanism; inputting the key space-time features into a classification module of the model to obtain a video action detection result. Compared with the prior art, the application has the advantages of improving the accuracy of action detection while reducing the amount of calculation.

Description

Technical Field

[0001] The present invention relates to the technical field of video action detection, and in particular to a video action detection method based on nonlinear spatiotemporal relative position bias. Background Art

[0002] Video action detection aims to identify and classify specific actions or activities in video sequences. The goal is to accurately locate the temporal and spatial locations of the actions in the video, thereby answering the questions of "what happened, where, and when did it happen". This task is of great significance in many application areas, such as human-computer interaction, video content analysis, and intelligent video surveillance. In recent years, methods based on Vision Transformer (ViT) have made significant progress in video classification tasks. ViT uses the self-attention mechanism to capture long-range spatiotemporal dependencies and shows unique advantages in modeling global context. However, although these features provide powerful representation capabilities for video tasks, the performance of ViT in video action detection still faces challenges, especially in the spatiotemporal localization of actions.

[0003] Existing ViTs typically use a one-dimensional position embedding approach, a design inspired by position encoding methods in natural language processing. However, unlike language data, video data contains complex structural relationships in both temporal and spatial dimensions, making it difficult for this one-dimensional encoding method to fully express the spatiotemporal position information in videos. In action detection tasks, the comprehensive and effective introduction of position information is crucial to improving detection performance. Although some studies have attempted to combine position encoding and relative position bias techniques, these methods are primarily designed for image tasks and fail to extend to the spatiotemporal dimensions of video analysis. Therefore, directly applying position encoding methods from image detection to video action detection often has limited results. This is because the position information in a video not only involves two-dimensional space but also requires modeling dynamic changes in the temporal dimension, which is much more complex than the analysis requirements of a single frame image, resulting in insufficient accuracy and high computational complexity in action detection. Summary of the Invention

[0004] The purpose of the present invention is to provide a video action detection method based on nonlinear spatiotemporal relative position bias in order to improve the accuracy of action detection while reducing the amount of calculation.

[0005] The purpose of the present invention can be achieved by the following technical solutions:

[0006] A video action detection method based on nonlinear spatiotemporal relative position bias, the method comprising the following steps:

[0007] Obtain video clips, convert the video clips into a series of tags, and obtain video data;

[0008] The video data is input into the Transformer-based video action detection model. Each frame of the video data in the model is divided into multiple cubes, and a grid of reference point coordinates is generated. The reference point coordinates (t1, x1, y1, t2, x2, y2) are constructed, where t1, x1, y1 are the time frame coordinates, height coordinates, and width coordinates of the bottom left point of the cube, and t2, x2, y2 are the time frame coordinates, height coordinates, and width coordinates of the top right point of the cube. The coordinate sequence of the target point is defined, and the relative coordinates of the cube are calculated using a nonlinear transformation based on the target point and the reference point.

[0009] Input the relative coordinates of the cube into the multi-layer perceptron of the video action detection model, output the nonlinear bias of each dimension, decouple and calculate the improved first relative position bias based on the nonlinear bias of each dimension, obtain the learnable vector v, pass the learnable vector v through the multi-layer perceptron, generate six-dimensional coordinates, calculate the relative coordinates of the learnable vector v based on the six-dimensional coordinates, generate the second relative position bias through the multi-layer perceptron, input the learnable vector and video data into the improved attention mechanism of the video action detection model, the attention mechanism outputs key spatiotemporal features, and the improved attention mechanism adopts the improved relative position bias B generated based on the first relative position bias and the second relative position bias;

[0010] Key spatiotemporal features are input into the classification module of the model to obtain video action detection results.

[0011] Furthermore, the reference point coordinates (t1,x1,y1,t2,x2,y2) are:

[0012] (t1,x1,y1,t2,x2,y2)=

[0013] stack(grid(t),grid(h),grid(w),grid(t)+1,grid(h)+1,grid(w)+1)

[0014] Among them, stack() is the overlay function, which overlays multiple arrays along a new axis and combines the time, width, and height ranges of each grid point into the six-dimensional coordinates of a cube. grid represents the grid, and t, h, and w represent the time, image height, and width dimensions, respectively.

[0015] Furthermore, the coordinate sequence of the target point is:

[0016] t c =linspace(0.5,t-0.5,t)

[0017] x c =linspace(0.5,h-0.5,h)

[0018] yc =linspace(0.5,w-0.5,w)

[0019] Among them, t c 、x c 、y c Respectively represent the time frame coordinate, height coordinate and width coordinate of the target point,

[0020] Furthermore, the relative coordinates of the cube are:

[0021]

[0022] Wherein, Δt1 represents the relative coordinate of the bottom left point in the relative coordinate of the cube in terms of the number of time frames, Δx1 represents the relative coordinate of the height of the bottom left point in the relative coordinate of the cube, and Δy1 represents the relative coordinate of the bottom left point in terms of the number of time frames in the width coordinate of the cube;

[0023] The relative coordinates of the learnable vector v are calculated using the same formula.

[0024] Furthermore, the nonlinear bias in each dimension is:

[0025] B t =MLP1(Δt1,Δt2)

[0026] B x =MLP2(Δx1,Δx2)

[0027] B y =MLP3(Δy1,Δy2)

[0028] Among them, B t is the nonlinear bias in the dimension of time frames, B x is the nonlinear bias in the height dimension, B y is the nonlinear bias of the width dimension, and MLP1, MLP2 and MLP3 are multi-layer perceptrons.

[0029] Furthermore, the first relative position offset is:

[0030] B′=unsqueeze(B t ,2,3)+unsqueeze(B x ,1,3)+unsqueeze(B y ,1,2)

[0031] Among them, unsqueeze(·) is the expansion dimension function.

[0032] Furthermore, the grid of the reference point coordinates is constructed based on a meshgrid function.

[0033] Furthermore, the key spatiotemporal features are input into the classification module of the model to obtain the video action detection results in the following specific steps:

[0034] The key spatiotemporal features are upsampled or downsampled to obtain multi-scale features and fused with a pyramid network to generate a bounding box. The fused features are passed to the decoder and then passed through the classification module to obtain the video action detection results.

[0035] Furthermore, the specific steps of converting the video clip into a series of tags are as follows:

[0036] The video clip is divided into several consecutive frames, which are sequentially formed into a frame sequence, and the frame sequence is encoded as a series of tags.

[0037] Furthermore, the input of the improved attention mechanism is [v,x], where v represents the learnable vector and x represents the video data. The improved attention mechanism is:

[0038]

[0039] Among them, Q, K, V are query, key and value matrices respectively, d represents the dimension, and B is the improved relative position bias, which is obtained by concatenating and expanding the first relative position bias and the second relative position bias.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] The present invention improves the shortcomings of traditional one-dimensional position encoding by introducing cube relative position bias, and can better capture the complex spatiotemporal relationships in videos. Through the nonlinear position bias design, the model pays more attention to key areas with closer distances, thereby improving the accuracy of motion detection. Compared with the traditional method of directly calculating the global bias between the reference point and the target point, the decomposition method of the nonlinear spatiotemporal relative position bias distributes the originally complex calculations into three dimensions, avoiding high computational overhead. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] 图1 It is a flowchart of the present invention. DETAILED DESCRIPTION

[0043] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0044] The present invention proposes a video action detection method based on nonlinear spatiotemporal relative position bias. First, the video frame sequence is encoded into a token sequence to construct a unified input feature representation. Then, a new type of nonlinear spatiotemporal relative position bias is introduced. The method first calculates the relative position relationship between each token, including horizontal, vertical and time dimensions, and generates position bias values ​​based on a nonlinear formula, thereby constructing a three-dimensional relative position encoding that is more in line with the characteristics of the video. Then, the nonlinear spatiotemporal relative position bias is integrated into the attention mechanism to improve the attention. The formula is improved, retaining the traditional bias term B′ while adding a learnable prefix vector to enhance the model's ability to capture spatiotemporal positions. Finally, under the improved attention mechanism, a multi-layer Transformer model is performed on the input features to extract key features within the spatiotemporal range of the video. Through the classification module and the bounding box regression module, the category and spatiotemporal position of the action in the video are output. Compared with the existing technology, the present invention has the advantages of strong spatiotemporal relationship modeling capabilities and high accuracy of action detection. The method includes the following steps:

[0045] Obtain video clips, convert the video clips into a series of tags, and obtain video data;

[0046] The video data is input into the Transformer-based video action detection model. Each frame of the video data in the model is divided into multiple cubes, and a grid of reference point coordinates is generated. The reference point coordinates (t1, x1, y1, t2, x2, y2) are constructed, where t1, x1, y1 are the time frame coordinates, height coordinates, and width coordinates of the bottom left point of the cube, and t2, x2, y2 are the time frame coordinates, height coordinates, and width coordinates of the top right point of the cube. The coordinate sequence of the target point is defined, and the relative coordinates of the cube are calculated using a nonlinear transformation based on the target point and the reference point.

[0047] The relative coordinates of the cube are input into the multi-layer perceptron of the video action detection model, which outputs the nonlinear bias B′ of each dimension. The improved relative position bias B is calculated based on the decoupling of the nonlinear bias of each dimension. The video data including the learnable vector is input into the improved attention mechanism of the video action detection model, and the attention mechanism outputs key spatiotemporal features.

[0048] The key spatiotemporal features are input into the classification module of the model to obtain the video action detection results. The flow chart is as follows 图1 shown.

[0049] The specific steps of the present invention are:

[0050] Video preprocessing: Split the input video sequence into several consecutive frames and form a frame sequence in order. Encode the frame sequence into spatiotemporal tags for subsequent Transformer model input.

[0051] The video clip is first encoded block by block using a cube embedding method, converting it into a series of tokens. This process represents the input video as a t×h×w three-dimensional grid, where t, h, and w represent the time, image height, and width dimensions, respectively. Positional embeddings are then added to these tokens, resulting in a sequence of n=t×h×w tokens that encode the spatiotemporal information of the video.

[0052] Nonlinear spatiotemporal relative position offset generation: This method calculates the three-dimensional relative position relationship between each marker in a frame sequence, including the displacement differences in the horizontal (x-axis), vertical (y-axis), and temporal (z-axis) dimensions. Based on this three-dimensional relative position relationship, and taking into account the characteristic that the influence of relative relationships decreases with increasing distance, a nonlinear function is designed to dynamically adjust the offset weights, emphasizing the correlation of closer regions.

[0053] (1) Define the reference coordinates of video data

[0054] The dimensions of the video data are set to (t,h,w), which represent the number of time frames, height, and width respectively. The spatial and temporal coordinates of each cube are defined as: the bottom left point (t1,x1,y1) and the top right point (t2,x2,y2). The meshgrid method is used to generate a grid of reference point coordinates, representing all possible cube coordinates:

[0055] grid=meshgrid(range(t),range(h),range(w))

[0056] Here, the range(t) function is used to generate a sequence of integers starting at 0 and ending at t with a step size of 1. The meshgrid function accepts multiple one-dimensional arrays as input and outputs a multidimensional array of grid coordinates, representing all possible coordinate combinations.

[0057] Construct the reference point coordinates (t1,x1,y1,t2,x2,y2) to represent the six-dimensional coordinates of each cube:

[0058] (t1,x1,y1,t2,x2,y2)=

[0059] stack(grid(t),grid(h),grid(w),grid(t)+1,grid(h)+1,grid(w)+1)

[0060] Here, the stack() function stacks multiple arrays along a new axis, combining the time, width, and height ranges of each grid point into the six-dimensional coordinates of a cube. The shape of this tensor is (n,1,6), where n = t × h × w, and each row represents the (t1,x1,y1,t2,x2,y2) of a cube.

[0061] (2) Define the coordinate sequence of the target point

[0062] Initialize the target point (t c ,x c ,y c )'s time, width, and height positions:

[0063] Time Series:

[0064] t c =linspace(0.5,t-0.5,t)

[0065] Height sequence:

[0066] x c =linspace(0.5,h-0.5,h)

[0067] Width sequence:

[0068] y c =linspace(0.5,w-0.5,w)

[0069] Here, linspace(0.5,t-0.5,t) is used to generate t values ​​uniformly distributed between 0.5 and t-0.5. The shapes of the above sequences are (1,t,1), (1,h,1), and (1,w,1), respectively.

[0070] (3) Calculate the relative coordinates of the cube

[0071] For each cube, calculate the relative coordinates of the bottom left point and the top right point to the target point:

[0072] Time dimension:

[0073] Δt1=t1-t c ,Δt2=t2-t c

[0074] Height dimension:

[0075] Δx1=x1-x c ,Δx2=x2-x c

[0076] Width dimension:

[0077] Δy1=y1-y c,Δy2=y3-y c

[0078] (4) Nonlinear transformation

[0079] In the above relative coordinate calculation process, the reference point (t c ,x c ,y c The relative coordinates of the target position (t1, t2, x1, y1, x2, y2) are defined by linear subtraction. This linear calculation means that the relative offset grows uniformly with increasing spatiotemporal distance. However, in practical applications, the influence of relative position relationships on the model often varies nonlinearly, meaning that the effect of the offset on attention decreases more rapidly with increasing distance.

[0080] To better capture this variation, we propose a nonlinear transformation that uses a Gaussian distribution to represent how the effect of position offset gradually decreases with increasing distance: it decreases slowly in the initial stage and decreases faster at longer distances.

[0081] Taking the relative coordinate Δt1 of the time axis as an example, we introduce the following nonlinear transformation:

[0082]

[0083] Where sign(x) returns the sign of x. The specific rules are:

[0084]

[0085] Used to determine the direction of the relative offset (positive or negative). Through the above formula, Δt1 describes the nonlinear change of the relative time offset: Initial stage: When |t1-t c When | is small, the offset decays slowly. Long distance stage: When |t1-t c | Beyond a certain range, the offset quickly stabilizes.

[0086] In order to make the Gaussian distribution of the time dimension reasonably reflect the relationship between frames of video data, we set Make sure to stay within the timeframe The frames within are assigned larger bias weights to better capture the similarity of temporally adjacent frames. Similarly, we set Make sure to stay within the height dimension Inner and width dimension range Markers within are assigned larger bias weights, which better capture the similarity of spatially neighboring markers.

[0087] (5) Construct bias mapping function

[0088] Three multi-layer perceptrons (MLPs) process the relative coordinates of time, width, and height respectively. The nonlinear bias of each dimension is calculated:

[0089] Time offset:

[0090] B t =MLP1(Δt1,Δt2)

[0091] The MLP input shape is (n×t×2) and the output is (n×t×m).

[0092] Width offset:

[0093] B x =MLP2(Δx1,Δx2)

[0094] The MLP input shape is (n×w×2) and the output is (n×h×m).

[0095] Height bias:

[0096] B y =MLP3(Δy1,Δy2)

[0097] The MLP input shape is (n×h×2) and the output is (n×w×m).

[0098] (6) Synthesize the final relative position offset

[0099] To improve computational efficiency, the spatiotemporal bias is decomposed into independent bias terms in the time (t), horizontal (x), and vertical (y) dimensions, and each term is processed independently, thereby significantly reducing computational complexity. The relative position bias B' is calculated as follows:

[0100] B'=unsqueeze(B t ,2,3)+unsqueeze(B x ,1,3)+unsqueeze(B y ,1,2),

[0101] The unsqueeze(·) function is used to expand the dimension, as follows:

[0102]

[0103] This decomposition makes the calculation of each dimension independent of each other, saving computational overhead.

[0104] Improved attention mechanism modeling: In the self-attention mechanism of the Transformer model, the nonlinear spatiotemporal relative position bias is used to improve the formula For the traditional bias term B′, the original relative position bias is retained as the global adjustment factor. The input adds a set of learnable vectors, enhancing the flexible modeling capability of spatio-temporal features. The nonlinear spatio-temporal relative position bias is integrated into the attention mechanism to extract key spatio-temporal features in the video through the multi-head attention mechanism. In the traditional self-attention mechanism, in order to realize the modeling of the position information, a relative position bias item B' is first introduced:

[0105]

[0106] wherein, is a relative position bias item, denote the query, key and value matrices respectively, d is the dimension thereof, and n is the number of image tokens. Q, K and V represent the query, key and value matrices obtained by linear transformation of the input sequence x through projection matrices W Q , W K and . At this time, the attention is calculated as Further, in order to more effectively introduce the relative position information, a learnable vector v is added before the input x. This process can be represented as [v,x]W = [vW,xW], wherein v is a learnable vector, and vW serves as an additional component of the relative position bias. The two parts of vW and xW correspond to the l x n and n x n parts in the attention weight graph, and in an (l+n) x (l+n) attention weight matrix, l represents the number of components of the additional input.

[0107] The initial shape of the learnable vector v is l x d, wherein l is the length of the vector. Through a multi-layer perceptron (MLP), the vector is converted from l x d to l x 6, and each vector corresponds to a six-dimensional coordinate (t1, x1, y1, t2, x2, y2). These coordinates represent the relative time and space positions, which are key to capturing spatio-temporal relationships.

[0108] Next, similar to steps (3) and (4), the relative coordinates (Δt1, Δt2), (Δx1, Δx2) and (Δy1, Δy2) are generated.

[0109]

[0110] wherein, t c , x c , y c , σ t , σ x , σ y have the same meanings as in steps (3) and (4).

[0111] These relative coordinates are processed by a multi-layer perceptron (MLP) to generate a relative position bias term B″ of shape l×n. Finally, the generated bias term B″ is concatenated with the original bias B′ to obtain an intermediate value of size (l+n)×n. This intermediate value is padded with zeros and expanded to a relative position code B of size (l+n)×(l+n), thereby integrating the relative position information into the attention mechanism and ultimately forming a feature map of size (l+n)×(l+n).

[0112] Action Detection and Classification: The classification module classifies the extracted features into action categories. The bounding box regression module predicts the spatiotemporal location of the action. Combining the classification results and location predictions, a complete action detection result is generated, including the action category and the corresponding spatiotemporal range.

[0113] The query-based localization module uses multi-scale keyframe features to predict actor bounding boxes. Multi-scale feature representations are obtained by upsampling or downsampling the intermediate feature maps generated by ViT. These features are fused via a feature pyramid network to capture detailed information at different resolutions. Sparse R-CNN is used to generate 100 bounding box proposals, providing the basis for the subsequent localization task. The classification module further refines the spatiotemporal features and models the relationship between the action regions and the contextual labels generated by the ViT encoder to achieve accurate action classification. Finally, the labels at the ViT encoder layer are reshaped into a spatiotemporal map. The bounding box proposals generated by the localization module are expanded to the entire video and subjected to a 3D RoIAlign operation to extract regional features of the action. The extracted features are then passed to the decoder for further refinement, outputting accurate action classification results.

[0114] The nonlinear spatiotemporal relative position bias solves the problem that traditional position encoding methods cannot capture complex spatiotemporal relationships by introducing six-dimensional cube coordinates and nonlinear spatiotemporal bias design.

[0115] The six-dimensional cube coordinates are defined as follows: each space-time volume is represented by the six-dimensional coordinates of the base and top corners (t1, x1, y1, t2, x2, y2). Each video frame is divided into multiple cubes, the total number of which is equal to the product of the number of time frames and the width and height of each frame. Each cube serves as a reference point or target point for subsequent offset calculations.

[0116] The relative space-time offset calculation step is to obtain the relative coordinates by calculating the difference between the reference point and the target point. The difference is calculated for the base angle and the top angle respectively. For the time dimension offset, the time offset of the base angle is equal to the start time of the reference point minus the center time of the target point, and the time offset of the top angle is the end time of the reference point minus the center time of the target point. The width and height offsets of the spatial dimension are calculated in a similar way. Finally, six relative offsets are obtained, corresponding to the base angle and top angle differences of time, width and height respectively. These offsets capture the relative relationship between the target point and the reference point.

[0117] The reference and target point generation steps involve constructing the six-dimensional cube coordinates of the reference points using a three-dimensional grid generator (e.g., the meshgrid function), where each coordinate represents the starting and ending positions of the bottom and top corners. The target points are initialized using a uniform distribution to ensure that the center positions of time, width, and height cover the entire video frame.

[0118] The nonlinear mapping step involves using a multi-layer perceptron (MLP) to perform a nonlinear transformation on the six biases, mapping them to a new space. The MLP consists of two fully connected layers and a ReLU activation function. Its input is a vector of six biases, and its output is the transformed weighted bias values, with dimensions matching the requirements of the multi-head attention mechanism.

[0119] The bias decoupling design involves breaking down the bias calculation into three parts: time, width, and height. These are then calculated independently and then superimposed. This decoupling approach significantly reduces the complexity of the original global calculation.

[0120] Compared with the traditional method of directly calculating the global bias between the reference point and the target point, the decomposition method of nonlinear spatiotemporal relative position bias distributes the originally complex calculation into three dimensions, avoiding high computational overhead.

[0121] This method can capture complex spatiotemporal relationships through nonlinear bias, significantly improving action detection performance. This method does not require additional predefined reference points and can be flexibly integrated into mainstream backbone networks such as the Vision Transformer (ViT).

[0122] Compared with the prior art, the present invention has the following advantages:

[0123] 1. Enhanced spatiotemporal relationship modeling capabilities: By introducing cube relative position bias, the shortcomings of traditional one-dimensional position encoding are improved, and the complex spatiotemporal relationships in videos can be better captured.

[0124] 2. Improved detection accuracy: Through nonlinear position bias design, the model pays more attention to key areas at close distances, improving the accuracy of action detection.

[0125] 3. Superior performance: Experimental verification on multiple mainstream datasets (such as AVA, UCF101-24, and JHMDB51-21) shows that this method significantly outperforms existing ViT-based technologies and has higher detection performance and robustness.

[0126] In order to verify the performance of the above method, the following experiments were designed in this embodiment.

[0127] This example evaluates the performance of the method of the present invention on three widely used video action detection datasets.

[0128] AVA: A large-scale benchmark dataset containing 299 15-minute videos, divided into 211,000 training segments and 57,000 validation segments. The dataset is based on keyframes sampled per second, with bounding boxes and action labels annotated at 1FPS.

[0129] UCF101-24: A subset of the UCF101 dataset, containing 3,207 videos from 24 sports-related action categories. Each video is annotated with an action category.

[0130] JHMDB51-21: This dataset contains 928 cropped videos covering 21 action categories.

[0131] The results of the entire experiment are as follows:

[0132] This example compares the proposed method with the most advanced methods. The results are shown in Table 1 and Table 2.

[0133] Table 1 Comparison with state-of-the-art methods on AVA

[0134]

[0135]

[0136] Table 2 Comparison with state-of-the-art methods on UCF101-24 and JHMDB51-21

[0137] Model Backbone JHMDB UCF24 AVA I3D-VGG 73.3 76.3 ACRN S3D-G 77.9 - YOW 3D-X101 80.4 75.7 WOO SF-R101-NL 80.5 - TubeR I3D 80.7 81.3 TubeR CSN-152 - 83.2 AIA R50-C2D - 78.8 STMixer SF-R101-NL 86.7 83.7 CubeRPB ViT-B 89.3 83.8

[0138] (1) Results on the AVA dataset: Table 1 compares the performance of this method with several state-of-the-art models on the AVAv2.2 benchmark. SlowFast and ACAR-Net use the SF-R101-NL backbone network and demonstrate strong performance; MeMViT significantly improves mAP by exploring different variants of the MViT backbone network; VideoMAE utilizes the ViT-B backbone and focuses on video mask modeling, while EVAD introduces spatiotemporal Token Dropout based on the same backbone and achieves excellent performance on both the K400 and K710+K400 pre-training sets; STMixer further improves performance on the CSN-152 and ViT-B backbones through its query-based adaptive feature design. In contrast, this method achieves excellent performance on the AVA v2.2 dataset, fully demonstrating its effectiveness.

[0139] (2) Results on the UCF101-24 and JHMDB51-21 datasets: Table 2 compares the performance of this method with other advanced methods on the JHMDB51-21 and UCF101-24 action recognition benchmarks. It is worth noting that TubeR, based on the I3D and CSN-152 backbones, performs well on both datasets, especially on UCF101-24; the YOWO method performs competitively on JHMDB, but performs slightly worse on UCF101-24; STMixer, based on the SF-R101-NL backbone, performs well on both datasets, reaching 86.7% (JHMDB) and 83.7% (UCF101-24) respectively. Our method, based on the ViT-B backbone, achieves 89.3% on JHMDB and 83.8% on UCF101-24, creating new benchmarks for these two datasets and demonstrating outstanding performance advantages.

[0140] The above describes in detail the preferred embodiments of the present invention. It should be understood that those skilled in the art can make numerous modifications and variations based on the concepts of the present invention without inventive effort. Therefore, any technical solutions that can be derived by those skilled in the art through logical analysis, reasoning, or limited experimentation based on the concepts of the present invention and the prior art should be within the scope of protection defined by the claims.

Claims

1. A video action detection method based on nonlinear spatiotemporal relative position bias, characterized in that: The method comprises the following steps: Obtain video clips, convert the video clips into a series of tags, and obtain video data; The video data is input into the Transformer-based video action detection model. Each frame of the video data in the model is divided into multiple cubes, and a grid of reference point coordinates is generated. The reference point coordinates (t1, x1, y1, t2, x2, y2) are constructed, where t1, x1, y1 are the time frame coordinates, height coordinates, and width coordinates of the bottom left point of the cube, and t2, x2, y2 are the time frame coordinates, height coordinates, and width coordinates of the top right point of the cube. The coordinate sequence of the target point is defined, and the relative coordinates of the cube are calculated using a nonlinear transformation based on the target point and the reference point. Input the relative coordinates of the cube into the multi-layer perceptron of the video action detection model, output the nonlinear bias of each dimension, decouple and calculate the improved first relative position bias based on the nonlinear bias of each dimension, obtain the learnable vector v, pass the learnable vector v through the multi-layer perceptron, generate six-dimensional coordinates, calculate the relative coordinates of the learnable vector v based on the six-dimensional coordinates, generate the second relative position bias through the multi-layer perceptron, input the learnable vector and video data into the improved attention mechanism of the video action detection model, the attention mechanism outputs key spatiotemporal features, and the improved attention mechanism adopts the improved relative position bias B generated based on the first relative position bias and the second relative position bias; Key spatiotemporal features are input into the classification module of the model to obtain video action detection results.

2. The video action detection method based on nonlinear spatiotemporal relative position bias according to claim 1, characterized in that: The coordinates of the reference point (t1,x1,y1,t2,x2,y2) are: (t1,x1,y1,t2,x2,y2)= stack(grid(t),grid(h),grid(w),grid(t)+1,grid(h)+1,grid(w)+1) Among them, stack() is the overlay function, which overlays multiple arrays along a new axis and combines the time, width, and height ranges of each grid point into the six-dimensional coordinates of a cube. grid represents the grid, and t, h, and w represent the time, image height, and width dimensions, respectively.

3. The video action detection method based on nonlinear spatiotemporal relative position bias according to claim 2, characterized in that: The coordinate sequence of the target point is: t c =linspace(0.5,t-0.5,t) x c =linspace(0.5,h-0.5,h) and c =linspace(0.5,w-0.5,w) Among them, t c 、x c 、y c Respectively represent the time frame coordinate, height coordinate and width coordinate of the target point, 4. The video motion detection method based on nonlinear spatiotemporal relative position offset according to claim 3, characterized in that: The relative coordinates of the cube are: Wherein, Δt1 represents the relative coordinate of the bottom left point in the relative coordinate of the cube in terms of the number of time frames, Δx1 represents the relative coordinate of the height of the bottom left point in the relative coordinate of the cube, and Δy1 represents the relative coordinate of the bottom left point in terms of the number of time frames in the width coordinate of the cube; The relative coordinates of the learnable vector v are calculated using the same formula.

5. The video action detection method based on nonlinear spatiotemporal relative position offset according to claim 4, characterized in that: The nonlinear bias for each dimension is: B t =MLP1(Δt1,Δt2) B x =MLP2(Δx1,Δx2) B y =MLP3(Δy1,Δy2) Among them, B t is the nonlinear bias in the dimension of time frames, B x is the nonlinear bias in the height dimension, B y is the nonlinear bias of the width dimension, and MLP1, MLP2 and MLP3 are multi-layer perceptrons.

6. The video action detection method based on nonlinear spatiotemporal relative position offset according to claim 5, characterized in that: The first relative position offset is: B′=unsqueeze(B t ,2,3)+unsqueeze(B x ,1,3)+unsqueeze(B y ,1,2) Among them, unsqueeze(·) is the expansion dimension function.

7. The video motion detection method based on nonlinear spatiotemporal relative position offset according to claim 1, characterized in that: The grid of the reference point coordinates is constructed based on the meshgrid function.

8. The video action detection method based on nonlinear spatiotemporal relative position offset according to claim 1, characterized in that: The key spatiotemporal features are input into the classification module of the model to obtain the video action detection results. The specific steps are as follows: The key spatiotemporal features are upsampled or downsampled to obtain multi-scale features and fused with a pyramid network to generate a bounding box. The fused features are passed to the decoder and then passed through the classification module to obtain the video action detection results.

9. The video motion detection method based on nonlinear spatiotemporal relative position offset according to claim 1, characterized in that: The specific steps to convert a video clip into a series of tags are: The video clip is divided into several consecutive frames, which are sequentially formed into a frame sequence, and the frame sequence is encoded as a series of tags.

10. The video action detection method based on nonlinear spatiotemporal relative position offset according to claim 1, characterized in that: The input of the improved attention mechanism is [v,x], where v represents the learnable vector and x represents the video data. The improved attention mechanism is: Among them, Q, K, V are query, key and value matrices respectively, d represents the dimension, and B is the improved relative position bias, which is obtained by concatenating and expanding the first relative position bias and the second relative position bias.

Images