Video motion detection method based on nonlinear space-time relative position bias

Abstract

The invention designs a video action detection method based on nonlinear space-time relative position offset, which is characterized by comprising the following steps of: acquiring video data; inputting the video data into a Transform-based video action detection model, and calculating relative coordinates of a cube by adopting nonlinear transformation based on a target point and a reference point; outputting a nonlinear offset of each dimension, calculating an improved relative position offset B, and outputting key spatio-temporal characteristics by an attention mechanism; and inputting the key spatial-temporal characteristics into a classification module of the model to obtain a video action detection result. Compared with the prior art, the method has the advantages of improving the accuracy of action detection, reducing the calculation amount and the like.

Description

Technical Field

[0001] The invention relates to the technical field of video action detection, and in particular to a video action detection method based on nonlinear time-space relative position bias. Background Art

[0002] Video action detection aims to identify and classify specific actions or activities in video sequences. The goal is to accurately locate the temporal and spatial locations of the actions in the video, thereby answering the question of "what happened, where, and when did it happen". This task is of great significance in many application areas, such as human-computer interaction, video content analysis, and intelligent video surveillance. In recent years, methods based on Vision Transformer (ViT) have made significant progress in video classification tasks. ViT uses the self-attention mechanism to capture long-range spatiotemporal dependencies and shows unique advantages in modeling global context. However, although these features provide powerful representation capabilities for video tasks, the performance of ViT in video action detection still faces challenges, especially in the spatiotemporal localization of actions.

[0003] Existing ViTs usually use a one-dimensional position embedding method, which is inspired by the position encoding method in natural language processing. However, unlike language data, video data contains complex structural relationships in both time and space dimensions. This one-dimensional encoding method is difficult to fully express the spatiotemporal position information in the video. In the task of action detection, the comprehensive and effective introduction of position information is crucial to improving detection performance. Although some studies have tried to combine position encoding and relative position bias techniques, these methods are mainly designed for image tasks and have not been extended to the spatiotemporal dimension of video analysis. Therefore, the direct application of position encoding methods in image detection to video action detection often has limited effect. This is because the position information of the video not only involves two-dimensional space, but also needs to model the dynamic changes in the time dimension, which is much more complex than the analysis requirements of a single frame image, resulting in insufficient accuracy and high computational complexity of action detection. Summary of the invention

[0004] The purpose of the present invention is to provide a video action detection method based on nonlinear spatiotemporal relative position bias in order to improve the accuracy of action detection while reducing the amount of calculation.

[0005] The purpose of the present invention can be achieved by the following technical solutions:

[0006] A video action detection method based on nonlinear spatiotemporal relative position bias, the method comprising the following steps:

[0007] Obtain a video clip, convert the video clip into a series of tags, and obtain video data;

[0008] The video data is input into the Transformer-based video action detection model. In the model, each frame of the video data is divided into multiple cubes, and a grid of reference point coordinates is generated to construct the reference point coordinates (t 1 ,x 1 ,y 1 ,t 2 ,x 2 ,y 2 ), where t 1 ,x 1 ,y 1 is the time frame coordinate, height coordinate and width coordinate of the left point at the bottom of the cube, t 2 ,x 2 ,y 2 The time frame number coordinates, height coordinates and width coordinates of the right point on the top of the cube are used to define the coordinate sequence of the target point, and the relative coordinates of the cube are calculated based on the target point and the reference point using nonlinear transformation;

[0009] Input the relative coordinates of the cube into the multi-layer perceptron of the video action detection model, output the nonlinear bias of each dimension, calculate the improved first relative position bias based on the nonlinear bias decoupling of each dimension, obtain the learnable vector v, pass the learnable vector v through the multi-layer perceptron, generate six-dimensional coordinates, calculate the relative coordinates of the learnable vector v based on the six-dimensional coordinates, generate the second relative position bias through the multi-layer perceptron, input the learnable vector and video data into the improved attention mechanism of the video action detection model, the attention mechanism outputs key spatiotemporal features, and the improved attention mechanism adopts the improved relative position bias B generated based on the first relative position bias and the second relative position bias;

[0010] The key spatiotemporal features are input into the classification module of the model to obtain the video action detection results.

[0011] Furthermore, the reference point coordinates (t 1 ,x 1 ,y 1 ,t 2 ,x 2 ,y 2 )for:

[0012] (t 1 ,x 1 ,y 1 ,t 2 ,x 2 ,y 2 )=

[0013] stack(grid(t),grid(h),grid(w),grid(t)+1,grid(h)+1,grid(w)+1)

[0014] Among them, stack() is the superposition function, which superimposes multiple arrays along the new axis and combines the time, width and height range of each grid point into the six-dimensional coordinates of a cube. grid represents the grid, and t, h and w represent the time, image height and width dimensions respectively.

[0015] Furthermore, the coordinate sequence of the target point is:

[0016] t c =linspace(0.5,t-0.5,t)

[0017] x c =linspace(0.5,h-0.5,h)

[0018] y c =linspace(0.5,w-0.5,w)

[0019] Among them, t c 、x c ,y c Respectively represent the time frame coordinate, height coordinate and width coordinate of the target point,

[0020] Furthermore, the relative coordinates of the cube are:

[0021]

[0022] Among them, Δt 1 Represents the relative coordinates of the bottom left point of the cube in time frames, Δx 1 Represents the relative coordinates of the height of the bottom left point in the relative coordinates of the cube, Δy 1 Represents the relative coordinates of the bottom left point in the width coordinates of the cube in terms of the number of time frames;

[0023] The relative coordinates of the learnable vector v are calculated using the same formula.

[0024] Furthermore, the nonlinear bias in each dimension is:

[0025] B t =MLP 1 (Δt 1 ,Δt 2 )

[0026] B x =MLP 2 (Δx 1 ,Δx 2 )

[0027] B y=MLP 3 (Δy 1 ,Δy 2 )

[0028] Among them, B t is the nonlinear bias in the dimension of the number of time frames, B x is the nonlinear bias in the height dimension, B y is the nonlinear bias of the width dimension, MLP 1 、MLP 2 and MLP 3 A multi-layer perceptron.

[0029] Further, the first relative position offset is:

[0030] B′=unsqueeze(B t ,2,3)+unsqueeze(B x ,1,3)+unsqueeze(B y ,1,2)

[0031] Among them, unsqueeze(·) is the expansion dimension function.

[0032] Furthermore, the grid of the reference point coordinates is constructed based on a meshgrid function.

[0033] Furthermore, the key spatiotemporal features are input into the classification module of the model to obtain the video action detection results in the following specific steps:

[0034] The key spatiotemporal features are upsampled or downsampled to obtain multi-scale features and fused with the pyramid network to generate a bounding box. The fused features are passed to the decoder and then passed through the classification module to obtain the video action detection results.

[0035] Furthermore, the specific steps of converting the video clip into a series of tags are as follows:

[0036] The video clip is divided into several consecutive frames, which are sequentially composed of frame sequences, and the frame sequences are encoded as a series of tags.

[0037] Furthermore, the input of the improved attention mechanism is [v, x], where v represents the learnable vector and x represents the video data. The improved attention mechanism is:

[0038]

[0039] Among them, Q, K, V are query, key and value matrices respectively, d represents the dimension, B is the improved relative position bias, and the improved relative position bias is obtained by concatenating and expanding the first relative position bias and the second relative position bias.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] The present invention improves the shortcomings of traditional one-dimensional position encoding by introducing cube relative position bias, and can better capture the complex spatiotemporal relationship in the video. Through the nonlinear position bias design, the model pays more attention to the key areas with closer distances, thereby improving the accuracy of action detection. Compared with the traditional method of directly calculating the global bias between the reference point and the target point, the decomposition method of the nonlinear spatiotemporal relative position bias distributes the originally complex calculation to three dimensions, thus avoiding high computational overhead. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a flowchart of the present invention. DETAILED DESCRIPTION

[0043] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0044] The present invention proposes a video action detection method based on nonlinear spatiotemporal relative position bias. First, the video frame sequence is encoded into a token sequence to construct a unified input feature representation. Then, a new type of nonlinear spatiotemporal relative position bias is introduced. This method first calculates the relative position relationship between each token, including horizontal, vertical and time dimensions, and generates position bias values ​​based on a nonlinear formula to construct a three-dimensional relative position encoding that is more in line with the characteristics of the video. Then, the nonlinear spatiotemporal relative position bias is integrated into the attention mechanism to improve the attention. The formula is improved, retaining the traditional bias term B′ while adding a learnable prefix vector to enhance the model's ability to capture spatiotemporal positions. Finally, under the improved attention mechanism, multi-layer Transformer modeling is performed on the input features to extract key features within the spatiotemporal range of the video. Through the classification module and the bounding box regression module, the category and spatiotemporal position of the action in the video are output. Compared with the prior art, the present invention has the advantages of strong spatiotemporal relationship modeling capabilities and high accuracy of action detection. The method comprises the following steps:

[0045] Obtain a video clip, convert the video clip into a series of tags, and obtain video data;

[0046] The video data is input into the Transformer-based video action detection model. In the model, each frame of the video data is divided into multiple cubes, and a grid of reference point coordinates is generated to construct the reference point coordinates (t 1 ,x 1 ,y 1 ,t2 ,x 2 ,y 2 ), where t 1 ,x 1 ,y 1 is the time frame coordinate, height coordinate and width coordinate of the left point at the bottom of the cube, t 2 ,x 2 ,y 2 The time frame number coordinates, height coordinates and width coordinates of the right point on the top of the cube are used to define the coordinate sequence of the target point, and the relative coordinates of the cube are calculated based on the target point and the reference point using nonlinear transformation;

[0047] The relative coordinates of the cube are input into the multi-layer perceptron of the video action detection model, and the nonlinear bias B′ of each dimension is output. The improved relative position bias B is calculated based on the nonlinear bias decoupling of each dimension. The video data including the learnable vector is input into the improved attention mechanism of the video action detection model, and the attention mechanism outputs key spatiotemporal features.

[0048] The key spatiotemporal features are input into the classification module of the model to obtain the video action detection results. The flowchart is as follows Figure 1 shown.

[0049] The specific steps of the present invention are:

[0050] Video preprocessing: Split the input video sequence into several consecutive frames and form a frame sequence in order. Encode the frame sequence into spatiotemporal tags for the input of the subsequent Transformer model.

[0051] First, the video clip is encoded in blocks using the cube embedding method and converted into a series of tokens. This process represents the input video as a t×h×w three-dimensional grid, where t, h, and w represent the time, image height, and width dimensions respectively. Then, position embedding is added to these tokens, resulting in a sequence of n=t×h×w tokens, which are used to encode the spatiotemporal information of the video.

[0052] Nonlinear spatiotemporal relative position offset generation: Calculate the three-dimensional relative position relationship between each marker in the frame sequence, including the displacement difference in the horizontal (x-axis), vertical (y-axis) and time (z-axis) dimensions. Based on the three-dimensional relative position relationship, according to the characteristic that the influence of the relative relationship decreases as the distance increases, a nonlinear function is designed to dynamically adjust the weight of the offset and emphasize the correlation of closer areas.

[0053] (1) Define the reference coordinates of video data

[0054] The dimensions of the video data are set to (t,h,w), which represent the number of time frames, height, and width respectively. The spatial and temporal coordinates of each cube are defined as: the bottom left point (t1 ,x 1 ,y 1 )Top right point (t 2 ,x 2 ,y 2 ). The meshgrid method generates a grid of reference point coordinates, representing all possible cube coordinates:

[0055] grid=meshgrid(range(t),range(h),range(w))

[0056] Here, the range(t) function is used to generate a sequence of integers starting at 0, ending at t, and with a step size of 1. The meshgrid function accepts multiple one-dimensional arrays as input and outputs a multidimensional array of grid coordinates, representing all possible coordinate combinations.

[0057] Construct reference point coordinates (t 1 ,x 1 ,y 1 ,t 2 ,x 2 ,y 2 ), representing the six-dimensional coordinates of each cube:

[0058] (t 1 ,x 1 ,y 1 ,t 2 ,x 2 ,y 2 )=

[0059] stack(grid(t),grid(h),grid(w),grid(t)+1,grid(h)+1,grid(w)+1)

[0060] Here, the stack() function stacks multiple arrays along a new axis, combining the time, width, and height ranges of each grid point into the six-dimensional coordinates of a cube. The shape of this tensor is (n,1,6), where n = t×h×w, and each row represents a cube (t 1 ,x 1 ,y 1 ,t 2 ,x 2 ,y 2 ).

[0061] (2) Define the coordinate sequence of the target point

[0062] Initialize the target point (t c ,x c ,y c )’s time, width, and height positions:

[0063] Time Series:

[0064] t c =linspace(0.5,t-0.5,t)

[0065] Height sequence:

[0066] x c =linspace(0.5,h-0.5,h)

[0067] Width sequence:

[0068] y c =linspace(0.5,w-0.5,w)

[0069] Here, linspace(0.5,t-0.5,t) is used to generate t values ​​uniformly distributed between 0.5 and t-0.5. The shapes of the above sequences are (1,t,1), (1,h,1), and (1,w,1).

[0070] (3) Calculate the relative coordinates of the cube

[0071] For each cube, calculate the relative coordinates of the bottom left point and the top right point to the target point:

[0072] Time Dimension:

[0073] Δt 1 =t 1 -t c ,Δt 2 =t 2 -t c

[0074] Height Dimension:

[0075] Δx 1 =x 1 -x c ,Δx 2 =x 2 -x c

[0076] Width dimension:

[0077] Δy 1 =y 1 -y c ,Δy 2 =y 3 -y c

[0078] (4) Nonlinear transformation

[0079] In the above relative coordinate calculation process, the reference point (tc ,x c ,y c ) and target position (t 1 ,t 2 ,x 1 ,y 1 ,x 2 ,y 2 ) is defined by linear subtraction. This linear calculation means that the relative offset will grow uniformly with the increase of spatiotemporal distance. However, in practical applications, the influence of relative position relationship on the model usually changes nonlinearly, that is, the greater the distance, the faster the influence of offset on attention decreases.

[0080] To better capture this variation, we propose a nonlinear transformation that uses a Gaussian distribution to represent how the effect of position offset gradually decreases with increasing distance: it decreases more slowly in the initial stage and decreases faster at longer distances.

[0081] The relative coordinate Δt of the time axis 1 As an example, we introduce the following nonlinear transformation:

[0082]

[0083] Where sign(x) returns the sign of x. The specific rules are:

[0084]

[0085] Used to determine the direction of the relative offset (positive or negative). Using the above formula, Δt 1 Describes the nonlinear change of relative time offset: Initial stage: When |t 1 -t c When | is small, the offset decays slowly. Long-distance stage: When |t 1 -t c | Beyond a certain range, the offset quickly stabilizes.

[0086] In order to make the Gaussian distribution of the time dimension reasonably reflect the inter-frame relationship of the video data, we set Make sure to stay within the time frame The frames within are assigned larger bias weights to better capture the similarity of temporally adjacent frames. Similarly, we set Make sure to stay within the height dimension Inner and width dimension range Markers within are assigned larger bias weights, thereby better capturing the similarity of spatially neighboring markers.

[0087] (5) Construct bias mapping function

[0088] Three multi-layer perceptrons (MLPs) process the relative coordinates of time, width, and height respectively. The nonlinear bias for each dimension is calculated:

[0089] Time offset:

[0090] B t =MLP 1 (Δt 1 ,Δt 2 )

[0091] The MLP input shape is (n×t×2) and the output is (n×t×m).

[0092] Width offset:

[0093] B x =MLP 2 (Δx 1 ,Δx 2 )

[0094] The MLP input shape is (n×w×2) and the output is (n×h×m).

[0095] Height Bias:

[0096] B y =MLP 3 (Δy 1 ,Δy 2 )

[0097] The MLP input shape is (n×h×2) and the output is (n×w×m).

[0098] (6) Synthesize the final relative position offset

[0099] In order to improve computational efficiency, the spatiotemporal bias is decomposed into independent bias terms in the time (t), horizontal (x), and vertical (y) dimensions, and each term is processed independently, thereby significantly reducing computational complexity. The relative position bias B' is calculated as follows:

[0100] B'=unsqueeze(B t ,2,3)+unsqueeze(B x ,1,3)+unsqueeze(B y ,1,2),

[0101] Among them, unsqueeze(·) is used to expand the dimension, as follows:

[0102]

[0103] This decomposition makes the calculation of each dimension independent of each other, saving computational overhead.

[0104] Improved attention mechanism modeling: In the self-attention mechanism of the Transformer model, the nonlinear spatiotemporal relative position bias is used to improve the formula For the traditional bias term B′, the original relative position bias is retained as the global adjustment factor. A set of learnable vectors are added to the input to enhance the flexible modeling capability of spatiotemporal features. The nonlinear spatiotemporal relative position bias is integrated into the attention mechanism, and the key spatiotemporal features in the video are extracted through the multi-head attention mechanism. In the traditional self-attention mechanism, in order to achieve the modeling of position information, a relative position bias term B′ is first introduced:

[0105]

[0106] in, is the relative position offset term, denote the query, key and value matrices respectively, d is its dimension and n is the number of image tokens. Q, K and V denote the projection matrix W Q , W K and The query, key, and value matrices obtained after linear transformation of the input sequence x. At this time, the attention is calculated as Furthermore, in order to more effectively introduce relative position information, a learnable vector v is added before the input x. This process can be expressed as [v,x]W = [vW,xW], where v is a learnable vector and vW is an additional component of the relative position bias. The two parts vW and xW correspond to the l×n and n×n parts in the attention weight map. In a (l+n)×(l+n) attention weight matrix, l represents the number of additional input components.

[0107] The initial shape of the learnable vector v is l×d, where l is the length of the vector. Through a multi-layer perceptron (MLP), the vector is transformed from l×d to l×6, where each vector corresponds to a six-dimensional coordinate (t 1 ,x 1 ,y 1 ,t 2 ,x 2 ,y 2 ). These coordinates represent relative positions in time and space and are key to capturing spatiotemporal relationships.

[0108] Next, similar to steps (3) and (4), generate the relative coordinates (Δt 1 ,Δt 2 )、(Δx 1 ,Δx 2 ) and (Δy 1 ,Δy 2 ).

[0109]

[0110] Among them, t c , x c ,y c , σ t , σ x , σ y The meaning is the same as in steps (3) and (4).

[0111] These relative coordinates are processed by a multi-layer perceptron (MLP) to generate a relative position bias term B″ of shape l×n. Finally, the generated bias term B″ is concatenated with the original bias B′ to obtain an intermediate value of size (l+n)×n. This intermediate value is padded with zeros and expanded to a relative position code B of size (l+n)×(l+n), thereby integrating the relative position information into the attention mechanism and ultimately forming a feature map of size (l+n)×(l+n).

[0112] Action detection and classification: Use the classification module to classify the extracted features into action categories. Use the bounding box regression module to predict the spatiotemporal location of the action. Combine the classification results and position prediction to generate a complete action detection result, including the action category and the corresponding spatiotemporal range.

[0113] The query-based localization module uses multi-scale keyframe features to predict the bounding boxes of actors. Multi-scale feature representations are obtained by upsampling or downsampling the intermediate feature maps generated by ViT. These features are fused through a feature pyramid network to capture detailed information at different resolutions. Sparse R-CNN is used to generate 100 bounding box proposals to provide a basis for subsequent localization tasks. The classification module further optimizes the spatiotemporal features and models the relationship between the action region and the contextual tags generated by the ViT encoder for accurate action classification. Finally, the tags of the ViT encoder layer are reshaped into a spatiotemporal map. The bounding box proposals generated by the localization module are extended to the entire video and subjected to a 3D RoIAlign operation to extract the regional features of the action. The extracted features are then passed to the decoder for further optimization to output accurate action classification results.

[0114] The nonlinear space-time relative position bias solves the problem that the traditional position encoding method cannot capture complex space-time relationships by introducing six-dimensional cube coordinates and nonlinear space-time bias design.

[0115] The six-dimensional cube coordinates are defined as follows: each space-time volume is represented by the six-dimensional coordinates of the bottom angle and the top angle (t 1 ,x 1 ,y 1 ,t 2 ,x 2 ,y 2). Each video frame is divided into multiple cubes, the total number of which is equal to the product of the number of time frames and the width and height of each frame. Each cube is used as a reference point or target point for subsequent offset calculation.

[0116] The relative space-time offset calculation step is to obtain the relative coordinates by calculating the difference between the reference point and the target point. The difference is calculated for the bottom angle and the top angle respectively. For the time dimension offset, the time offset of the bottom angle is equal to the start time of the reference point minus the center time of the target point, and the time offset of the top angle is the end time of the reference point minus the center time of the target point. The width and height offsets of the spatial dimension are calculated in a similar way. Finally, six relative offsets are obtained, corresponding to the bottom angle and top angle differences of time, width and height respectively. These offsets capture the relative relationship between the target point and the reference point.

[0117] The reference point and target point generation steps are as follows: the six-dimensional cube coordinates of the reference point are constructed by a three-dimensional grid generator (such as meshgrid function), and each coordinate is represented as the starting and ending positions of the bottom angle and the top angle. The target point is initialized by uniform distribution to ensure that the center position of time, width and height can cover the entire video frame.

[0118] The nonlinear mapping step is to use a multi-layer perceptron (MLP) to perform nonlinear transformation on the six biases and map them to a new space. The MLP includes two fully connected layers and a ReLU activation function. Its input is a vector composed of six biases, and its output is the transformed weight bias value. The dimension matches the requirements of the multi-head attention mechanism.

[0119] The bias decoupling design step is that the bias calculation is decomposed into three parts: time axis, width axis and height axis, which are calculated independently and superimposed. This decoupling method significantly reduces the complexity of the original global calculation.

[0120] Compared with the traditional method of directly calculating the global bias between the reference point and the target point, the decomposition method of the nonlinear spatiotemporal relative position bias distributes the originally complex calculation into three dimensions, avoiding high computational overhead.

[0121] This method can capture complex spatiotemporal relationships through nonlinear bias, significantly improving the performance of action detection. This method does not require additional predefined reference points and can be flexibly integrated into mainstream backbone networks such as Vision Transformer (ViT).

[0122] Compared with the prior art, the present invention has the following advantages:

[0123] 1. Enhanced spatiotemporal relationship modeling capabilities: By introducing the relative position bias of the cube, the shortcomings of traditional one-dimensional position encoding are improved, and the complex spatiotemporal relationships in the video can be better captured.

[0124] 2. Improved detection accuracy: Through nonlinear position bias design, the model pays more attention to key areas at close distances, thereby improving the accuracy of action detection.

[0125] 3. Superior performance: Experimental verification on multiple mainstream datasets (such as AVA, UCF101-24, and JHMDB51-21) shows that this method is significantly better than existing ViT-based technologies and has higher detection performance and robustness.

[0126] In order to verify the performance of the above method, the following experiment was designed in this embodiment.

[0127] This embodiment evaluates the performance of the method of the present invention on three widely used video action detection datasets.

[0128] AVA: A large-scale benchmark dataset containing 299 15-minute videos, divided into 211,000 training clips and 57,000 validation clips. The dataset is based on key frames sampled per second, with bounding boxes and action labels annotated at 1FPS.

[0129] UCF101-24: A subset of the UCF101 dataset containing 3,207 videos from 24 sports-related action categories. Each video is annotated with one action category.

[0130] JHMDB51-21: This dataset contains 928 cropped videos covering 21 action categories.

[0131] The results of the entire experiment are as follows:

[0132] This example compares the proposed method with the most recent state-of-the-art methods. The results are shown in Tables 1 and 2.

[0133] Table 1. Comparison with state-of-the-art methods on AVA

[0134]

[0135]

[0136] Table 2 Comparison with state-of-the-art methods on UCF101-24 and JHMDB51-21

[0137] Model Backbone JHMDB UCF24 AVA I3D-VGG 73.3 76.3 ACRN S3D-G 77.9 - YOWO 3D-X101 80.4 75.7 WOO SF-R101-NL 80.5 - TubeR I3D 80.7 81.3 TubeR CSN-152 - 83.2 AIA R50-C2D - 78.8 STMixer SF-R101-NL 86.7 83.7 CubeRPB ViT-B 89.3 83.8

[0138] (1) Results on the AVA dataset: Table 1 compares the performance of this method with several state-of-the-art models on the AVAv2.2 benchmark. SlowFast and ACAR-Net use the SF-R101-NL backbone network and show strong performance; MeMViT has significantly improved mAP by exploring different MViT backbone network variants; VideoMAE uses the ViT-B backbone and focuses on video mask modeling, while EVAD introduces spatiotemporal Token Dropout based on the same backbone and achieves excellent performance on both the K400 and K710+K400 pre-training sets; STMixer further improves performance on the CSN-152 and ViT-B backbones through its query-based adaptive feature design. In contrast, this method achieves excellent performance on the AVA v2.2 dataset, fully demonstrating its effectiveness.

[0139] (2) Results on UCF101-24 and JHMDB51-21 datasets: Table 2 compares the performance of this method with other advanced methods on the JHMDB51-21 and UCF101-24 action recognition benchmarks. It is worth noting that TubeR, based on the I3D and CSN-152 backbones, performs well on both datasets, especially on UCF101-24; the YOWO method performs competitively on JHMDB, but performs slightly worse on UCF101-24; STMixer, based on the SF-R101-NL backbone, performs well on both datasets, reaching 86.7 (JHMDB) and 83.7 (UCF101-24) respectively. This method, based on the ViT-B backbone, achieves 89.3 on JHMDB and 83.8 on UCF101-24, creating new benchmarks for these two datasets and demonstrating outstanding performance advantages.

[0140] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A video action detection method based on nonlinear spatiotemporal relative position bias, characterized in that: The method comprises the following steps: Obtain a video clip, convert the video clip into a series of tags, and obtain video data; The video data is input into the Transformer-based video action detection model. Each frame of the video data in the model is divided into multiple cubes. A grid of reference point coordinates is generated, and the reference point coordinates (t1, x1, y1, t2, x2, y2) are constructed. Among them, t1, x1, y1 are the time frame coordinates, height coordinates and width coordinates of the left point at the bottom of the cube, and t2, x2, y2 are the time frame coordinates, height coordinates and width coordinates of the right point at the top of the cube. The coordinate sequence of the target point is defined, and the relative coordinates of the cube are calculated based on the target point and the reference point using nonlinear transformation; Input the relative coordinates of the cube into the multi-layer perceptron of the video action detection model, output the nonlinear bias of each dimension, calculate the improved first relative position bias based on the nonlinear bias decoupling of each dimension, obtain the learnable vector v, pass the learnable vector v through the multi-layer perceptron, generate six-dimensional coordinates, calculate the relative coordinates of the learnable vector v based on the six-dimensional coordinates, generate the second relative position bias through the multi-layer perceptron, input the learnable vector and video data into the improved attention mechanism of the video action detection model, the attention mechanism outputs key spatiotemporal features, and the improved attention mechanism adopts the improved relative position bias B generated based on the first relative position bias and the second relative position bias; The key spatiotemporal features are input into the classification module of the model to obtain the video action detection results.

2. The video action detection method based on nonlinear spatiotemporal relative position bias according to claim 1, characterized in that: The coordinates of the reference point (t1,x1,y1,t2,x2,y2) are: (t1,x1,y1,t2,x2,y2)= stack(grid(t),grid(h),grid(w),grid(t)+1,grid(h)+1,grid(w)+1) Among them, stack() is the superposition function, which superimposes multiple arrays along the new axis and combines the time, width and height range of each grid point into the six-dimensional coordinates of a cube. grid represents the grid, and t, h and w represent the time, image height and width dimensions respectively.

3. The video action detection method based on nonlinear spatiotemporal relative position bias according to claim 2, characterized in that: The coordinate sequence of the target point is: t c =linspace(0.5,t-0.5,t) x c =linspace(0.5,h-0.5,h) and c =linspace(0.5,w-0.5,w) Among them, t c 、x c ,y c Respectively represent the time frame coordinate, height coordinate and width coordinate of the target point, 4. The video action detection method based on nonlinear spatiotemporal relative position bias according to claim 3, characterized in that: The relative coordinates of the cube are: Wherein, Δt1 represents the relative coordinate of the time frame number of the bottom left point in the relative coordinate of the cube, Δx1 represents the relative coordinate of the height of the bottom left point in the relative coordinate of the cube, and Δy1 represents the relative coordinate of the time frame number of the bottom left point in the width coordinate of the cube; The relative coordinates of the learnable vector v are calculated using the same formula.

5. The video action detection method based on nonlinear spatiotemporal relative position offset according to claim 4, characterized in that: The nonlinear bias for each dimension is: B t =MLP1(Δt1,Δt2) B x =MLP2(Δx1,Δx2) B y =MLP3(Δy1,Δy2) Among them, B t is the nonlinear bias in the dimension of the number of time frames, B x is the nonlinear bias in the height dimension, B y is the nonlinear bias of the width dimension, and MLP1, MLP2 and MLP3 are multi-layer perceptrons.

6. The video action detection method based on nonlinear spatiotemporal relative position offset according to claim 5, characterized in that: The first relative position offset is: B′=unsqueeze(B t ,2,3)+unsqueeze(B x ,1,3)+unsqueeze(B y ,1,2) Among them, unsqueeze(·) is the expansion dimension function.

7. The video action detection method based on nonlinear spatiotemporal relative position offset according to claim 1, characterized in that: The grid of reference point coordinates is constructed based on the meshgrid function.

8. The video action detection method based on nonlinear spatiotemporal relative position offset according to claim 1, characterized in that: The key spatiotemporal features are input into the classification module of the model to obtain the video action detection results as follows: The key spatiotemporal features are upsampled or downsampled to obtain multi-scale features and fused with the pyramid network to generate a bounding box. The fused features are passed to the decoder and then passed through the classification module to obtain the video action detection results.

9. The video action detection method based on nonlinear spatiotemporal relative position offset according to claim 1, characterized in that: The specific steps to convert a video clip into a series of tags are: The video clip is divided into several consecutive frames, which are sequentially composed of frame sequences, and the frame sequences are encoded as a series of tags.

10. The video action detection method based on nonlinear spatiotemporal relative position offset according to claim 1, characterized in that: The input of the improved attention mechanism is [v, x], where v represents the learnable vector and x represents the video data. The improved attention mechanism is: Among them, Q, K, V are query, key and value matrices respectively, d represents the dimension, B is the improved relative position bias, and the improved relative position bias is obtained by concatenating and expanding the first relative position bias and the second relative position bias.

Images