A real-time action localization method based on spatiotemporal cross attention

Abstract

The present invention relates to a real-time action positioning method based on spatiotemporal cross attention. First, a video clip is sampled to obtain an input video, and key frames are extracted by frame set division. The obtained video clip is input into an R(2+1)D network to extract temporal features, and the key frames are input into a CSPNet network to extract spatial features. Secondly, the temporal features are compressed and encoded and converted with spatial features. The potential adaptation between temporal features and spatial features is calculated, and the features are embedded in transformer self-attention for cross-representation of features, and the features are spliced ​​and fused using a contextual attention module. Finally, the bounding box position and motion category are predicted by regression and classification networks, and the link scores of each category of adjacent frames are calculated according to the prediction results, and the Viterbi algorithm is used to find the best path for generating action pipes.

Description

Technical Field

[0001] The invention relates to the field of pattern recognition and computer vision, and in particular to a real-time action positioning method based on spatiotemporal cross attention. Background Art

[0002] With the development of science and technology, spatiotemporal action localization has been a hot research topic in recent years. In the fields of unmanned driving, security monitoring, transportation, human-computer interaction systems, etc., the application of real-time spatiotemporal action localization has become more and more extensive. Previous spatiotemporal action localization methods directly spliced ​​the temporal features extracted by the deep network with the spatial features, ignoring the different data sources between the temporal features and the spatial features, and the different correlation between the elements in the features. 2D features focus on background information, focusing on the importance of local structure and contextual information for recognition. 3D features focus on temporal information, focusing on the temporal dependency between a single frame and adjacent frames. Simply splicing 2D and 3D features cannot effectively integrate spatiotemporal features. Instead, it makes the temporal features and spatial features mutually exclusive, confusing the motion area that the video action should focus on. In response to the above situation, we propose a real-time action localization method based on spatiotemporal cross attention. Summary of the invention

[0003] The purpose of the present invention is to provide a real-time action localization method based on spatiotemporal cross attention, which can effectively locate spatiotemporal actions.

[0004] To achieve the above object, the technical solution of the present invention is: a real-time action positioning method based on spatiotemporal cross attention, comprising the following steps:

[0005] Step S1, sampling the video clip to obtain an input video, and extracting key frames by dividing the frame set;

[0006] Step S2: Input the acquired video clips into the R(2+1)D network to extract temporal features, and input the key frames into the CSPNet network to extract spatial features;

[0007] Step S3: compress the temporal features, encode the spatial features, calculate the potential adaptation between the temporal features and the spatial features, and perform cross-representation between the features through transformer self-attention;

[0008] Step S4, concatenate features and fuse them, predict the bounding box position and motion category through regression and classification, and use IOU loss and binary cross entropy loss to calculate the bounding box loss and category loss respectively;

[0009] Step S5: Calculate the link scores of each category of adjacent frames according to the prediction results, and use the Viterbi algorithm to find the best path for generating action pipes.

[0010] Compared with the prior art, the present invention has the following beneficial effects:

[0011] 1. The present invention proposes a method for extracting key frames. Compared with the previous method of using only the current frame as the key frame, the method of the present invention can better represent the process of an action.

[0012] 2. The present invention proposes to fuse attention based on spatial cross features, encode spatiotemporal features using spatiotemporal cross transformers, guide the fusion of temporal features and spatial features, enhance the correlation between features, and avoid the influence of the repulsion between temporal features and spatial features on action recognition.

[0013] 3. The present invention uses channel splicing and attention mechanism to smoothly aggregate spatiotemporal features. Different from the previous method of attention fusion, the attention mechanism effectively enhances the effective information of the features and further integrates the spatiotemporal features. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a schematic diagram of the principle of the present invention. DETAILED DESCRIPTION

[0015] It should be noted that the following detailed descriptions are illustrative and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0016] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0017] like Figure 1 As shown, this embodiment provides a real-time action positioning method based on spatiotemporal cross attention, which specifically includes the following steps:

[0018] Step S1, sampling the video clip to obtain an input video, and extracting key frames by dividing the frame set;

[0019] Step S2: Input the acquired video clips into the R(2+1)D network to extract temporal features, and input the key frames into the CSPNet network to extract spatial features;

[0020] Step S3: compress the temporal features, encode the spatial features, calculate the potential adaptation between the temporal features and the spatial features, and perform cross-representation between the features through transformer self-attention;

[0021] Step S4, concatenating features to fuse features, predicting bounding box positions and motion categories through regression and classification, using IOU loss and binary cross entropy loss, and calculating bounding box loss and category loss respectively;

[0022] Step S5: Calculate the link scores of each category of adjacent frames according to the prediction results, and use the Viterbi algorithm to find the best path for generating action pipes.

[0023] In this embodiment, step S1 specifically includes the following steps:

[0024] Step S11, uniformly sampling the input video segments at intervals of l frames as the input video segments;

[0025] Step S12: Divide the input video clip into n frames of equal length, that is, S = {s 1 ,s 2 ,…,s n}, each frame set s fi It consists of a sequence of video frames of equal length;

[0026] Step S13: Extract s 1 、s n / 2 、s n Frameset, select the middle frame of the frameset as the keyframe to briefly represent the action.

[0027] In this embodiment, step S2 specifically includes the following steps:

[0028] Step S21: Input the video clip into the 3D backbone network R(2+1)D network to extract the temporal features M(X M ∈R C ×I×H×W ), where I is the number of input frames, H and W are the height and width of the input image, and C is the number of output channels;

[0029] Step S22, the R(2+1)D network is one of the mainstream backbone networks in action recognition, which models temporal motion information by explicitly decomposing 3D convolution into two independent and continuous operations, namely 2D spatial convolution and 1D temporal convolution.

[0030] Step S23: Input the key frame into the 2D backbone network CSPNet network to extract the spatial feature P(X P ∈R C×H×W );

[0031] In this embodiment, step S3 specifically includes the following steps:

[0032] Step S31, to match the output feature map of the 2D backbone network, reduce the depth dimension of the R(2+1)D output feature M to 1 and compress it to [C×H×W];

[0033] Step S32: Convert the compressed temporal features and spatial features from [C×H×W] to [H×W×C], and flatten the features to obtain M'(X M' ∈R T×C ), P'(X P' ∈R T×C ), where T = H × W;

[0034] Step S33: respectively analyze the time series features X M' ∈R T×C With spatial feature X P' ∈R T×C Perform position embedding to carry time information. The specific formula is as follows:

[0035]

[0036]

[0037] Among them, PE is the position embedding matrix, and its calculation formula is as follows:

[0038]

[0039]

[0040] Among them, PE[i,2j] means that the PE position is embedded in the i-th row of the matrix, and the value of the 2j column is PE[i,2j+1] means that the PE position is embedded in the i-th row of the matrix, and the value of the 2j+1 column is

[0041] Step S34: Calculate the time series feature X M' ∈R T×C To spatial feature X P' ∈R T×C The latent adaptive representation Y P' And the spatial feature X P' ∈R T×C To the time series feature X M' ∈R T×C The latent adaptive representation Y M' , calculated as follows:

[0042] Calculate from the time series feature X M' ∈R T×C To spatial feature XP' ∈R T×C The potential adaptive performance is Y P' =CM M'→P' (X P' , X M' )∈R T×C :

[0043]

[0044] Among them, the definition in, is the weight. P' , X M' The score matrix is ​​calculated by softmax (softmax(·)∈R T×T ), V M' Y is obtained by weighted aggregation through the score matrix P' ;

[0045] Calculate from the spatial feature X P' ∈R T×C To the time series feature X M' ∈R T×C The potential adaptation is represented by Y M' =CM P'→M' (X M' , X P′ )∈R T×C :

[0046]

[0047] Among them, the definition in, is the weight; X M′ , X P' Calculate the score matrix softmax(·)∈R through softmax T×T , V P' Y is obtained by weighted aggregation through the score matrix M' .

[0048] Step S35: Respectively represent the feature potential adaptive representation Y P' , Y M' Embedded in transformer self-attention, one feature can receive information from another feature. For example, we make the temporal (M') feature pass to the spatial (P') feature, which is represented by "M'→P'":

[0049]

[0050]

[0051]

[0052] where f θ is a positional feed-forward sublayer parameterized by θ, g represents the number of layers, and LN represents a normalization layer.

[0053] Finally, the features And features

[0054] In this embodiment, step S4 specifically includes the following steps:

[0055] Step S41: Splicing features Then the feature encoding [T×2C] is modified to [2C×T] to concatenate the feature A∈R 2C×T , and reshape the result back to three-dimensional space to obtain A'∈R 2C×H×W ;

[0056] Step S42: remap the A' feature through two convolution blocks to obtain A' res :

[0057] A' res =F(A';θ,Ω)

[0058] Where F(·) represents the residual function, APool(·) represents the average pooling layer, and θ and Ω represent the structures of the convolutional layers, respectively.

[0059] Step S43: The feature A' is subjected to a non-complete compression operation through a pooling layer, and then remapped through two convolution blocks, and then upsampled to obtain A'. attn :

[0060] A′ attn =Up(F(APool(A′);θ,Ω))

[0061] Step S44: A′ res , A′ attn Fusion to get A' out :

[0062] A′ out =A′ attn *A′ res +A′ attn

[0063] Step S45: Fusion feature A′ outA 1×1 convolutional layer is used to generate an output channel of size [(5×(NumCls+5))×H×W], where (NumCls+5) includes NumCls action scores Score, 4 coordinates [bx, by, bw, bh] and a confidence score Conf;

[0064] Step S46, predict the bounding box position through regression, classify by fully connected layer and Softmax layer, use IOU loss, binary cross entropy loss and calculate bounding box loss and category loss respectively;

[0065] Step S47: select a confidence threshold, take out the boxes and scores of each category with scores greater than a certain threshold, sort them, filter out low-threshold prediction boundary boxes, and obtain the prediction results.

[0066] In this embodiment, step S5 specifically includes the following steps:

[0067] Step S51: Assume that Box t , Box t+1 It is the two predicted boxes of consecutive frames t and t+1, and calculates Box in action category cls t , Box t+1 Total score of the two boxes Sum cls (Box t , Box t+1 )as follows:

[0068] Sum cls (Box t , Box t+1 )=Score cls (Box t )+Score cls (Box t+1 )+α·Score cls (Box t )·Score cls (Box t+1 )

[0069] Among them, Score cls (Box t ), Score cls (Box t+1 ) is the prediction box Box in the prediction category cls t and Box t+1 Action score, α is a custom parameter;

[0070] Step S52: Calculate Box t , Box t+1 The overlap ov(Box t , Boxt+1 )as follows:

[0071] ov(Box t , Box t+1 )=1-IOU(Box t , Box t+1 )

[0072] Among them, IOU is the ratio of the overlapping area of ​​the bounding box to the total area.

[0073] Step S53: According to Box t , Box t+1 The total score of the two boxes and the degree of overlap calculation in action class c Box t , Box t+1 The link scores are as follows:

[0074] Score cls (Box t , Box t+1 )

[0075] =ψ(ov(Box t , Box t+1 ))·[Sum cls (Box t , Box t+1 )+β·ov(Box t , Box t+1 )]

[0076] Among them, β is a custom parameter, ψ(ov(Box t , Box t+1 )) is the indicator function. If ov(Box t , Box t+1 ) is greater than 0, the output is 1, otherwise the output is 0;

[0077] Step S54: After all connection scores are calculated, the Viterbi algorithm is used to find the best path for generating the action pipe.

[0078] The Viterbi algorithm uses dynamic programming to solve the path with the highest probability, and is a dynamic programming algorithm that is widely used in machine learning.

[0079] In particular, the present invention is a real-time spatiotemporal positioning method based on spatiotemporal cross attention, which can locate spatiotemporal actions in real time. The present invention proposes a method for extracting key frames. Compared with the previous method of only using the current frame as the key frame, the method of the present invention can better represent the process of the action. The present invention proposes to fuse attention based on spatial cross features, encode spatiotemporal features using spatiotemporal cross transformers, guide the fusion of temporal features and spatial features, enhance the correlation between features, and avoid the influence of the repulsion of temporal features and spatial features on action recognition. The present invention uses channel splicing and attention mechanisms to smoothly aggregate spatiotemporal features. Different from the previous method of attention fusion, the attention mechanism effectively enhances the effective information of the features and further fuses the spatiotemporal features.

[0080] The above are preferred embodiments of the present invention. Any changes made according to the technical solution of the present invention, as long as the resulting functions do not exceed the scope of the technical solution of the present invention, belong to the protection scope of the present invention.

Claims

1. A real-time action localization method based on spatiotemporal cross attention, characterized in that: The steps include: Step S1, sampling the video clip to obtain an input video, and extracting key frames by dividing the frame set; Step S2: Input the acquired video clips into the R(2+1)D network to extract temporal features, and input the key frames into the CSPNet network to extract spatial features; Step S3: compress the temporal features, encode the spatial features, calculate the potential adaptation between the temporal features and the spatial features, and perform cross-representation between the features through transformer self-attention; Step S4, concatenate features and fuse them, predict the bounding box position and motion category through regression and classification, and use IOU loss and binary cross entropy loss to calculate the bounding box loss and category loss respectively; Step S5, calculating the link scores of each category of adjacent frames according to the prediction results, and using the Viterbi algorithm to find the best path for generating actions; The step S2 is specifically implemented as follows: Step S21: Input the video clip into the 3D backbone network, i.e., the R(2+1)D network, to extract the temporal features M(X M ∈R C ×I×H×W ), where I is the number of input frames, H and W are the height and width of the input image, and C is the number of output channels; Step S22, the R(2+1)D network is one of the mainstream backbone networks in action recognition, which models temporal motion information by explicitly decomposing 3D convolution into two independent and continuous operations, namely 2D spatial convolution and 1D temporal convolution; Step S23: Input the key frame into the 2D backbone network, namely the CSPNet network, to extract the spatial feature P(X P ∈R C×H×W ); The step S3 is specifically implemented as follows: Step S31, to match the output feature map of the 2D backbone network, reduce the depth dimension of the R(2+1)D network output feature M to 1 and compress it to [C×H×W]; Step S32: Convert the compressed temporal features and spatial features from [C×H×W] to [H×W×C], and flatten the features to obtain M'(X M' ∈R T×C ), P'(X P' ∈R T×C ), where T = H × W; Step S33: respectively analyze the time series features X M' ∈R T×C With spatial feature X P' ∈R T×C Perform position embedding to carry time information. The specific formula is as follows: Among them, PE is the position embedding matrix, and its calculation formula is as follows: Among them, PE[i,2j] means that the PE position is embedded in the i-th row of the matrix, and the value of the 2j column is PE[i,2j+1] means that the PE position is embedded in the i-th row of the matrix, and the value of the 2j+1 column is Step S34: Calculate the time series feature X M' ∈R T×C To spatial feature X P' ∈R T×C The latent adaptive representation Y P' And the spatial feature X P' ∈R T×C To the time series feature X M' ∈R T×C The latent adaptive representation Y M' , calculated as follows: Calculate from the time series feature X M' ∈R T×C To spatial feature X P' ∈R T×C The potential adaptation is represented by Y P' =CM M'→P' (X P' , X M' )∈R T×C : Among them, the definition in, is the weight; X P' , X M' Calculate the score matrix softmax(·)∈R through softmax T×T , V M' Y is obtained by weighted aggregation through the score matrix P' ; Calculate from the spatial feature X P' ∈R T×C To the time series feature X M' ∈R T×C The potential adaptation is represented by Y M' =CM P'→M' (X M' , X P' )∈R T×C : Among them, the definition in, is the weight; X M' , X P' Calculate the score matrix softmax(·)∈R through softmax T×T , V P' Y is obtained by weighted aggregation through the score matrix M' ; Step S35: Respectively represent the feature potential adaptive representation Y P' , Y M' Embedded in transformer self-attention, one feature can receive information from another feature; the temporal feature is passed to the spatial feature, which is represented by "M'→P'": where f θ is the positional feed-forward sublayer parameterized by θ, g represents the number of layers, and LN represents the normalization layer; Finally, the features And features The step S4 is specifically implemented as follows: Step S41: Splicing features Then the feature encoding [T×2C] is modified to [2C×T] to concatenate the feature A∈R 2C×T , and reshape the result back to three-dimensional space to obtain A'∈R 2C×H×W ; Step S42: remap the A' feature through two convolution blocks to obtain A' res : A' res =F(A';θ,Ω) Where F(·) represents the residual function, θ and Ω represent the structure of the convolutional layer; Step S43: The feature A' is subjected to a non-complete compression operation through a pooling layer, and then remapped through two convolution blocks, and then upsampled to obtain A'. attn : A′ attn =Up(F(APool(A');θ,Ω)) Where APool(·) represents the average pooling layer, Step S44: A' res , A' attn Fusion to get A' out : A′ out =A′ attn *A′ res +A′ attn Step S45: Fusion feature A' out A 1×1 kernel convolutional layer is used to generate an output channel of size [(5×(NumCls+5))×H×W], where (NumCls+5) includes NumCls class action scores Score c , 4 coordinates [bx, by, bw, bh] and confidence score Conf; Step S46, predicting the bounding box position through regression, classifying by the fully connected layer and the softmax layer, and calculating the bounding box loss and category loss by using the IOU loss and the binary cross entropy loss respectively; Step S47, selecting a confidence threshold, taking out the boxes and scores of each category with scores greater than the predetermined threshold, sorting them, filtering out the predicted boundary boxes with scores lower than the predetermined threshold, and obtaining the prediction results; The step S5 is specifically implemented as follows: Step S51: Assume that Box t , Box t+1 It is the two predicted boxes of consecutive frames t and t+1, and calculates Box in action category cls t , Box t+1 Total score of the two boxes Sum cls (Box t , Box t+1 )as follows: Sum cls (Box t ,Box t+1 )=Score cls (Box t )+Score cls (Box t+1 )+α·Score cls (Box t )·Score cls (Box t+1 ) Among them, Score cls (Box t ), Score cls (Box t+1 ) is the prediction box Box in the prediction category cls t and Box t+1 Action score, α is a custom parameter; Step S52: Calculate Box t , Box t+1 The overlap ov(Box t , Box t+1 )as follows: ov(Box t ,Box t+1 )=1-IOU(Box t ,Box t+1 ) Among them, IOU is the ratio of the overlapping area of ​​the bounding box to the total area; Step S53: According to Box t , Box t+1 The total score of the two boxes and the degree of overlap calculation in action class c Box t , Box t+1 The link scores are as follows: Score cls (Box t ,Box t+1 ) =ψ(ov(Box t ,Box t+1 ))·[Sum cls (Box t ,Box t+1 )+β·ov(Box t ,Box t+1 )] Among them, β is a custom parameter, ψ(ov(Box t , Box t+1 )) is the indicator function. If ov(Box t , Box t+1 ) is greater than 0, the output is 1, otherwise the output is 0; Step S54: After all connection scores are calculated, the Viterbi algorithm is used to find the best path for generating actions.

2. A real-time action localization method based on spatiotemporal cross attention according to claim 1, characterized in that: The step S1 is specifically implemented as follows: Step S11, uniformly sampling the input video segment at intervals of 1 frame as the input video segment; Step S12: Divide the input video clip into n frames of equal length, that is, S = {s1, s2, ..., s n }, each frame set s fi It consists of a sequence of video frames of equal length, f i ∈[1,n]; Step S13: Extract s1, s n / 2 、s n Frameset, select the middle frame of the frameset as the keyframe to briefly represent the action.

Images