Online action detection method and system based on non-autoregressive attention mechanism

Abstract

The embodiment of the application discloses a kind of online action detection method and system based on non-autoregressive attention mechanism, comprising: step 1) utilize vehicle-mounted camera system to collect human action video data, further adopt equal interval sparse sampling to obtain action fragment unit to video;Step 2) adopt double-flow network to extract fixed dimension feature representation for each action fragment unit;Step 3) adopt recurrent neural network to capture local human action features of observation part;Step 4) based on the local features captured, use non-autoregressive attention model to predict the action feature representation of future time point;Step 5) local features and predicted future features are spliced, and sent to classifier to obtain human action online detection result.The system and method can effectively detect the occurrence of human action in real time, and have wide application prospect in automatic driving, human-computer interaction and the like.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of action recognition, and particularly relates to an online action detection method and system based on a non-autoregressive attention mechanism. BACKGROUND

[0002] The purpose of the online action detection task is to correctly identify the human action being performed from a video stream. In recent years, the online action detection task has attracted more and more attention, mainly due to its broad application scenarios and market value in automatic driving, video-only surveillance and anomaly monitoring.

[0003] The object to be processed by the online action detection task is an online video stream, so the model used by the online action detection task can only effectively infer according to the observed local information. The key challenge of this task is to monitor the action at the time when the video frame arrives in real time according to incomplete video information. To solve this problem, the long-term action relationship needs to be learned.

[0004] At present, the mainstream online action detection method adopts an encoding model based on a recurrent neural network to model and predict long-term action features respectively. However, the recurrent neural network adopts a progressive serial way to predict in the action prediction process, and the prediction at the next time point depends on the prediction result at the last time, which will cause the accumulation of error information. SUMMARY

[0005] In view of the technical defects mentioned in the background, the purpose of the embodiments of the present application is to provide an online action detection method and system based on a non-autoregressive attention mechanism.

[0006] To achieve the above-mentioned purpose, in a first aspect, the embodiments of the present application provide an online action detection method based on a non-autoregressive attention mechanism, comprising:

[0007] S1, collecting human action video data by using a vehicle-mounted camera system, and performing equal-interval sparse sampling on the human action video data to obtain a plurality of continuous time segment units;

[0008] S2, dividing the plurality of continuous time segment units into an observation part and a prediction part;

[0009] S3, performing spatiotemporal feature extraction on the observation part by using a double-flow network, and performing channel splicing to obtain first data;

[0010] S4, sending the first data into a bidirectional recurrent neural network, and capturing the context semantic information between the continuous frames of the video in the observation part by using the bidirectional recurrent neural network;

[0011] S5, adopting a non-autoregressive attention model to process the captured context semantic information to obtain a local human action feature;

[0012] S6, adopting a double-flow network to perform spatiotemporal feature extraction on the prediction part and perform channel splicing to obtain second data;

[0013] S7, sending the second data into a non-autoregressive attention model to predict the action feature at a future time point to obtain a predicted action feature;

[0014] S8, sending the local human action feature and the predicted action feature into a classifier to obtain a human action online detection result.

[0015] As a specific implementation, the double-flow network includes a spatial branch network constructed by a RestNet-200 network and a temporal branch network constructed by a BN-Inception network, the input of the spatial branch network is an RGB image, and the input of the temporal branch network is an optical flow image between adjacent two frames.

[0016] As a specific implementation, the non-autoregressive attention model includes a multi-head self-attention module, a multi-head cross-attention module, and a feedforward network.

[0017] In a second aspect, an embodiment of the present application provides another online action detection method based on a non-autoregressive attention mechanism, comprising:

[0018] S1, collecting human action video data by using a vehicle-mounted camera system;

[0019] S2, performing equal-interval sparse sampling on the human action video data to obtain a plurality of continuous time segment units;

[0020] S3, dividing the plurality of continuous time segment units into an observation part and a prediction part;

[0021] S4, sending the observation part into a double-flow network to extract spatiotemporal features and perform channel splicing, and sending into a bidirectional recurrent neural network to capture context semantic feature representation h between continuous frames of the video;

[0022] S5, sending the prediction part and the context semantic feature representation h into a non-autoregressive attention model to obtain a human action online detection result.

[0023] The non-autoregressive attention model includes a multi-head self-attention module, a multi-head cross-attention module, and a feedforward network, and step S5 includes:

[0024] S51, sending a token vector into a multi-head cross-attention module to obtain a feature representation x;

[0025] S52, input the context semantic feature representation h obtained in step S4 and the feature representation x obtained in step 51 into the cross attention module to perform linear transformation on the feature representation x;

[0026] S53, input the feature representation x obtained in step S52 into the feedforward network to obtain the final output feature

[0027] S54, in the training phase, according to the output feature obtained confidence score;

[0028] S55, in the inference phase, repeat steps S2-S4, S51-S54 to obtain the prediction class confidence score of each frame, and take the action class with the highest score as the online detection result of the current human action.

[0029] In a third aspect, an online action detection system based on a non-autoregressive attention mechanism is provided, comprising:

[0030] A first unit is configured to collect human action video data by using a vehicle-mounted camera system, and perform equal-interval sparse sampling on the human action video data to obtain a plurality of continuous time segment units;

[0031] A second unit is configured to divide the plurality of continuous time segment units into an observation part and a prediction part;

[0032] A third unit is configured to perform spatio-temporal feature extraction on the observation part by using a dual-stream network, and perform channel splicing to obtain first data;

[0033] A fourth unit is configured to input the first data into a bidirectional recurrent neural network, and capture context semantic information between continuous frames of the observation part by using the bidirectional recurrent neural network;

[0034] A fifth unit is configured to process the captured context semantic information by using a non-autoregressive attention model to obtain local human action features;

[0035] A sixth unit is configured to perform spatio-temporal feature extraction on the prediction part by using a dual-stream network, and perform channel splicing to obtain second data;

[0036] A seventh unit is configured to input the second data into a non-autoregressive attention model to predict action features at a future time point, and obtain predicted action features;

[0037] An eighth unit is configured to input the local human action features and the predicted action features into a classifier to obtain an online detection result of human action.

[0038] In a fourth aspect, the embodiments of the present application further provide an online action detection system based on a non-autoregressive attention mechanism, comprising a processor, an input device, an output device and a memory, which are connected to each other, wherein the memory is used to store a computer program, the computer program comprises program instructions, and the processor is configured to invoke the program instructions to execute the method steps of the second aspect.

[0039] Compared with the prior art, the present application has the following beneficial effects:

[0040] For the online action detection task, the previous method adopts an autoregressive mode based on a recurrent neural network to serially predict the action features at future time points. In the serial structure, the prediction result at the current time point depends on the result at the previous time point, and therefore, this will cause the accumulation of error information. Meanwhile, the recurrent neural network cannot perform parallel processing on the sequence, resulting in a significant reduction in prediction efficiency. The present application uses a non-autoregressive model to solve the drawbacks of the above method. On the one hand, the attention mechanism can effectively capture useful information in the video sequence and support parallel processing, thereby improving the prediction efficiency. On the other hand, the length of the predicted feature is modeled by using a length-learnable vector, and the future action features are predicted in parallel by using a non-autoregressive mode, thereby effectively reducing the influence of the accumulation of error information on the final prediction result. BRIEF DESCRIPTION OF DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the specific embodiments or prior art description will be briefly introduced as follows.

[0042] Figure 1 is a flowchart of the online action detection method based on a non-autoregressive attention mechanism provided by the first embodiment of the present application;

[0043] Figure 2 is a method flowchart of the second embodiment of the present application;

[0044] Figure 3 is a structural schematic diagram of a Bi-LSTM network;

[0045] Figure 4 is a structural schematic diagram of a non-autoregressive attention model;

[0046] Figure 5 is a structural diagram of the online action detection system based on a non-autoregressive attention mechanism provided by the first embodiment of the present application;

[0047] Figure 6 is a system structural diagram provided by the second embodiment of the present application. DETAILED DESCRIPTION

[0048] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of the present application.

[0049] It should be understood that, when used in the specification and the appended claims, the terms "comprise" and "include" indicate the presence of the described features, integers, steps, operations, elements, and / or components, but do not exclude one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0050] The inventive concept of the present application is: first, using a long short-term memory network LSTM to aggregate effective information of observed partial actions, the LSTM network can effectively capture the context information of the actions; based on the captured context information, using multi-head self-attention, multi-head cross-attention, and a feedforward neural network, in combination with a length-learnable token vector, to effectively predict the action feature representation of a certain length in the future in a parallel non-autoregressive manner.

[0051] The online action detection scheme based on the non-autoregressive attention mechanism provided by the embodiments of the present application can be summarized as follows: step 1) using a vehicle-mounted camera system to collect human action video data, and further performing equal-interval sparse sampling on the video to obtain action segment units; step 2) using a double-stream network to extract a fixed-dimensional feature representation for each action segment unit; step 3) using a recurrent neural network to capture local human action features of the observed part; step 4) based on the captured local features, using a non-autoregressive attention model to predict the action feature representation at a future time point; step 5) concatenating the local features and the predicted future features, and sending them to a classifier to obtain the online human action detection result. The system and method can effectively detect the occurrence of human actions in real time, and have wide application prospects in automatic driving, human-computer interaction, etc.

[0052] Next, the online action detection scheme based on the non-autoregressive attention mechanism described above will be described in detail in two specific embodiments.

[0053] Embodiment 1

[0054] Please refer to Figure 1 The online action detection method based on the non-autoregressive attention mechanism provided by the first embodiment of the present application comprises:

[0055] S101, using a vehicle-mounted camera system to collect human action video data, and performing equal-interval sparse sampling on the human action video data to obtain a plurality of continuous time segment units.

[0056] S102, divide the plurality of continuous time segment units into an observation part and a prediction part.

[0057] S103, adopt a double-flow network to extract spatio-temporal features of the observation part, and perform channel splicing to obtain first data.

[0058] The double-flow network comprises a spatial branch network constructed by a RestNet-200 network and a time branch network constructed by a BN-Inception network, the input of the spatial branch network is an RGB image, and the input of the time branch network is an optical flow image between adjacent two frames.

[0059] S104, input the first data into a bidirectional recurrent neural network, and adopt the bidirectional recurrent neural network to capture context semantic information between continuous frames in the observation part.

[0060] S105, adopt a non-autoregressive attention model to process the captured context semantic information to obtain local human action features.

[0061] The non-autoregressive attention model comprises a multi-head self-attention module, a multi-head cross-attention module and a feedforward network.

[0062] S106, adopt a double-flow network to extract spatio-temporal features of the prediction part, and perform channel splicing to obtain second data.

[0063] S107, input the second data into a non-autoregressive attention model, predict action features at a future time point to obtain predicted action features.

[0064] S108, input the local human action features and the predicted action features into a classifier to obtain an online human action detection result.

[0065] Embodiment 2:

[0066] Please refer to Figure 2 The online action detection method based on the non-autoregressive attention mechanism provided by the second embodiment of the present application comprises:

[0067] Step one:

[0068] Human action video data is collected by using a vehicle-mounted camera system, and is divided into a training set and a test set. For the training set and the test set, equal-interval sparse sampling is adopted to obtain a plurality of continuous time segment units.

[0069] Specifically, the collected human action video data is THUMOS2014 dataset, which is divided into 200 training sets and 213 test sets, and each video has annotation information in time sequence, that is, the start and end time points of the action are labeled. Each video is composed of continuous human motion images, which can be divided into multiple continuous time segment units (which can be called video feature sequence).

[0070] It should be noted that in step one, the frame rate of the video sampling is set to 24FPS, that is, 24 frames per second, and the sampling frequency of the video frame is 4FPS, that is, 4 frames of images are sampled per second.

[0071] Step two:

[0072] The dual-stream network is constructed as a feature extractor.

[0073] Specifically, the dual-stream network includes two branches: the spatial branch is constructed by ResNet-200 network, and the temporal branch is constructed by BN-Inception network, which are respectively used to extract the static information of the spatial dimension of the video image and the motion information of the time dimension; the input of the spatial branch network is the RGB image, and the input of the temporal branch network is the optical flow image between adjacent two frames.

[0074] It should be noted that in step two, the network used for feature extraction is pre-trained on the ActivityNet-V1.3 human action dataset, and the feature dimension after splicing the outputs of the time and space networks is 3072. Among them, the output dimension of the Flatten 673 layer of the ResNet200 network is 2048, and the output dimension of the global pool layer of the BN-Inception network is 1024.

[0075] Step three:

[0076] The multiple continuous time segment units are divided into observation part and prediction part, and the observation part is sent into the dual-stream network to extract the space-time features, which are further sent into the bidirectional recurrent neural network to capture the context semantic feature representation h between the continuous frames of the video.

[0077] Specifically, the multiple continuous time segment units are divided into observation part and prediction part. Assuming that the first t frames are the observable part F seen ={f1,f2,...,f t}, t+1 to T are invisible part F unseen ={f t+1 ,f t+2 ,...,f TThe images of the first t frames are processed through a dual-stream network to extract spatiotemporal features and then concatenated into channels. These are then fed into a bidirectional recurrent neural network (Bi-LSTM) to capture the contextual semantic information between consecutive video frames. For example... Figure 3 As shown, this process can be represented as:

[0078]

[0079]

[0080] Among them, LSTM f Represents a feedforward recurrent neural network, LSTM b This represents a reverse recurrent neural network. The contextual features at time i can be derived from the forward hidden states. and reverse hidden state The features are concatenated to obtain the semantic feature representation of the video. [;] indicates a channel splicing operation on the feature.

[0081] It should be noted that in step three, the video prediction portion is primarily used as a supervisory signal during the training phase. The model needs to perform effective inference based on the observation portion, compare the predicted results obtained from the inference with the corresponding ground truth values, and thus better optimize the network model. In a specific embodiment, the observation portion is set to 12 time segments, and the prediction portion is set to 8 time segments.

[0082] Furthermore, such as Figure 3 As shown, Bi-LSTM consists of two LSTM neural networks in opposite directions. Each network contains one hidden layer with 256 neurons per layer, and the concatenated feature dimension is 512.

[0083] Step Four:

[0084] The semantic feature representation h obtained in the previous step is fed into the non-autoregressive model to obtain the feature representation x.

[0085] like Figure 4 As shown, the non-autoregressive model consists of a multi-head self-attention module (MSA), a multi-head cross-attention module (MCA), and a feedforward network (FFN). Non-autoregressive means that during the prediction process, a certain length of future prediction segment can be obtained at once, while predictions are made step-by-step in a cyclical, progressive manner. Clearly, non-autoregressive models can significantly improve prediction efficiency while effectively avoiding prediction bias caused by the accumulation of erroneous information.

[0086] First, perform a triplet linear transformation on the input feature h.

[0087] Q h =W qh + b q K h = W k h + b k V h = W v h + b v ,

[0088] where W q , W k , W v , b q , b k , b v are learnable parameters.

[0089] The non-autoregressive model directly predicts the future feature representation by a length learnable vector token ∈ R L×d where L denotes the length of predicted feature and d is the dimension of predicted feature. Thus, similarly, a triple linear transformation is applied to token,

[0090] Q token = W q h + b q K token = W k h + b k V token = W v h + b v

[0091] Further, the token vector is fed into a multi-head self-attention mechanism module to obtain the feature representation x,

[0092] x = token + MSA(token, token, token)

[0093] MSA = Stack(Head1, Head2,..., Head n )

[0094]

[0095]

[0096] Step five:

[0097] The feature representation h obtained in step three and the feature representation x obtained in step four are fed into a cross-attention module to linearly transform the feature representation x.

[0098] Specifically, x is linearly transformed,

[0099] Q x = W q h + b q

[0100] and compute Q of x x and K of h h , V h between them,

[0101] x = x + MSA(x, h, h)

[0102] MSA = Stack(Head1, Head2,..., Head n )

[0103]

[0104] where the multi-head is to concatenate the single heads Head i by channel.

[0105] It should be noted that in the multi-head attention mechanism of steps four and five, the number of heads is set to 8.

[0106] Step six:

[0107] The feature representation x obtained in step five is sent to the feedforward neural network, and the final output feature

[0108]

[0109] Step seven:

[0110] In the training phase, the predicted feature is averaged as the feature representation at the current time, that is, Further, the confidence score at the current time is obtained by a fully connected layer and a softmax function where W c represents the parameters of the fully connected layer for classification, and C represents the number of predicted action categories. In the training phase, the predicted action features at future time are known as supervision information, and the corresponding confidence scores of the predicted features at future L length are:

[0111]

[0112] The entire network adopts a supervised training method, and the optimization loss function is as follows:

[0113]

[0114] where CE is the cross-entropy loss function, is the true action category at future i time, and λ is the loss adjustment term.

[0115] Step eight:

[0116] In the reasoning stage, the prediction category confidence score of each frame is obtained by using the foregoing steps, and the action category with the highest score is taken as the current recognition result.

[0117] It should be noted that the parts similar to the steps in Embodiment 1 in Embodiment 2 can refer to the detailed description of the part of Embodiment 2.

[0118] From the above description, it can be seen that the embodiments of the present application:

[0119] For the online action detection task, the previous method adopts an autoregressive manner based on a recurrent neural network to serially predict the action features at future time. In the serial structure, the prediction result at the current time depends on the result at the previous time, and therefore, this will cause the accumulation of error information. Meanwhile, the recurrent neural network cannot be used for parallel processing of the sequence, resulting in a great reduction in prediction efficiency. The non-autoregressive model is used to solve the drawbacks of the foregoing method, on the one hand, the attention mechanism can effectively capture useful information in the video sequence, and can support parallel processing and improve the prediction efficiency; on the other hand, the length-learnable vector is used to model the length of the prediction feature, and the non-autoregressive manner is used to parallelly predict the future action feature, effectively reducing the influence of the accumulation of error information on the final prediction result.

[0120] Based on the same inventive concept, the embodiments of the present application provide an online action detection system based on a non-autoregressive attention mechanism. As shown in Figure 5 The system comprises:

[0121] The first unit is configured to collect human action video data by using a vehicle-mounted camera system, and perform equal-interval sparse sampling on the human action video data to obtain a plurality of continuous time segment units.

[0122] The second unit is configured to divide the plurality of continuous time segment units into an observation part and a prediction part.

[0123] The third unit is configured to perform spatiotemporal feature extraction on the observation part by using a double-flow network, and perform channel splicing to obtain first data.

[0124] The fourth unit is configured to send the first data into a bidirectional recurrent neural network, and capture context semantic information between continuous frames in the observation part by using the bidirectional recurrent neural network.

[0125] The fifth unit is configured to process the captured context semantic information by using a non-autoregressive attention model to obtain local human action features.

[0126] The sixth unit is configured to perform spatio-temporal feature extraction on the predicted part by using a double-flow network, and perform channel splicing to obtain second data;

[0127] The seventh unit is configured to input the second data into a non-autoregressive attention model to predict the action feature at the future time point, and obtain a predicted action feature.

[0128] The eighth unit is configured to input the local human body action feature and the predicted action feature into a classifier to obtain a human body action online detection result.

[0129] As another preferred embodiment of the present application, as shown in Figure 6 The detection system can include one or more processors 101, one or more input devices 102, one or more output devices 103, and a memory 104, which are connected to each other through a bus 105. The memory 104 is configured to store a computer program, the computer program including program instructions, and the processor 101 is configured to invoke the program instructions to execute the method steps as described in Embodiment 1 or Embodiment 2.

[0130] It should be understood that, in the embodiments of the present application, the processor 101 can be a central processing unit (CPU), and the processor can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0131] The input device 102 can include a keyboard, etc., and the output device 103 can include a display (LCD, etc.), a speaker, etc.

[0132] The memory 104 can include read-only memory and random access memory, and provide instructions and data to the processor 101. A portion of the memory 104 can also include non-volatile random access memory. For example, the memory 104 can also store device type information.

[0133] In specific implementation, the processor 101, the input device 102 and the output device 103 described in the embodiments of the present application can perform the implementation described in the embodiments of the online action detection method based on the non-autoregressive attention mechanism provided by the embodiments of the present application, which will not be described here.

[0134] It should be noted that for more specific workflow description of the detection system part, please refer to the foregoing method embodiment part, which will not be described here.

[0135] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. An online action detection method based on a non-autoregressive attention mechanism, characterized in that, The method comprises the following steps: S1, collecting human action video data by using a vehicle-mounted camera system, and performing equal-interval sparse sampling on the human action video data to obtain a plurality of continuous time segment units; S2, dividing the plurality of continuous time segment units into an observation part and a prediction part; S3, performing spatiotemporal feature extraction on the observation part by using a double-flow network, and performing channel splicing to obtain first data; S4, inputting the first data into a bidirectional recurrent neural network, and capturing context semantic information between continuous frames of the observation part by using the bidirectional recurrent neural network; S5, processing the captured context semantic information by using a non-autoregressive attention model to obtain local human action features; the non-autoregressive attention model comprises a multi-head self-attention module, a multi-head cross-attention module and a feedforward network; S6, performing spatiotemporal feature extraction on the prediction part by using a double-flow network, and performing channel splicing to obtain second data; S7, inputting the second data into the non-autoregressive attention model to predict action features at a future time point, and obtaining predicted action features; S8, inputting the local human action features and the predicted action features into a classifier to obtain a human action online detection result.

2. The method of claim 1, wherein, The double-flow network comprises a spatial branch network constructed by a RestNet-200 network and a temporal branch network constructed by a BN-Inception network, an input of the spatial branch network is an RGB image, and an input of the temporal branch network is an optical flow image between adjacent two frames.

3. An online action detection method based on a non-autoregressive attention mechanism, characterized in that, The method comprises the following steps: S1, collecting human action video data by using a vehicle-mounted camera system; S2, performing equal-interval sparse sampling on the human action video data to obtain a plurality of continuous time segment units; S3, dividing the plurality of continuous time segment units into an observation part and a prediction part; S4, inputting the observation part into a double-flow network, extracting spatiotemporal features and performing channel splicing, and inputting into a bidirectional recurrent neural network to capture context semantic feature representation h between continuous frames of the video; S5, inputting the prediction part and the context semantic feature representation h into a non-autoregressive attention model to obtain a human action online detection result; The non-autoregressive attention model comprises a multi-head self-attention module, a multi-head cross-attention module and a feedforward network, and step S5 comprises: S51, inputting a token vector into the multi-head cross-attention module to obtain feature representation x; S52, inputting the context semantic feature representation h obtained in step S4 and the feature representation x obtained in step 51 into the cross-attention module to perform linear transformation on the feature representation x; S53, send the feature representation x obtained in step S52 into the feedforward network, and obtain the final output feature through the skip connection operation S54, in the training phase, according to the output features obtain a confidence score; S55, in the inference stage, repeating steps S2-S4, S51-S54 to obtain a prediction class confidence score of each frame, and taking the action class with the highest score as the current human action online detection result.

4. The method of claim 3, wherein, The double-flow network comprises a spatial branch network constructed by a RestNet-200 network and a temporal branch network constructed by a BN-Inception network, an input of the spatial branch network is an RGB image, and an input of the temporal branch network is an optical flow image between adjacent two frames.

5. An online action detection system based on non-autoregressive attention mechanism, characterized in that, The method comprises the following steps: The first unit is configured to collect human action video data by using a vehicle-mounted camera system, and to perform equal-interval sparse sampling on the human action video data to obtain a plurality of continuous time segment units. The second unit is configured to divide the plurality of continuous time segment units into an observation part and a prediction part. The third unit is configured to perform spatiotemporal feature extraction on the observation part by using a double-flow network, and to perform channel splicing to obtain first data. The fourth unit is configured to input the first data into a bidirectional recurrent neural network, and to capture context semantic information between continuous video frames in the observation part by using the bidirectional recurrent neural network. The fifth unit is configured to process the captured context semantic information by using a non-autoregressive attention model to obtain local human action features; the non-autoregressive attention model includes a multi-head self-attention module, a multi-head cross-attention module, and a feedforward network. The sixth unit is configured to perform spatiotemporal feature extraction on the prediction part by using a double-flow network, and to perform channel splicing to obtain second data. The seventh unit is configured to input the second data into a non-autoregressive attention model, to predict action features at a future time point, and to obtain predicted action features. The eighth unit is configured to input the local human action features and the predicted action features into a classifier to obtain an online human action detection result.

6. An online action detection system based on non-autoregressive attention mechanism, characterized in that, The device comprises a processor, an input device, an output device, and a memory, which are connected to each other, wherein the memory is configured to store a computer program, the computer program comprises program instructions, and the processor is configured to invoke the program instructions to execute the method steps of claim 3 or 4.

Images