Video motion detection system and video motion detection method

Abstract

The application provides a video action detection system, comprising: a video backbone feature extraction module configured to output a time sequence video continuous segment as a corresponding 5D feature map; a person positioning module configured to output a corresponding person positioning frame according to the 5D feature map; and an action recognition module configured to output an action category corresponding to each person positioning frame according to the 5D feature map and the person positioning frame. The preferred embodiment of the application provides a video action detection system combining a 3D-CNN with a transformer model, and two encoder-decoder structures are used for two subtasks of video action detection to perform person positioning and action recognition respectively. The video action detection system provided by the application not only has a simple structure and is easy to implement, but also achieves optimal performance on multiple benchmarks, and is a preferred solution in the field of video person action detection.

Description

TECHNICAL FIELD

[0001] The present application relates generally to the field of image recognition, and more particularly, to a video action detection system and a method of video action detection. BACKGROUND

[0002] Video action detection (also called spatio-temporal action localization) is a key task in video understanding. Existing techniques are all inspired from the two-stage object detection framework of Faster-RCNN. Specifically, these techniques first need a 3D convolution based backbone network and a person detector. Then, the bounding box of the person and the features of the video are extracted by a RoI (Regions of Interest) Pooling layer. Finally, the features are passed through an action classification network and a position refinement network to get the action class and the spatial position of the person. Figure One

[0003] After 2018, Non-Local is added to the feature extraction backbone network and the classification network for action recognition. After 2019, the memory bank represented by long-term feature bank is proposed to encode and store long-distance video feature information. The stored video features can be indexed by temporal information. This technology of exchanging memory for GPU memory can significantly increase the length of the action recognition context.

[0004] Video action detection has two sub-tasks, namely person localization and action recognition. Person localization needs a relatively long spatial context, and action recognition needs a very long temporal-spatial context. Existing techniques rely on customized modules, such as pre-trained person / object detection models, region proposal networks (RPN), memory banks, etc., to obtain the context of the localization and detection tasks. These complex customized modules make the video person action detection system bulky and extremely complex.

[0005] The content of the background art section merely represents the knowledge of the inventor and does not necessarily represent the state of the art. SUMMARY

[0006] In view of at least one defect of the prior art, the present application provides a video action detection system, characterized in that it comprises:

[0007] a video backbone feature extraction module configured to output a corresponding 5D feature map for a continuous segment of a time sequence video;

[0008] ​a person positioning module configured to output a person positioning box according to the 5D feature map;

[0009] an action recognition module configured to output an action category corresponding to each person positioning box according to the 5D feature map and the person positioning box.

[0010] According to an aspect of the present application, wherein the video backbone feature extraction module is configured to: divide the continuous time video segment into a plurality of single video segments with equal frame numbers, the dimensions of the 5D feature map include: the length of the continuous time video segment, the frame number of the single video segment, the channel number of the video, the height, the width, and the video backbone feature extraction module is further configured to:

[0011] reduce the channel number by 3D convolution.

[0012] According to an aspect of the present application, wherein the person positioning module is further configured to:

[0013] output the person positioning box according to the middle slice on the frame number dimension of the 4D feature map corresponding to the single video segment.

[0014] According to an aspect of the present application, wherein the person positioning module is further configured to:

[0015] extract the action query option of the person positioning box by a spatio-temporal pooling layer.

[0016] According to an aspect of the present application, wherein the action recognition module is further configured to:

[0017] reduce the time dimension of the single video segment by a temporal pooling layer.

[0018] According to an aspect of the present application, wherein the action recognition module is further configured to:

[0019] output the action category corresponding to the action query option of each person positioning box according to the concatenated pooled 5D feature map.

[0020] According to an aspect of the present application, wherein the person positioning module further comprises:

[0021] a first encoder configured to encode the spatial information of the middle slice on the frame number dimension of the 4D feature map corresponding to the single video segment;

[0022] a first decoder configured to output the coordinates of the person positioning box according to the person query option and the spatial information encoding.

[0023] According to an aspect of the present application, wherein the person positioning module further comprises:

[0024] The first video cutting module is configured to cut the 4D feature map corresponding to the single video segment.

[0025] According to an aspect of the present application, the action recognition module further comprises:

[0026] The second encoder is configured to encode the 5D feature map corresponding to the continuous video segment in time and space information, and output scene context and character time sequence context.

[0027] The second decoder is configured to output the corresponding action category according to the action query option, the scene context, and the character time sequence context.

[0028] According to an aspect of the present application, the action recognition module further comprises:

[0029] The second video cutting module is configured to cut the 5D feature map corresponding to the continuous video segment.

[0030] The present application also provides a video action detection method, which is completed by the video action detection system as described above.

[0031] The preferred embodiment of the present application provides a video action detection system combining 3D-CNN and transformer model. For two sub-tasks of video action detection, two encoder-decoder structures are respectively used to perform character positioning and action recognition. The video action detection system provided by the present application uses convolutional neural network to model low-level visual structure, and uses transformer model to model high-level semantics. Experiments prove that the two have complementarity. The video action detection system provided by the present application not only has simple structure and is easy to implement, but also achieves optimal performance on multiple benchmarks, and is the preferred solution in the field of video character action detection. BRIEF DESCRIPTION OF DRAWINGS

[0032] The accompanying drawings are included to provide a further understanding of the present application, and constitute a part of the specification, illustrate the present application, and are used to explain the present application together with the embodiments of the present application, and do not constitute a limitation on the present application. In the drawings:

[0033] Figure 1 A video action detection system according to an embodiment of the present application is shown;

[0034] Figure 2 A video action detection system according to an embodiment of the present application is shown;

[0035] Figure 3 A character positioning module according to an embodiment of the present application is shown;

[0036] Figure 4 An action recognition module according to an embodiment of the present application is shown;

[0037] Figure 5 A video action detection method according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0038] Hereinafter, certain exemplary embodiments will be described simply. As can be recognized by those skilled in the art, the described embodiments can be modified in various different ways without departing from the spirit or scope of the present application. Therefore, the accompanying drawings and description are to be regarded as illustrative in nature rather than restrictive.

[0039] In the description of the present application, it is to be understood that the orientations or positional relationships indicated by the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", and the like are based on the orientations or positional relationships shown in the drawings, and are merely for the purpose of facilitating the description of the present application and simplifying the description, and do not indicate or imply that the devices or elements indicated thereby must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application. In addition, the terms "first", "second" are merely for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the technical features indicated thereby. Therefore, the features defined as "first", "second" can explicitly or implicitly include one or more of the features. In the description of the present application, the meaning of "a plurality of" is two or more, unless otherwise explicitly specified and limited.

[0040] In the description of the present application, it should be noted that, unless otherwise explicitly specified and limited, the terms "mounting", "connection", "connecting" should be understood broadly, for example, can be fixed connection, can be detachable connection, or integrally connected; can be mechanical connection, or electrical connection or can communicate with each other; can be directly connected, or indirectly connected through an intermediate medium; can be the internal communication of two elements or the interaction relationship between two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0041] In the present application, unless specifically stated and limited otherwise, a first feature "on" or "under" a second feature can include the first and second features being directly in contact, or can include the first and second features not being directly in contact but being in contact through another feature between them. Also, a first feature "over", "above" and "on top of" a second feature includes the first feature being directly above and obliquely above the second feature, or simply means that the first feature is higher in level than the second feature. A first feature "under", "below" and "underneath" a second feature includes the first feature being directly above and obliquely above the second feature, or simply means that the first feature is lower in level than the second feature.

[0042] The disclosure hereafter provides many different embodiments or examples for implementing different structures of the present application. For the purpose of simplification of the present application disclosure, the components and arrangements of the specific examples are described hereafter. Of course, they are merely examples and the purpose is not to limit the present application. In addition, the present application can repeatedly refer to the reference numerals and / or reference letters in different examples, and such repetition is for the purpose of simplification and clarity, which itself does not indicate the relationship between the various embodiments and / or arrangements discussed. In addition, the present application provides various specific examples of processes and materials, but those of ordinary skill in the art can realize the application of other processes and / or the use of other materials.

[0043] The embodiments of the present application will be described in conjunction with the drawings hereafter, and it should be understood that the embodiments described here are only for the purpose of illustration and explanation of the present application, and are not intended to limit the present application.

[0044] Since 2017, the Transformer model based on attention mechanism has revolutionized the traditional model based on RNN and CNN in the field of natural language processing and machine vision. For example, BERT based on Transformer combined with unsupervised pre-training has become a standard model in language understanding tasks. DETR, which serializes CNN features and uses Transformer to perform set prediction, is a new paradigm in object detection that rivals Faster-RCNN. Compared with CNN, which needs to be stacked deeper to obtain a larger receptive field, the attention mechanism of Transformer can associate any long-distance tokens in a given sequence. In the two sub-tasks of video human action detection, whether it is human positioning or action recognition, long context is needed, and the long-distance relationship modeling capability of Transformer is very suitable for the task of video human action detection. The present application combines the low-level visual structure modeling capability of CNN and the high-level semantic modeling capability of Transformer, and proposes a video human action detection system based on the Transformer model to simplify the structure and implementation of the video action detection system.

[0045] Figure 1 A video motion detection system 100 according to an embodiment of the present invention is shown, including a video backbone feature extraction module 110, a person localization module 120, and a motion recognition module 130. Wherein:

[0046] The video backbone feature extraction module 110 is configured to output continuous segments of time-series video as corresponding 5D feature maps.

[0047] The character positioning module 120 is configured to output the corresponding character positioning box based on the 5D feature map.

[0048] The action recognition module 130 is configured to output the action category corresponding to each person's location box based on the 5D feature map and the person's location box.

[0049] That is, the video backbone feature extraction module 110 is used to extract the video backbone, the character localization module 120 is used to locate characters by modeling spatial relationships, and the action classification module 130 is used to classify actions by modeling time-space relationships.

[0050] According to a preferred embodiment of the present invention, the video backbone extraction module 110 includes a 3D-CNN backbone network. For example... Figure 2 As shown, a sequential video segment v (Video Trunk as shown in the figure) with a certain length (a certain number of frames) is divided into n individual video segments V (V as shown in the figure). k-L …V k …V k+L The n individual video clips V are input into the 3D-CNN backbone network (as shown in the figure). The feature dimensions of the input clips include: the duration / number of frames T of each individual video clip V, the number of individual video clips V n, the number of channels C of the temporally consecutive video clip v (e.g., three RGB channels), the height H of the temporally consecutive video clip v, and the width W. That is, v∈R n*T*C*H*W The output of the 3D-CNN backbone network is the corresponding 5D feature map f(v)∈R. n*T*C*H*W Where T, C, H, and W represent the duration / number of frames, number of channels, height, and width of each individual video segment V, respectively, and n represents the number of individual video segments V.

[0051] According to a preferred embodiment of the present invention, in the video motion detection system 100, the video backbone feature extraction module 110 is configured to divide the sequential video continuous segment v into multiple individual video segments V with an equal number of frames.

[0052] According to a preferred embodiment of the present application, each single video segment V is a fixed number of frames, and adjacent single video segments V do not overlap, for example: the first single video segment V1 is frames 1-5 of the time-continuous video segment v, the second single video segment V2 is frames 6-10 of the time-continuous video segment v, the third single video segment V3 is frames 11-15 of the time-continuous video segment v, and so on. According to another preferred embodiment of the present application, adjacent single video segments V can also overlap, for example: the first single video segment V1 is frames 1-5 of the time-continuous video segment v, the second single video segment V2 is frames 5-9 of the time-continuous video segment v, the third single video segment V3 is frames 9-13 of the time-continuous video segment v, and so on.

[0053] The feature dimension of the convolution result output by the 3D-CNN backbone network includes: the number n of single video segments V, the time length / frame number T of the single video segment, the channel number C of the video, the height H of the video, and the width W of the video. The video backbone feature extraction module 110 is further configured to reduce the channel number C by 3D convolution.

[0054] According to a preferred embodiment of the present application, the 5D feature map f(v) corresponding to the time-continuous video segment v is output by the 3D-CNN backbone network, and then the channel number is reduced from C to d by a 3D convolution with a 1x1x1 convolution kernel (not shown in the figure). Figure 2 m d m for the model dimension of the subsequent person positioning module 120 and the action recognition module 130.

[0055] According to a preferred embodiment of the present application, in the video action detection system 100, the person positioning module 120 is configured to output the person positioning frame according to the middle slice of the 4D feature map f(V) of each single video segment V in the frame number dimension T.

[0056] As shown in Figure 2 , the person positioning module 120 includes n Transformer models for person positioning, as shown in the figure. Figure 2 The n PTx models of the person positioning module 120 correspond to the 4D feature map f(V) output after the n single video segments V pass through the 3D-CNN backbone network. The 4D feature map f(V) corresponding to each single video segment V is shown in the figure. The middle frame of the 4D feature map f(V) is input into the PTx model as a key frame. The PTx model generates a person detection frame BOXES according to the key frame of each single video segment V.

[0057] ​According to a preferred embodiment of the present invention, in the video motion detection system 100, the person positioning module 120 is further configured to extract the motion query option Qa of the person positioning box through a time-space pooling layer ST-Rol-Pool.

[0058] Repeat the person detection boxes over time to form an action tube surrounding the person, such as... Figure 2 The area within the dashed ellipse is the action tube, which is then subjected to dimensionality reduction using a temporal-spatial pooling layer (ST-Rol-Pool). This is used to decode the action query (Q) for the character's action category. a The feature map f(V) is initialized as the feature map f(V) corresponding to the action tube in the intermediate segment of the video backbone. The temporal-spatial pooling layer ST-Rol-Pooling is used to pool the Q-values. a The dimension is reduced to d m .

[0059] According to a preferred embodiment of the present invention, in the video motion detection system 100, the motion recognition module 130 is configured to perform temporal dimensionality reduction on the single video segment V through a time pooling layer T-Pool.

[0060] like Figure 2 As shown, according to a preferred embodiment of the present invention, the action recognition module 130 includes a transformer model for action recognition. Figure 2 The ATx shown is used to obtain the feature map corresponding to the video backbone. A temporal pooling layer T-Pool is used to perform temporal dimensionality reduction on the feature map f(V) of each individual video segment V, resulting in individual video segments V1, V2...V... n The input is rearranged and fed into the ATx module. The ATx module receives the pooled video backbone feature maps concatenated in time and outputs the corresponding action category for each person bounding box (BOXES).

[0061] According to a preferred embodiment of the present invention, in the motion detection system 100, the motion recognition module 130 is further configured to: output the motion category corresponding to the motion query option of each character positioning box based on the concatenated pooled 5D feature maps.

[0062] That is, based on the concatenated pooled 5D feature maps, the action recognition module 130 is provided with the scene context required for the attention mechanism, and based on the feature maps corresponding to the action tubes on the non-intermediate segments in the video backbone, the action recognition module 130 is provided with the temporal context required for the attention mechanism.

[0063] The application of the character positioning module 120 and the action recognition module 130 to the transformer model is specifically explained below.

[0064] The core of the transformer model is the attention module. Given queries, keys and values, where the query is a query option, the key is a query result including the query option, and the value is information content corresponding to the query result. The attention module aggregates values in a weighted sum, and for each query, the size of the weight is measured by the compatibility of all query-key pairs, that is:

[0065] Attention(Q, K, V) = ∑ i (Similarity(Q, K u )*V i )

[0066] Specifically, given an X q matrix wrapped by N q queries and an X kv matrix wrapped by N kv keys / values, the feature of each single-head attention can be obtained as follows:

[0067]

[0068] Where X′ q = X q W q , X′ k = X kv W k , and X′ v = X kv W v .W q , W k and W v are embedding matrices with dimensions d m xd′ m .

[0069] The character positioning module adopted by the application includes a Person Transformer (PTx) and an action recognition module including an Action Transformer (ATx), which are both variants of the standard Transformer model structure. The two transformer structures are combined and used together to perform character positioning and action recognition, respectively.

[0070] According to a preferred embodiment of the application, as Figure 3As shown, in the video action detection system 100, the person localization module 120 further comprises:

[0071] a first encoder PTx Encoder configured to encode spatial information of the intermediate slices of the 4D feature map f(V) corresponding to the single video segment V in the frame dimension.

[0072] a first decoder PTx Decoder configured to decode the coordinates of the person localization box from the person query option Q p and the spatial information encoding M C output the coordinates of the person localization box.

[0073] According to a preferred embodiment of the present application, as Figure 3 shown, in the video action detection system 100, the person localization module 120 further comprises:

[0074] a first video cutting module Tokenize-1 configured to cut the 4D feature map f(V) corresponding to the single video segment V.

[0075] Figure 3 A specific implementation structure of the person localization module 120 is shown. Among them, Tokenize-1 represents the first video cutting module, PTx Encoder represents the first encoder, and PTx Decoder represents the first decoder.

[0076] In the transformer model PTx of the person localization module 120, the first encoder PTx Encoder encodes the spatial information of the time sequence intermediate frame in the video segment to obtain M C , and the first decoder PTx Decoder decodes the coordinates of the person localization box from the person query option Q p and the spatial information encoding M C .

[0077] Preferably, the model PTx generates the person localization box on the time sequence intermediate frame of each sub-segment in the video backbone. Because the original video without cropping is divided into separate and continuous single video segments of the same size, each segment is centered on the annotated "key frame", so the time sequence intermediate slice of the 4D feature map generated by the 3D-CNN backbone network corresponds to the feature map containing the annotated "key frame". As Figure 2 shown, PTx receives the time sequence intermediate slice of the 4D feature map as input and generates the person localization box BOXES on the intermediate frame of the video segment. PTx has two stages, Tokenization and Transformer.

[0078] As Figure 3As shown, in the Tokenization phase, the first video segmentation module Tokenize-1, for a given i-th segment,

[0079]

[0080] Folded into one dimension, generating a dimension d m ×HW's visual token sequence T p .

[0081] In the Transformer stage, a standard Transformer encoder-decoder architecture is used to transform the T... p Convert to person queries (Q) p Since the Transformer has neither recursion nor convolution, a fixed sinusoidal "positional encoding" is added to the visual label T. p In China. Based on T p Using its self-attention mechanism, the encoder PTx Encoder roughly distinguishes different characters through reasoning on spatial scenes.

[0082] The decoder PTx Decoder uses both Q and Q. p The self-attention mechanism on the surface is used to obtain the relationship between them, and then through Q p and spatial information encoding M C Cross-attention is used to focus on key peripheral organs of a person, such as the head and legs. Specifically, in the cross-attention of the decoder (PTx Decoder), T is updated by the encoder (PTx Encoder). p As contextual memory, in Figure 3 The middle is represented as M c A fixed number of people can query Q. p It is initialized as an all-zero vector and supplemented with a learnable "positional code" E as an anchor.

[0083] Finally, a multi-layer fully connected network (FFN) was used to process the person query (Q). p Decode into the coordinates of the positioning frame.

[0084] According to a preferred embodiment of the present invention, such as Figure 4 As shown, in the video motion detection system 100, the motion recognition module 130 further includes:

[0085] The second encoder, PTx Encoder, is configured to encode the spatiotemporal information of the 5D feature maps corresponding to the continuous segments of the temporal video, and output the scene context and the character temporal context.

[0086] a second decoder PTx Decoder configured to decode the action query Q a , the scene context M S , the person temporal context M L to output the corresponding action category.

[0087] According to a preferred embodiment of the present application, in the video action detection system 100, the action recognition module 130 further comprises:

[0088] a second video tokenization module Tokenize-2 configured to tokenize the 5D feature map of the corresponding continuous video segment.

[0089] Figure 4 A specific implementation structure of the action recognition module 130 is shown. Where Tokenize-2 represents the second video tokenization module, ATx Encoder represents the second encoder, and ATx Decoder represents the second decoder.

[0090] The ATx transformer model of the action recognition module 130 also adopts an encoder-decoder structure and has "position encoding" and parallel decoding. Given the temporally discrete person bounding boxes BOXES output by the PTx model on the continuous video segments, these person bounding boxes BOXES are repeated in time to form an action tube (as shown in the dashed oval in Figure 2 ). Unlike the all-zero initialization of the person query (Q p ) used to decode the person bounding box, the action query (Q a ) used to decode the person action category is initialized as the feature map f(V) corresponding to the action tube on the middle segment in the video backbone. The time-space pooling layer ST-Rol-Pooling is used to reduce the dimension of Q a to d m . ATx also has two stages, Tokenization and Transformer.

[0091] As shown in Figure 4 , in the Tokenization stage, the second video tokenization module Tokennize-2 is given the 5D feature map corresponding to the video backbone. The H and W dimensions are first folded into one dimension, and then the T dimension is averaged into one dimension, finally obtaining the visual token sequence T a of the video backbone. The dimension of T a is d mHW x n. To compensate for the lost spatio-temporal information, fixed spatial "position encoding" for the HW dimension and learnable temporal "position encoding" E for the n dimension are added to T a In the first cross-attention module in the decoder ATx Decoder, the updated T a is provided with the context needed by the attention module, i.e. the scene context, as scene context memory (M S ). In the second cross-attention module in the decoder ATx Decoder, the feature maps corresponding to the action tube on the non-intermediate segments in the video backbone are provided as the context needed by the attention module, i.e. the temporal context, as long-term context memory (M L ).

[0092] In the Transformer stage, T a is transformed into action query (Q a ) using a standard Transformer model. Q a is supplemented with extra information by position encoding (coordinates of the bounding boxes of the persons) and group encoding (identity labels of the persons). First, the encoder ATx Encoder is able to roughly associate the interaction information of different persons and objects in the spatio-temporal scene using self-attention mechanism on T a . Then, the decoder ATx Decoder obtains the relationship between them (such as the interaction action between persons) using self-attention mechanism on Q a , and aggregates the context information of other persons and objects (such as the body posture action, the person-object interaction action, and the person-person interaction action) using two cross-attention mechanisms on the scene context memory (M s ) and the long-term context memory (M L ). Finally, a multi-layer fully connected network FFN is used to decode the action query (Q a ) into action categories.

[0093] Figure 3 、 Figure 4The skilled in the art can understand that the number of the encoder and the decoder in the transformer model is not limited to the illustration, and the skilled in the art can set different numbers of the encoder and / or the decoder to achieve the best detection effect. Table 1 shows the performance of the person positioning (Loc) and the action classification (Cls) obtained by setting different numbers of the encoder and the decoder according to one preferred embodiment of the present application.

[0094]

[0095] According to one preferred embodiment of the present application, the number of the PTx transformer model in the person positioning module 120 and the number of the ATx transformer model in the action recognition module 130 in the video action detection system 100 provided by the present application can be adjusted to obtain the best detection performance. The number of the PTx transformer model can correspond to the number n of the time-sequential video segment v divided into a plurality of single video segments V, or can be less than n, that is, the person detection result of the intermediate frame of the video segment V is output by the PTx transformer model.

[0096] According to one preferred embodiment of the present application, as shown in Figure 5 The present application further provides a video action detection method 10, which is completed by the video action detection system as described above.

[0097] In step S101, the time-sequential video segment is output as a corresponding 5D feature map by the video backbone feature extraction module;

[0098] In step S102, the person positioning module outputs a corresponding person positioning frame according to the 5D feature map;

[0099] In step S103, the action recognition module outputs an action category corresponding to each person positioning frame according to the 5D feature map and the person positioning frame.

[0100] The preferred embodiment of the present application provides a video action detection system combining 3D-CNN and transformer model, for two sub-tasks of video action detection, two encoder-decoder structures are used respectively to perform person positioning and action recognition respectively. The video action detection system provided by the present application uses a convolutional neural network to model low-level visual structures, and uses a transformer model to model high-level semantics. Experiments show that the two have complementarity. The video action detection system provided by the present application not only has a simple structure and is easy to implement, but also achieves optimal performance on multiple benchmarks, and is the preferred solution in the field of video person action detection.

[0101] Finally, it should be noted that: the above only for the preferred embodiments of the present application, and not for limiting the present application, although the foregoing embodiments of the present application are described in detail, for those skilled in the art, the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A video motion detection system, characterized by, Comprise: a video backbone feature extraction module comprising a 3D-CNN backbone network configured to output a corresponding 5D feature map for a continuous segment of a time-series video, and obtain a 4D feature map by reducing the channel number of the 5D feature map through 3D convolution; a person positioning module configured to output a corresponding person positioning frame according to the 5D feature map, the person positioning module comprising a plurality of Transformer models for person positioning, configured to generate the person positioning frame according to a key frame of each single video segment, wherein the key frame is a middle frame of the 4D feature map, and to extract an action query option of the person positioning frame through a space-time pooling layer; the person positioning module comprises: a first encoder configured to encode spatial information of a middle slice in the frame number dimension of the 4D feature map corresponding to the single video segment; a first decoder configured to output coordinates of the person positioning frame according to a person query option and the spatial information encoding; and a first video cutting module configured to cut the 4D feature map corresponding to the single video segment; an action recognition module configured to output an action category corresponding to each person positioning frame according to the 5D feature map and the person positioning frame, the action recognition module comprising a plurality of Transformer models for action recognition, configured to output an action category corresponding to an action query option of each person positioning frame according to the concatenated pooled 5D feature map; the action recognition module comprises: a second encoder configured to encode time-space information of the 5D feature map corresponding to the continuous segment of the time-series video, outputting a scene context and a person time-series context; a second decoder configured to output a corresponding action category according to the action query option, the scene context, and the person time-series context; and a second video cutting module configured to cut the 5D feature map corresponding to the continuous segment of the time-series video.

2. The video motion detection system of claim 1, wherein the video backbone feature extraction module is configured to divide the time-sequential video continuous segment into a plurality of single video segments with equal number of frames, the dimensions of the 5D feature map comprising: The length of the continuous segment of the time-series video, the frame number of the single video segment, the channel number, height, and width of the video.

3. The video action detection system of claim 2, wherein the person positioning module is further configured to: output the person positioning frame according to the middle slice in the frame number dimension of the 4D feature map corresponding to the single video segment.

4. The video action detection system of claim 3, wherein the action recognition module is further configured to: perform time-series dimension reduction on the single video segment through a time pooling layer.

5. A method for video action detection, accomplished by the video action detection system of any one of claims 1-4.

Images