Spatiotemporal action detection method and system based on center alignment and motion mining

Abstract

The application discloses a kind of spatio-temporal action detection method and system based on center alignment and motion mining, wherein the method comprises: inputting T frame video frame into pre-trained backbone network, and extracting features;The extracted features are input into three branches, respectively to obtain boundary box size, action vector, center point spatial position and action category;Wherein, in the center point positioning branch and motion branch, center alignment model is used for center alignment, and the action center on adjacent frame is aligned with key frame;Discriminative motion features are mined using motion mining model, and motion information unrelated to the action is filtered and focused using mask mechanism.The application aligns the action center between frames, and obtains discriminative motion features, solves the false detection problem caused by rapid motion, and significantly reduces the computational complexity, and is more suitable for application in actual scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video processing, in particular to a spatiotemporal action detection method and system based on center alignment and motion mining, and simultaneously provides a corresponding computer terminal and computer readable storage medium. BACKGROUND

[0002] The goal of spatiotemporal action detection is to localize action instances in space and time and recognize action labels in frame sequences. Current mainstream spatiotemporal action detection methods can be divided into 2D skeleton-based small tube detectors and 3D skeleton-based methods. The 3D skeleton-based method generally requires a separate actor detector, resulting in a dense candidate, which is difficult to achieve real-time inference in practical application scenarios. Thanks to the development of anchor-free methods, 2D skeleton-based tube detectors show potential in speed and accuracy. A recently proposed 2D-CNN-based accurate action detector, MOC detector (Actions as Moving Points, 2020), treats each action instance as a moving point. The MOC detector provides a strong baseline for real-time action detection and achieves impressive performance.

[0003] Although the MOC detector has achieved good performance, the quality of the action instance is still not satisfactory, especially for fast action instances. First, affected by fast motion, 2D-CNN is prone to produce chaotic features that are inconsistent in space and time. Considering the dramatic changes in the spatiotemporal features of the actor, directly aggregating chaotic features to predict the action center is prone to mutual interference between inter-frame features. In a dramatic motion scene, mutual interference occurs between adjacent frames, resulting in incorrect action positions. Therefore, in order to prevent mutual interference, the action center must be calibrated between adjacent frames.

[0004] Second, spatiotemporal features require efficient and sufficient motion mining to improve action classification performance. Self-attention is widely used because it can model long-term dependencies. However, self-attention focuses on all feature dependencies, including irrelevant features. Focusing on global information also leads to high computational complexity. SUMMARY

[0005] In view of the above-mentioned deficiencies of the prior art spatiotemporal action detection technology, the present application provides a spatiotemporal action detection method and system based on center alignment and motion mining, and simultaneously provides a corresponding computer terminal and computer readable storage medium.

[0006] According to one aspect of the present application, a spatiotemporal action detection method based on center alignment and motion mining is provided, comprising:

[0007] acquire T continuous video frames;

[0008] extract spatial features from the T video frames;

[0009] perform frame-by-frame processing on the spatial features to obtain the size and position of the bounding box of each frame of action;

[0010] align the action centers of adjacent frames in the spatial features with the key frame in a center alignment manner, and remove irrelevant responses to obtain center-aligned motion features;

[0011] perform discriminative motion feature mining on the center-aligned motion features in a motion mining manner, and filter motion features irrelevant to the action of interest to obtain spatiotemporal features;

[0012] calculate the motion vector of the motion center point of each frame in the spatiotemporal features from the motion center point of the key frame;

[0013] detect the spatial position and action category of the motion center point of the key frame in the spatiotemporal features;

[0014] based on the size and position of the bounding box, the motion vector of the motion center point of each frame from the motion center point of the key frame, and the spatial position and action category of the motion center point of the key frame, construct the detection result of each frame of video frame to complete spatiotemporal action detection.

[0015] Preferably, the spatial feature extraction of the T video frames comprises:

[0016] extract features of the T video frames using a pre-trained backbone network;

[0017] stack the extracted features to obtain spatial features.

[0018] Preferably, the center alignment manner of aligning the action centers of adjacent frames with the key frame and removing irrelevant responses to obtain center-aligned motion features comprises:

[0019] construct a center alignment model based on deformable convolution;

[0020] obtain effective foreground features in the field range as learning samples to pre-train the center alignment model;

[0021] input the i-th frame and the key frame into the center alignment model to adaptively guide the action centers of adjacent frames to align with the key frame;

[0022] remove irrelevant responses using the response intensity of each pixel of the i-th frame to the key frame to obtain center-aligned motion features.

[0023] Preferably, the motion mining method is used to mine discriminative motion features from the center-aligned motion features, and filter out motion features irrelevant to the action of interest, to obtain discriminative spatiotemporal features, including:

[0024] Constructing a motion mining model;

[0025] Pre-training the motion mining model using the target detection COCO dataset;

[0026] Inputting the center-aligned motion features into the motion mining model to obtain valuable motion regions of the motion features;

[0027] Filtering out valuable motion regions irrelevant to the action of interest to obtain discriminative spatiotemporal features.

[0028] Preferably, the filtering out of valuable motion regions irrelevant to the action of interest includes:

[0029] Generating a bias quantity using a time variance feature map to capture motion regions;

[0030] Using a deformable attention to mine motion regions inconsistent with time representation to obtain one or more offset branches;

[0031] Generating S independent masks for each offset branch, using incoherent motion masks to filter out irrelevant noise, to obtain discriminative spatiotemporal features.

[0032] Preferably, the calculation of the motion vector of the motion center point on each frame in the initial spatiotemporal features from the motion center point of the key frame includes:

[0033] Constructing a fully connected layer network model for regression of the motion vector of the motion center point on each frame from the motion center point of the key frame;

[0034] Constructing an L1 loss function for constraining the fully connected layer network model to learn the motion vector of the motion center point on each frame from the motion center point of the key frame;

[0035] The motion vector of the motion center point on each frame in the initial spatiotemporal features from the motion center point of the key frame is obtained through the above steps.

[0036] Preferably, the detection of the spatial position and action category of the motion center point of the key frame includes:

[0037] Constructing a fully connected layer network model for learning class-related motion center point heat maps;

[0038] Constructing a Focal-Loss loss function for constraining the fully connected layer network model to learn class-related motion center point heat maps;

[0039] According to the obtained correlation motion center point heat map, the spatial position and action category of the key frame motion center point are detected.

[0040] According to another aspect of the present application, a spatio-temporal action detection system based on center alignment and motion mining is provided, comprising:

[0041] a data acquisition module, configured to acquire continuous T frames of video frames;

[0042] a backbone network module, configured to extract features from the T frames of video frames;

[0043] a bounding box regression module, configured to perform frame-by-frame processing on the spatial features to obtain the size and position of the bounding box of each frame of action;

[0044] a center alignment module, configured to align the action centers of adjacent frames in the spatial features to the key frame and remove irrelevant responses in a center alignment manner, to obtain the center-aligned motion features;

[0045] a motion mining module, configured to perform discriminative motion feature mining on the center-aligned motion features in a motion mining manner, to obtain the spatio-temporal features with discrimination;

[0046] a motion vector calculation module, configured to calculate the motion vector of the motion center point of each frame in the spatio-temporal features from the motion center point of the key frame;

[0047] a key frame motion center point detection module, configured to detect the spatial position and action category of the motion center point of the key frame in the spatio-temporal features;

[0048] a spatio-temporal action detection module, configured to construct the detection result of each frame of video frame based on the size and position of the bounding box, the motion vector of the motion center point of each frame from the motion center point of the key frame, and the spatial position and action category of the motion center point of the key frame, to complete the spatio-temporal action detection.

[0049] According to a third aspect of the present application, a computer terminal is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to perform the method of any one of the above, or runs the system of the above.

[0050] According to a fourth aspect of the present application, a computer readable storage medium is provided, which stores a computer program executable by a processor to perform the method of any one of the above, or to run the system of the above.

[0051] Compared with the prior art, the present application has at least one of the following beneficial effects:

[0052] The present application adopts the spatio-temporal action detection technology of center alignment and motion mining, solves the representation confusion and action pipeline dislocation problems caused by the misalignment of spatio-temporal context features, and thus obtains more accurate action detection.

[0053] The center alignment branch model proposed by the present application aligns adjacent frame features and key frame features at the pixel level by using deformable convolution, and obtains a more accurate action center.

[0054] The motion mining branch model proposed by the present application captures discriminative motion features, uses time variance guided deformable self-attention, fully mines the motion regions with inconsistent time representations, and has lower computational complexity compared with the original self-attention method which captures all spatial dependencies. BRIEF DESCRIPTION OF DRAWINGS

[0055] Other features, objects and advantages of the present application will become more apparent from the following detailed description of non-limiting embodiments with reference to the attached drawings:

[0056] Figure 1 The working flow chart of the spatio-temporal action detection method based on center alignment and motion mining in a preferred embodiment of the present application.

[0057] Figure 2 The working principle diagram of the spatio-temporal action detection method and system based on center alignment and motion mining in a preferred embodiment of the present application.

[0058] Figure 3 The working schematic diagram of the center alignment branch model in a preferred embodiment of the present application.

[0059] Figure 4 The working schematic diagram of the motion mining branch model in a preferred embodiment of the present application.

[0060] Figure 5 The component module schematic diagram of the spatio-temporal action detection system based on center alignment and motion mining in a preferred embodiment of the present application. DETAILED DESCRIPTION

[0061] The present application will be described in detail below with reference to specific embodiments. The following embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any form. It should be noted that, for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made. These all belong to the protection scope of the present application.

[0062] The embodiment of the present application provides a spatio-temporal action detection method based on center alignment and motion mining.

[0063] As shown in Figure 1 and Figure 2 The embodiment provides a spatio-temporal action detection method based on center alignment and motion mining, which comprises the following steps.

[0064] S1, acquiring continuous T frames of video frames;

[0065] S2, performing spatial feature extraction on the T frames of video frames;

[0066] S3, performing frame-by-frame processing on the spatial features to obtain the size and position of the boundary box of each frame of action;

[0067] S4, adopting the center alignment mode, aligning the action centers on adjacent frames in the spatial features to the key frame, and removing irrelevant responses to obtain the center-aligned motion features;

[0068] S5, adopting the motion mining mode, performing discriminative motion feature mining on the center-aligned motion features, and filtering the motion features irrelevant to the concerned action to obtain the spatio-temporal features;

[0069] S6, calculating the motion vector of the motion center point on each frame in the spatio-temporal features to the motion center point of the key frame;

[0070] S7, detecting the spatial position and action category of the motion center point of the key frame;

[0071] S8, based on the size and position of the boundary box, the motion vector of the motion center point on each frame to the motion center point of the key frame, and the spatial position and action category of the motion center point of the key frame, constructing the detection result of each frame of video frame to complete the spatio-temporal action detection.

[0072] The spatio-temporal action detection result is composed of a series of single-frame detection results, and the single-frame detection result is the size, position and action category of each frame. The motion center position of each frame is obtained by adding the position of the key frame motion center point and the motion vector of each frame relative to the key frame. The center position, size and category of each frame jointly constitute the detection result of each frame.

[0073] In a preferred embodiment of S2, the spatial feature extraction on the T frames of video frames comprises the following steps.

[0074] S21, a pre-trained backbone network is used to extract features of the T-frame video frame; in a preferred embodiment, the backbone network can use DLA-34 or RESNET, further, using the DLA-34 network with a hollow convolution can obtain better performance.

[0075] S22, the extracted features are stacked to obtain spatial features.

[0076] In a preferred embodiment of S4, the action center of the adjacent frame is aligned with the key frame in a center alignment manner, and irrelevant responses are removed to obtain the center-aligned motion features, including:

[0077] S41, a center alignment model is constructed based on deformable convolution; in a preferred embodiment, the center alignment model includes a splicing module, a multi-layer convolution module, and a deformable convolution module; the splicing module is used to splice the spatial features of the key frame and the spatial features of the adjacent frame; the multi-layer convolution module is used to learn the learnable parameter offset and modulation scalar of the deformable convolution module kernel; the center alignment process is completed by modulating the learnable parameters of the deformable convolution module kernel; the obtained alignment features are added to the adjacent frame features in a residual manner;

[0078] S42, effective foreground features in the field range are obtained as learning samples to pre-train the center alignment model;

[0079] S43, the i-th frame and the key frame are input into the center alignment model to adaptively guide the action center of the adjacent frame to align with the key frame;

[0080] S44, the response intensity of each pixel of the i-th frame to the key frame is used to remove irrelevant responses to obtain the center-aligned motion features.

[0081] In a preferred embodiment of S5, the center-aligned motion features are discriminative motion feature mined in a motion mining manner, and the motion features irrelevant to the concerned action are filtered to obtain discriminative spatiotemporal features, including:

[0082] S51, a motion mining model is constructed; in a preferred embodiment, the motion mining model includes a multi-layer convolution layer module, a sampling function module, and a self-attention mechanism module; the multi-layer convolution module is used to project the input features to three linear spaces respectively to obtain query (query vector q), key (index vector k), and value (content vector v), wherein the query vector q will be subjected to variance operation in time sequence to obtain a time sequence variance map, and the time sequence variance map is subjected to the multi-layer convolution module to obtain the input bias of the sampling function, and the selection of the discriminative motion point is completed; the sampling function module is used to sample the index vector k and the content vector v respectively under the guidance of the input bias to obtain the sampled index vector with the content vector The self-attention mechanism module is used to perform matrix multiplication on the query vector q and the sampled index vector to obtain an attention map, and the attention map is multiplied with the sampled content vector to obtain the motion mining feature.

[0083] S52, the motion mining model is pre-trained using the target detection COCO dataset.

[0084] S53, the center-aligned motion feature is input into the motion mining model to obtain a valuable motion region of the motion feature.

[0085] S54, filtering the valuable motion region irrelevant to the action of interest to obtain a discriminative spatiotemporal feature.

[0086] In a preferred embodiment of S54, filtering the valuable motion region irrelevant to the action of interest includes:

[0087] S541, generating a bias quantity using a time variance feature map to capture a motion region.

[0088] S542, using a deformable attention to mine a motion region inconsistent with a time representation to obtain one or more offset branches.

[0089] S543, generating S independent masks for each offset branch, and using the incoherent motion mask to filter out irrelevant noise to obtain a discriminative spatiotemporal feature. Further, the mask is directly multiplied with the input feature to filter out irrelevant noise.

[0090] In a preferred embodiment of S6, the motion vector of the motion center point on each frame in the initial spatiotemporal feature from the motion center point of the key frame is calculated, including:

[0091] A fully connected layer network model is constructed for regression of the motion vector of the motion center point on each frame from the motion center point of the key frame.

[0092] An L1 loss function is constructed for constraining the fully connected layer network model to learn the motion vector of the motion center point on each frame from the motion center point of the key frame.

[0093] The motion vector of the motion center point on each frame in the initial spatiotemporal feature from the motion center point of the key frame is obtained through the above steps.

[0094] In a preferred embodiment of S7, the spatial position and action category of the key frame motion center point are detected, including:

[0095] A fully connected layer network model is constructed for learning a class-related motion center point heat map.

[0096] A Focal-Loss loss function is constructed for constraining the fully connected layer network model to learn the class-related motion center point heat map.

[0097] According to the obtained related motion center point heat map, the spatial position and action category of the key frame motion center point are detected.

[0098] In the above steps:

[0099] The key frame refers to the center frame in the T frames, i.e., the T / 2 frame, and T is generally taken as an odd number, and T / 2 is rounded down. The adjacent frame refers to other frames in the T frames except the key frame.

[0100] Discriminative refers to the gain for the action classification and positioning task. Generally, the more discriminative the feature is, the higher the precision performance can be obtained. The discriminative motion feature refers to the effective feature extracted from the motion region, such as the swing of the thigh and foot and the movement of the football for the kicking football action, which is considered as a discriminative motion feature.

[0101] The motion vector is the difference value of the two-dimensional coordinates of the key frame motion center and the adjacent frame motion center.

[0102] The action category is various categories defined by the data set, such as shooting and combing hair in the UCF101-24 data set.

[0103] The technical solutions provided by the above embodiments of the application are further described in detail as follows.

[0104] As shown in Figure 1 and Figure 2 The spatiotemporal action detection method based on center alignment and motion mining in the above embodiments of the application includes the following steps:

[0105] First, obtain continuous T frames of video frames.

[0106] Second, input the T frames of video frames into a pre-trained backbone network for feature extraction.

[0107] Third, input the features extracted in S2 into three branches, which are a frame branch, a motion regression branch, and a class-aware action center point positioning branch (referred to as a center point positioning branch), for obtaining the size and position of the bounding box, the motion vector of the center point, and the spatial position and category of the center point.

[0108] In the third step:

[0109] The center alignment model and the motion mining model are used in the motion branch and the center point positioning branch. Among them:

[0110] The center alignment model is used for center alignment to align the motion center key frames on adjacent frames;

[0111] The motion mining model is used for mining discriminative motion features and filtering motion information irrelevant to the motion of interest by using a mask mechanism.

[0112] Further, the extracted spatial features are input into three branches to obtain the size and position of the bounding box, the motion vector of the center point, and the spatial position and category of the center point. In a specific application embodiment, the MOC detector can be used as the basic architecture of the three branches. Of course, in other embodiments, other detector architectures, such as the ACT detector, can also be used.

[0113] In a specific embodiment, three branches are constructed for spatiotemporal action detection, which can be implemented as follows:

[0114] Step 3.1, a bounding box regression branch is designed for frame-by-frame processing to obtain the size and position of the action bounding box of each frame.

[0115] Step 3.2, a motion regression branch is designed to estimate the motion vector of the motion center point of each frame relative to the motion center point of the key frame.

[0116] Step 3.3, a class-aware action center point localization branch is designed to detect the spatial position and category of the key frame center point.

[0117] The MOC detector is used as the basic architecture, and the anchor-free structure is used to make the precision and speed of the method meet the real-time requirements.

[0118] Further, the center alignment model and the motion mining model are used in the working process of the center point localization branch and the motion branch. These two models can be arranged in the two branch parts or as independent parts arranged at the front end of the two branch parts.

[0119] In a specific embodiment, the center alignment model and the motion mining model are used in the center point localization branch and the motion branch, which can be implemented as follows:

[0120] Step 3.i, the extracted spatial features are used as the input of the center alignment model to obtain the center-aligned motion features.

[0121] Step 3.ii, the center-aligned motion features are used as the input of the motion mining model to obtain discriminative spatiotemporal features.

[0122] Step 3.iii, the discriminative spatiotemporal features are used as the input of the motion regression branch and the center point localization branch, respectively.

[0123] The center alignment model is adopted to implicitly learn motion cues and obtain effective foreground features in a domain range to adaptively guide the alignment of inter-frame features. The motion mining module is adopted to mine discriminative spatiotemporal features to filter out irrelevant motion information.

[0124] Further, the center alignment model performs center alignment to align the action centers on adjacent frames to the key frame. Of course, in this specific application example, the center alignment model based on modulation (i.e., the center alignment model based on deformable convolution) can be adopted, or other center alignment models can be adopted.

[0125] In a specific embodiment, the implementation of the center alignment model can refer to the following operations:

[0126] The center alignment model based on modulation is constructed. The deformable convolution is used to obtain local pixel-level alignment. As shown in Figure 3 the i-th frame and the key frame are taken as inputs of the center alignment model, where the Offset reflects the position movement. The Scalars are modulation levels, which reflect the response strength of each pixel of the i-th frame to the key frame, and are taken as filters to remove irrelevant responses.

[0127] The center alignment model based on modulation adaptively guides the alignment of inter-frame features, implicitly learns motion cues, and obtains effective foreground features in a neighborhood range.

[0128] Further, the motion mining model mines discriminative motion features, obtains discriminative spatiotemporal features under the guidance of valuable motion regions, and filters out motion information irrelevant to the action of interest by using a mask mechanism. In a specific application example, the sparse valuable motion mining model can be adopted, or other motion mining models can be adopted. The bias is generated by using a temporal variance feature map; the deformable self-attention mechanism is used to reduce the computational complexity; and the irrelevant motion mask is used to filter out irrelevant noise.

[0129] In a specific embodiment, the implementation of the motion mining model can refer to the following operations:

[0130] A sparse valuable motion mining model is constructed to obtain discriminative spatio-temporal features under the guidance of valuable motion regions. Figure 4

[0131] Specifically,

[0132] In the third step, a deformable attention module is used to effectively reduce the computational complexity.

[0133] In the third step, a deformable attention module is used to effectively reduce the computational complexity.

[0134] In the third step, a kind of irrelevant motion mask is designed to filter out irrelevant noise, so as to obtain more discriminative motion information.

[0135] The above-mentioned embodiments of the present application effectively solve the problem of false detection caused by fast motion by using a center alignment model (preferably a modulation-based center alignment model).

[0136] The above-mentioned embodiments of the present application effectively solve the problem of false detection caused by fast motion by using a center alignment model (preferably a modulation-based center alignment model).

[0137] An embodiment of the present application provides a spatio-temporal action detection system based on center alignment and motion mining.

[0138] As shown in Figure 5 The spatio-temporal action detection system based on center alignment and motion mining provided by the embodiment can include:

[0139] ​a data acquisition module configured to acquire continuous T-frame video frames;

[0140] a backbone network module configured to extract features from the T-frame video frames;

[0141] a bounding box regression module configured to perform frame-by-frame processing on the spatial features to obtain the size and position of a bounding box of each frame of action;

[0142] a center alignment module configured to align the action centers of adjacent frames in the spatial features with a key frame in a center alignment manner and remove irrelevant responses to obtain center-aligned motion features;

[0143] a motion mining module configured to perform discriminative motion feature mining on the center-aligned motion features in a motion mining manner to obtain spatiotemporal features with discrimination;

[0144] a motion vector calculation module configured to calculate a motion vector of a motion center point of each frame in the spatiotemporal features from a motion center point of a key frame;

[0145] a key frame motion center point detection module configured to detect the spatial position action and category of a key frame motion center point in the spatiotemporal features;

[0146] a spatiotemporal action detection module configured to construct a detection result of each frame of video based on the size and position of the bounding box, the motion vector of the motion center point of each frame from the motion center point of the key frame, and the spatial position and action category of the key frame motion center point, and complete spatiotemporal action detection.

[0147] In a preferred embodiment, the bounding box regression module serves as a bounding box regression branch; the center alignment module, the motion mining module, and the motion vector calculation module jointly serve as a motion regression branch; and the center alignment module, the motion mining module, and the key frame motion center point detection module jointly serve as a center point positioning branch.

[0148] In a preferred embodiment, the bounding box regression module serves as a bounding box regression branch; the motion vector calculation module serves as a motion regression branch; the key frame motion center point detection module serves as a center point positioning branch; and the center alignment module and the motion mining module are located at the front end of the motion regression branch and the center point positioning branch.

[0149] It should be noted that the steps in the method provided by the present application can be implemented by using corresponding components in the system, and those skilled in the art can refer to the technical solution of the system to implement the step flow of the method, or refer to the technical solution of the method to implement the composition of the system, that is, the embodiments in the system and the embodiments in the method can be understood as preferred examples, which will not be described here.

[0150] An embodiment of the present application provides a computer terminal, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor is configured to execute the computer program to implement the method of any one of the above-mentioned embodiments of the present application or run the system of any one of the above-mentioned embodiments of the present application.

[0151] Optionally, the memory is configured to store the program, and the memory can comprise a volatile memory, for example, a random-access memory (RAM), such as a static random-access memory (SRAM), a Double Data Rate Synchronous Dynamic Random Access Memory (DDR SDRAM) and the like, or a non-volatile memory, for example, a flash memory. The memory is configured to store a computer program (for example, an application program or a functional module for implementing the above-mentioned method), computer instructions and the like, and the above-mentioned computer program, computer instructions and the like can be stored in one or more memories in a partitioned manner. Furthermore, the above-mentioned computer program, computer instructions and data can be invoked by the processor.

[0152] The above-mentioned computer program, computer instructions and the like can be stored in one or more memories in a partitioned manner. Furthermore, the above-mentioned computer program, computer instructions and data can be invoked by the processor.

[0153] The processor is configured to execute the computer program stored in the memory to implement each step of the method or each module of the system. Details can be referred to the above-mentioned method and system embodiments.

[0154] The processor and the memory can be an independent structure or an integrated structure. When the processor and the memory are an independent structure, the memory and the processor can be coupled and connected through a bus.

[0155] An embodiment of the present application further provides a computer readable storage medium, which stores a computer program, and the program is configured to be executed by the processor to implement the method of any one of the above-mentioned embodiments of the present application or run the system of any one of the above-mentioned embodiments of the present application.

[0156] Those skilled in the art will appreciate that embodiments of the application can be provided as a method, or as a computer program product. Accordingly, the application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the application can take the form of a computer program product on one or more computer readable storage media (including, but not limited to, disk memory, CD-ROMs, optical storage devices, etc.) embodying computer readable program code.

[0157] The present application is described in reference to the flowchart and / or block diagrams of the method, apparatus (system) and computer program product according to embodiments of the application. It should be understood that each flow and / or block in the flowchart and / or block diagrams, as well as combinations of flows and / or blocks in the flowchart and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, a special purpose computer, an embedded processor or other programmable data processing device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing device, create means for implementing the functions specified in the flowchart and / or block diagrams block or blocks. Figure 1 one or more flows and / or blocks Figure 1 means for carrying out the function specified by the flow or flows and / or block or blocks.

[0158] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instruction means that implement the function specified in the flowchart and / or block diagrams flow or flows and / or block or blocks. Figure 1 one or more flows and / or blocks Figure 1 means for carrying out the function specified by the flow or flows and / or block or blocks.

[0159] These computer program instructions can also be loaded onto a computer or other programmable data processing device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer implemented process such that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in the flowchart and / or block diagrams flow or flows and / or block or blocks. Figure 1 one or more flows and / or blocks Figure 1 means for carrying out the function specified by the flow or flows and / or block or blocks.

[0160] The space-time action detection method and system provided by the above embodiments of the application can input the extracted features into three branches to obtain the size of the boundary box, the action vector, the spatial position of the center point and the category, wherein the center alignment model is used to align the action center between frames, and the motion mining model is used to obtain the discriminative motion features, so as to solve the false detection problem caused by fast motion, and significantly reduce the calculation complexity, and the method is more suitable for application in actual scenes.

[0161] While the preferred embodiments of the application have been described, additional variations and modifications can be made to these embodiments by those skilled in the art once they have the benefit of the present disclosure without departing from the spirit and scope of the application. Accordingly, it is intended that the appended claims include all such modifications and variations as fall within the scope of the present application.

[0162] It is apparent that those skilled in the art can make various changes and modifications to the application without departing from the spirit and scope of the application. It is therefore intended that the present application cover all such changes and modifications that are within its scope.

Claims

1. A spatio-temporal action detection method based on center alignment and motion mining, characterized in that, The method comprises the following steps: obtaining continuous T frames of video frames; extracting spatial features of the T frames of video frames; performing frame-by-frame processing on the spatial features to obtain the size and position of the bounding box of each frame of action; aligning the action centers of adjacent frames in the spatial features with the key frame in a center alignment manner, and removing irrelevant responses to obtain center-aligned motion features; performing discriminative motion feature mining on the center-aligned motion features in a motion mining manner, and filtering motion features irrelevant to the action of interest to obtain spatiotemporal features; calculating the motion vector of the motion center point of each frame in the spatiotemporal features from the motion center point of the key frame; detecting the spatial position and action category of the motion center point of the key frame in the spatiotemporal features; based on the size and position of the bounding box, the motion vector of the motion center point of each frame from the motion center point of the key frame, and the spatial position and action category of the motion center point of the key frame, constructing the detection result of each frame of video frame to complete spatiotemporal action detection.

2. The spatio-temporal action detection method based on center alignment and motion mining according to claim 1, characterized in that, The method comprises the following steps: extracting features of the T frames of video frames by using a pre-trained backbone network; stacking the extracted features to obtain spatial features.

3. The spatio-temporal action detection method based on center alignment and motion mining of claim 1, wherein, The method comprises the following steps: based on deformable convolution, constructing a center alignment model; obtaining effective foreground features in a domain range as learning samples to pre-train the center alignment model; inputting the i-th frame and the key frame into the center alignment model to adaptively guide the action center of the adjacent frame to align with the key frame; using the response intensity of each pixel of the i-th frame to the key frame to remove irrelevant responses to obtain center-aligned motion features.

4. The spatio-temporal action detection method based on center alignment and motion mining of claim 1, wherein, The method comprises the following steps: constructing a motion mining model; pre-training the motion mining model by using a target detection COCO dataset; inputting the center-aligned motion features into the motion mining model to obtain valuable motion regions of the motion features; filtering the valuable motion regions irrelevant to the action of interest to obtain discriminative spatiotemporal features.

5. The spatio-temporal action detection method based on center alignment and motion mining of claim 4, wherein, The method comprises the following steps: using a time variance feature map to generate a bias quantity to capture motion regions; using deformable attention to mine motion regions inconsistent with time representation to obtain one or more offset branches; generating S independent masks for each offset branch, using incoherent motion masks to filter out irrelevant noise to obtain discriminative spatiotemporal features.

6. The spatio-temporal action detection method based on center alignment and motion mining of claim 1, wherein, The method comprises the following steps: constructing a fully connected layer network model for regression of the motion vector of the motion center point of each frame from the motion center point of the key frame; An L1 loss function is constructed to constrain the fully connected layer network model to learn the motion vector of the motion center point on each frame relative to the motion center point of the key frame; The motion vector of the motion center point on each frame relative to the motion center point of the key frame in the spatio-temporal feature is obtained through the above steps.

7. The spatio-temporal action detection method based on center alignment and motion mining of claim 1, wherein, The spatial position and action category of the motion center point of the key frame are detected, including: A fully connected layer network model is constructed to learn the class-related motion center point heat map; A Focal-Loss loss function is constructed to constrain the fully connected layer network model to learn the class-related motion center point heat map; According to the obtained related motion center point heat map, the spatial position and action category of the motion center point of the key frame are detected.

8. A spatio-temporal action detection system based on center alignment and motion mining, characterized in that, It comprises: A data acquisition module is configured to acquire continuous T frames of video frames; A backbone network module is configured to extract spatial features from the T frames of video frames; A bounding box regression module is configured to process the spatial features frame by frame to obtain the size and position of the bounding box of each frame of action; A center alignment module is configured to align the motion center of adjacent frames in the spatial features to the key frame in a center alignment manner and remove irrelevant responses to obtain the center-aligned motion features; A motion mining module is configured to mine discriminative motion features from the center-aligned motion features to obtain spatio-temporal features with discrimination; A motion vector calculation module is configured to calculate the motion vector of the motion center point on each frame relative to the motion center point of the key frame in the spatio-temporal feature; A key frame motion center point detection module is configured to detect the spatial position and action category of the motion center point of the key frame in the spatio-temporal feature; A spatio-temporal action detection module is configured to construct the detection result of each frame of video frame based on the size and position of the bounding box, the motion vector of the motion center point on each frame relative to the motion center point of the key frame, and the spatial position and action category of the motion center point of the key frame, and complete spatio-temporal action detection.

9. A computer terminal comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to execute the method of any one of claims 1-7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to execute the method of any one of claims 1-7.

Images