Video action recognition method, device and system and storage medium
The video features are sparsely processed through the sparse temporal pyramid module and sparse attention mechanism, which solves the problem of high computational complexity of the video action recognition model and achieves efficient action recognition and memory optimization.
Patent Information
- Application Number
- CN202510852573.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-20
- Publication Date
- 2025-10-17
AI Technical Summary
Existing video action recognition models have high computational complexity, resulting in low action recognition efficiency. In particular, the global attention mechanism needs to calculate the interactions between all frame pairs, resulting in the amount of computation growing quadratically with the length of the video.
The sparse temporal pyramid module and sparse attention mechanism are adopted to perform sparse processing on video features through sparse feature processing and sparse tensor core modules, and multi-level feature processing is performed in combination with the sparse feature fusion module to reduce the amount of feature calculation.
It effectively reduces the feature calculation amount of the model, improves the efficiency of action recognition, and achieves exponential acceleration of fine-grained action recognition through the sparse feature fusion module, reducing memory usage and computational overhead.
Smart Images
Figure CN120808431A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of artificial intelligence and the technical field of financial technology, and in particular to a video action recognition method, device and system and a storage medium. BACKGROUND
[0002] Video action recognition is one of the core tasks in the field of computer vision, and its goal is to identify and classify specific actions or behaviors from continuous video frames. For example, in the financial field, specific action verification is performed on the verification videos uploaded by users in account registration, large payment and other businesses, or the abnormal actions of customers in the monitoring videos between automatic teller machines (ATMs) are identified. For another example, in the medical and health field, health detection videos uploaded by users are identified, which require specific actions such as squatting, kicking legs and jumping.
[0003] In recent years, attention mechanisms and Transformer architectures have been introduced into the field of video action recognition, and long-range temporal dependencies can be captured through self-attention. However, the standard global attention mechanism needs to calculate the interaction between all pairs of frames, resulting in a quadratic increase in computational complexity with the length of the video, which leads to a large amount of model computation and low action recognition efficiency of the model. SUMMARY
[0004] The embodiments of the present application provide a video action recognition method, device, system and storage medium, which can reduce the amount of model computation and improve the action recognition efficiency of the model.
[0005] In a first aspect, the embodiments of the present application provide a video action recognition method, which is applied to a video action recognition system, wherein a video action recognition model is deployed in the video action recognition system, and the video action recognition model comprises a feature extraction module, a sparse temporal pyramid module and an output module. The method comprises the following steps:
[0006] obtaining a time sequence feature sequence of a to-be-recognized video through the feature extraction module;
[0007] determining sparse attention of the time sequence feature sequence through a sparse attention mechanism in the sparse temporal pyramid module;
[0008] performing feature sparsification processing on the time sequence feature sequence according to the sparse attention and a pruning rule in the sparse temporal pyramid module, to obtain a target sparse feature;
[0009] performing action recognition processing on the target sparse feature through the output module, and outputting an action recognition result of the to-be-recognized video.
[0010] In some embodiments, the sparse temporal pyramid module comprises a multi-layer pyramid sub-module, the video action recognition model further comprises a sparse tensor core module and a sparse feature fusion module, after the feature sparse processing of the time sequence features according to the sparse attention and the pruning rule in the sparse temporal pyramid module to obtain the target sparse features, the method further comprises:
[0011] performing feature compression processing on the target sparse features by the sparse tensor core module to obtain target compressed features;
[0012] taking the target compressed features as the input of the time sequence features of the sparse temporal pyramid module, returning to perform the step of determining the sparse attention of the time sequence features by the sparse attention mechanism in the sparse temporal pyramid module until the target sparse features corresponding to each layer of the pyramid sub-module are obtained;
[0013] the action recognition processing of the target sparse features by the output module to output the action recognition result of the video to be recognized, comprising:
[0014] performing feature fusion processing on the target sparse features corresponding to each layer of the pyramid sub-module according to the sparse feature fusion module to obtain target fusion features;
[0015] performing action recognition processing on the target fusion features by the output module to output the action recognition result of the video to be recognized.
[0016] In some embodiments, the feature sparse processing of the time sequence features according to the sparse attention and the pruning rule in the sparse temporal pyramid module to obtain the target sparse features, comprising:
[0017] performing feature sparse processing on the time sequence features based on a preset feature sparse processing formula to obtain target sparse features, the feature sparse processing formula is:
[0018]
[0019] wherein, H l is the target sparse feature corresponding to the l-th layer of pyramid sub-module, SparseAttn l (F l ) is the sparse attention corresponding to the l-th layer of pyramid sub-module, M l is the binary mask corresponding to the l-th layer of pyramid sub-module generated dynamically based on the pruning rule, and M l ∈{0,1} T×T , T is the length of time, F lthe time sequence feature sequence input into the lth pyramid sub-module, represents element-wise multiplication.
[0020] In some embodiments, the pruning rule includes a dynamic pruning formula, and the dynamic pruning formula is:
[0021]
[0022] wherein, RELU is an activation function, and TopK is the top K maximum values, wherein the value of K is is a preset pruning ratio, and W l is a projection weight corresponding to the lth pyramid sub-module.
[0023] In some embodiments, the feature compression processing of the target sparse feature by the sparse tensor core module includes:
[0024] The feature compression processing of the target sparse feature based on a preset sparse tensor compression formula, and the sparse tensor compression formula is:
[0025] F l+1 = STC(H l ) = U l · (S l *H l )·V l ;
[0026] wherein, F l+1 is the target compression feature, U l and V l are low-rank projection matrices, * represents sparse tensor contraction operation, S l is a block-diagonal sparse matrix, and STC is the sparse tensor core module.
[0027] In some embodiments, the feature fusion processing of the target sparse feature corresponding to each layer of the pyramid sub-module by the sparse feature fusion module includes:
[0028] The feature fusion processing of the target sparse feature corresponding to each layer of the pyramid sub-module based on a preset sparse feature fusion formula, and the sparse feature fusion formula is:
[0029]
[0030] wherein, Y is the target fusion feature, L is the number of layers of the pyramid sub-module, and α l is a fusion weight corresponding to the lth pyramid sub-module, and pad is a feature concatenation function.
[0031] In some embodiments, the video action recognition model further comprises an input module, and before the time sequence feature sequence of the to-be-identified video is acquired by the feature extraction module, the method further comprises:
[0032] acquiring the to-be-identified video by the input module;
[0033] frame sampling is performed on the to-be-identified video to obtain a plurality of video frames;
[0034] normalization processing is performed on the plurality of video frames to obtain a video frame sequence of the to-be-identified video;
[0035] the time sequence feature sequence of the to-be-identified video is acquired by the feature extraction module, comprising:
[0036] frame-level feature extraction processing is performed on the video frame sequence by the feature extraction module to obtain the time sequence feature sequence.
[0037] In a second aspect, the embodiments of the present application further provide a video action recognition device, which is deployed in a video action recognition system, wherein a video action recognition model is deployed in the video action recognition system, the video action recognition model comprises a feature extraction module, a sparse time sequence pyramid module and an output module, and the video action recognition device comprises a transceiver unit and a processing unit, wherein:
[0038] the transceiver unit is configured to acquire a to-be-identified video;
[0039] the processing unit is configured to acquire a time sequence feature sequence of the to-be-identified video by the feature extraction module; determine a sparse attention of the time sequence feature sequence by a sparse attention mechanism in the sparse time sequence pyramid module; perform feature sparse processing on the time sequence feature sequence according to the sparse attention and a pruning rule in the sparse time sequence pyramid module to obtain a target sparse feature; and perform action recognition processing on the target sparse feature by the output module to output an action recognition result of the to-be-identified video.
[0040] In a third aspect, the embodiments of the present application further provide a video action recognition system, which comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the above method when executing the computer program.
[0041] In a fourth aspect, the embodiments of the present application further provide a computer readable storage medium, wherein the storage medium stores a computer program, the computer program comprises program instructions, and the program instructions can implement the above method when executed by a processor.
[0042] Embodiments of the present application provide a video action recognition method, device, system and storage medium. The method is applied to a video action recognition system, the video action recognition system is deployed with a video action recognition model, the video action recognition model comprises a feature extraction module, a sparse time sequence pyramid module and an output module, and the method comprises: acquiring a time sequence feature sequence of a to-be-recognized video through the feature extraction module; determining sparse attention of the time sequence feature sequence through a sparse attention mechanism in the sparse time sequence pyramid module; performing feature sparse processing on the time sequence feature sequence according to the sparse attention and a pruning rule in the sparse time sequence pyramid module, to obtain a target sparse feature; and performing action recognition processing on the target sparse feature through the output module, and outputting an action recognition result of the to-be-recognized video. The video action recognition model in the embodiments of the present application will perform feature sparse processing on features in combination with a sparse attention mechanism and a pruning rule before performing action recognition processing, so as to reduce the feature calculation amount of the model and improve the action recognition efficiency of the model. BRIEF DESCRIPTION OF DRAWINGS
[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.
[0044] Figure 1 A structural schematic diagram of a video action recognition model provided by the embodiments of the present application is shown in the figure.
[0045] Figure 2 Another structural schematic diagram of a video action recognition model provided by the embodiments of the present application is shown in the figure.
[0046] Figure 3 A flowchart of a video action recognition method provided by the embodiments of the present application is shown in the figure.
[0047] Figure 4 A detailed structural block diagram of a dynamic sparse time sequence aggregation network in a video action recognition model provided by the embodiments of the present application is shown in the figure.
[0048] Figure 5 A schematic block diagram of a video action recognition device provided by the embodiments of the present application is shown in the figure.
[0049] Figure 6 A schematic block diagram of a video action recognition system provided by the embodiments of the present application is shown in the figure. DETAILED DESCRIPTION
[0050] With reference to the drawings and the embodiments of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all the other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.
[0051] It should be understood that, when used in the specification and the appended claims, the terms "comprise" and "include" indicate the presence of the described features, integers, steps, operations, elements, and / or components, but do not exclude one or more other features, integers, steps, operations, elements, components, and / or groups thereof.
[0052] It should also be understood that the terms used in the specification of the present application are only for the purpose of describing the specific embodiments and are not intended to limit the present application. As used in the specification and the appended claims of the present application, the singular forms "a", "an" and "the" are intended to include the plural forms unless the context clearly indicates otherwise.
[0053] It should be further understood that the term "and / or" used in the specification and the appended claims of the present application refers to any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.
[0054] The embodiments of the present application provide a video action recognition method, device, system and storage medium.
[0055] The execution subject of the video action recognition method can be a video action recognition device provided by the embodiments of the present application, or a video action recognition system integrated with the video action recognition device, wherein the video action recognition device can be realized in the form of hardware or software, and the video action recognition system can be a terminal or a server, and the terminal can be a smart phone, a tablet computer, a palm computer, or a notebook computer, etc.
[0056] The video action recognition method provided in the application is applied to a video action recognition system, which can be a video monitoring system in the financial field or the medical health field, or an action recognition system of a financial application background in the financial field, for example, when a user performs a business operation such as account registration or large payment through a financial application, a verification video with certain action requirements needs to be uploaded, at this time, the action recognition system of the financial application background identifies the specific action of the verification video uploaded by the user, or an action recognition system of a medical health application background in the medical health field, the user can upload a health detection video containing a specific action through the medical health application, at this time, the action recognition system of the medical health application background identifies the specific action of the health detection video uploaded by the user.
[0057] The video action recognition system provided in the embodiment is deployed with a video action recognition model, please refer to Figure 1 , the video action recognition model includes an input module, a dynamic sparse time sequence aggregation network and an output module, wherein the dynamic sparse time sequence aggregation network includes a feature extraction module (for example, a video Swin Transformer (VST), a Sparse Temporal Pyramid (STP) module.
[0058] In some other embodiments, please refer to Figure 2 , the dynamic sparse time sequence aggregation network in the video action recognition model provided in the embodiment further includes a Sparse Tensor Core (STC) module and a Sparse Feature Fusion (SFF) module.
[0059] Figure 3 is a flowchart of the video action recognition method provided in the embodiment of the application. As Figure 3 shown, the method includes the following steps S110-S140.
[0060] S110, obtaining a time sequence feature sequence of a to-be-identified video through the feature extraction module.
[0061] In some embodiments, the video action recognition system can directly obtain the time sequence feature sequence of the to-be-identified video.
[0062] In some other embodiments, the video action recognition system can obtain an original to-be-identified video and extract a time sequence feature sequence from the original to-be-identified video, at this time, before step S110, the method further includes:
[0063] The video to be identified is obtained through the input module; frame sampling is performed on the video to be identified to obtain multiple video frames; the multiple video frames are normalized to obtain a video frame sequence of the video to be identified; at this time, the temporal feature sequence of the video to be identified is obtained through the feature extraction module, including: frame-level feature extraction processing is performed on the video frame sequence through the feature extraction module to obtain the temporal feature sequence.
[0064] In this embodiment, the video to be identified may be a surveillance video acquired in real time by a video acquisition device, or a video to be identified uploaded by a user.
[0065] For example, in the financial sector, to ensure financial security, surveillance equipment (video capture devices) is installed in key areas such as bank branches, ATM rooms, vaults, and data centers. These devices collect real-time video to be recognized. For another example, when a user registers an account through a financial application, they are required to upload a video of their head shaking. The uploaded video is then sent to the backend video motion recognition system for motion recognition, completing the user's real-name verification.
[0066] For example, in the healthcare field, while a doctor is performing surgery, a surveillance device can capture the doctor's surgical video as the video to be identified. The video action recognition system can automatically identify the doctor's operating steps based on the movements in the video to be identified. For another example, a patient can shoot a video following specific action requirements and upload it through a healthcare application. At this time, the video uploaded by the patient through the healthcare application will be sent to the back-end video action recognition system for action recognition. The video action recognition system can then use the action recognition results to conduct rehabilitation assessments and rehabilitation action corrections.
[0067] Among them, the time series feature sequence extracted in this embodiment is F∈R T×D , where R represents a set of real numbers, T represents the time length, and D represents the feature dimension.
[0068] S120. Determine the sparse attention of the temporal feature sequence through the sparse attention mechanism in the sparse temporal pyramid module.
[0069] In this embodiment, Figure 4 As shown in the figure, the sparse time series pyramid module includes a sparse attention mechanism and a differentiable N:M pruning rule.
[0070] Specifically, the sparse attention mechanism in this embodiment determines the sparse attention by calculating the sparse attention weights between frames (between two frames) in the temporal feature sequence.
[0071] S130, performing feature sparsification on the time sequence feature sequence according to the sparse attention and a pruning rule in the sparse time sequence pyramid module, to obtain target sparse features.
[0072] Specifically, in some embodiments, step S130 comprises:
[0073] performing feature sparsification on the time sequence feature sequence according to a preset feature sparsification formula, to obtain target sparse features, the feature sparsification formula being:
[0074]
[0075] wherein H l is the target sparse feature corresponding to the lth layer pyramid submodule, SparseAttn l (F l ) is the sparse attention corresponding to the lth layer pyramid submodule, M l is a binary mask corresponding to the lth layer pyramid submodule generated dynamically based on the pruning rule, and M l ∈{0, 1} T×T , T is the length of time, F l is the time sequence feature sequence input into the lth layer pyramid submodule, denotes element-wise multiplication.
[0076] wherein the pruning rule comprises a dynamic pruning formula, and the dynamic pruning formula is:
[0077]
[0078] wherein RELU is an activation function, and TopK is the top K maximum values, wherein the value of K is is a preset pruning ratio, W l is a learnable projection weight corresponding to the lth layer pyramid submodule.
[0079] It should be noted that, in some embodiments, if the sparse time sequence pyramid module only comprises one layer of pyramid submodule, such as only the lth layer pyramid submodule, then the lth layer pyramid submodule (l is 1 at this time), at this time, the output module only needs to perform action recognition processing according to the output H l .
[0080] In some embodiments, the sparse temporal pyramid module comprises a plurality of pyramid sub-modules, and the video action recognition model further comprises a sparse tensor core module and a sparse feature fusion module, the sparse tensor core module is configured to perform cross-level feature compression, and the sparse feature fusion module is configured to perform sparse feature fusion.
[0081] The target compressed feature is obtained by performing feature compression processing on the target sparse feature through the sparse tensor core module, and the target compressed feature is taken as input of the time sequence feature sequence of the sparse temporal pyramid module, and the step of determining the sparse attention of the time sequence feature sequence through the sparse attention mechanism in the sparse temporal pyramid module is executed until the target sparse feature corresponding to each layer of the pyramid sub-module is obtained.
[0082] In this embodiment, the target sparse feature of each layer of the pyramid sub-module is achieved through the above feature sparsification formula and the dynamic pruning formula.
[0083] Specifically, in some embodiments, the target compressed feature is obtained by performing feature compression processing on the target sparse feature through the sparse tensor core module, comprising:
[0084] The target compressed feature is obtained by performing feature compression processing on the target sparse feature based on a preset sparse tensor compression formula, and the sparse tensor compression formula is:
[0085] F l+1 =STC(H l )=U l ·(S l *H l )·V l ;
[0086] Wherein, F l+1 is the target compressed feature, U l and V l are low-rank projection matrices, * represents sparse tensor contraction operation, S l is a block diagonal sparse matrix, and STC is the sparse tensor core module, wherein the low-rank projection matrix is used to reduce the feature dimension, and the block diagonal sparse matrix is used to define the connection mode.
[0087] Through the sparse tensor core module of this embodiment, the memory occupation of data can be reduced.
[0088] For example, the sparse temporal pyramid module includes three pyramid sub-modules, i.e., a first pyramid sub-module, a second pyramid sub-module, and a third pyramid sub-module, and the first pyramid sub-module outputs a first target sparse feature; the first target sparse feature is input into the sparse tensor core module for feature compression processing to obtain a first target compressed feature, the first target compressed feature is taken as a time sequence of features input into the second pyramid sub-module, and a second target sparse feature is output by the second pyramid sub-module; the second target sparse feature is input into the sparse tensor core module for feature compression processing to obtain a second target compressed feature, the second target compressed feature is taken as a time sequence of features input into the third pyramid sub-module, and a third target sparse feature is output by the third pyramid sub-module. In this way, the target sparse features of the pyramid sub-modules are obtained.
[0089] In S140, the output module is used to perform action recognition processing on the target sparse feature, and an action recognition result of the video to be recognized is output.
[0090] In the embodiment, when the sparse temporal pyramid module includes multiple pyramid sub-modules, the sparse feature fusion module is used to perform feature fusion processing on the target sparse features respectively corresponding to the pyramid sub-modules to obtain a target fusion feature; and the output module is used to perform action recognition processing on the target fusion feature, and an action recognition result of the video to be recognized is output.
[0091] In some embodiments, the feature fusion processing performed by the sparse feature fusion module on the target sparse features respectively corresponding to the pyramid sub-modules includes:
[0092] The feature fusion processing performed by the sparse feature fusion module on the target sparse features respectively corresponding to the pyramid sub-modules is based on a preset sparse feature fusion formula, and the sparse feature fusion formula is:
[0093]
[0094] wherein Y is the target fusion feature, L is the number of layers of the pyramid sub-modules, α l is a learnable fusion weight corresponding to the lth pyramid sub-module, and pad is a feature concatenation function.
[0095] Further, in the embodiment, the output module is trained end to end by using a composite loss function.
[0096] The composite loss function is:
[0097]
[0098] wherein Losscls For classification loss, cross-entropy is used to calculate the loss between Y and the artificially labeled actions during training, and λ controls the sparsity regularization strength.
[0099] It can be seen that the sparse temporal pyramid module in this embodiment realizes multi-level temporal feature processing through a learnable dynamic mask, and through the learnable dynamic mask M l The multi-level temporal feature processing is realized, and only the key motion features are retained to reduce redundancy; moreover, the system adopts an end-to-end optimized N:M sparse connection mode, and realizes efficient feature compression across levels in combination with a sparse tensor core module, thereby reducing memory occupation. Finally, the multi-level output is integrated through a sparse feature fusion module, and the existing feature extraction module and temporal comparison classification head are compatible, thereby realizing a fold acceleration in the fine-grained action recognition task and an accuracy higher than that of a traditional 3D convolutional network. This method does not require artificial design of a temporal receptive field, and can automatically learn a dynamic sparse connection mode related to the input.
[0100] The video action recognition method provided in this embodiment is described below with a specific example:
[0101] A video to be recognized uploaded by a video monitoring system in a financial field or a medical and health field is acquired, the video to be recognized is a 10-second 30fps video (a total of 300 frames), and a temporal feature sequence F ∈ R 300×512 is obtained through feature extraction; then, according to the first layer pyramid sub-module, a sparse attention SparseAttn1(F) of F ∈ R 300×512 is calculated, a mask M1 is generated, 30% of the connections are retained, and an output H1 is obtained. Then, feature compression is performed through the sparse tensor core module: H1 is compressed from 300x512 to 150x256; the compressed feature is input into the second layer pyramid sub-module as a temporal feature sequence, and the above operation is repeated until the processing of each layer pyramid sub-module is completed, and finally Y is obtained through fusion. Y is input into a classifier in the output layer, and an action recognition result of the video to be recognized is output.
[0102] For example, the video to be recognized is a monitoring video between ATMs in the financial field, at this time, the video action recognition system discovers actions such as kicking and punching by analyzing the monitoring video, and the output action recognition result is that there is kicking and punching behavior, at this time, the video action recognition system can also automatically trigger an alarm according to the action recognition result.
[0103] For example, the video to be recognized is a monitoring video of a doctor's operation in the medical and health field, at this time, the video action recognition system automatically analyzes the operation steps in the monitoring video, such as incision, suture, hemostasis, and use of specific instruments, at this time, the video action recognition system can also automatically generate a structured operation report according to the action recognition result.
[0104] To sum up, the method provided in the embodiment is applied to a video action recognition system, the video action recognition system is deployed with a video action recognition model, the video action recognition model comprises a feature extraction module, a sparse temporal pyramid module and an output module, and the method comprises: acquiring a time sequence of features of a to-be-identified video through the feature extraction module; determining sparse attention of the time sequence of features through a sparse attention mechanism in the sparse temporal pyramid module; performing feature sparsification processing on the time sequence of features according to the sparse attention and a pruning rule in the sparse temporal pyramid module, to obtain a target sparse feature; and performing action recognition processing on the target sparse feature through the output module, and outputting an action recognition result of the to-be-identified video. The video action recognition model in the embodiment performs feature sparsification processing on features in combination with a sparse attention mechanism and a pruning rule before performing action recognition processing, so as to reduce the feature calculation amount of the model and improve the action recognition efficiency of the model.
[0105] In addition, the sparse temporal pyramid module provided in the embodiment is a multi-level time sequence feature processing structure, feature selection is realized through a learnable dynamic mask, and redundant calculation can be reduced while key motion patterns are reserved. In addition, the sparse temporal pyramid module provided in the embodiment comprises an N:M structured pruning, an optimal sparse connection mode is automatically learned in a training process, and only M most relevant connections are reserved in every N connections. Finally, the sparse tensor core module provided in the embodiment can efficiently realize a cross-level feature communication module, and greatly reduces memory occupation and calculation overhead.
[0106] Figure 5 is a schematic block diagram of a video action recognition device provided in the embodiment of the present application. As shown in Figure 5 Corresponding to the above video action recognition method, the present application also provides a video action recognition device 500. The video action recognition device 500 comprises units for executing the above video action recognition method. Specifically, the video action recognition device 500 is deployed in a video action recognition system, the video action recognition system is deployed with a video action recognition model, the video action recognition model comprises a feature extraction module, a sparse temporal pyramid module and an output module, please refer to Figure 5 The video action recognition device 500 comprises a transceiver unit 501 and a processing unit 502, wherein:
[0107] The transceiver unit 501 is configured to acquire a to-be-identified video.
[0108] The processing unit 502 is configured to acquire a time sequence of features of a video to be recognized by the feature extraction module; determine sparse attention of the time sequence of features by a sparse attention mechanism in the sparse time sequence pyramid module; perform feature sparsification processing on the time sequence of features according to the sparse attention and a pruning rule in the sparse time sequence pyramid module to obtain target sparse features; and perform action recognition processing on the target sparse features by the output module, and output an action recognition result of the video to be recognized.
[0109] In some embodiments, the sparse time sequence pyramid module comprises a plurality of pyramid sub-modules, and the video action recognition model further comprises a sparse tensor core module and a sparse feature fusion module. After performing the feature sparsification processing on the time sequence of features according to the sparse attention and the pruning rule in the sparse time sequence pyramid module to obtain the target sparse features, the processing unit 502 is further configured to:
[0110] perform feature compression processing on the target sparse features by the sparse tensor core module to obtain target compressed features; input the target compressed features as the time sequence of features of the sparse time sequence pyramid module, and return to perform the step of determining the sparse attention of the time sequence of features by the sparse attention mechanism in the sparse time sequence pyramid module until target sparse features corresponding to each layer of the pyramid sub-modules are obtained;
[0111] At this time, when performing the step of performing the action recognition processing on the target sparse features by the output module and outputting the action recognition result of the video to be recognized, the processing unit 502 is specifically configured to:
[0112] perform feature fusion processing on the target sparse features corresponding to each layer of the pyramid sub-modules according to the sparse feature fusion module to obtain target fusion features; and perform the action recognition processing on the target fusion features by the output module, and output the action recognition result of the video to be recognized.
[0113] In some embodiments, when performing the step of performing the feature sparsification processing on the time sequence of features according to the sparse attention and the pruning rule in the sparse time sequence pyramid module to obtain the target sparse features, the processing unit 502 is specifically configured to:
[0114] perform the feature sparsification processing on the time sequence of features based on a preset feature sparsification formula to obtain the target sparse features, and the feature sparsification formula is:
[0115]
[0116] wherein, H lSparseAttn l (F l ) is the sparse attention corresponding to the l-th pyramid sub-module, M l is the binary mask dynamically generated based on the pruning rule corresponding to the l-th pyramid sub-module, and M l ∈{0,1} T×T , T is the length of time, F l is the time sequence feature sequence input into the l-th pyramid sub-module, represents element-wise multiplication.
[0117] In some embodiments, the pruning rule includes a dynamic pruning formula, and the dynamic pruning formula is:
[0118]
[0119] wherein, RELU is an activation function, and TopK is the top K maximum values, wherein the value of K is is a preset pruning ratio, W l is the projection weight corresponding to the l-th pyramid sub-module.
[0120] In some embodiments, when the processing unit 502 performs the step of performing feature compression processing on the target sparse feature through the sparse tensor core module to obtain a target compressed feature, the processing unit 502 is specifically configured to:
[0121] performing feature compression processing on the target sparse feature based on a preset sparse tensor compression formula to obtain the target compressed feature, wherein the sparse tensor compression formula is:
[0122] F l+1 =STC(H l )=U l ·(S l *H l )·V l ;
[0123] wherein, F l+1 is the target compressed feature, U l and V l are low-rank projection matrices, * represents sparse tensor contraction operation, S l is a block-diagonal sparse matrix, and STC is the sparse tensor core module.
[0124] In some embodiments, when the processing unit 502 performs the step of performing feature fusion processing on the target sparse feature corresponding to each layer of the pyramid sub-module according to the sparse feature fusion module to obtain a target fusion feature, the processing unit 502 is specifically configured to:
[0125] Based on the preset sparse feature fusion formula, the target sparse features corresponding to the pyramid sub-modules of each layer are fused to obtain the target fusion features. The sparse feature fusion formula is:
[0126]
[0127] Among them, Y is the target fusion feature, L is the number of layers of the pyramid submodule, α l is the fusion weight corresponding to the l-th layer pyramid sub-module, and pad is the feature splicing function.
[0128] In some embodiments, the video action recognition model further includes an input module. Before executing the step of obtaining the temporal feature sequence of the video to be recognized through the feature extraction module, the transceiver unit 501 is further configured to:
[0129] Acquire the video to be identified through the input module; perform frame sampling on the video to be identified through the processing unit 502 to obtain a plurality of video frames; perform normalization processing on the plurality of video frames through the processing unit 502 to obtain a video frame sequence of the video to be identified;
[0130] At this time, when the transceiver unit 501 executes the step of obtaining the temporal feature sequence of the video to be identified through the feature extraction module, it is specifically used to:
[0131] The processing unit 502 performs frame-level feature extraction processing on the video frame sequence through the feature extraction module to obtain the temporal feature sequence.
[0132] To sum up, the video action recognition model in the embodiment of the present application will combine the sparse attention mechanism and pruning rules to perform feature sparse processing on the features before performing action recognition processing, so as to reduce the feature calculation amount of the model and improve the action recognition efficiency of the model.
[0133] In addition, the sparse temporal pyramid module provided by this embodiment is a multi-level temporal feature processing structure that implements feature selection through learnable dynamic masks, which can reduce redundant calculations while retaining key motion patterns. In addition, the sparse temporal pyramid module provided by this embodiment includes N:M structured pruning, which automatically learns the optimal sparse connection pattern during training, ensuring that only the M most relevant connections are retained for every N connections. Finally, the sparse tensor core module provided by this embodiment can efficiently implement a module for cross-level feature communication, significantly reducing memory usage and computational overhead.
[0134] It should be noted that the specific implementation process of the video action recognition device 500 and each unit can be clearly understood by those skilled in the art, and can refer to the corresponding description in the foregoing method embodiments. For the convenience and brevity of description, it will not be repeated here.
[0135] The video action recognition device described above can be implemented in the form of a computer program, which can run on a video action recognition system as shown in the figure. Figure 6
[0136] Please refer to Figure 6 , Figure 6 is a schematic block diagram of a video action recognition system provided by an embodiment of the present application. The video action recognition system 600 can be a terminal or a server. The terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a personal digital assistant, a wearable device, or other electronic devices with communication functions. The server can be a stand-alone server or a server cluster composed of multiple servers. The video action recognition system is deployed with a video action recognition model, which includes a feature extraction module, a sparse temporal pyramid module, and an output module.
[0137] Referring to Figure 6 , the video action recognition system 600 includes a processor 602, a memory, and a network interface 605 connected through a system bus 601. The memory can include a non-volatile storage medium 603 and an internal memory 604.
[0138] The non-volatile storage medium 603 can store an operating system 6031 and a computer program 6032. The computer program 6032 includes program instructions that, when executed, can cause the processor 602 to perform a video action recognition method.
[0139] The processor 602 is configured to provide computing and control capabilities to support the operation of the entire video action recognition system 600.
[0140] The internal memory 604 provides an environment for the running of the computer program 6032 in the non-volatile storage medium 603. When the computer program 6032 is executed by the processor 602, the processor 602 can perform a video action recognition method.
[0141] The network interface 605 is configured to perform network communication with other devices. Those skilled in the art can understand that Figure 6 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the video action recognition system 600 to which the scheme of the present application is applied. The specific video action recognition system 600 can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0142] The processor 602 is configured to run the computer program 6032 stored in the memory to implement the following steps:
[0143] The time sequence feature sequence of the video to be recognized is obtained by the feature extraction module;
[0144] The sparse attention of the time sequence feature sequence is determined by the sparse attention mechanism in the sparse time sequence pyramid module.
[0145] The target sparse feature is obtained by performing feature sparse processing on the time sequence feature sequence according to the sparse attention and the pruning rule in the sparse time sequence pyramid module.
[0146] The action recognition result of the video to be recognized is output by performing action recognition processing on the target sparse feature by the output module.
[0147] It should be understood that, in the embodiments of the present application, the processor 602 can be a central processing unit (CPU), and the processor 602 can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.
[0148] It can be understood by those skilled in the art that all or part of the processes in the method of the above-mentioned embodiments can be completed by instructing related hardware by a computer program. The computer program includes program instructions, and the computer program can be stored in a storage medium, which is a computer readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the above-mentioned embodiments of the method.
[0149] Therefore, the application further provides a storage medium. The storage medium can be a computer readable storage medium. The storage medium stores a computer program, where the computer program includes program instructions. The program instructions are executed by a processor to enable the processor to perform the following steps:
[0150] The time sequence feature sequence of the to-be-identified video is acquired through the feature extraction module;
[0151] The sparse attention of the time sequence feature sequence is determined through the sparse attention mechanism in the sparse time sequence pyramid module;
[0152] The time sequence feature sequence is subjected to feature sparse processing according to the sparse attention and a pruning rule in the sparse time sequence pyramid module, to obtain a target sparse feature;
[0153] The target sparse feature is subjected to action recognition processing through the output module, to output an action recognition result of the to-be-identified video.
[0154] The storage medium can be a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk, and various computer readable storage media that can store program codes.
[0155] Those skilled in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been described in general terms in the above description. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the application.
[0156] In several embodiments provided in the application, it should be understood that the disclosed apparatus and method can be implemented in other ways. For example, the apparatus embodiments described above are merely schematic. For example, the division of each unit is only a logical function division, and actual implementation can have another division manner. For example, a plurality of units or components can be combined or integrated into another system, or some features can be omitted or not executed.
[0157] The steps in the method embodiments of the present application can be adjusted in sequence, combined and reduced according to actual needs. The units in the device embodiments of the present application can be combined, divided and reduced according to actual needs. In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit.
[0158] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a storage medium. Based on such understanding, the technical solutions of the present application essentially or say the part of the prior art that contributes to the technical solutions, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application.
[0159] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.
Claims
1. A video action recognition method, characterized in that: The method is applied to a video action recognition system, wherein a video action recognition model is deployed in the video action recognition system, and the video action recognition model includes a feature extraction module, a sparse temporal pyramid module, and an output module. The method includes: Acquire a temporal feature sequence of the video to be identified through the feature extraction module; Determining the sparse attention of the temporal feature sequence through the sparse attention mechanism in the sparse temporal pyramid module; Perform feature sparse processing on the temporal feature sequence according to the sparse attention and the pruning rules in the sparse temporal pyramid module to obtain target sparse features; The output module performs action recognition processing on the target sparse features and outputs the action recognition result of the video to be recognized.
2. The method according to claim 1, characterized in that The sparse temporal pyramid module includes a multi-layer pyramid submodule, and the video action recognition model also includes a sparse tensor core module and a sparse feature fusion module. After performing feature sparse processing on the temporal feature sequence according to the sparse attention and pruning rules in the sparse temporal pyramid module to obtain target sparse features, the method further includes: Performing feature compression processing on the target sparse features through the sparse tensor core module to obtain target compressed features; Using the target compressed features as a temporal feature sequence input into the sparse temporal pyramid module, returning to the step of determining the sparse attention of the temporal feature sequence through the sparse attention mechanism in the sparse temporal pyramid module until the target sparse features corresponding to the pyramid sub-modules at each layer are obtained; The step of performing action recognition processing on the target sparse features by the output module and outputting an action recognition result of the video to be recognized includes: Performing feature fusion processing on the target sparse features corresponding to the pyramid sub-modules of each layer according to the sparse feature fusion module to obtain target fusion features; The output module performs action recognition processing on the target fusion features and outputs the action recognition result of the video to be recognized.
3. The method according to claim 2, characterized in that The step of performing feature sparse processing on the temporal feature sequence according to the sparse attention and the pruning rules in the sparse temporal pyramid module to obtain target sparse features includes: The time series feature sequence is subjected to feature sparse processing based on a preset feature sparse processing formula to obtain target sparse features. The feature sparse processing formula is: Among them, H l is the target sparse feature corresponding to the l-th layer pyramid submodule, SparseAttn l (F l ) is the sparse attention corresponding to the l-th layer pyramid submodule, M l is the binary mask corresponding to the l-th layer pyramid submodule dynamically generated based on the pruning rule, and M l ∈{0,1} T×T , T is the time length, F l is the temporal feature sequence input to the l-th layer pyramid submodule, Represents element-wise multiplication.
4. The method according to claim 3, characterized in that The pruning rule includes a dynamic pruning formula, which is: Among them, RELU is the activation function, TopK is the top K maximum value, and the value of K is is the preset pruning ratio, W l is the projection weight corresponding to the l-th layer pyramid sub-module.
5. The method according to claim 2, characterized in that The step of performing feature compression processing on the target sparse features by the sparse tensor core module to obtain target compressed features includes: The target sparse feature is compressed based on a preset sparse tensor compression formula to obtain the target compressed feature. The sparse tensor compression formula is: F l+1 =STC(H l )=U l ·(S l *H l )·V l ; Among them, F l+1 is the target compression feature, U l and V l is the low-rank projection matrix, * represents the sparse tensor contraction operation, S l is a block diagonal sparse matrix, and STC is the Sparse Tensor Core module.
6. The method according to claim 2, characterized in that The step of performing feature fusion processing on the target sparse features corresponding to the pyramid sub-modules at each layer according to the sparse feature fusion module to obtain target fusion features includes: Based on the preset sparse feature fusion formula, the target sparse features corresponding to the pyramid sub-modules of each layer are fused to obtain the target fusion features. The sparse feature fusion formula is: Among them, Y is the target fusion feature, L is the number of layers of the pyramid submodule, α l is the fusion weight corresponding to the l-th layer pyramid sub-module, and pad is the feature splicing function.
7. The method according to any one of claims 1 to 6, characterized in that The video action recognition model further includes an input module. Before obtaining the temporal feature sequence of the video to be recognized through the feature extraction module, the method further includes: Acquire the video to be identified through the input module; Performing frame sampling on the video to be identified to obtain multiple video frames; Normalizing the plurality of video frames to obtain a video frame sequence of the video to be identified; The step of obtaining a temporal feature sequence of a video to be identified by the feature extraction module includes: The feature extraction module performs frame-level feature extraction processing on the video frame sequence to obtain the temporal feature sequence.
8. A video action recognition device, characterized in that: The video action recognition device is deployed in a video action recognition system. A video action recognition model is deployed in the video action recognition system. The video action recognition model includes a feature extraction module, a sparse temporal pyramid module, and an output module. The video action recognition device includes a transceiver unit and a processing unit, wherein: The transceiver unit is used to obtain the video to be identified; The processing unit is used to obtain a temporal feature sequence of a video to be identified through the feature extraction module; determine the sparse attention of the temporal feature sequence through the sparse attention mechanism in the sparse temporal pyramid module; perform feature sparse processing on the temporal feature sequence according to the sparse attention and the pruning rules in the sparse temporal pyramid module to obtain target sparse features; perform action recognition processing on the target sparse features through the output module, and output the action recognition result of the video to be identified.
9. A video action recognition system comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: A video action recognition model is deployed in the video action recognition system, and the video action recognition model includes a feature extraction module, a sparse temporal pyramid module, and an output module. When the processor executes the computer program, the video action recognition method according to any one of claims 1 to 7 is implemented.
10. A storage medium, characterized in that: The storage medium stores a computer program, which includes program instructions. When the program instructions are executed by a processor, the processor executes the video action recognition method according to any one of claims 1 to 7.