Slow-fast diversity perception prototype learning method for egocentric action recognition

Abstract

This invention discloses a slow-fast diversity-aware prototype learning method for egocentric action recognition. This method utilizes a part-to-prototype (P2P) module and a slow-fast contextualization (SFC) mechanism. In P2P, partial tokens obtained from each frame by the video converter are further interacted with several initial prototypes through diversity-aware attention, refining / extracting the semantic information of the interacting objects from the partial level to the prototype level. In SFC, slow and fast awareness attention is introduced to interact with the original prototypes to finely supplement semantic information. Three streaming features are aggregated using slow and fast gates, ultimately yielding classification results for nouns and verbs.

Description

Technical Field

[0001] This invention relates to a first-person action recognition method in the field of computer vision, specifically a slow-fast diversity perception prototype learning method for egocentric action recognition, belonging to the field of computer vision technology. Background Technology

[0002] With the emergence of smart wearable devices, egocentric videos have become very popular on social media, and due to their wide range of applications, such as human-computer interaction, motion prediction, and virtual reality, they have attracted increasing attention and become another research hotspot in the field of action recognition. Egocentric action recognition (EAR) requires identifying interacting objects and movements against a cluttered background with distracting objects.

[0003] Compared to third-person videos shot from a distance and a fixed perspective, egocentric videos offer naturalistic insights (such as the human gaze) and perspective shifts caused by frequent body movements of the actors. This illustrates that, unlike conventional action recognition, egocentric action recognition (EAR) aims to identify the actor's behavioral perspective from a first-person viewpoint, including interacting objects (nouns) and motion (verbs). Therefore, most existing work breaks down EAR into two sub-tasks: identifying interacting objects and identifying motion. A straightforward solution for identifying interacting objects is to detect objects appearing in the video scene; however, this is labor-intensive. Furthermore, the inherent variation in motion duration across egocentric videos of varying lengths makes motion recognition in egocentric videos extremely challenging.

[0004] Effectively capturing interactive objects and adaptively capturing motion of varying durations are key factors in solving egocentric video action recognition. Previous research has attempted to use object annotators or detectors to capture interactive objects, or to add additional optical flow information to recognize motion. However, these studies have all come at a high cost and generated excessive redundant information. Summary of the Invention

[0005] Purpose of the invention: To address the problems and shortcomings of existing technologies, this invention provides a Slow Diversity-Aware Prototype Learning (SDP) method for first-person action recognition (EAR) to effectively capture interactive objects in cluttered backgrounds without a detector and adaptively capture motion in long or short videos.

[0006] Technical solution: A slow-fast diversity perception prototype learning method for egocentric action recognition, comprising the following steps:

[0007] Step 1: Obtain partial tokens from the input video that generate spatiotemporal features using Transformer, and initialize the prototype in each frame, while enhancing the prototype through a multi-head self-attention mechanism;

[0008] Step 2: Enhance the prototype and some tokens using diversity-aware attention to obtain a semantically compressed prototype and calculate the diversity loss;

[0009] Step 3: The obtained prototype sequence is subjected to up-enhancement and down-enhancement operations on fast flow and slow flow respectively to obtain the down-enhancement prototype and the down-enhancement prototype.

[0010] Step 4: The upper and lower augmented prototypes interact with the prototypes through slow and fast perceptual attention to obtain semantic supplementation;

[0011] Step 5: Use a slow-shutter strategy to measure the correlation between the upper augmented prototype, the lower augmented prototype, and the normal prototype to adaptively adjust the weights of the upper and lower augmented prototypes in slow and fast flows.

[0012] Step 6: Aggregate the upper augmented prototype, lower augmented prototype, normal prototype, and slow shutter to obtain the final action representation. After obtaining the action representation, calculate the noun recognition loss and verb recognition loss using the form of cross-entropy loss.

[0013] Step 7: Combine the noun loss, verb loss, and diversity loss according to their weights to optimize the network.

[0014] Step 1 includes the following specific steps:

[0015] The input video is processed by a video converter f(·) to generate spatiotemporal features Z∈R. T×HW×D Where T×HW represents the total number of tokens, D represents the dimension of each token, T is the number of frames, and H×W represents the image size of the input frame; for the t-th frame, a set of L learnable prototypes is initialized. To enhance the relationships between tokens, multi-head self-attention and layer normalization operations are employed to provide an enhanced prototype. Right now:

[0016] Q:=PW Q K:=PW K V:=PW V ,

[0017]

[0018] Where {Q, K, V} represent the query, key, and value, respectively. P is the initial learnable prototype. W Q W K WV , are the projection matrices of the query, key, and value, respectively. LN is layer normalization. MHSA is a multi-head attention mechanism.

[0019] Step 2 includes the following specific steps:

[0020] Obtain the enhanced prototype Then, using diversity-aware attention, we learn a compact but diverse prototype covering the interactive objects by extracting higher purity semantic information from some tokens.

[0021] Take a set of partial tokens And a set of enhanced prototypes As input to diversity-perceived attention, its output is a set of semantically compressed prototypes. Right now:

[0022]

[0023]

[0024]

[0025] Where {Q, K, V} represent the query, key, and value, respectively. P is the initial learnable prototype. The ~ symbol indicates enhancement, and the ^ symbol indicates semantic compression. W * is the projection matrix. LN is layer normalization. MHSA is the multi-head attention mechanism. This represents an excessive semantic compression prototype. LN stands for Layer Normalization. MHSA stands for Multi-head Attention Mechanism. MLP stands for Multilayer Perceptron.

[0026] To further expand the semantic compression prototype To address the diversity issue, a more focused diversity loss function L is designed. div ,Right now:

[0027]

[0028] Where sim(·) represents the cosine similarity function. Represents all negative sample prototypes in frame t, and l [l'≠l] ∈{0,1} is an indicator function set to 1 when l'≠l, and log is the logarithmic function with base e.

[0029] Step 3 includes the following specific steps:

[0030] The obtained prototype sequence Up-down enhancement is performed to capture semantic-level motion. This is achieved by upsampling the prototype along the time dimension using deconvolution operations, capturing motion in slow-moving streams. Timescale extended to slow flow Tslow In the temporal resolution, enrich the temporal context of short-duration actions and enhance the transient dynamic information in short-duration videos. In the downsampling enhancement, use convolutional operations to compress the temporal scale of the prototype sequence along the temporal dimension to the fast stream T fast in the temporal resolution, thereby eliminating redundant information in long-duration videos.

[0031] The motion enhancement prototype can be expressed through upsampling enhancement and downsampling enhancement as:

[0032]

[0033]

[0034] where T slow > T and T fast < T are the temporal resolutions of the slow stream and the fast stream respectively. DECONV and CONV represent upsampling and downsampling of three-dimensional convolution in the temporal dimension. BN represents batch normalization.

[0035] Step 4 includes the following specific steps:

[0036] The prototypes of up / downsampling enhancement interact with the original prototypes in the normal flow through two collaborative attentions, namely slow perception attention and fast perception attention to finely supplement semantic information into the slow / fast stream. In slow perception attention, set the semantically enhanced prototype obtained from the fast stream as the query, and set as the key / value for calculating the cross-attention map, and obtain the semantically supplemented prototype features in the slow stream That is:

[0037]

[0038]

[0039]

[0040] where {Q, K, V} are the query, key, and value respectively. The subscripts slow and s represent operations performed in the slow stream. W * represents the projection matrix. SlowAtt is the slow perception attention mechanism. LN is layer normalization. MHSA is the multi-head attention mechanism. AvgPool represents average pooling. represents the excessive semantically supplemented prototype.

[0041] In fast perception attention, set the semantically enhanced prototype obtained from the slow stream as the query, as the key and value, and obtain the semantically supplemented prototype features in the fast stream Right now:

[0042]

[0043]

[0044]

[0045] Where {Q, K, V} represent the query, key, and value, respectively. The subscripts fast and f indicate operations performed in a fast stream. W * This represents the projection matrix. FastAtt is the fast perceptual attention mechanism. LN is layer normalization. MHSA is the multi-head attention mechanism. AvgPool represents average pooling. The prototype for indicating excessive semantic supplementation.

[0046] Step 5 includes the following specific steps:

[0047] Using a slow-shutter strategy, by measuring and The correlation between them adaptively adjusts the slow and fast flows. and The weights, i.e.:

[0048]

[0049]

[0050]

[0051] Where σ(·) represents the sigmoid function, π S / f Indicates slow / fast shutter speed. U * and V * This represents the weight matrix. These represent the semantic supplementary prototypes obtained in fast, slow, and normal streams, respectively.

[0052] By and With π S and π f Aggregation yields the final action representation P. action ,Right now:

[0053]

[0054] Here, ⊙ represents the dot product of an element's direction.

[0055] Step 6 includes the following specific steps:

[0056] Obtain action representation P actionThen, the cross-entropy loss was used for the noun recognition loss. and verb recognition loss Right now:

[0057]

[0058]

[0059] in FC * Indicates a fully connected layer. Is with action (y N y V The truth label is a combination of ) and log; log is the logarithmic function with base e.

[0060] The obtained noun recognition loss And verb recognition loss Combining diversity loss L div The final model loss is obtained to optimize the network, i.e.:

[0061]

[0062] Here, the parameter λ is used to control the penalty for the diversity loss term.

[0063] Beneficial effects: The Slow-Fast Diversity-Aware Prototype Learning (SDP) framework proposed in this invention is used for egocentric action recognition. It can automatically capture interactive objects and motions of various durations in a detector-free manner, while better enhancing instantaneous dynamic information in short videos and eliminating redundant information in long videos, achieving the best performance to date. Attached Figure Description

[0064] Figure 1 This is a flowchart of an embodiment of the present invention;

[0065] Figure 2 This is a model framework diagram provided in the embodiments of the present invention. Detailed Implementation

[0066] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading the present invention, any modifications of the present invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0067] like Figure 1 , Figure 2 As shown, a slow-fast diversity perception prototype learning method for egocentric action recognition includes three processes: part-to-prototype learning, slow / fast context mechanism, and loss calculation.

[0068] Partial to prototype learning includes the following steps:

[0069] Step 1: Extract partial tokens from the input video to generate spatiotemporal features using a Transformer. The input video is converted to spatiotemporal features Z∈R using a video transformer f(·). T×HW×D Where T×HW represents the total number of tokens and D represents the dimension of each token.

[0070] Step 2: For frame t, a set of L-learnable prototypes is initialized. To strengthen their relationship, multi-head self-attention operations and layer normalization operations are employed to provide enhanced prototypes. Right now:

[0071] Q:=PW Q K:=PW K V:=PW V ,

[0072]

[0073] Where {Q, K, V} are the query, key, and value, respectively, and are the projection matrix in the attention operation.

[0074] Step 3: Use a set of partial tokens and a set of enhanced prototypes in step 2 As input to diversity-perceived attention, its output is a semantically compressed prototype. Right now:

[0075]

[0076]

[0077]

[0078] MLP stands for Multilayer Perceptron.

[0079] Step 4: To further expand the semantic compression model To address the diversity issue, a more focused diversity loss function L is designed. div ,Right now:

[0080]

[0081] Where sim(·) represents the cosine similarity function. Represents all negative sample prototypes in frame t, and l [l'≠l] ∈{0,1} is an indicator function set to 1 when l'≠l.

[0082] The slow / fast context mechanism includes the following steps:

[0083] Step 5: The prototype sequence obtained in Step 3 is enhanced up / down to capture the semantic-level motion. By using deconvolution operations to upsample the prototypes along the time dimension, the time scale in the slow stream is extended to T slow , so as to obtain richer temporal context information in short-term actions and enhance the transient dynamic information in short-term videos. The motion-enhanced prototypes can be expressed as:

[0084]

[0085] where T slow >T is the time resolution of the slow stream, DECONV is the upsampling and downsampling of the three-dimensional convolution in the time dimension. BN represents batch normalization.

[0086] Step 6: In the down enhancement, use the convolution operation to compress the time scale of the prototype sequence along the time dimension to T in the fast stream fast , eliminating the redundant information in long-term videos. After motion enhancement can be expressed as:

[0087]

[0088] where T fast <T is the time resolution of the fast stream, CONV is the upsampling and downsampling of the three-dimensional convolution in the time dimension. BN represents batch normalization.

[0089] Step 7: Set the obtained in Step 5 as the query in the slow perceptual attention, and set as the key / value for calculating the cross-attention map, and obtain the prototype features with semantic complement in the slow stream that is:

[0090]

[0091]

[0092]

[0093] where, AvgPool represents average pooling.

[0094] Step 8: Set the obtained in Step 6 as the query in the fast perceptual attention, Prototype features for keys and values, and for obtaining semantic complementation in fast streams. Right now:

[0095]

[0096]

[0097]

[0098] Step 9: Using the slow-shutter strategy, measure the values ​​obtained in Step 7 and Step 8. and as well as The correlation between them adaptively adjusts the slow and fast flows. and The weights, i.e.:

[0099]

[0100]

[0101]

[0102] Where σ(·) represents the sigmoid function, π S / f Indicates slow / fast shutter speed. U * and V * This represents the weight matrix.

[0103] Step 10: By and The π obtained in step 9 S and π f Aggregation yields the final action representation P. action ,Right now:

[0104]

[0105] Here, ⊙ represents the dot product of an element's direction.

[0106] Step 11: Use the action representation P obtained in Step 10. action Cross-entropy loss is used for noun recognition loss. And verb recognition loss Right now:

[0107]

[0108]

[0109] in FC * Indicates a fully connected layer. Is with action (yN y V Truth labels that combine )

[0110] Step 12: Calculate the noun recognition loss obtained in Step 11. And verb recognition loss Combined with the diversity loss L obtained in step 4 div The final model loss is obtained to optimize the network, i.e.:

[0111]

[0112] Here, the parameter λ is used to control the penalty for the diversity loss term.

[0113] Step 13: Using the above model, output the classification and recognition results of nouns, verbs, and actions.

[0114] Table 1 compares our method with state-of-the-art methods on EPIC-KITCHENS-100. “A”, “V”, and “N” represent the recognition of actions, verbs, and nouns, respectively. “Obj.” indicates the use of object annotations. This indicates the use of optical flow. "Average" refers to the average accuracy of action, verb, and noun recognition.

[0115] Table 1

[0116]

[0117] Table 2 compares our method with state-of-the-art methods on EGTEA. "A" indicates action recognition. "De." indicates the use of an object detector. "Gz." indicates the use of eye gaze data provided by this dataset. This indicates the use of optical flow.

[0118] Table 2

[0119]

[0120] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A slow-fast diversity perception prototype learning method for egocentric action recognition, characterized in that, Includes the following steps: Step 1: Obtain partial tokens from the input video that generate spatiotemporal features using Transformer, and initialize the prototype in each frame, while enhancing the prototype through a multi-head self-attention mechanism; Step 2: The enhanced prototype and some tokens are used with diversity-aware attention to obtain a semantically compressed prototype and the diversity loss is calculated; Step 3: The obtained prototype sequence is subjected to up-enhancement and down-enhancement operations on fast flow and slow flow respectively to obtain the up-enhancement prototype and the down-enhancement prototype; Step 4: The upper and lower augmented prototypes interact with the prototypes through slow and fast perceptual attention to obtain semantic supplementation; Step 5: Use a slow-shutter strategy to measure the correlation between the upper augmented prototype, the lower augmented prototype, and the normal prototype to adaptively adjust the weights of the upper and lower augmented prototypes in slow and fast flows. Step 6: Aggregate the upper enhanced prototype, lower enhanced prototype, normal prototype, and slow shutter speed to obtain the final motion representation; After obtaining the action representation, the noun recognition loss and verb recognition loss are calculated using the form of cross-entropy loss; Step 7: Add the noun recognition loss, verb recognition loss, and diversity loss according to their weights to optimize the network; Step 8: Use the optimized network to classify and identify verbs, nouns, and actions; In step 6, the action representation is obtained. Then, the cross-entropy loss was used for the noun recognition loss. And verb recognition loss ,Right now: in , FC * Indicates a fully connected layer. / Is with action ( , Truth labels that combine ) The obtained noun recognition loss And verb recognition loss Combined with diversity loss The final model loss is obtained to optimize the network, i.e.: Here, the parameter λ is used to control the penalty for the diversity loss term.

2. The slow-fast diversity perception prototype learning method for egocentric action recognition according to claim 1, characterized in that, Step 1 includes the following specific implementation process: The input video was processed by a video converter f(·) to generate spatiotemporal features. Where T×HW represents the total number of tokens, and D represents the dimension of each token; for frame t, a set of L learnable prototypes is initialized. Multi-head self-attention and layer normalization operations are employed to provide enhanced prototyping. ,Right now: Where {Q, K, V} are the query, key, and value, respectively, and W... Q W K and W V These are the projection matrices for the query, key, and value, respectively.

3. The slow-fast diversity perception prototype learning method for egocentric action recognition according to claim 2, characterized in that, The implementation process of step 2 is as follows: Obtain the enhanced prototype Then, using diversity-aware attention, we learn a compact but diverse prototype covering the interactive objects by extracting higher purity semantic information from some tokens. Take partial tokens and the enhanced prototype As input to diversity-perceived attention, its output is a semantically compressed prototype. ,Right now: MLP stands for Multilayer Perceptron; Design a diversity loss function , Where sim(·) represents the cosine similarity function. Represents all negative sample prototypes in frame t, and at the same time Is The indicator function is set to 1 under certain conditions.

4. The slow-fast diversity perception prototype learning method for egocentric action recognition according to claim 3, characterized in that, The specific implementation process of step 3 is as follows: The obtained prototype sequence Up / down enhancement is performed to capture semantic-level motion; by using deconvolution operations to upsample the prototype along the time dimension, the motion in the slow stream is captured. Time scale extended In the lower enhancement, convolution operations are used to compress the time scale of the prototype sequence along the time dimension into the fast stream. ; The motion enhancement prototype is represented by up-enhancement and down-enhancement as follows: , ,in, > T , < T , and , representing the temporal resolutions of slow flow and fast flow, respectively; DECONV and CONV represent the upsampling and downsampling of 3D convolution in the temporal dimension, respectively; BN represents batch normalization.

5. The slow-fast diversity perception prototype learning method for egocentric action recognition according to claim 4, characterized in that, The implementation process of step 4 is as follows: The up / down augmented prototype, through two collaborative attentional processes—slow-perception attention and fast-perception attention—is integrated with the original prototype in the normal flow. To interact; in slow-perception attention, Set as a query, Set up to compute key / value pairs for cross-attention mapping and obtain semantically supplemented prototype features in slow stream. ,Right now: Where AvgPool represents average pooling; Setting up fast-sensing attention For query, Prototype features for keys and values, and for obtaining semantic complementation in fast streams. ,Right now: 。 6. The slow-fast diversity perception prototype learning method for egocentric action recognition according to claim 1, characterized in that, The specific implementation process of step 5 is as follows: Using a slow-shutter strategy, by measuring / and The correlation between them adaptively adjusts the slow and fast flows. and The weights, i.e.: in, This represents the sigmoid function. Indicates slow / fast shutter speed; and Represents the weight matrix; By , and and and Aggregation yields the final action representation. ,Right now: Here, ⊙ represents the dot product of an element's direction.

Images