An action recognition method introducing frame clustering and self-attention time shift module

Abstract

The application relates to an action recognition method introducing a frame clustering and self-attention time shift module, a model of the recognition method comprising a frame clustering module, a local detail enhancement module and a self-attention time shift network; the frame clustering module divides and clusters video frames in the early stage of the model; the local detail enhancement module provides the model with action detail features in a small time neighborhood; and the self-attention time shift network further extracts the spatiotemporal features of actions for a final classification task. The application proposes a video division algorithm based on the maximum entropy principle, which optimizes the video clustering result in frame clustering, thereby reducing the error caused by clustering; and the TSM module and the multi-head self-attention mechanism are combined to solve the problem of low time modeling capability of a lightweight action recognition model caused by the naive time shift.

Description

Technical Field

[0001] This invention belongs to the field of action recognition technology, specifically relating to an action recognition method that incorporates frame clustering and a self-attention time-shifting module. Background Technology

[0002] Frame clustering is a method to address the information loss problem caused by video sampling. It clusters video frames early in the network by directly stacking frame activations together. Typically, if a deep learning model inputs all frames into the network, it needs to retain the activations and gradients of each frame at each layer for backpropagation. Therefore, clustering frames early reduces the size of the network and significantly reduces memory consumption. However, merging frames inevitably introduces errors into the network. Frame clustering also relies on the prior knowledge that if the signs of activations using the ReLU function are consistent, then accumulating gradients after frame activation is equivalent to accumulating activations first and then calculating gradients. Therefore, the error in frame clustering depends on the proportion of sign consistency among the clustered frame activations. Thus, the method of video clustering determines the quality of the frame clustering results.

[0003] The Temporal Shift Module (TSM) is a channel fusion method based on shift operations. Shift operations were initially proposed to replace convolution operations. A shift operation moves each channel of its input tensor along different spatial directions; for example, in two-dimensional data, this could be horizontal, vertical, or diagonal. Unlike spatial and depthwise convolutions, shift operations themselves do not require parameters or floating-point operations, but are implemented through a series of memory operations. Due to the parameterless and FLOP-free nature of shift operations, the complexity and computational cost of the model can be significantly reduced. To address the high complexity of feature extraction in action recognition using 3D convolutions, the Temporal Shift Module (TSM) was proposed and applied to action recognition models. The core idea of ​​TSM is to introduce temporal operations into the network. Typically, a CNN model can efficiently process two-dimensional images and extract their features. To enable the network to capture temporal features, TSM segments the input video feature tensor along the temporal dimension, shifting some channels along the temporal dimension to features from past and future adjacent frames. After shifting the channels, the feature map of each frame will simultaneously contain most of the original features, as well as some features from the previous frame and some features from the next frame. However, time shifting is too simplistic and difficult to model complex actions.

[0004] Multi-head self-attention (MSA) is a mechanism for modeling sequence data, initially introduced into neural networks for natural language processing tasks and widely used in Transformer models. MSA allows the model to interact and correlate between different positions within a sequence, thus capturing long-range dependencies. The basic idea of ​​MSA is to model the sequence by calculating the correlation between each position and other positions. In MSA, given an input sequence X and its corresponding feature representation matrix, the model obtains the representation of each position by calculating attention weights between each pair of elements. These attention weights are used to weight and sum the representations of other positions in the sequence to obtain a context-aware representation of each position. However, Transformer models are extremely complex, quadratically proportional to the number of vectors applied to the Transformer, making it difficult for typical devices to support large Transformer models.

[0005] In summary, frame clustering is resource-efficient but is affected by video segmentation strategies, inevitably introducing errors. Temporal shifting modules are low-complexity, but their performance is hampered by the difficulty of performing effective temporal modeling through naive shifting. Multi-head self-attention mechanisms support fully connected communication between several vectors, but suffer from excessively high complexity. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art and provide an action recognition method that introduces frame clustering and a self-attention time-shifting module. It proposes a video segmentation algorithm based on the maximum entropy principle to optimize the video clustering results in frame clustering, thereby reducing the error caused by clustering. At the same time, it combines the TSM module and multi-head self-attention mechanism to solve the problem of low time modeling capability of lightweight action recognition models caused by naive time shifting.

[0007] The technical problem solved by this invention is achieved through the following technical solution:

[0008] An action recognition method incorporating frame clustering and a self-attention time-shifting module is characterized in that: the model of the recognition method includes a frame clustering module, a local detail enhancement module, and a self-attention time-shifting network; the frame clustering module divides and clusters video frames in the early stage of the model; the local detail enhancement module provides the model with action detail features in a small temporal neighborhood; and the self-attention time-shifting network further extracts the spatiotemporal features of the action for the final classification task;

[0009] The frame clustering module aims to divide the video into a fixed number of segments and aggregate the features of each segment's frames. That is, for a given video input V = {f1, f2, ... f...} n Divide it into several segments p = {p1, p2, ..., p} T}, where n represents the number of frames in the video, T is the hyperparameter of the model, and T represents the number of segments. For one segment and This represents the start and end frames of the segment, and ∑ 1≤i≤T p i =V; then the frame clustering module will generate merged features F = [F1, F2, ..., F] for each video segment. T The generated merged features will then be input into the next module.

[0010] Furthermore, the frame clustering module uses a video segmentation algorithm based on the maximum entropy principle to segment the video. The optimization objective formula for this method is:

[0011]

[0012] Where: d is the distance metric, used to measure the difference between two video segments;

[0013] The algorithm is greedy in structure. Its goal is to find the global optimum that maximizes the value of D. It greedily searches for the segmentation points of the video segments to achieve a local optimum, so that the value of D is the maximum value that can be achieved at the current time.

[0014] (1) First, initialize the video frame and process it to obtain N segmentation points as candidate segmentation points for dividing the video segment. Select the N adjacent frames with the largest differences as candidate segmentation points. After the frame is activated and binarized, calculate the Hamming distance of all adjacent frames.

[0015] (2) Select N frame splitting points based on the largest N Hamming distances, where N is usually one-eighth of the total number of video frames n;

[0016] (3) Find the current local optimum, consider N segmentation points in sequence, attempt to divide the video segment at each segmentation point, and calculate the D value after the division. Calculate the mean of each segmented segment, and then calculate the cosine similarity between different video segments as the D value. The calculation method is shown in the formula:

[0017]

[0018] After comparing and finding the dividing point that maximizes the current D value, the algorithm finds the current local optimum. It then divides the video frame using this dividing point and records that the dividing point has been used. The algorithm then continues to search for the next local optimum by greedily selecting the dividing point that maximizes the D value from the remaining N-1 dividing points. This process is repeated T-1 times until all T-1 dividing points are determined.

[0019] Furthermore, the frame clustering module aggregates the segmented frame activations by averaging the frame activations of each segment, resulting in T activations for the video, expressed by the formula:

[0020] F = [F1, F2, ..., F] T ]

[0021]

[0022] Wherein: F i This represents the merging feature of the i-th segment;

[0023] I j Indicates that it belongs to fragment f i Activation of video frame j;

[0024] |f i | indicates the frame number of the segment;

[0025] The aggregated features will be fed into a subsequent self-attention time-shifting network to further extract the overall motion features of the video.

[0026] Furthermore, the local detail enhancement module obtains the segmentation and merging features of each video segment from the frame aggregation module as input, and performs detailed feature extraction on each segment, expressed as a formula:

[0027] F′=[F′1,F′2,…F′ T ]

[0028] F′ i =F i +H(I[I1,I2,I3,I4,I5])

[0029] Where: H represents the extraction of motion features from the segment, I represents the set of representative frames for the segment, and each video segment in the model uses five representative frames;

[0030] The local detail enhancement module selects several sampled frames, stacks their RGB differences, and uses a CNN to extract local motion detail features from the video clip. A uniform sampling strategy is designed: for a video clip of length m, the model uniformly selects five video frames and stacks their RGB differences along the channel direction, as shown in the formula:

[0031] D = [f mid / 2 -f1,f mid -f mid / 2 ,f 3mid / 2 -f mid ,f m -f 3mid / 2 ]

[0032] Where: f mid This indicates the middle frame of the segment;

[0033] Design a lightweight CNN module to process stacked RGB differences. This module employs a low-resolution processing strategy: first, it downsamples the data using average pooling, then extracts features using a CNN, and finally upsamples the data to match the extracted spatial features of the segment. This can be expressed as the formula:

[0034] H(p i )=Upsample(CNN(Downsample(D(p i ))))

[0035] Where: D represents the i-th video segment p i The RGB differences represent the stacked frames.

[0036] Furthermore, the self-attention time-shifting network performs a linear transformation on a portion of the feature map of each frame in the TSM network and adds positional encoding to obtain a time sequence X = x1, x2, ..., x T The model then performs a linear transformation on the obtained time series to obtain the query, key, and value for each element. Next, the model calculates the attention score for each query q. i Calculate its relationship with all keys k i The similarity score is calculated using the dot product attention method, refer to the formula:

[0037]

[0038] Next, the attention scores are converted into attention weights using the softmax function, and these weights are used to calculate a weighted sum of the value vectors to obtain the attention vector at position i in the sequence. The calculation method is shown in the formula below.

[0039]

[0040] The multi-head self-attention mechanism applies the above process to combinations of multiple queries, keys, and values ​​to obtain multiple different attention representations. Then, the attention vectors of multiple heads are concatenated and integrated to obtain the final output MultiHead. Finally, it is time-shifted again and concatenated with the original feature map. The multi-head attention output will pass through a linear layer to adjust its shape back to the feature map.

[0041] The advantages and beneficial effects of this invention are as follows:

[0042] 1. This invention greatly reduces the complexity of the model by inputting all video frames into the model;

[0043] 2. This invention improves the model's temporal modeling capability, thereby increasing the model's accuracy in action recognition;

[0044] 3. This invention supplements the local detail features in the video, which can better identify actions. Attached Figure Description

[0045] Figure 1 This is a schematic diagram of the overall structure of the model of the present invention;

[0046] Figure 2 This is a schematic diagram of the frame clustering module of the present invention;

[0047] Figure 3 This is a schematic diagram of the self-attention time-shifting network of the present invention. Detailed Implementation

[0048] The present invention will be further described in detail below through specific embodiments. The following embodiments are merely descriptive and not limiting, and should not be used to limit the scope of protection of the present invention.

[0049] This invention discloses an action recognition method that incorporates frame clustering and a self-attention time-shifting module. Its innovation lies in the following: the core of this invention is an action recognition model incorporating frame clustering and a self-attention time-shifting module. The model mainly consists of three parts: a frame clustering module, a local detail enhancement module, and a self-attention time-shifting network. The overall structure of the model is shown in Figure (1). The frame clustering module divides and clusters video frames early in the model, reducing memory pressure on subsequent models and minimizing errors. The local detail enhancement module provides action detail features in a small temporal neighborhood, while the self-attention time-shifting network further extracts the spatiotemporal features of the action for the final classification task.

[0050] 1) Frame clustering module

[0051] To minimize additional memory usage and prevent information loss due to frame sampling, this invention chooses to aggregate video frame information early in the model's development process instead of frame sampling. This preserves the original video information while reducing the model's memory consumption. Therefore, the frame clustering module aims to divide the video into a fixed number of segments and aggregate the features of each segment's frames. That is, for a given video input V = {f1, f2, ... f...} n The present invention aims to divide it into several segments p = {p1, p2, ... p}. T}, where n represents the number of frames in the video, T is the hyperparameter of the model, and T represents the number of segments. For one segment and This represents the start and end frames of the segment, and ∑ 1≤i≤T p i =V. Then the frame clustering module will generate merged features F = [F1, F2, ..., F...] for each video segment. T The generated merged features will be input into the next module.

[0052] To better segment videos, this invention proposes a video segmentation algorithm based on the maximum entropy principle. Video data is a type of time-series data, and due to the instability of time-series data, its distribution changes constantly. Therefore, deep learning networks need to learn common knowledge from these different distributions to accurately determine the video category. The maximum entropy principle is a criterion for selecting the statistical properties of random variables that best reflect objective reality. According to the maximum entropy principle, the model with the highest entropy is the best model. When segmenting time-series data without any prior assumptions, it is most reasonable to make each segment as diverse as possible to maximize the entropy of the overall distribution. This allows the model to model the data more generally and flexibly in the future. Segmenting video data according to the case of maximum entropy also means that the segmented video segments do not contain any default priors, which is equivalent to training the model under the "worst-case" scenario, thereby giving the trained model better generalization ability. The model can increase the entropy of the overall distribution by maximizing the differences between the segmented segments. Based on this, this invention designs a video segmentation algorithm in the frame aggregation module, which can be achieved by solving an optimization problem. The objective of this optimization problem can be expressed as the formula:

[0053]

[0054] Where d is the distance metric, used to measure the difference between two video segments, and can be Euclidean distance, cosine similarity, etc.

[0055] The algorithm is greedy in structure, and its goal is to find the globally optimal solution that maximizes the value of D. The basic process of the algorithm is to continuously and greedily search for segmentation points to divide the video segment in order to achieve a local optimum, such that the value of D is the maximum achievable value at that point.

[0056] The algorithm first initializes by processing the video frames to obtain N segmentation points as candidate segmentation points for dividing the video segments. Therefore, selecting these segmentation points is crucial. Considering the segmentation point between two frames, the algorithm selects the N adjacent frames with the greatest difference as candidate segmentation points. This invention argues that Hamming distance is a good choice for measuring the similarity between frames. Therefore, the model first binarizes the frame activation and then calculates the Hamming distance between all adjacent frames.

[0057] Then, the algorithm selects N frame splitting points based on the N largest Hamming distances. N is typically one-eighth of the total number of video frames, n. Not all n splitting points are used because the algorithm's time complexity is positively correlated with the number of candidate splitting points. The model usually clusters the video into 8 frames, while a ten-second video often has between 200 and 300 frames. Therefore, the size of N is sufficient for the model's requirements.

[0058] Next, the algorithm begins searching for the current local optimum. It considers N segmentation points sequentially, attempting to divide the video segment at each point and calculating the D-value after each segmentation. This invention uses cosine similarity as the metric. First, it calculates the mean of each segmented segment, and then calculates the cosine similarity between different video segments as the D-value. The calculation method is shown in the formula:

[0059]

[0060] After comparing and finding the dividing point that maximizes the current D value, the algorithm finds the current local optimum, divides the video frame using this dividing point, and records that the dividing point has been used. The algorithm then continues to search for the next local optimum, greedily selecting the dividing point that maximizes the D value from the remaining N-1 dividing points. This process is repeated T-1 times until all T-1 dividing points are determined.

[0061] The structure of the frame clustering algorithm is shown in Figure (2). This invention combines a video frame segmentation algorithm based on the maximum entropy principle with a frame clustering algorithm. The clustering module aggregates the segmented frame activations by averaging the frame activations of each segment, thus generating T activations for the video. This can be expressed as the formula:

[0062] F = [F1, F2, ..., F] T ]

[0063]

[0064] Among them, F i I represents the merging feature of the i-th segment. j Indicates that it belongs to fragment f i Activation of video frame j, |f i | indicates the frame number of the segment. These aggregated features will be fed into the subsequent TSM network to further extract the overall motion features of the video.

[0065] 2) Local detail enhancement module

[0066] This part takes the segmentation and merging features of each video segment as input from the frame aggregation module, and performs detailed feature extraction on each segment, which can be expressed as a formula:

[0067] F′=[F′1,F′2,…F′ T ]

[0068] F′ i =F i +H(I[I1,I2,I3,I4,I5])

[0069] Where H represents the extraction of motion features from the segment, and I represents the set of representative frames for the segment. Each video segment in the model uses five representative frames.

[0070] Since the frame clustering module divides the video based on the maximum entropy principle, frames within a given video segment are often similar. Calculating the RGB difference between two similar frames can emphasize the local regions where changes occur. Therefore, the LDEM module selects several sampled frames, stacks their RGB differences, and uses a CNN to extract local motion detail features from the video segment. Regarding the selection of sampled frames, both excessively long and short time spans are detrimental to the network's motion modeling. Excessively long time spans tend to overlook subtle changes in motion, and the large differences between frames can cause the CNN to lose focus when extracting features. Insufficiently short time spans result in overly redundant information in the video frames, leading to extremely sparse features. Therefore, a uniform sampling strategy is designed. For a video segment of length m, the model uniformly selects five video frames and stacks their RGB differences along the channel direction, as shown in the formula:

[0071] D = [f mid / 2 -f1,f mid -f mid / 2 ,f 3mid / 2 -f mid ,f m -f 3mid / 2 ]

[0072] Where f mid This indicates the middle frame of the segment.

[0073] Due to the similarity between video frames, the calculated RGB differences often produce high responses locally, but low responses in most areas. Therefore, this invention designs a lightweight CNN module to process stacked RGB differences without sacrificing too much accuracy. This module employs a low-resolution processing strategy: first, it downsamples the data using average pooling, then extracts features using a CNN, and finally upsamples the data to match the extracted spatial features of the segment. This can be expressed as the formula:

[0074] H(p i )=Upsample(CNN(Downsample(D(p i ))))

[0075] Where D represents the i-th video segment p i The RGB differences represent the stacked frames.

[0076] Furthermore, regarding the spatial features of video segments, compared to encoding video segments using individual video frames, using aggregated features as spatial features of video segments allows the model to retain richer video semantic information. Therefore, this invention directly uses the aggregated features obtained in the frame aggregation module as the spatial features of the segment, and connects them with motion features to obtain the merged features of the segment.

[0077] 3) Self-attention time-shifting network

[0078] The self-attention time-shifting module is shown in Figure (3). The naive time-shifting module cannot model complex action changes, so this invention designs a self-attention time-shifting module to improve TSM. Due to the fully connected nature of the self-attention mechanism, its high complexity is often unacceptable for lightweight networks. However, since the complexity of the self-attention mechanism is positively correlated with the quadratic power of the sequence length, and the model proposed in this invention aggregates video frames through frame clustering in the early stages, the number of frames is greatly reduced. Therefore, it is feasible to use the self-attention mechanism in the model in this paper. In order to optimize the time-shifting module in the TSM network, this invention applies the self-attention mechanism to the feature map of the network that will be time-shifted. This approach can help the model capture global dependencies in the time dimension, thereby better understanding the relationship between different frames in the video. When performing time shifting, the model can better consider the relationship between each video frame and other video frames. At the same time, the self-attention mechanism can dynamically learn the relationship between each video frame, which enhances the model's temporal modeling ability and further improves the model's ability to extract temporal features of video actions, thereby improving the model's expressive power and performance. To apply the self-attention mechanism to the time-shifting module, the model first performs a linear transformation on a portion of the feature map of each frame in the TSM network and adds positional encoding to obtain the time sequence X = x1, x2, ..., x T The model then performs a linear transformation on the obtained time series to obtain the query, key, and value for each element.

[0079] Next, the model will calculate the attention score for each query q. i Calculate its relationship with all keys k i The similarity score is calculated using the dot product attention method, refer to the formula:

[0080]

[0081] Next, the attention scores are converted into attention weights using the softmax function, and these weights are used to calculate a weighted sum of the value vectors to obtain the attention vector at position i in the sequence. The calculation method is shown in the formula below.

[0082]

[0083] The multi-head self-attention mechanism applies the above process to multiple queries, key-value combinations to obtain multiple different attention representations. The attention vectors from multiple heads are then concatenated and integrated to obtain the final output, MultiHead. Finally, to restore the result to a feature map, it undergoes a time shift and is concatenated with the original feature map. The multi-head attention output passes through a linear layer to reshape it back into a feature map.

[0084] Although embodiments and drawings of the present invention have been disclosed for illustrative purposes, those skilled in the art will understand that various substitutions, variations and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the scope of the present invention is not limited to the contents disclosed in the embodiments and drawings.

Claims

1. An action recognition method incorporating frame clustering and a self-attention time-shifting module, characterized in that: The recognition method's model includes a frame clustering module, a local detail enhancement module, and a self-attention time-shifting network. The frame clustering module divides and clusters video frames in the early stages of the model. The local detail enhancement module provides the model with action detail features in a small temporal neighborhood. The self-attention time-shifting network further extracts the spatiotemporal features of the action for the final classification task. The frame clustering module aims to divide the video into a fixed number of segments and aggregate the features of each segment's frames, that is, for a given video input... Divide it into several paragraphs Where n represents the number of frames in the video, T is the hyperparameter of the model, and T represents the number of segments. For one segment... , and This indicates the start and end frames of the segment, and ; The frame clustering module will then generate merged features for each video segment. The generated merged features will then be input into the next module.

2. The action recognition method incorporating frame clustering and a self-attention time-shifting module according to claim 1, characterized in that: The frame clustering module uses a video segmentation algorithm based on the maximum entropy principle to segment the video. The optimization objective formula for this method is: ； Where: d is the distance metric, used to measure the difference between two video segments; The algorithm is greedy in structure. Its goal is to find the global optimum that maximizes the value of D. It greedily searches for the segmentation points of the video segments to achieve a local optimum, so that the value of D is the maximum value that can be achieved at the current time. (1) First, initialize the video frame and process it to obtain N segmentation points as candidate segmentation points for dividing the video segment. Select the N adjacent frames with the largest differences as candidate segmentation points. After the frame is activated and binarized, calculate the Hamming distance of all adjacent frames. (2) Select N frame splitting points based on the largest N Hamming distances, where N is usually one-eighth of the total number of video frames n; (3) Find the current local optimum, consider N segmentation points in sequence, attempt to divide the video segment with each segmentation point, and calculate the D value after the division. Calculate the mean of each segment and then calculate the cosine similarity between different video segments as the d value. The calculation method is shown in the formula: ； After comparing and finding the dividing point that maximizes the current D value, the algorithm finds the current local optimum. It then divides the video frame using this dividing point and records that the dividing point has been used. The algorithm then continues to search for the next local optimum by greedily selecting the dividing point that maximizes the D value from the remaining N-1 dividing points. This process is repeated T-1 times until all T-1 dividing points are determined.

3. The action recognition method incorporating frame clustering and a self-attention time-shifting module according to claim 1, characterized in that: The frame clustering module aggregates the segmented frame activations by averaging the frame activations of each segment, resulting in T activations for the video, represented by the formula: ； ； in: This represents the merging feature of the i-th segment; Indicates that it belongs to a fragment video frames Activation; Indicates the frame number of the segment; The merged features will be fed into a subsequent self-attention time-shifting network to further extract the overall motion features of the video.

4. The action recognition method incorporating frame clustering and a self-attention time-shifting module according to claim 1, characterized in that: The local detail enhancement module obtains the segmentation and merging features of each video segment from the frame clustering module as input, and performs detailed feature extraction on each segment, expressed as the formula: ； ； Where: H represents the extraction of motion features from the segment, I represents the set of representative frames for the segment, and each video segment in the model uses five representative frames; The local detail enhancement module selects several sampled frames, stacks their RGB differences, and uses CNN to extract the local detail features of motion in the video clip. A uniform sampling strategy is designed. For a video clip of length m, the model uniformly selects five video frames and stacks their RGB differences along the channel direction. Design a lightweight CNN module to process stacked RGB differences. This module employs a low-resolution processing strategy: first, it downsamples the data using average pooling, then extracts features using a CNN, and finally upsamples the data to match the extracted spatial features of the segment. This can be expressed as the formula: ； in: Represents the i-th video segment The RGB differences represent the stacked frames.

5. The action recognition method incorporating frame clustering and a self-attention time-shifting module according to claim 1, characterized in that: The self-attention time-shifting network performs a linear transformation on a portion of the feature map of each frame in the TSM network and adds positional encoding to obtain a time sequence. The model then performs a linear transformation on the obtained time series to obtain the query, key, and value for each element. Next, the model calculates an attention score for each query. Calculate its relationship with all keys The similarity score is calculated using the dot product attention method, refer to the formula: ； Next, the attention scores are converted into attention weights using the softmax function, and these weights are used to calculate a weighted sum of the value vectors to obtain the attention vector at position i in the sequence. The calculation method is shown in the formula below. ； The multi-head self-attention mechanism applies the above process to combinations of multiple queries, keys, and values ​​to obtain multiple different attention representations. Then, the attention vectors of multiple heads are concatenated and integrated to obtain the final output MultiHead. Finally, it is time-shifted again and concatenated with the original feature map. The multi-head attention output will pass through a linear layer to adjust its shape back to the feature map.

Images