A lightweight transform video action recognition method based on video and text fusion

Abstract

The present application relates to the field of video action recognition, and particularly relates to a lightweight Transformer video action recognition method based on video and text fusion, comprising the following steps executed in sequence: S1: acquiring a video dataset, acquiring a key frame image set and a text set from the video dataset; S2: extracting the spatio-temporal visual features of the key frame image set by using a three-dimensional convolutional neural network, obtaining structured video visual features, and extracting semantic features of the text set by using a pre-trained bidirectional Transformer language model (BERT); S3: inputting the video visual features and the semantic features into a lightweight Transformer fusion layer for feature fusion, and acquiring fusion features; S4: inputting the fusion features into a fully connected layer, mapping the fusion features to an action category space, and performing a Softmax normalization operation, inputting the normalized features into a classifier for action prediction, and outputting an action category and a corresponding probability value; thereby improving the accuracy of video action recognition.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of video action recognition, and in particular to a lightweight Transformer video action recognition method based on video and text fusion. BACKGROUND

[0002] With the popularity of short video platforms and smart fitness applications, video action recognition technology plays a key role in human behavior understanding, human-computer interaction, exercise guidance, and content recommendation. Existing video action recognition methods mainly rely on visual modalities, extracting spatio-temporal features from video frame sequences and performing action classification through three-dimensional convolutional neural networks such as I3D. However, single-modal methods using only visual features have limited accuracy in handling certain categories, especially in the following situations:

[0003] Actions with highly similar appearances are difficult to distinguish visually, such as "jogging", "jumping jacks", and "jumping jacks", whose visual patterns are very similar, and traditional models are prone to confusion between these categories;

[0004] Action details rely on semantic context, but visual cues are difficult to identify certain actions that depend on subtitles, such as "ready", "start", and "squat", which cannot obtain these semantic clues solely relying on frame images;

[0005] Short video scenes change quickly and are often obscured, resulting in missing visual information. When key actions are obscured or the screen is shaking, single-modal models are prone to misjudging action categories.

[0006] Existing research attempts to use video description text to assist in recognition, but still has the following shortcomings: lack of structured video-text paired samples, making multi-modal training difficult; multi-modal model structure is complex and computationally intensive; cross-modal semantic alignment capability is insufficient, making it difficult to effectively improve action classification performance.

[0007] Existing technologies cannot accurately output action categories in complex short video scenarios, such as "jumping rope", "squatting", and "squatting", especially when actions are similar, visual is unclear, or semantic dependence is high. Therefore, improving the accuracy, robustness, and real-time performance of action classification has become a technical problem that needs to be solved. SUMMARY

[0008] The purpose of the present application is to provide a lightweight Transformer video action recognition method based on video and text fusion that improves classification accuracy.

[0009] To achieve the above purpose, the technical solution adopted by the present application is as follows:

[0010] A lightweight Transformer video action recognition method based on video and text fusion, comprising the following sequentially executed steps:

[0011] S1: Obtain the video dataset and sample each video segment in the dataset frame by frame;

[0012] Based on the principle of average time distribution, images of a first preset number of frames are selected at equal intervals from each video segment as a keyframe image set for action representation, and action labels are assigned to the keyframe image set according to the category information corresponding to the video.

[0013] A set of image frames for text parsing is constructed by selecting images of a second preset number of frames at fixed time intervals from each video segment;

[0014] Extract subtitles and / or screen text from the set of image frames to obtain a text set corresponding to the video;

[0015] S2: A three-dimensional convolutional neural network is used to extract the spatiotemporal visual features of the keyframe image set. Global average pooling and linear transformation are applied to the spatiotemporal visual features to enhance them and obtain structured video visual features. A pre-trained bidirectional Transformer language model is used to extract the semantic features of the text set.

[0016] S3: Input the visual features and semantic features of the video into the lightweight Transformer fusion layer for feature fusion. The specific fusion steps are as follows:

[0017] S3-1: Perform feature mapping between the visual features and semantic features of the video;

[0018] S3-2: Add modality type labels to the mapped visual and semantic features respectively;

[0019] S3-3: A token-level joint encoding strategy is used to sequentially concatenate the video visual features and semantic features embedded with modality type tags to obtain joint features;

[0020] S3-4: A single-layer shared Transformer encoder is used to perform bidirectional interaction between video and text on the joint features. The specific interaction process is as follows: cross-modal self-attention is used to perform bidirectional semantic interaction on the joint features, outputting interactive features. The interactive features are then residually connected with the joint features. The residually connected features are then normalized. A lightweight feedforward network module is used to perform feature transformation and nonlinear enhancement on the normalized features, outputting fused features.

[0021] S4: Input the fused features into the fully connected layer, map the fused features to the action category space, perform Softmax normalization, input the normalized features into the classifier for action prediction, and output the action category and the corresponding probability value.

[0022] Preferably, in step S1, action labels are assigned to the image set according to the video category information. The action labels will serve as supervision signals for the subsequent training of the video-text fusion model, guiding the model to learn the discriminative features of different action categories.

[0023] Preferably, step S1 further includes cleaning the subtitles and / or screen text using regularization cleaning, duplicate line removal, symbol filtering, noise character removal, redundant and duplicate deletion, and BERT semantic similarity or edit distance merging algorithms.

[0024] By adopting the aforementioned design scheme, the beneficial effects of the present invention are: the single-layer shared Transformer encoder introduced in this application replaces the traditional multi-layer stacked Transformer structure, which reduces the amount of computation in the forward and backward propagation process from the model structure, significantly reduces the parameter scale and computational complexity, and enables the model to maintain stable operation in a low-latency, low-resource environment.

[0025] In the fusion stage, this application adopts a token-level joint coding strategy and a modality type embedding mechanism, enabling fine-grained bidirectional interaction between video visual features and text semantic features within a unified attention space. This is no longer limited to the "global feature splicing" of traditional methods. This fine-grained cross-modal dependency modeling approach can enhance the model's ability to focus on action category discrimination features and improve the model's ability to distinguish similar actions, such as "squat" and "deadlift," thereby improving classification accuracy at the structural level.

[0026] This application designs a video token aggregation output mechanism, which enables the model to automatically select the visual-semantic combination features most relevant to action recognition, reduce interference from irrelevant background information, and further enhance the representation ability of action categories. This mechanism can make the fused semantic representation more discriminative, provide higher quality input features for the final classification, and thus improve the overall recognition performance.

[0027] In summary, this application not only optimizes the computational overhead in terms of structure, but also effectively improves the accuracy and robustness of video action recognition through an improved cross-modal fusion mechanism. Attached Figure Description

[0028] Figure 1 This is a flowchart of the video action recognition method of the present invention;

[0029] Figure 2 This is a schematic diagram of the single-layer shared Transformer encoder structure of the present invention. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0031] The terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0032] A lightweight Transformer video action recognition method based on video and text fusion, such as... Figure 1 As shown, the steps are executed sequentially as follows:

[0033] S1: Obtain the video dataset and sample each video segment in the dataset frame by frame;

[0034] Based on the principle of average time distribution, images of a first preset number of frames are selected at equal intervals from each video segment as a keyframe image set for action representation. Action labels are assigned to the keyframe image set according to the category information corresponding to the video. These action labels serve as supervision signals for the subsequent training of the video-text fusion model, guiding the model to learn the discriminative features of different action categories.

[0035] Based on this, a set of image frames for text parsing is constructed by selecting images of a second preset number of frames at fixed time intervals from each video segment; optical character recognition (OCR) is performed on the set of image frames to extract subtitles and / or screen text, and the recognition results are cleaned and matched to obtain a text set corresponding to the video.

[0036] In this embodiment, the first preset frame rate is set to 16 frames and the second preset frame rate is set to 10 frames. Other settings can also be made according to actual needs.

[0037] In this embodiment, the extracted subtitles or screen text are cleaned and paired to eliminate OCR noise in the image frame set and integrate cross-frame subtitles. Conventional rule-based or text similarity-based methods can be used for cleaning and pairing, such as regularization cleaning, duplicate line removal, symbol filtering, noise character removal, redundant and duplicate deletion, BERT semantic similarity or edit distance merging algorithms, etc. The same subtitle recognized by OCR in multiple key image frames is automatically paired and merged to obtain a continuous and correct subtitle semantic sequence.

[0038] S2: A three-dimensional convolutional neural network (I3D) is used to extract the spatiotemporal visual features of the keyframe image set, resulting in a 2048-dimensional video global feature vector. Global average pooling and linear transformation are performed on the 2048-dimensional feature vector to obtain a structured video visual feature representation. This video visual feature is used as the input for the subsequent cross-modal fusion stage to participate in the calculation of the joint semantic representation of video and text.

[0039] A pre-trained bidirectional Transformer language model (BERT) is used to extract semantic features from the text set, resulting in a text feature sequence composed of several 768-dimensional text tokens. This text feature sequence, along with video visual features, is input into a lightweight Transformer fusion module to achieve cross-modal interaction and semantic alignment.

[0040] After completing S1 and S2, a multimodal sample set consisting of "video frame sequences + subtitle text + action tags" will be constructed, providing the model with dual-channel input of visual and semantic meaning. Action tags originate from manual annotation during the dataset collection phase, meaning that action categories are manually assigned based on video themes or action types when collecting raw videos. Subsequently, during model training, action tags serve as supervisory signals in loss function calculations, such as cross-entropy loss, to guide the model in learning the mapping relationship between video-text fusion features and action categories. Therefore, although action tags are not shown separately in the structure diagram, they are present throughout the training process after the feature extraction phase and serve as the training target for the final classification module.

[0041] S3: Input the video visual feature vector and the semantic feature into the Lightweight Transformer Fusion Layer for feature fusion. The specific fusion steps are as follows:

[0042] S3-1: Perform feature mapping on the visual and semantic features of the video, projecting the two modalities onto a unified latent space representation to eliminate dimensional differences between different modalities, making them comparable and interactive, and laying the foundation for subsequent cross-modal attention computation and joint modeling.

[0043] S3-2: Add modality type labels to the mapped visual and semantic features respectively. These labels identify whether the current feature originates from a video modality or a text modality. By adding learnable modality type embedding vectors, the Transformer can distinguish the feature sources of different modalities during subsequent attention calculations, thereby avoiding modality confusion and improving the effectiveness of cross-modal interactions.

[0044] S3-3: A token-level joint encoding strategy is adopted to sequentially concatenate the visual and semantic features of the video with embedded modality type labels to obtain joint features. Here, sequential concatenation means that the video features with added modality labels are constructed into a single video token and concatenated with multiple text tokens in a fixed order, with the video token placed at the beginning of the sequence and the text tokens arranged sequentially after it, thus forming a unified cross-modal input sequence.

[0045] S3-4: Adopt as follows Figure 2 The single-layer shared Transformer encoder shown performs bidirectional interaction between video and text on the joint features. The specific interaction process is as follows: Cross-modal self-attention is used to perform bidirectional semantic interaction on the joint features, outputting interactive features, enabling the model to automatically focus on key content in the text related to the current video action, thereby completing cross-modal alignment; the interactive features are residually connected with the joint features to avoid information loss, enhance feature stability, and make gradient propagation smoother during training; the features after residual connection are subjected to layer normalization to balance the feature scale differences between different modalities, improving training stability and fusion effect; a lightweight feed-forward network (FFN) module is used to perform feature transformation and nonlinear enhancement on the normalized features, outputting fused features.

[0046] S4: Input the fused features into the Fully Connected Classification Layer, map the fused features to the action category space, perform Softmax normalization, input the normalized features into the classifier for action prediction, and output the action category and the corresponding probability value.

[0047] Table 1. Accuracy comparison of the video action recognition method of this application with other models.

[0048]

[0049] As shown in Table 1, under the same dataset conditions, the classification performance of various methods shows a gradual improvement from unimodal to multimodal. Firstly, in the text modality, the accuracies of the pure text models TF-IDF+Regression and BERT are 0.4920 and 0.4751, respectively. Their overall performance is limited by the sparsity and limited semantic information of the caption text, making it difficult to accurately distinguish action categories. Secondly, in the video modality, models such as TSN, R3D, and I3D all rely on visual spatiotemporal information for discrimination. Among them, I3D, relying on the stronger modeling ability of 3D convolution for spatiotemporal structure, achieves an accuracy of 0.5306, significantly outperforming other unimodal models. In multimodal fusion methods, cross-modal models such as LXMERT and VisualBERT further improve the model's ability to distinguish action semantics by introducing text descriptions to assist visual understanding. VisualBERT achieves an accuracy of 0.5783, indicating that multimodal information can compensate for the shortcomings of unimodal video or unimodal text. Compared to the aforementioned models, the lightweight Transformer fusion method proposed in this invention achieves significant performance advantages while maintaining low computational complexity, achieving an accuracy of 0.6143, with optimal performance in precision, recall, and F1 score. This is attributed to the efficient cross-modal interaction structure of feature mapping, modality type embedding, and a single-layer shared Transformer, enabling the model to more fully utilize the fine-grained correlation between video visual information and subtitle semantic information, thereby obtaining more accurate action recognition results. In summary, the method of this invention demonstrates the best overall performance in multimodal action recognition tasks, proving the effectiveness and superiority of its design.

[0050] In summary, this application uses a single-layer Transformer encoder instead of a multi-layer stacked structure to reduce the number of parameters and computational complexity, and achieves bidirectional interaction of video-text features and global semantic dependency modeling through a single-layer self-attention mechanism.

[0051] By introducing type embedding vectors for video and text modalities, the Transformer can retain modal differentiation information within a unified attention space;

[0052] A token-level joint encoding strategy is adopted, which represents video features as a single video token and text features as multiple text tokens. These tokens are then concatenated into a unified input sequence in the order of "video token first, text token last". This sequence serves as the input to the Transformer encoder, enabling the model to simultaneously focus on video and text information in the self-attention mechanism, thereby achieving fine-grained cross-modal semantic combination and feature fusion.

[0053] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A lightweight Transformer video action recognition method based on video and text fusion, characterized in that: The steps are as follows, performed sequentially: S1: Obtain the video dataset and sample each video segment in the dataset frame by frame; Based on the principle of average time distribution, images of a first preset number of frames are selected at equal intervals from each video segment as a keyframe image set for action representation, and action labels are assigned to the keyframe image set according to the category information corresponding to the video. A set of image frames for text parsing is constructed by selecting images of a second preset number of frames at fixed time intervals from each video segment; Extract subtitles and / or screen text from the set of image frames to obtain a text set corresponding to the video; S2: A three-dimensional convolutional neural network is used to extract the spatiotemporal visual features of the keyframe image set. Global average pooling and linear transformation are applied to the spatiotemporal visual features to enhance them and obtain structured video visual features. A pre-trained bidirectional Transformer language model is used to extract the semantic features of the text set. S3: Input the visual features and semantic features of the video into a lightweight Transformer fusion layer for feature fusion. The specific fusion steps are as follows: S3-1: Perform feature mapping between the visual features and semantic features of the video; S3-2: Add modality type labels to the mapped visual and semantic features respectively; S3-3: A token-level joint encoding strategy is used to sequentially concatenate the video visual features and semantic features embedded with modality type tags to obtain joint features; S3-4: A single-layer shared Transformer encoder is used to perform bidirectional interaction between video and text on the joint features. The specific interaction process is as follows: cross-modal self-attention is used to perform bidirectional semantic interaction on the joint features, outputting interactive features. The interactive features are then residually connected with the joint features. The residually connected features are then normalized. A lightweight feedforward network module is used to perform feature transformation and nonlinear enhancement on the normalized features, outputting fused features. S4: Input the fused features into the fully connected layer, map the fused features to the action category space, perform Softmax normalization, input the normalized features into the classifier for action prediction, and output the action category and the corresponding probability value.

2. The lightweight Transformer video action recognition method based on video and text fusion as described in claim 1, characterized in that: In step S1, action labels are assigned to the image set according to the video category information. These action labels will serve as supervision signals for the subsequent training of the video-text fusion model, guiding the model to learn the discriminative features of different action categories.

3. The lightweight Transformer video action recognition method based on video and text fusion as described in claim 1, characterized in that: Step S1 also includes cleaning the subtitles and / or screen text using regularization cleaning, duplicate line removal, symbol filtering, noise character removal, redundant and duplicate deletion, and BERT semantic similarity or edit distance merging algorithms.

Images