Space-time multi-mode video understanding method based on Video LLeMA2

Abstract

The invention discloses a time-space multi-mode video understanding method based on Video LLeMA2. The time-space multi-mode video understanding method comprises the following steps: S1, extracting an image frame sequence and an audio stream sequence of video data; s2, carrying out pretreatment; s3, inputting the image frame sequence into a visual encoder to generate a visual initial feature; s4, inputting the visual initial features into a space-time convolution connector, and generating visual modal features by using three-dimensional convolution and a RegStage module; s5, inputting the audio stream sequence into an audio encoder to obtain audio modal features; s6, aligning the audio modal features with the visual modal features; s7, performing feature fusion; and S8, obtaining a video content understanding result through a language decoder. The video content semantic understanding method is based on the Video LLeMA2 model, integrates multi-modal space-time modeling and language generation technologies, realizes video content semantic understanding, and has the advantages of natural expression and high precision.

Description

Technical Field

[0001] This invention relates to the field of video understanding technology, and in particular to a spatiotemporal multimodal video understanding method based on VideoLLeMA2. Background Technology

[0002] With the development of artificial intelligence technology, video understanding, as one of the key tasks in the intersection of computer vision and natural language processing, has been widely used in scenarios such as intelligent monitoring, human-computer interaction, content retrieval, assisted driving, education and entertainment. Video understanding aims to simulate human cognitive ability to video content, automatically identify scenes, events, objects and their semantic relationships in videos, and express them in structured information or natural language. Since video data contains multimodal information such as visual, audio and time series, how to efficiently model multimodal features, capture spatiotemporal dynamic changes and generate high-quality semantic expressions has always been an important research direction in this field.

[0003] Traditional video understanding methods are mostly based on structures such as convolutional neural networks and recurrent neural networks for modeling. Typical methods include using 2D CNNs to extract static frame features and combining them with temporal networks such as LSTM for dynamic modeling, or using 3D CNNs to directly extract spatiotemporal features from video blocks. In addition, some methods use dual-stream network structures to process optical flow and image frames separately to improve the ability to recognize actions. Although these methods have achieved tasks such as video classification and action recognition to a certain extent, they still have several limitations: First, the network structure is usually fixed and it is difficult to adapt to multimodal inputs; second, the fusion of visual and audio modalities is mostly shallow splicing and lacks semantic alignment mechanisms; third, the model output is mainly classification labels and lacks the ability to understand and generate content at the natural language level, making it difficult to meet the application requirements of complex semantic expressions.

[0004] In recent years, with the successful application of the Transformer architecture in the field of natural language processing, large-scale pre-trained language models have been widely introduced into multimodal modeling tasks, giving rise to a new generation of video understanding models represented by "visual-language pre-training". On this basis, architectures such as CLIP, VideoCLIP, and BLIP, which integrate large language models with visual semantic modeling capabilities, have emerged. These methods improve the generalization ability of models in tasks such as description generation and question answering reasoning by constructing a unified semantic space and establishing alignment relationships between multimodalities.

[0005] Although existing large language model frameworks have achieved multimodal modeling and semantic generation of video content, significant technical bottlenecks remain in the following aspects. First, some methods focus only on visual modal information modeling, ignoring key semantic cues contained in the audio modality, such as speech content, background sounds, and abnormal noises, thus limiting the accuracy of overall semantic understanding. Second, the fusion of visual and audio modalities often adopts simple splicing or attention-weighted strategies, lacking a unified feature alignment mechanism and failing to fully address the issues of dimensional inconsistency and inaccurate temporal alignment between modalities. Third, in terms of visual modeling, traditional image encoders struggle to capture the temporal evolution patterns between image frames, resulting in an inability to effectively parse action changes and dynamic semantics between consecutive frames. Although some models introduce 3D convolutional structures or temporal Transformer modules, their connection with the large language model is not yet tight, leading to insufficient feature transfer and semantic decoupling. Finally, while language decoders possess powerful language generation capabilities, they lack a hierarchical semantic guidance mechanism when processing fused features from complex multimodal inputs, easily resulting in output text lacking logical coherence and event consistency.

[0006] In terms of audio modeling, although self-supervised pre-trained audio models such as Wav2Vec and BEATs have emerged in recent years, their use in multimodal video understanding tasks is still relatively rudimentary. In particular, the BEATs model, as a high-performance audio representation model based on Transformer, has good frequency and time domain modeling capabilities, but current methods mostly treat it as a separate module to extract audio features, failing to form an effective synergy with visual features, resulting in the audio semantics being difficult to be fully conveyed to the final language output.

[0007] Therefore, there is an urgent need for a video understanding method that can achieve unified modeling of visual and audio information, construction of spatiotemporal dynamic structures, accurate alignment and deep fusion of multimodal features, and natural language decoding capabilities. Summary of the Invention

[0008] One objective of this invention is to propose a spatiotemporal multimodal video understanding method based on VideoLLeMA2. This invention is based on the VideoLLeMA2 model and integrates a visual encoder, an audio encoder, a spatiotemporal convolutional connector, and a large language decoder to construct an end-to-end multimodal spatiotemporal modeling and natural language generation process. This enables accurate understanding and natural expression of complex semantic content in videos, and has the advantages of fluent expression, high understanding accuracy, and rich adaptability to various tasks.

[0009] A spatiotemporal multimodal video understanding method based on VideoLLeMA2 according to an embodiment of the present invention includes the following steps: S1. Obtain the raw video data, perform image frame decoding on the raw video data, and extract the image frame sequence and audio stream sequence; S2. Perform image preprocessing and audio preprocessing on the image frame sequence and audio stream sequence respectively to generate the image input sequence and audio input sequence; S3. Input the image input sequence into the visual encoder, and use the CLIP ViT-Large-Patch14-336 model to encode the image frames and generate initial visual features; S4. Input the initial visual features into the spatiotemporal convolution connector, use 3D convolution to extract the temporal features between image frames, and use the RegStage module to perform semantic enhancement on the temporal features to generate visual modal features; S5. Input the audio input sequence into the audio encoder, use the BEATs model to model the audio in the frequency domain and time domain, and output the audio modal features; S6. Input the audio modal features and visual modal features into the multilayer perceptron, and perform feature alignment by projecting the features onto a unified feature dimension space. S7. Based on the multimodal fusion module, the aligned features are fused to obtain multimodal features; S8. Input the multimodal features into the language decoder, use a large language model for semantic decoding, and generate video content understanding results in natural language form.

[0010] Optionally, the image preprocessing includes image denoising, size normalization, inter-frame synchronization, and normalization.

[0011] Optionally, the audio preprocessing includes amplitude normalization, silence removal, boundary clipping, and smoothing.

[0012] Optionally, S3 specifically includes: S31. The image input sequence is input into the visual encoder of the VideoLLaMA2 model. The visual encoder uses the CLIP ViT-Large-Patch14-336 model. The VideoLLaMA model is a video understanding model based on a large language model, which integrates visual and audio information and performs spatiotemporal modeling and semantic alignment through a unified Transformer framework. The VideoLLaMA model combines a visual encoder, an audio encoder, a spatiotemporal convolutional connector, a multilayer perceptron, a multimodal fusion module, and a language decoder. The CLIP ViT-Large-Patch14-336 model is an image encoding model based on a visual Transformer architecture. It divides the input image into fixed-size image blocks and performs linear projection, and extracts global semantic features through a multilayer self-attention mechanism. Each image block is 14×14 in size, the input image resolution is 336×336, and the model contains 24 Transformer encoder layers to generate image feature representations with a dimension of 1024. S32. Perform a segmentation operation on each frame of the image input sequence, dividing it into... Non-overlapping image patches; S33. After expanding each image patch into a vector, perform a linear transformation using a linear mapping weight matrix and a bias vector: ; in, Indicates the first The first image frame Feature vectors of image patches Represents the linear mapping weight matrix. This represents the bias vector. Representing feature dimension, Indicates the first The first image frame Image blocks, This means flattening the image patch into a vector. Represents the set of real numbers; S34. Add two-dimensional positional coding and frame sequence temporal coding to the feature vector of the image patch to obtain the embedding vector, and concatenate the embedding vectors of all image patches in each frame to obtain the image embedding representation. ,in Represents the vector transpose operation: ; in, This represents the embedding vector after adding two-dimensional positional coding and frame sequence coding. Indicates the first The first frame Feature vectors of image patches Represents the two-dimensional location code value. Indicates the frame sequence time-coded value; Define two-dimensional position encoding: ; in, Indicates the first In the dimension of the first Frame sequence time coding of each image block Indicates the sequence number of the image patch after it has been flattened in the image. Indicates the dimension of the encoding; Define frame sequence time coding: ; in, Indicates the first In the dimension of the first Two-dimensional positional encoding values ​​of each image frame, Indicates the dimension of the encoding; S35. Input the embedding representation matrix into the Transformer encoding module to obtain the encoded output of each frame of the image, and perform average pooling operation on the encoded result of each frame to extract the overall semantic feature vector of the image: ; in, Indicates the first The overall semantic feature vector of an image frame This indicates the number of blocks in each frame of the image. Indicates the first The first image frame The output of the Transformer encoding module for each image patch; S36. Concatenate all the overall semantic feature vectors of the images in chronological order to form the initial visual features.

[0013] Optionally, S4 specifically includes: S41. Input the initial visual features into the spatiotemporal convolution connector, and construct a three-dimensional tensor representation based on the initial visual features. The number of channels in the three-dimensional tensor representation is the feature dimension of the initial visual features, the time step is the number of image frames of the initial visual features, and the spatial dimension is a single point. The spatiotemporal convolution connector is a feature enhancement module that models the temporal dynamic changes and spatial local structures of visual features. It consists of a three-dimensional convolution and a RegStage module. The three-dimensional convolution is used to perform temporal and spatial joint modeling on the input initial visual features, extracting the inter-frame temporal dependencies and spatial transformation patterns. The RegStage module performs nonlinear transformation and structural optimization on the output of the three-dimensional convolution, thereby improving the semantic expressiveness and temporal continuity and stability of the features. S42. Apply a three-dimensional convolution operation to the three-dimensional tensor representation, perform convolution processing along the time axis on the three-dimensional tensor, and generate a three-dimensional convolution output tensor. S43. Divide the three-dimensional convolution output tensor according to the time dimension to construct a time step feature sequence. Each time step feature of the time step feature sequence is a one-dimensional vector with the dimension being the number of channels of the three-dimensional convolution output. S44. Input the time step feature sequence into the RegStage module. The RegStage module consists of two linear mapping layers. The first linear mapping layer projects the time step feature sequence to the intermediate hidden layer dimension and adds a first bias parameter. After the activation function transformation, the second linear mapping layer restores the original output dimension and adds a second bias parameter to obtain the enhanced features of each time step. S45. Concatenate the enhanced features of all time steps in chronological order into a two-dimensional feature matrix to form a visual modality feature. The time dimension of the visual modality feature is the number of time steps of the three-dimensional convolution output, and the feature dimension is the number of channels of the three-dimensional convolution output.

[0014] Optionally, the BEATs model includes an audio processing layer, a patch layer, a position encoding layer, a Transformer encoder layer, and an output projection layer.

[0015] Optionally, S5 specifically includes: S51. Represent the audio input sequence as a sequence of length... A one-dimensional audio signal sequence, where each audio signal represents the audio amplitude at a sampling point; S52. Perform frame segmentation on the one-dimensional audio signal sequence. S53. Apply short-time Fourier transform to each frame of audio signal sequence to obtain the spectral representation: ; in, The spectrum is represented to describe the first... Frame number Complex spectral values ​​at each frequency point Indicates the length of each frame. Indicates the first Frame number The amplitude of each sampling point Represents the window function. Represents the imaginary unit. Represents natural parameters; S54. Stack all frame spectrum representations to form a two-dimensional spectrogram, and divide it according to the time direction into... One non-overlapping spectrogram; S55. Flatten each spectrogram and perform a linear mapping, while adding time position encoding to generate an embedded representation; S56. Input the embedded representation into the Transformer encoder of the BEATs model and perform attention mechanism and feedforward network operations. S57. Extract all frame features from the last layer output of the Transformer encoder and concatenate them into audio modal features.

[0016] Optionally, S8 specifically includes: S81. The multimodal feature sequence is used as the initial input to the large language model and initialized as the input representation of the 0th layer of the large language model. S82. In the large language model, a decoder structure consisting of multiple stacked layers is constructed. Each layer receives the output of the previous layer as the current input and completes hierarchical propagation in sequence to gradually extract semantic features. S83. Within each decoder layer, multi-head attention mechanism and feedforward neural network processing are executed sequentially to capture long-term dependencies and cross-modal semantic information in the time series. S84. Output the token representation sequence in the last decoder layer, representing the embedding form of natural language; S85. Perform linear mapping and normalization on the token representation sequence to convert it into a natural language vocabulary index, thereby obtaining a text token sequence; S86. Decode the text token sequence into a video understanding result in natural language form, wherein the video understanding result includes event description, behavior category, semantic label, or question-and-answer text.

[0017] Optionally, the decoder is a neural network structure composed of multiple layers of stacked Transformer units.

[0018] Optionally, the token representation sequence is a vector combination of tokens, whereby the token represents the smallest semantic unit of the text.

[0019] The beneficial effects of this invention are: First, this invention provides a spatiotemporal multimodal video understanding method based on VideoLLeMA2. Addressing the problems of insufficient modal fusion, incomplete spatiotemporal modeling, and limited language expression capabilities in existing technologies, it constructs a unified processing framework from raw video data to natural language semantic output. This improves the performance of multimodal video understanding systems in terms of semantic expression completeness, modal collaboration accuracy, and task adaptability. By extracting image frame sequences and audio stream sequences separately at the raw video data level and employing dedicated preprocessing mechanisms for images and audio respectively, the quality and structural consistency of the input data are ensured, providing a high-quality input foundation for subsequent modeling.

[0020] Secondly, in terms of visual modeling, this invention introduces the CLIP ViT-Large-Patch14-336 model as a visual encoder, which combines image block segmentation, linear embedding and Transformer structure to extract global visual semantics. At the same time, it combines a three-dimensional convolutional structure and the RegStage module to build a spatiotemporal convolutional connector, which enhances the ability to model dynamic features between image frames and effectively captures cross-frame behavioral evolution patterns and local semantic change trends.

[0021] Furthermore, in terms of audio modeling, this invention fully utilizes the BEATs model to uniformly encode the frequency and time domain structures of audio signals. It combines short-time Fourier transform, patch partitioning, and Transformer coding mechanisms to extract high-resolution audio semantic representations, thereby supplementing the expressive capabilities of information such as background sound, language content, and abnormal sounds in videos. To address the inconsistencies between visual and audio modalities in terms of time synchronization, semantic granularity, and dimensional structure, this invention introduces a multilayer perceptron module to project and unify the dimensions of features from each modality. Based on the fusion module, it achieves feature alignment and deep fusion, effectively improving the accuracy of information collaboration and semantic complementarity between modalities.

[0022] Finally, based on the fusion representation, this invention further introduces a large language model as a language decoder to perform semantic decoding on multimodal feature sequences. It supports various video understanding output methods such as event description, behavior recognition, semantic label generation, and natural language question answering. This breaks through the limitation of traditional models that can only generate classification labels or single descriptions, significantly improving the model's semantic expression ability and user-friendliness. At the same time, since the decoder adopts a hierarchical Transformer structure, it has the ability to model long sequences and preserve context, resulting in better performance in terms of logical coherence and semantic consistency of the generated results.

[0023] In summary, through the fusion and optimization of multi-level model structures, this invention achieves an integrated processing flow for spatiotemporal joint modeling of visual and audio modalities, unified feature space alignment, multimodal semantic expression, and natural language generation. It overcomes the problems of modality separation, structural fragmentation, and limited expression in existing methods, significantly improves the application capability of video understanding systems in complex semantic scenarios, and has good practicality, accuracy, and scalability. Attached Figure Description

[0024] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0025] Figure 1 The flowchart shows a spatiotemporal multimodal video understanding method based on VideoLLeMA2 proposed in this invention. Figure 2 This is a schematic diagram of the VideoLLeMA2 model, which is the basis for the spatiotemporal multimodal video understanding method proposed in this invention. Figure 3 This is a schematic diagram of the visual modality modeling process of a spatiotemporal multimodal video understanding method based on VideoLLeMA2 proposed in this invention. Detailed Implementation

[0026] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0027] refer to Figure 1-3 A spatiotemporal multimodal video understanding method based on VideoLLeMA2 includes the following steps: S1. Obtain the raw video data, perform image frame decoding on the raw video data, and extract the image frame sequence and audio stream sequence; S2. Perform image preprocessing and audio preprocessing on the image frame sequence and audio stream sequence respectively to generate the image input sequence and audio input sequence; S3. Input the image input sequence into the visual encoder, and use the CLIP ViT-Large-Patch14-336 model to encode the image frames and generate initial visual features; S4. Input the initial visual features into the spatiotemporal convolution connector, use 3D convolution to extract the temporal features between image frames, and use the RegStage module to perform semantic enhancement on the temporal features to generate visual modal features; S5. Input the audio input sequence into the audio encoder, use the BEATs model to model the audio in the frequency domain and time domain, and output the audio modal features; S6. Input the audio modal features and visual modal features into the multilayer perceptron, and perform feature alignment by projecting the features onto a unified feature dimension space. S7. Based on the multimodal fusion module, the aligned features are fused to obtain multimodal features; S8. Input the multimodal features into the language decoder, use a large language model for semantic decoding, and generate video content understanding results in natural language form.

[0028] This invention constructs a unified video understanding process that integrates a visual encoder, an audio encoder, a spatiotemporal modeling module, a multimodal alignment and fusion module, and a language decoder. This enables end-to-end modeling from raw video data to natural language output, improving the accuracy and expressive power of video semantic understanding and providing excellent cross-modal collaboration and task generalization capabilities.

[0029] In this embodiment, the image preprocessing includes image denoising, size normalization, inter-frame synchronization, and normalization.

[0030] This invention effectively improves the consistency and modeling stability of image frame input by introducing image preprocessing steps such as image denoising, size standardization, inter-frame synchronization and normalization, reduces the impact of image noise interference on the subsequent visual encoder, and enhances the model's ability to capture details and changes in the video.

[0031] In this embodiment, the audio preprocessing includes amplitude normalization, silence segment removal, boundary clipping, and smoothing.

[0032] This invention employs an audio preprocessing method that includes amplitude normalization, silence segment removal, boundary clipping, and smoothing. This method helps improve the quality and effectiveness of audio features, removes meaningless information and boundary interference, enhances the audio encoder's ability to extract time-frequency variations and semantic cues, and provides a more accurate speech signal foundation for multimodal understanding.

[0033] In this embodiment, S3 specifically includes: S31. The image input sequence is input into the visual encoder of the VideoLLaMA2 model. The visual encoder uses the CLIP ViT-Large-Patch14-336 model. The VideoLLaMA model is a video understanding model based on a large language model, which integrates visual and audio information and performs spatiotemporal modeling and semantic alignment through a unified Transformer framework. The VideoLLaMA model combines a visual encoder, an audio encoder, a spatiotemporal convolutional connector, a multilayer perceptron, a multimodal fusion module, and a language decoder. The CLIP ViT-Large-Patch14-336 model is an image encoding model based on a visual Transformer architecture. It divides the input image into fixed-size image blocks and performs linear projection, and extracts global semantic features through a multilayer self-attention mechanism. Each image block is 14×14 in size, the input image resolution is 336×336, and the model contains 24 Transformer encoder layers to generate image feature representations with a dimension of 1024. S32. Perform a segmentation operation on each frame of the image input sequence, dividing it into... Non-overlapping image patches; S33. After expanding each image patch into a vector, perform a linear transformation using a linear mapping weight matrix and a bias vector: ; in, Indicates the first The first image frame Feature vectors of image patches Represents the linear mapping weight matrix. This represents the bias vector. Representing feature dimension, Indicates the first The first image frame Image blocks, This means flattening the image patch into a vector. Represents the set of real numbers; S34. Add two-dimensional positional coding and frame sequence temporal coding to the feature vector of the image patch to obtain the embedding vector, and concatenate the embedding vectors of all image patches in each frame to obtain the image embedding representation. ,in Represents the vector transpose operation: ; in, This represents the embedding vector after adding two-dimensional positional coding and frame sequence coding. Indicates the first The first frame Feature vectors of image patches Represents the two-dimensional location code value. Indicates the frame sequence time-coded value; Define two-dimensional position encoding: ; in, Indicates the first In the dimension of the first Frame sequence time coding of each image block Indicates the sequence number of the image patch after it has been flattened in the image. Indicates the dimension of the encoding; Define frame sequence time coding: ; in, Indicates the first In the dimension of the first Two-dimensional positional encoding values ​​of each image frame, Indicates the dimension of the encoding; S35. Input the embedding representation matrix into the Transformer encoding module to obtain the encoded output of each frame of the image, and perform average pooling operation on the encoded result of each frame to extract the overall semantic feature vector of the image: ; in, Indicates the first The overall semantic feature vector of an image frame This indicates the number of blocks in each frame of the image. Indicates the first The first image frame The output of the Transformer encoding module for each image patch; S36. Concatenate all the overall semantic feature vectors of the images in chronological order to form the initial visual features.

[0034] This invention encodes image frames based on the CLIP ViT-Large-Patch14-336 model. By combining image block partitioning, linear transformation, position and frame order encoding, Transformer extraction and average pooling strategies, it comprehensively extracts spatial and semantic information from image frames, improving the completeness of visual feature expression and the ability to model temporal context.

[0035] In this embodiment, S4 specifically includes: S41. Input the initial visual features into the spatiotemporal convolution connector, and construct a three-dimensional tensor representation based on the initial visual features. The number of channels in the three-dimensional tensor representation is the feature dimension of the initial visual features, the time step is the number of image frames of the initial visual features, and the spatial dimension is a single point. The spatiotemporal convolution connector is a feature enhancement module that models the temporal dynamic changes and spatial local structures of visual features. It consists of a three-dimensional convolution and a RegStage module. The three-dimensional convolution is used to perform temporal and spatial joint modeling on the input initial visual features, extracting the inter-frame temporal dependencies and spatial transformation patterns. The RegStage module performs nonlinear transformation and structural optimization on the output of the three-dimensional convolution, thereby improving the semantic expressiveness and temporal continuity and stability of the features. S42. Apply a three-dimensional convolution operation to the three-dimensional tensor representation, perform convolution processing along the time axis on the three-dimensional tensor, and generate a three-dimensional convolution output tensor. S43. Divide the three-dimensional convolution output tensor according to the time dimension to construct a time step feature sequence. Each time step feature of the time step feature sequence is a one-dimensional vector with the dimension being the number of channels of the three-dimensional convolution output. S44. Input the time step feature sequence into the RegStage module. The RegStage module consists of two linear mapping layers. The first linear mapping layer projects the time step feature sequence to the intermediate hidden layer dimension and adds a first bias parameter. After the activation function transformation, the second linear mapping layer restores the original output dimension and adds a second bias parameter to obtain the enhanced features of each time step. S45. Concatenate the enhanced features of all time steps in chronological order into a two-dimensional feature matrix to form a visual modality feature. The time dimension of the visual modality feature is the number of time steps of the three-dimensional convolution output, and the feature dimension is the number of channels of the three-dimensional convolution output.

[0036] This invention effectively realizes temporal modeling and spatial local structure extraction between image frames by inputting initial visual features into a spatiotemporal convolution connector composed of 3D convolution and RegStage modules. This enhances the dynamic perception capability and semantic continuity of visual features and improves the system's recognition accuracy for complex actions and event changes.

[0037] In this embodiment, the BEATs model includes an audio processing layer, a patch layer, a position encoding layer, a Transformer encoder layer, and an output projection layer.

[0038] This invention clarifies that the BEATs model structure includes an audio processing layer, a patch layer, a positional encoding layer, a Transformer encoder layer, and an output projection layer. This facilitates the modularization of the audio modeling process, improves the controllability and scalability of feature extraction, and enhances the overall system's depth of audio semantic analysis.

[0039] In this embodiment, S5 specifically includes: S51. Represent the audio input sequence as a sequence of length... A one-dimensional audio signal sequence, where each audio signal represents the audio amplitude at a sampling point; S52. Perform frame segmentation on the one-dimensional audio signal sequence. S53. Apply short-time Fourier transform to each frame of audio signal sequence to obtain the spectral representation: ; in, The spectrum is represented to describe the first... Frame number Complex spectral values ​​at each frequency point Indicates the length of each frame. Indicates the first Frame number The amplitude of each sampling point Represents the window function. Represents the imaginary unit. Represents natural parameters; S54. Stack all frame spectrum representations to form a two-dimensional spectrogram, and divide it according to the time direction into... One non-overlapping spectrogram; S55. Flatten each spectrogram and perform a linear mapping, while adding time position encoding to generate an embedded representation; S56. Input the embedded representation into the Transformer encoder of the BEATs model and perform attention mechanism and feedforward network operations. S57. Extract all frame features from the last layer output of the Transformer encoder and concatenate them into audio modal features.

[0040] This invention refines the audio modeling process of BEATs, clarifying the short-time Fourier transform, patch partitioning, time position encoding, and Transformer encoding processes, thereby improving the time-frequency structure modeling effect of audio signals and enabling audio semantics to be expressed in the final features with more fine granularity.

[0041] In this embodiment, S8 specifically includes: S81. The multimodal feature sequence is used as the initial input to the large language model and initialized as the input representation of the 0th layer of the large language model. S82. In the large language model, a decoder structure consisting of multiple stacked layers is constructed. Each layer receives the output of the previous layer as the current input and completes hierarchical propagation in sequence to gradually extract semantic features. S83. Within each decoder layer, multi-head attention mechanism and feedforward neural network processing are executed sequentially to capture long-term dependencies and cross-modal semantic information in the time series. S84. Output the token representation sequence in the last decoder layer, representing the embedding form of natural language; S85. Perform linear mapping and normalization on the token representation sequence to convert it into a natural language vocabulary index, thereby obtaining a text token sequence; S86. Decode the text token sequence into a video understanding result in natural language form, wherein the video understanding result includes event description, behavior category, semantic label, or question-and-answer text.

[0042] This invention constructs a multi-layer decoder structure and combines a multi-head attention mechanism with feedforward network processing to achieve deep semantic modeling and abstract expression of multimodal features, thereby improving the ability of large language models to capture the semantics of video content and the contextual consistency of language output.

[0043] In this embodiment, the decoder is a neural network structure composed of multiple layers of stacked Transformer units.

[0044] This invention enhances the ability to preserve context and model long-distance dependencies during language generation by explicitly adopting a multi-layered Transformer stacked structure for the decoder, effectively improving the accuracy and coherence of the decoder in long text generation and multi-turn expression.

[0045] In this embodiment, the token representation sequence is a vector combination of tokens, and the token represents the smallest semantic unit of text.

[0046] This invention, by explicitly defining the token representation sequence as a vector combination of tokens, enables the model to process natural language in a structured form during the language output stage, thereby improving the flexibility of lexical representation and the precision of language output during language decoding.

[0047] Example 1: To verify the feasibility of this invention in practice, it was applied to the video review and semantic annotation tasks of a large-scale educational short video platform. The platform has an average daily upload volume of over 8,000 videos, covering multiple dimensions such as course explanations, student interactions, and campus life. It urgently needs an efficient video content understanding system to automatically identify core events, semantic themes, character behaviors, and audio environments in videos, thereby achieving automatic video classification, summary generation, and content security review.

[0048] Taking the "student lab videos" collected by the platform as a typical application scenario, the videos were filmed in a biology lab at a university in Nanjing, Jiangsu Province. They were simultaneously captured by a camera and microphone, containing both visual and audio modal information. Traditional video review systems mainly rely on image frame classification models and keyword recognition algorithms to process this type of data, but they have significant shortcomings in the following aspects: First, they cannot effectively determine whether students are operating according to regulations during the experiment; second, they cannot identify whether the experimental instructors are present and providing explanations; third, the audio modality is only used to detect whether there is silence, without semantic extraction of the speech content and background noise; and fourth, the classification labels are too coarse, failing to generate high-quality structured or natural language descriptions.

[0049] This invention is deployed in the cloud-based multimodal processing module of the platform. First, video data of a laboratory scene is acquired through a video acquisition device, and the video is decoded into image frame sequences and audio streams are extracted. The image frame sequences are denoised, sized, and synchronized between frames by an image preprocessing module. The audio stream is normalized in amplitude, silence is removed, and the boundaries are cropped. Subsequently, the image frame sequences are input into the CLIP ViT-Large-Patch14-336 visual encoder, where each frame is divided into image blocks and linearly mapped, and two-dimensional positional encoding and frame sequence temporal encoding are added to extract global visual features. The visual features of all frames are modeled into inter-frame temporal relationships through three-dimensional convolution and optimized by the RegStage module to form a spatiotemporal feature representation of the visual modality.

[0050] The audio signal is processed by a BEATs encoder, first generating a spectrogram through a short-time Fourier transform, then segmenting it into time patches and adding positional encoding. This is then input into a multi-layer Transformer structure to extract audio semantic features. After the visual and audio features are mapped to the same dimension via a multi-layer perceptron module, they enter a fusion module for feature alignment and fusion operations. Finally, the fused features are input into a large language model to generate video semantic understanding results in natural language format.

[0051] Table 1. Experimental Comparison of the Invention Method and Traditional Video Understanding Methods

[0052] Regarding the accuracy of video behavior recognition, traditional methods are limited by static image feature extraction and coarse-grained classification, with an accuracy of only 72.5%. However, this invention effectively captures dynamic behavior features between image frames by combining CLIP visual encoder and three-dimensional convolutional spatiotemporal modeling, improving the accuracy to 91.3%, an increase of 18.8%, which significantly enhances the system's ability to recognize student operation behaviors and action processes.

[0053] Regarding the coverage of speech content comprehension, this invention introduces the BEATs audio encoder and performs time-frequency modeling on the audio signal, enabling the system to recognize key information such as teacher explanations and voice commands with higher quality, achieving a coverage rate of 88.9%, which is 40.7% higher than the 48.2% of the traditional method. This improvement allows the system to not only recognize "what was done" in the image, but also understand "what was said", laying the foundation for complete semantic expression.

[0054] In terms of background noise recognition and semantic modeling, traditional methods cannot model audio signals such as motor noise and ambient sound. However, this invention effectively supports low-frequency noise modeling and semantic association processing through the BEATs model, enabling the video semantics to include natural language expressions such as "the background is accompanied by continuous motor running sound", thereby greatly improving the realism and subtlety of video understanding.

[0055] In terms of the language quality of the multimodal generated text, the BLEU-4 score improved from 25.4 to 43.7, and the ROUGE-L score improved from 38.9 to 67.2, representing increases of 18.3 and 28.3 points respectively. This indicates that the descriptive text generated by this invention is closer to manually annotated text in terms of language structure, key phrase restoration, and contextual consistency. In addition, the subjective score for the naturalness of the video summary improved from 3.1 points in the traditional method to 4.6 points, which also verifies the improvement in language output effect from the user's perspective.

[0056] In terms of efficiency and deployability, this invention achieves a 93.4% automation rate for video review, a 36.7% improvement compared to the 56.7% of traditional systems, while reducing the need for manual intervention from 43.2% to 12.5%. This means that the system can significantly reduce the cost of manual intervention and is suitable for large-scale concurrent processing scenarios. Furthermore, the average processing time for a single video is reduced from 5.4 seconds to 3.7 seconds, and the number of videos that the system can process per minute is increased from 190 to 310, resulting in a 63.2% improvement in processing efficiency.

[0057] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A spatiotemporal multimodal video understanding method based on VideoLLeMA2, characterized in that, Includes the following steps: S1. Obtain the raw video data, perform image frame decoding on the raw video data, and extract the image frame sequence and audio stream sequence; S2. Perform image preprocessing and audio preprocessing on the image frame sequence and audio stream sequence respectively to generate the image input sequence and audio input sequence; S3. Input the image input sequence into the visual encoder, and use the CLIP ViT-Large-Patch14-336 model to encode the image frames and generate initial visual features; S4. Input the initial visual features into the spatiotemporal convolution connector, use 3D convolution to extract the temporal features between image frames, and use the RegStage module to perform semantic enhancement on the temporal features to generate visual modal features; S5. Input the audio input sequence into the audio encoder, use the BEATs model to model the audio in the frequency domain and time domain, and output the audio modal features; S6. Input the audio modal features and visual modal features into the multilayer perceptron, and perform feature alignment by projecting the features onto a unified feature dimension space. S7. Based on the multimodal fusion module, the aligned features are fused to obtain multimodal features; S8. Input the multimodal features into the language decoder, use a large language model for semantic decoding, and generate video content understanding results in natural language form.

2. The spatiotemporal multimodal video understanding method based on VideoLLeMA2 according to claim 1, characterized in that, The image preprocessing includes image denoising, size normalization, inter-frame synchronization, and normalization.

3. The spatiotemporal multimodal video understanding method based on VideoLLeMA2 according to claim 1, characterized in that, The audio preprocessing includes amplitude normalization, silence removal, boundary clipping, and smoothing.

4. The spatiotemporal multimodal video understanding method based on VideoLLeMA2 according to claim 1, characterized in that, S3 specifically includes: S31. Input the image input sequence into the visual encoder in the VideoLLaMA2 model, wherein the visual encoder uses the CLIP ViT-Large-Patch14-336 model. S32. Perform a segmentation operation on each frame of the image input sequence, dividing it into... Non-overlapping image patches; S33. After expanding each image patch into a vector, perform a linear transformation using a linear mapping weight matrix and a bias vector: ； in, Indicates the first The first image frame Feature vectors of image patches Represents the linear mapping weight matrix. This represents the bias vector. Representing feature dimension, Indicates the first The first image frame Image blocks, This means flattening the image patch into a vector. Represents the set of real numbers; S34. Add two-dimensional positional coding and frame sequence temporal coding to the feature vector of the image patch to obtain the embedding vector, and concatenate the embedding vectors of all image patches in each frame to obtain the image embedding representation. ,in Represents the vector transpose operation: ； in, This represents the embedding vector after adding two-dimensional positional coding and frame sequence coding. Indicates the first The first frame Feature vectors of image patches Represents the two-dimensional location code value. Indicates the frame sequence time-coded value; S35. Input the embedding representation matrix into the Transformer encoding module to obtain the encoded output of each frame of the image, and perform average pooling operation on the encoding result of each frame to extract the overall semantic feature vector of the image. S36. Concatenate all the overall semantic feature vectors of the images in chronological order to form the initial visual features.

5. The spatiotemporal multimodal video understanding method based on VideoLLeMA2 according to claim 1, characterized in that, S4 specifically includes: S41. Input the initial visual features into the spatiotemporal convolutional connector, and construct a three-dimensional tensor representation based on the initial visual features. The number of channels in the three-dimensional tensor representation is the feature dimension of the initial visual features, the time step is the number of image frames of the initial visual features, and the spatial dimension is a single point. S42. Apply a three-dimensional convolution operation to the three-dimensional tensor representation, perform convolution processing along the time axis on the three-dimensional tensor, and generate a three-dimensional convolution output tensor. S43. Divide the three-dimensional convolution output tensor according to the time dimension to construct a time step feature sequence. Each time step feature of the time step feature sequence is a one-dimensional vector with the dimension being the number of channels of the three-dimensional convolution output. S44. Input the time step feature sequence into the RegStage module. The RegStage module consists of two linear mapping layers. The first linear mapping layer projects the time step feature sequence to the intermediate hidden layer dimension and adds a first bias parameter. After the activation function transformation, the second linear mapping layer restores the original output dimension and adds a second bias parameter to obtain the enhanced features of each time step. S45. Concatenate the enhanced features of all time steps in chronological order into a two-dimensional feature matrix to form a visual modality feature. The time dimension of the visual modality feature is the number of time steps of the three-dimensional convolution output, and the feature dimension is the number of channels of the three-dimensional convolution output.

6. The spatiotemporal multimodal video understanding method based on VideoLLeMA2 according to claim 1, characterized in that, The BEATs model includes an audio processing layer, a patch layer, a position encoding layer, a Transformer encoder layer, and an output projection layer.

7. The spatiotemporal multimodal video understanding method based on VideoLLeMA2 according to claim 1, characterized in that, S5 specifically includes: S51. Represent the audio input sequence as a sequence of length... A one-dimensional audio signal sequence, where each audio signal represents the audio amplitude at a sampling point; S52. Perform frame segmentation on the one-dimensional audio signal sequence. S53. Apply short-time Fourier transform to each frame of audio signal sequence to obtain the spectral representation: ； in, The spectrum is represented to describe the first... Frame number Complex spectral values ​​at each frequency point Indicates the length of each frame. Indicates the first Frame number The amplitude of each sampling point Represents the window function. Represents the imaginary unit. Represents natural parameters; S54. Stack all frame spectrum representations to form a two-dimensional spectrogram, and divide it according to the time direction into... One non-overlapping spectrogram; S55. Flatten each spectrogram and perform a linear mapping, while adding time position encoding to generate an embedded representation; S56. Input the embedded representation into the Transformer encoder of the BEATs model and perform attention mechanism and feedforward network operations. S57. Extract all frame features from the last layer output of the Transformer encoder and concatenate them into audio modal features.

8. The spatiotemporal multimodal video understanding method based on VideoLLeMA2 according to claim 1, characterized in that, S8 specifically includes: S81. The multimodal feature sequence is used as the initial input to the large language model and initialized as the input representation of the 0th layer of the large language model. S82. In the large language model, a decoder structure consisting of multiple stacked layers is constructed. Each layer receives the output of the previous layer as the current input and completes hierarchical propagation in sequence to gradually extract semantic features. S83. Within each decoder layer, multi-head attention mechanism and feedforward neural network processing are executed sequentially to capture long-term dependencies and cross-modal semantic information in the time series. S84. Output the token representation sequence in the last decoder layer, representing the embedding form of natural language; S85. Perform linear mapping and normalization on the token representation sequence to convert it into a natural language vocabulary index, thereby obtaining a text token sequence; S86. Decode the text token sequence into a video understanding result in natural language form, wherein the video understanding result includes event description, behavior category, semantic label, or question-and-answer text.

9. The spatiotemporal multimodal video understanding method based on VideoLLeMA2 according to claim 8, characterized in that, The decoder is a neural network structure composed of multiple stacked Transformer units.

10. A spatiotemporal multimodal video understanding method based on VideoLLeMA2 according to claim 8, characterized in that, The token representation sequence is a vector combination of tokens, and the token represents the smallest semantic unit of the text.

Images