A multi-modal fusion combined multi-layer attention video content description method
By combining multimodal fusion with multi-layer attention, the problem of incomplete information extraction in video descriptions is solved, and more accurate text generation is achieved.
Patent Information
- Application Number
- CN202111420124.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-11-26
- Publication Date
- 2025-11-18
- Estimated Expiration
- 2041-11-26
AI Technical Summary
Existing video description methods are incomplete in extracting overall video representation information, resulting in inaccurate text descriptions and a tendency to lose hidden information.
A multimodal fusion combined with multi-layer attention method is adopted. Video frames are converted into JPEG images and audio into WAV files using FFmpeg. Visual features are extracted using ResNet-152 and I3D. Feature extraction and fusion are performed by combining channel attention and self-attention mechanisms. LSTM is used for encoding and decoding to generate descriptions.
It improves the accuracy of video content description, effectively reduces information loss in the feature encoding stage, and generates more accurate text descriptions.
Smart Images
Figure CN115661697B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the field of computer vision and natural language processing, and in particular to a video content description method combining multi-modal fusion and multi-layer attention. BACKGROUND
[0002] With the rapid development of mobile Internet and the rapid popularization of intelligent devices, the human informationization process has entered a new stage. As a new type of user-generated content, short videos have been widely used on various social platforms. How to obtain the content of short videos and represent it in text form for the analysis and understanding of video content. The present application has broad application prospects, such as answering image questions, applying images and video search, and assisting visually impaired patients in understanding media content.
[0003] Most existing video description tasks use an encoding-decoding framework as the main description framework. The encoder is usually composed of a convolutional neural network (CNN), and the decoder is composed of a long short-term memory neural network (LSTM). This method directly uses the final state of the encoder as the input of the decoder, which can achieve partial video description tasks, but the extraction of overall video representation information is not perfect, and the intermediate hidden information is easily lost during the video feature encoding stage, resulting in inaccurate text description. SUMMARY
[0004] To overcome the shortcomings of the prior art, the present application provides a video content description method combining multi-modal fusion and multi-layer attention, comprising the steps of
[0005] S1: Video multi-modal data extraction, using FFmpeg to convert each frame of video in the data set into a single JPEG image; the audio information extracted from the video is stored as a wav format audio file;
[0006] S2: Feature extraction and preprocessing, using a ResNet-152 network combined with a channel attention mechanism to extract the single-frame modal static features of the converted single JPEG image; using a dual-stream inflated 3D convolution network I3D (Inflated 3DConvNet, I3D) to extract the single modal motion timing features of the continuous frames obtained by converting the video using FFmpeg; according to the Mel frequency cepstral coefficient MFCC (Mel Frequency Cepstral Coefficient, MFCC), the audio signal carried by the audio file is extracted to obtain the audio modal MFCC feature;
[0007] S3: feature embedding, embedding the single-frame modal static features, single-modal motion time sequence features and audio modal MFCC features into an embedding layer neural network designed using a self-attention mechanism to learn single-modal feature parameters;
[0008] S4: feature fusion, performing multi-modal feature fusion on the single-frame modal static features, single-modal motion time sequence features and audio modal MFCC features using a collaborative representation structure to obtain fused features;
[0009] S5: feature encoding, encoding the fused features using two layers of LSTM;
[0010] S6: feature decoding, decoding the encoded feature vector using a multi-layer LSTM network to generate a description corresponding to the video.
[0011] Further, the feature extraction and preprocessing of step S2 includes steps
[0012] S21: using the Squeeze operation of the channel attention SE module to compress the single-frame modal convolutional features extracted by the ResNet-152 network along the spatial dimension to form a real number Z with a global receptive field C , as shown in formula (1),
[0013]
[0014] wherein Z C represents the compressed result, H represents the height, and W represents the width;
[0015] S22: using an Excitation module constructed using two fully connected FC structures to capture channel dependence of the information aggregated in the Squeeze operation, reduce the model complexity and improve the generalization ability of the model, wherein the first FC layer serves to reduce the dimension, compressing C channels into channels, and the reduction coefficient r is a hyperparameter; the second FC layer is used to restore the original dimension of the feature map, and finally obtains a weight coefficient S, as shown in formula (2),
[0016] S=F ex (z,w)=σ(g(z,w))=σ(w2δ(w1z))) (2)
[0017] wherein F ex function represents the entire Excitation operation, sigma represents the sigmoid function, and delta represents the Relu function,
[0018] S23: Reweight operation is performed on the weight coefficient S, which is weighted to the feature map U channel by channel, to complete the reweighting of the original feature in the feature map channel dimension, to obtain the final attention feature The calculation is shown in formula (3),
[0019]
[0020] Wherein, F scale represents the channel multiplication between the attention weight s c and the feature map u c ;
[0021] S24: Each video in the data set is preprocessed into a fixed frame of 224*224, and is sent into the I3D model pre-trained by ImageNet and Kinetics to extract single-mode motion timing features;
[0022] S25: The audio signal in the audio file is preprocessed by pre-emphasis, framing and windowing, and the single-frame signal after framing is subjected to discrete Fourier transform to obtain frequency domain data, as shown in formula (4),
[0023]
[0024] Wherein, x(n) is the input speech signal, and N represents the number of Fourier transform points;
[0025] S26: The frequency domain data is filtered by w Mel frequency filters to extract the frequency spectrum, Mel filter bank and frequency envelope, and the frequency domain response H w (k) is shown in formula (5),
[0026]
[0027] Wherein, ∑H W (k)=1; f(w) represents the filter center frequency;
[0028] S27: The logarithmic energy s(m) output by each filter bank is calculated as shown in formula (6),
[0029]
[0030] Wherein, s(m) is the logarithmic energy, and H m (k) is the frequency response of the triangular filter;
[0031] S28: The MFCC coefficient of the logarithmic energy is calculated by Discrete Cosine Transform (DCT), and the calculation is shown in formula (6),
[0032]
[0033] Wherein, w indicates the wth Mel filter, i indicates the ith frame, n is the spectrum line obtained after DCT, and the audio modal MFCC feature is extracted.
[0034] Further, the step S3 comprises the steps of
[0035] S31: after the single-frame modal static features, the single-modal motion time sequence features and the audio modal MFCC features of the video are normalized, the long short-term memory network is used for context feature processing;
[0036] S32: the single-modal features are calculated by using the self-attention mechanism, as shown in formula (7),
[0037]
[0038] Wherein, Q, K and V are query vector matrix, key vector matrix and value vector matrix respectively,
[0039] S33: the single-frame modal static features, the single-modal motion time sequence features and the audio modal MFCC features extracted by the self-attention mechanism are sent into the long short-term memory network for dimension reduction processing.
[0040] Further, the step S6 of generating the description corresponding to the video comprises the steps of: calculating 20 groups of prediction probability values corresponding to each video, and taking the word corresponding to the maximum value in each group of probability values to generate the final description text.
[0041] The beneficial effects of the present application are that, compared with the prior art, the present application overcomes the problem that the prior art is not perfect in extracting the overall representation information of the video, which is easy to cause the loss of hidden information in the video feature coding stage, and leads to the low accuracy of the generated description text, and effectively improves the accuracy of the video content description. BRIEF DESCRIPTION OF DRAWINGS
[0042] Figure 1 It is a multi-modal fusion combined with multi-layer attention video content description method flow chart. DETAILED DESCRIPTION
[0043] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.
[0044] Please refer to the abstract drawings andFigure 1 The technical solution idea of the present application is: first, using FFmpeg to convert each frame of video in the data set into a single JPEG image, and storing the audio information of each video as an audio file in wav format; second, using a ResNet-152 network combined with a channel attention mechanism to extract the static features of the video, using a dual-stream dilated 3D convolution network I3D to extract the motion timing features of the video, and extracting the audio signal features carried by the video according to the Mel frequency cepstrum coefficient MFCC; third, embedding the single-modal features into a network structure designed using a self-attention mechanism, learning the single-modal feature parameters, and fusing the modal features using a collaborative representation structure; finally, using two layers of LSTM to encode the fused features, and using a multi-layer LSTM network to decode the encoded feature vectors to generate the description corresponding to the video.
[0045] A video content description method combining multi-modal fusion and multi-layer attention, comprising the steps of
[0046] S1: video multi-modal data extraction, using FFmpeg to convert each frame of video in the data set into a single JPEG image; storing the extracted audio information in the video as an audio file in wav format;
[0047] S2: feature extraction and preprocessing, using a ResNet-152 network combined with a channel attention mechanism to extract the single-modal static features of the converted single JPEG image; using a dual-stream dilated 3D convolution network I3D to extract the single-modal motion timing features of the converted multiple JPEG images; extracting the audio signal carried by the audio file according to the Mel frequency cepstrum coefficient MFCC to obtain the single-modal MFCC feature;
[0048] S3: feature embedding, embedding the single-modal static features, single-modal motion timing features and single-modal MFCC features into a neural network designed using a self-attention mechanism to learn the single-modal feature parameters;
[0049] S4: feature fusion, using a collaborative representation structure to perform multi-modal feature fusion on the single-modal static features, single-modal motion timing features and single-modal MFCC features to obtain fused features;
[0050] S5: feature encoding, using two layers of LSTM to encode the fused features;
[0051] S6: feature decoding, using a multi-layer LSTM network to decode the encoded feature vectors to generate the description corresponding to the video.
[0052] The feature extraction and preprocessing of step S2 comprises the steps of
[0053] S21: The single-frame convolutional features extracted by the ResNet-152 network are compressed along the spatial dimension by the Squeeze operation of the channel attention SE module to form a real number Z with a global receptive field C , as shown in equation (1),
[0054]
[0055] where Z C represents the compressed result, H represents the height, and W represents the width;
[0056] S22: The Excitation module constructed by two fully connected FC structures captures the channel dependence of the information aggregated in the Squeeze operation, reduces the model complexity, and improves the generalization ability of the model, where the first FC layer serves as a dimension reduction function, compressing C channels into channels, and the dimension reduction coefficient r is a hyperparameter; the second FC layer is used to restore the original dimension of the feature map, and finally a weight coefficient S is obtained, as shown in equation (2),
[0057] S = F ex (z, w) = σ (g (z, w)) = σ (w2δ (w1z)) ) (2)
[0058] where F ex function represents the entire Excitation operation, σ represents the sigmoid function, and δ represents the Relu function,
[0059] S23: The weight coefficient S is reweighted and weighted to the feature map U channel by channel to complete the reweighting of the original features in the channel dimension of the feature map, and the final attention feature is obtained , as shown in equation (3),
[0060]
[0061] where F scale represents the channel-wise multiplication between the attention weight s c and the feature map u c ;
[0062] S24: Each video in the data set is preprocessed into a fixed frame of 224*224 and sent into the I3D model pre-trained by ImageNet and Kinetics to extract single-modal motion timing features;
[0063] S25: Pre-emphasis, frame and window pre-processing operations are performed on the audio signal in the audio file, and discrete Fourier transform is performed on the single frame signal after frame, to obtain frequency domain data, as shown in formula (4),
[0064]
[0065] Wherein, x(n) is the input voice signal, N represents the number of Fourier transform points;
[0066] S26: The frequency domain data is filtered by w Mel frequency filters to extract the spectrum, Mel filter set and frequency envelope, and the frequency domain response H w (k) as shown in formula (5),
[0067]
[0068] Wherein, ∑H W (k)=1;f(w) represents the filter center frequency;
[0069] S27: The logarithmic energy s(m) output by each filter set is calculated as shown in formula (6),
[0070]
[0071] Wherein, s(m) is the logarithmic energy, H m (k) is the frequency response of the triangular filter;
[0072] S28: The MFCC coefficient of the logarithmic energy is calculated by using Discrete Cosine Transform (DCT), and the calculation is as shown in formula (6),
[0073]
[0074] Wherein, w refers to the wth Mel filter, i refers to the ith frame, n is the spectrum line obtained after DCT, and the audio modal MFCC feature is extracted.
[0075] Further, step S3 comprises the steps of
[0076] S31: After the single frame modal static features, single modal motion time sequence features and audio modal MFCC features of the video are normalized, the long short-term memory network is used for context feature processing;
[0077] S32: The single modal features are calculated by using self-attention mechanism, as shown in formula (7),
[0078]
[0079] Wherein Q, K, V are query vector matrix, key vector matrix and value vector matrix respectively,
[0080] S33: The single-frame modal static features, the single-modal motion time sequence features and the audio modal MFCC features extracted through the self-attention mechanism are sent into a long short-term memory network for dimension reduction processing.
[0081] Further, the step S6 of generating the description corresponding to the video comprises the steps of: calculating 20 groups of prediction probability values corresponding to each video, and taking the word corresponding to the maximum value in each group of probability values to generate the final description text.
[0082] The above describes the preferred embodiments of the present application. It should be noted that, for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, which are also considered within the scope of protection of the present application.
Claims
1. A video content description method combining multimodal fusion and multi-layer attention, characterized in that, Including steps S1: Video multimodal data extraction, using FFmpeg to convert each frame of the video in the dataset into a single JPEG image; storing the extracted audio information from the video as a WAV format audio file; S2: Feature extraction and preprocessing: The ResNet-152 network and channel attention mechanism are used to extract the single-modal static features of the transformed single JPEG image. Single-modal motion temporal features of multiple JPEG images obtained by dual-stream dilated 3D convolutional network (I3D) are extracted; audio signals carried by audio files are extracted based on Mel-frequency cepstral coefficients (MFCCs) to obtain single-modal MFCC features. S3: Feature embedding, embedding the single-modal static features, single-modal motion temporal features, and single-modal MFCC features into an embedding layer neural network designed using a self-attention mechanism, to learn the single-modal feature parameters, including the following steps: S31: After normalizing the single-frame modal static features, single-modal motion temporal features, and audio modal MFCC features of the video, a long short-term memory network is used for contextual feature processing. S32: The single-modal features are calculated using a self-attention mechanism as shown in formula (7). Where Q, K, and V are the query vector matrix, key vector matrix, and value vector matrix, respectively. S33: The single-frame modal static features, single-modal motion temporal features, and audio modal MFCC features extracted through the self-attention mechanism are fed into the long short-term memory network for dimensionality reduction. S4: Feature fusion. The collaborative representation structure is used to fuse the single-modal static features, single-modal motion temporal features and single-modal MFCC features from the previous step through the embedding layer structure to obtain fused features. S5: Feature encoding, using two layers of LSTM to encode the fused features; S6: Feature Decoding. The encoded feature vector is decoded using a multi-layer LSTM network to generate a description corresponding to the video. This includes the following steps: calculating 20 sets of predicted probability values for each video and extracting the word corresponding to the maximum value in each set of probability values to generate the final description text.
2. The video content description method combining multimodal fusion and multi-layer attention as described in claim 1, characterized in that, The feature extraction and preprocessing described in step S2 includes the following steps: S21: The single-frame convolutional features extracted by the ResNet-152 network are processed along the spatial dimension using the Squeeze operation of the channel attention SE module. U Compress the data to form a real number with a global receptive field. The calculation is shown in formula (1). in, This represents the result after compression. Represents height, Represents width; S22: Utilizing two fully connected layers FC The constructed Excitation module relies on the information capture channels gathered in the Squeeze operation, reducing model complexity and improving the model's generalization ability. The first... FC Layers play a role in dimensionality reduction, C Each channel was compressed into Each channel, dimensionality reduction coefficient r It's a hyperparameter; the second one. FC The layer is used to restore the original dimensions of the feature map, ultimately yielding a weight coefficient. S The calculation is shown in formula (2). in, The function represents the entire Excitation operation. This represents the sigmoid function. Represents the ReLU function. , ; S23: Weighting coefficients S Perform a reweighting operation, weighting the feature map channel by channel. U The above steps involve recalibrating the original features along the feature map channel dimension to obtain the final attention features. The calculation is shown in formula (3). in, Represents attention weights and feature map Channel-based multiplication between; S24: Preprocess the frames extracted from each video in the dataset into... 224*224 The fixed frames are fed into the I3D model pre-trained with ImageNet and Kinetics to extract single-modal motion temporal features; S25: Perform pre-emphasis, framing, and windowing preprocessing operations on the audio signal in the audio file, and perform discrete Fourier transform on the single-frame signal after framing to obtain frequency domain data, as shown in formula (4). in, The input voice signal. S26: Indicates the number of points in the Fourier transform; S26: Transforms the frequency domain data through... w The filter is filtered using a Mel frequency filter, and the spectrum, Mel filter bank, and frequency envelope are extracted. The frequency domain response of the filter is then determined. As shown in formula (5), in,: ; S27: The center frequency of the filter; S28: The logarithmic energy of the output of each filter bank. The calculation is shown in formula (6). in, Logarithmic energy, This is the frequency response of the triangular filter; S28: The MFCC coefficients of the logarithmic energy are calculated using the Discrete Cosine Transform (DCT), as shown in Equation (6). in, w Refers to the first w Mel filters, Refers to the first frame, n For the spectral lines obtained after DCT, extract the audio modal MFCC features.