Inter-modal joint encoding method, device and equipment based on transformer

Abstract

This invention discloses a Transformer-based intermodal joint encoding method, apparatus, and device, relating to the field of multimodal fusion technology. The technical problem it addresses is "how to fuse intermodal information to achieve better sentiment classification results." The method includes: acquiring a video to be analyzed containing multimodal information; extracting text features, audio features, and video image features from the video; unifying the text features, audio features, and video image features to the same dimension based on fully connected layers and LSTM layers; performing multimodal attention joint encoding on the text features, audio features, and video image features based on a Transformer model; and processing and weighting the representation features based on a multilayer perceptron classification model to obtain the classification result of the video to be analyzed. This method can simultaneously perform joint attention encoding on different modalities using a Transformer model, achieving better classification results.

Description

Technical Field

[0001] This invention relates to the field of multimodal fusion technology. Background Technology

[0002] Numerous studies have focused on multimodal analysis of videos, particularly in recent years with the advancements in deep learning, leading to significant progress in this area. Videos typically exist in three modalities: text, audio, and video. Text-based analysis includes the accompanying text, subtitles, and dialogue; audio primarily comprises the auditory information, including dialogue and background music; and video mainly consists of the visual information.

[0003] Current multimodal sentiment analysis primarily relies on deep learning techniques to model information within different modalities and interactions between modalities. Intramodal modeling refers to modeling information within a specific modality, independently of other modalities. Intermodal modeling involves modeling information across different modalities, including both synchronous and asynchronous information modeling. For video multimodal analysis tasks, the main challenge lies in finding feature representations within modalities and fusing features between different modalities. See also Figure 1 The foundation of video multimodal analysis is to use appropriate feature extraction mechanisms to extract visual, auditory, and textual features, and then fuse the features extracted from the three different modalities for subsequent analysis.

[0004] For multimodal tasks involving video, a crucial aspect is exploring the information fusion of feature representations across different modalities. Some works primarily rely on early-stage intermodal information fusion, fusing features across different modal dimensions. These features are simply concatenated and used as input to a prediction model to predict sentiment and attitude. The prediction models in these approaches are mostly widely used traditional methods such as Hidden Markov Models, Support Vector Machines, or Conditional Random Fields. Other works address intermodal information fusion at a later stage, designing and training a separate model for each modality, relying on voting and weighting methods to predict sentiment and attitude. These methods often train models separately for each modality, without considering intermodal interactions during model training, and failing to combine the prediction results from multiple models to predict the final outcome.

[0005] In neural network-based multimodal emotion understanding tasks, models that build upon multimodal information and their interactions often exhibit low interpretability, with most neural network-based models remaining a black box in terms of intermodal information fusion. Although many models have achieved success on certain tasks, researchers are still attempting to understand and safely apply these models. Summary of the Invention

[0006] To address the technical problems existing in the prior art, this invention provides a method, apparatus, and device for joint inter-modal encoding based on Transformer, which introduces an interactive attention mechanism. This allows for joint attention encoding of different modalities using the Transformer model simultaneously, resulting in better classification performance.

[0007] A Transformer-based inter-modal joint coding method includes:

[0008] Acquire the video to be analyzed, which contains multimodal information;

[0009] Extract the text features of the video to be analyzed;

[0010] Extract the audio features of the video to be analyzed;

[0011] Extract the video frame features of the video to be analyzed;

[0012] Based on fully connected layers and LSTM layers, the text features, audio features, and video image features are unified into the same dimension;

[0013] Based on the Transformer model, the text features, audio features, and video image features are jointly encoded using multimodal attention to obtain text representation features, audio representation features, and video image representation features.

[0014] Based on the multilayer perceptron classification model, the text representation features, audio representation features, and video image representation features are weighted to obtain the classification result of the video to be analyzed.

[0015] Furthermore, the text features are extracted based on a pre-trained Chinese-BERT-wmm model;

[0016] The Chinese-BERT-wmm model is loaded using a Hugging face Transformer;

[0017] The Chinese-BERT-wmm model employs a full-word masking mechanism during training.

[0018] Furthermore, the R(2+1)D model is used to extract video frame features from the video to be analyzed;

[0019] Extracting video frame features from the video to be analyzed includes:

[0020] Extract the three-dimensional sequence features from the video to be analyzed;

[0021] Flatten the three-dimensional sequence features into two-dimensional sequence features;

[0022] The two-dimensional sequence features are downsampled, and one frame is selected from every 16 frames to extract features. The network output features of the R(2+1)D model in the spatiotemporal pooling layer are selected as the extracted video frame features.

[0023] Furthermore, during the multimodal attention joint encoding process, the text modality is used as the primary modality, and the audio modality and video image modality are modulated and encoded.

[0024] The Transformer model includes three joint coding units, each of which includes a multi-head attention module, a first residual connection normalization module, a feed-forward module, a second residual connection normalization module, a soft-attention module, and a third residual connection normalization module connected in sequence.

[0025] Based on the Transformer model, multimodal attention joint encoding is performed on the text features, audio features, and video image features to obtain text representation features, audio representation features, and video image representation features, including:

[0026] The text features are input into the first joint coding unit to obtain the text representation features;

[0027] The text features and audio features are input into the second joint coding unit to obtain audio representation features;

[0028] The text features and video image features are input into the third joint coding unit to obtain video image representation features.

[0029] Furthermore, the soft attention module includes multiple soft attention layers;

[0030] The soft attention module performs operations on the input features, including:

[0031] Soft attention operations are performed on the input features in each soft attention layer;

[0032] The results obtained from the operations of each soft attention layer are summed to obtain the output of the soft attention module, which is expressed by the following formula:

[0033] S M = stack(m1, ..., mG) m );

[0034] Among them, S MThis represents the output of the soft attention module, where stack indicates stacking computation, m1 represents the one-dimensional vector obtained from the soft attention operation, and mG represents the output of the soft attention module. m This indicates that the vector obtained from the soft attention operation is weighted.

[0035] Furthermore, based on a multilayer perceptron classification model, the text representation features, audio representation features, and video image representation features are weighted to obtain the classification result of the video to be analyzed, including:

[0036] The text representation features, audio representation features, and video image representation features are respectively input into the first fully connected layer, the ReLU activation function layer, and the second fully connected layer, and a random deactivation Dropout mechanism is introduced to obtain intermediate text representation features, intermediate audio representation features, and intermediate video image representation features.

[0037] Based on the soft-attention mechanism, soft attention vectors are calculated according to the intermediate text representation features, intermediate audio representation features, and intermediate video frame representation features, respectively.

[0038] Based on the soft attention vector, the text representation features, audio representation features, and video image representation features are weighted respectively to obtain the final text representation features, final audio representation features, and final video image representation features.

[0039] Based on the pre-obtained fusion weight matrix, the final text representation features, final audio representation features, and final video image representation features are subjected to layer normalization weighted fusion and input into the third fully connected layer to obtain the classification result of the video to be analyzed.

[0040] Furthermore, based on the soft-attention mechanism, soft attention vectors are calculated according to the intermediate text representation features, intermediate audio representation features, and intermediate video image representation features, respectively. The text representation features, audio representation features, and video image representation features are then weighted based on these soft attention vectors, as expressed by the following formula:

[0041]

[0042]

[0043]

[0044]

[0045]

[0046]

[0047] Where L° represents the weighted final text representation feature, A° represents the weighted final audio representation feature, V° represents the weighted final video image representation feature, and α... L α represents the soft attention vector representing the text representation features. A α represents the soft attention vector representing the audio representation features. V This represents the soft attention vector that represents the features of the video frame, where Softmax represents the normalized exponential function. This represents the characteristics of the intermediate text. This represents the intermediate audio characteristics. This indicates the representational features of the intermediate video frame. This represents the features of the i-th sentence in the text. This represents the feature of the i-th audio segment. Let l represent the features of the i-th video frame. i Let a represent the weighted hyperparameter of the i-th text. i v represents the i-th audio weighting hyperparameter. i This represents the weighted hyperparameter of the i-th video frame.

[0048] Furthermore, based on the pre-obtained fusion weight matrix, the final text representation features, final audio representation features, and final video image representation features are subjected to layer-normalized weighted fusion, expressed by the following formula:

[0049] predictedy = LayerNorm(W L L o +W A A o +W V V o );

[0050] Among them, predicted y To predict the classification results, LayerNorm represents the layer normalization operation, W L W A W V The weights corresponding to the text modality, audio modality, and video modality satisfy the following conditions: Where d z It is a hyperparameter representing the dimension of the vector after weighted fusion.

[0051] A Transformer-based intermodal joint coding device includes:

[0052] The video acquisition module is used to acquire videos containing multimodal information to be analyzed.

[0053] The text feature extraction module is used to extract the text features of the video to be analyzed;

[0054] An audio feature extraction module is used to extract the audio features of the video to be analyzed;

[0055] The visual feature extraction module is used to extract video frame features from the video to be analyzed.

[0056] The dimension unification module is used to unify the text features, audio features, and video image features into the same dimension based on the fully connected layer and LSTM layer;

[0057] The representation feature calculation module is used to perform multimodal attention joint encoding on the text features, audio features and video image features based on the Transformer model to obtain text representation features, audio representation features and video image representation features;

[0058] The classification result calculation module is used to weight the text representation features, audio representation features, and video image representation features based on the multilayer perceptron classification model to obtain the classification result of the video to be analyzed.

[0059] An electronic device includes a processor and a storage device, the storage device storing a plurality of instructions, the processor being configured to read the plurality of instructions from the storage device and execute the method described above.

[0060] The inter-modal joint coding method, apparatus, and device based on Transformer provided by this invention have at least the following beneficial effects:

[0061] (1) Based on Transformer, a neural network model is constructed, and the features between multiple modalities are jointly encoded using an interactive attention mechanism. Information fusion between modalities is achieved through the attention mechanism, allowing the neural network to obtain the correlation information related to the emotional attitude to be analyzed between modalities through the attention mechanism. By introducing an interactive attention module, the Transformer model can be used to perform joint attention encoding for different modalities at the same time. The features of different modalities do not need to be input into the Transformer model in sequence. The powerful representation ability of Transformer can be used to mine the information of interest in the features, and the correlation between specific modalities can be modeled at the same time, thereby achieving better classification results.

[0062] (2) When the classification model performs weighted fusion of representation features containing sentiment, the random deactivation Dropout mechanism is introduced to prevent overfitting during the training process of the neural network; the soft-attention mechanism is introduced to better extract key information and achieve better classification results. Attached Figure Description

[0063] Figure 1 A flowchart of one embodiment of the Transformer-based inter-modal joint coding method provided by the present invention;

[0064] Figure 2 A schematic diagram of the structure of an embodiment of the text feature extraction model provided by the present invention;

[0065] Figure 3 A schematic diagram comparing the structures of the C3D model and the R(2+1)D model provided in this invention;

[0066] Figure 4 A schematic diagram of the structure of one embodiment of the R(2+1)D model provided by the present invention;

[0067] Figure 5 A schematic diagram of the structure of an embodiment of the Transformer model for joint encoding of three modalities provided by the present invention;

[0068] Figure 6 This is a schematic diagram of the structure of one embodiment of the Transformer model for joint encoding of two modalities provided by the present invention. Detailed Implementation

[0069] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0070] See Figure 1 In some embodiments, a Transformer-based inter-modal joint coding method is provided, including:

[0071] S1. Obtain the video to be analyzed, which contains multimodal information;

[0072] S2. Extract the text features of the video to be analyzed;

[0073] S3. Extract the audio features of the video to be analyzed;

[0074] S4. Extract the video frame features of the video to be analyzed;

[0075] S5. Based on the fully connected layer and LSTM layer, the text features, audio features and video image features are unified into the same dimension;

[0076] S6. Based on the Transformer model, perform multimodal attention joint encoding on the text features, audio features and video image features to obtain text representation features, audio representation features and video image representation features;

[0077] S7. Based on the multilayer perceptron classification model, the text representation features, audio representation features, and video image representation features are weighted to obtain the classification result of the video to be analyzed.

[0078] In some embodiments, in step S2, the text features are extracted based on a pre-trained Chinese-BERT-wmm model, which employs a full-word masking mechanism during training. The full-word masking mechanism means that in training samples, if a portion of a word is masked, other words belonging to the same word segment will also be masked.

[0079] See Figure 2 The BERT model is a popular pre-trained language model based on Transformer. BERT constructs a bidirectional language model through the encoder structure in the Transformer model. Figure 2 In this context, TokN represents the Nth word in the sentence, Class Label represents the text feature classification label, and E N T represents the embedding of the Nth word. N This represents the feature vector of the Nth word output after processing by the model. The advantage of the BERT model is its ability to directly learn contextual information. The Chinese-BERT-wmm model is a pre-trained BERT model specifically for Chinese. The original BERT model uses a mask mechanism during training, randomly masking each subword in the training samples. However, the BERT-wmm model uses a full-word masking mechanism; if a part of a subword is masked, all other subwords belonging to the same segment are also masked—this is the full-word masking mechanism. Chinese-BERT-wmm is a powerful language model pre-trained on a large-scale Chinese corpus dataset, capable of accurately extracting semantic information from text.

[0080] As a preferred implementation, we use Hugging Face Transformer to load the pre-trained Chinese-BERT-wmm model. Hugging Face Transformer provides an API that is easy to download and use quickly, enabling the pre-trained model to be conveniently used for text feature processing, thus improving work efficiency.

[0081] In step S3, the audio features are extracted using Mel-spectrogram features based on the librosa audio signal processing library. The audio information in the video contains a large amount of human speech and music, along with many non-verbal expressions (laughter, sighs, etc.). This information, along with linguistically meaningful information, plays a significant role in video analysis. In the field of speech processing, features such as F0 and MFCCs are widely used, but higher-level features often overlook a large amount of information. To overcome this problem, this embodiment uses low-level auditory features as the audio feature input for subsequent multimodal analysis, selecting Mel-spectrogram features for extraction. Mel-spectrogram features are suitable for real-time speech recognition tasks and achieve good results; therefore, Mel-spectrograms can serve as a good representation of auditory information and are applied in this embodiment. As a preferred implementation, downsampling is also used during the extraction process to reduce the temporal feature dimension, thereby reducing complexity, accelerating the inference process, and improving the model's feature extraction efficiency.

[0082] See Figure 3 This paper provides a structural comparison between the C3D model (a 3D version of the VGG model) and the R(2+1)D model (a 3D variant of ResNet). This embodiment uses the R(2+1)D network, i.e., a 3D convolutional model, as the network model for video image feature extraction. Three-dimensional convolutional kernels inherently have higher computational costs and are prone to overfitting compared to two-dimensional convolutional kernels. To reduce complexity, the R(2+1)D network performs the three-dimensional convolution step-by-step, decomposing the three-dimensional convolution operation into one two-dimensional spatial convolution and one temporal one-dimensional convolution. This separates the three-dimensional temporal information from the spatial information of the video frame content, making it easier to optimize and resulting in a smaller loss function.

[0083] See Figure 4This document provides an R(2+1)D model structure, and this embodiment uses a pre-trained R(2+1)D-152 network. Here, `clip` represents a video clip, `(2+1)D conv` represents a combination of 2D spatial convolution and 1D temporal convolution, `pace-time pool` represents a spatiotemporal pooling layer, `fc` represents a fully connected layer, and `s` represents a fully connected layer. The model is trained on the Sports-1M dataset, which includes one million videos collected from YouTube, covering 487 motion-related video categories, with 1000-3000 videos in each category. During pre-training, the model receives 32 RGB video frames as input each time, with the frame size scaled to 128×171, and then a 112×112 video frame is randomly cropped from the original frame. The network output features from the spatiotemporal pooling layer are selected as the extracted features; the output of the fully connected layer is not selected. This allows the model to have only convolutional layers, making the length and width of the network input unrestricted and more flexible.

[0084] In step S4, the R(2+1)D model is used to extract the video frame features of the video to be analyzed;

[0085] Extracting video frame features from the video to be analyzed includes:

[0086] S41. Extract the three-dimensional sequence features from the video to be analyzed;

[0087] S42. Flatten the three-dimensional sequence features into two-dimensional sequence features;

[0088] S43. The two-dimensional sequence features are scaled and randomly cropped.

[0089] S44. Downsample the two-dimensional sequence features, selecting one frame out of every 16 frames to extract features, and using the network output features from the spatiotemporal pooling layer as the extracted video frame features. This reduces the complexity of the sampled video features, speeding up the model's video processing without compromising its performance.

[0090] Furthermore, the first eight video frames from which features are extracted are used as input for video features to ensure that the features in the video remain aligned.

[0091] In step S5, before inputting the extracted text features, audio features, and video features into the Transformer model, the features of different modalities are respectively input into a fully connected layer and an LSTM layer to ensure that each modality has the same vector dimension, thereby realizing the multimodal attention joint encoding in subsequent steps.

[0092] In the method provided in this embodiment, the video to be analyzed includes three modalities: text modality, audio modality, and video frame modality.

[0093] In step S6, during the multimodal attention joint encoding process, the text modality is used as the primary modality, and the audio modality and video image modality are modulated and encoded.

[0094] See Figure 5 The Transformer model includes three joint coding units, each of which includes a multi-head attention module, a first residual connection normalization module (Add&Norm), a feed-forward module, a second residual connection normalization module (Add&Norm), a soft attention module, and a third residual connection normalization module (Add&Norm) connected in sequence.

[0095] Based on the Transformer model, multimodal attention joint encoding is performed on the text features, audio features, and video image features to obtain text representation features, audio representation features, and video image representation features, including:

[0096] S61. Input the text features into the first joint coding unit to obtain text representation features;

[0097] S62. Input the text features and audio features into the second joint coding unit to obtain audio representation features;

[0098] S63. Input the text features and video image features into the third joint coding unit to obtain video image representation features.

[0099] The Transformer model provided in this embodiment includes an interactive attention module consisting of a multi-head attention module (Multi-Head), a first residual connection normalization module (Add&Norm), a feed-forward module (Feed-Forward), and a second residual connection normalization module (Add&Norm) connected in sequence. A soft attention module is also introduced to map the features output by the interactive attention module to a new representation space.

[0100] See Figure 6Taking the joint attention encoding of two modalities as an example, this paper introduces the working principle of multimodal joint attention encoding. First, assume there are two modalities, X and Y, where modality X is the dominant modality used to modulate and encode information from modality Y. The joint attention encoding process introduces the idea of ​​the Guide-attention unit from the interactive attention module, replacing the K and V matrices of the multi-head attention mechanism in the Transformer model with X instead of Y. After the replacement, it is similar to QK in the Transformer. T The calculation yields an attention matrix, which can be understood as a similarity matrix of row vectors in the features of modes X and Y, mining the feature correlation between modes X and Y. After obtaining this similarity matrix, a dot product is performed between this similarity matrix and X. Similar to the Transformer model, residual connections and LayerNorm models are introduced. The entire calculation process can be represented as follows:

[0101] f=LayerNorm(y+MA(y,x,x));

[0102] Here, LayerNorm represents the layer normalization operation, y represents the input feature, and MA(y,x,x) represents multi-head attention.

[0103] The LayerNorm operation is used to make the changes in the loss function more stable during training, thereby achieving better joint encoding results.

[0104] To ensure the smooth execution of matrix dot product and residual join, the vector dimensions of X and Y should be the same, as expressed by the following formula:

[0105]

[0106]

[0107] Among them, R N×K Let represent a linear space containing all matrices, where each matrix is ​​an N-row, K-column matrix.

[0108] In some embodiments, the soft attention module in the Transformer model includes multiple soft attention layers;

[0109] The soft attention module performs operations on the input features, including:

[0110] Soft attention operations are performed on the input features in each soft attention layer;

[0111] The results obtained from the operations of each soft attention layer are summed to obtain the output of the soft attention module, which is expressed by the following formula:

[0112] SM = stack(m1, ..., mG) m );

[0113] Among them, S M This represents the output of the soft attention module, where stack indicates stacking computation, m1 represents the one-dimensional vector obtained from the soft attention operation, and mG represents the output of the soft attention module. m This indicates that the vector obtained from the soft attention operation is weighted.

[0114] In some embodiments, soft attention operations are performed on the input features in each soft attention layer, as expressed by the following formula:

[0115]

[0116]

[0117]

[0118] Among them, W m Let be a transformation matrix of shape 2k×k, representing the network weights of a fully connected layer in the neural network. is a shared parameter across all soft attention layers. Softmax is a normalized exponential function, and soft-attention is the soft attention operation function performed by each soft attention layer. Let m be the weight vector of the i-th 1×2k. i Let M be the output of the i-th soft attention layer, and M be the input feature matrix. j Let be the feature matrix of the j-th input.

[0119] In step S7, based on the multilayer perceptron classification model, the text representation features, audio representation features, and video image representation features are weighted to obtain the classification result of the video to be analyzed, including:

[0120] S71. The text representation features, audio representation features and video image representation features are respectively input into the first fully connected layer, the ReLU activation function layer and the second fully connected layer, and a random deactivation Dropout mechanism is introduced to obtain intermediate text representation features, intermediate audio representation features and intermediate video image representation features.

[0121] S72. Based on the soft-attention mechanism, calculate the soft attention vectors according to the intermediate text representation features, intermediate audio representation features, and intermediate video image representation features respectively.

[0122] S73. Based on the soft attention vector, the text representation features, audio representation features and video image representation features are weighted respectively to obtain the final text representation features, final audio representation features and final video image representation features.

[0123] S74. Based on the pre-obtained fusion weight matrix, perform layer normalization weighted fusion on the final text representation features, final audio representation features, and final video image representation features, and input the result into the third fully connected layer to obtain the classification result of the video to be analyzed.

[0124] After multimodal joint encoding, the outputs are the modal features represented by the network modules, including text representation features, audio representation features, and video image representation features, with each feature dimension remaining the same, as expressed by the following formula:

[0125]

[0126]

[0127]

[0128] in, Representing text representation features, audio representation features, and video image representation features, l m Let a represent the m-th text feature. m Let v represent the m-th audio feature. m R represents the feature of the m-th video frame. N×K Let represent the linear space consisting of all N rows and K columns of matrices.

[0129] After multimodal joint encoding, the text representation features, audio representation features, and video image representation features already contain a large amount of emotional attitude features and corresponding attention information. These representations need to be dimensionality reduced to output the final prediction result of the neural network. Specifically, in step S71, a random deactivation mechanism, Dropout, is introduced to prevent overfitting during the neural network training process.

[0130] In steps S72 and S73, based on the soft-attention mechanism, soft attention vectors are calculated according to the intermediate text representation features, intermediate audio representation features, and intermediate video image representation features, respectively. The text representation features, audio representation features, and video image representation features are then weighted based on these soft attention vectors, as expressed by the following formula:

[0131]

[0132]

[0133]

[0134]

[0135]

[0136]

[0137] Where L° represents the weighted final text representation feature, A° represents the weighted final audio representation feature, V° represents the weighted final video image representation feature, and α... L α represents the soft attention vector representing the text representation features. A α represents the soft attention vector representing the audio representation features. V This represents the soft attention vector that represents the features of the video frame, where Softmax represents the normalized exponential function. This represents the characteristics of the intermediate text. This represents the intermediate audio characteristics. This indicates the representational features of the intermediate video frame. This represents the features of the i-th sentence in the text. This represents the feature of the i-th audio segment. Let l represent the features of the i-th video frame. i Let a represent the weighted hyperparameter of the i-th text. i v represents the i-th audio weighting hyperparameter. i This represents the weighted hyperparameter of the i-th video frame.

[0138] When the classification model performs weighted fusion of representational features containing sentiment, it introduces a soft-attention mechanism, which can better extract key information.

[0139] In step S74, based on the pre-obtained fusion weight matrix, the final text representation features, final audio representation features, and final video image representation features are subjected to layer-normalized weighted fusion, expressed by the following formula:

[0140] predicted y =LayerNorm(W L L o +W A A o +W V V o );

[0141] Among them, predicted y To predict the classification results, LayerNorm represents the layer normalization operation, W L W AW V The weights corresponding to the text modality, audio modality, and video modality are learned by the network and satisfy the following conditions: Where d z It is a hyperparameter representing the dimension of the vector after weighted fusion.

[0142] As a preferred implementation method, the multilayer perceptron model employs a binary cross-entropy loss function.

[0143] As a preferred implementation, the obtained results are input into the third fully connected layer. In order to map the fused features into a final output prediction vector, the vector has a dimension of 1.

[0144] In a specific application scenario, the CPU is a 6-core Intel(R) Core(TM) i5-9600K CPU@3.70GHz, the memory is 16GB 2400MHz DDR4, the graphics card is an Nvidia GTX 1080Ti, the operating system is Ubuntu 20.04.1LTS, the code is mainly implemented in Python, version 3.6.13, the neural network part mainly relies on PyTorch 1.2.0, and Anaconda is used to manage the required third-party libraries.

[0145] For feature extraction, the 3D sequence features extracted from the video were flattened to 2D, and downsampling was used, selecting one frame out of every 16 video frames for feature extraction. Furthermore, to ensure feature alignment within the video, the first 8 video frames from which features were extracted were used as input for the video features. For text feature input, a pre-trained Chinese-BERT-wmm was loaded using Hugging Face. During text feature extraction, start and end markers were added to the beginning and end of the text, respectively. <sos>and <eos>The embedding vector of ).

[0146] Similar to video features, audio features also undersampling was used. Furthermore, the Python-based third-party library pickle saved the extracted features from different modalities as .pkl files. This eliminates the modal feature extraction portion from the neural network computation, allowing focus only on the forward and backward propagation, ensuring efficient neural network training. In the network implementation, the number of multi-head attention nodes was set to 4, the hidden layer size of the Transformer was 1024, and there were 6 Transformer modules. The input size of the fully connected layers was 1024. The total number of parameters was approximately 140M.

[0147] During the model training phase, the Adam optimizer is used to optimize the network. The Adam optimizer is a popular optimizer in deep learning, with advantages such as simplicity, efficiency, and suitability for large-scale data. The initial learning rate of the neural network is set to 1e-3, and if the accuracy on the validation set does not improve, the learning rate will decrease by a factor of 0.2. The node retention probability for Dropout random deactivation is set to 0.1. Dropout can regularize the model and reduce overfitting in deep learning. Due to the limitations of the graphics card's memory, the batch size is set to 16.

[0148] In some embodiments, a Transformer-based intermodal joint coding apparatus is provided, comprising:

[0149] The video acquisition module is used to acquire videos containing multimodal information to be analyzed.

[0150] The text feature extraction module is used to extract the text features of the video to be analyzed;

[0151] An audio feature extraction module is used to extract the audio features of the video to be analyzed;

[0152] The visual feature extraction module is used to extract video frame features from the video to be analyzed.

[0153] The dimension unification module is used to unify the text features, audio features, and video image features into the same dimension based on the fully connected layer and LSTM layer;

[0154] The representation feature calculation module is used to perform multimodal attention joint encoding on the text features, audio features and video image features based on the Transformer model to obtain text representation features, audio representation features and video image representation features;

[0155] The classification result calculation module is used to weight the text representation features, audio representation features, and video image representation features based on the multilayer perceptron classification model to obtain the classification result of the video to be analyzed.

[0156] In some embodiments, an electronic device is provided, including a processor and a storage device, wherein the storage device stores a plurality of instructions, and the processor is configured to read the plurality of instructions from the storage device and execute the method described above.

[0157] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention. Clearly, those skilled in the art can make various alterations and modifications to the invention without departing from its spirit and scope. Thus, if these modifications and modifications of the invention fall within the scope of the claims and their equivalents, the invention is also intended to include these modifications and modifications.< / eos> < / sos>

Claims

1. A method for inter-modal joint encoding based on Transformer, characterized in that, include: Acquire the video to be analyzed, which contains multimodal information; Extract the text features of the video to be analyzed; Extract the audio features of the video to be analyzed; Extract the video frame features of the video to be analyzed; Based on fully connected layers and LSTM layers, the text features, audio features, and video image features are unified into the same dimension; Based on the Transformer model, the text features, audio features, and video image features are jointly encoded using multimodal attention to obtain text representation features, audio representation features, and video image representation features. Based on the multilayer perceptron classification model, the text representation features, audio representation features, and video image representation features are weighted to obtain the classification result of the video to be analyzed. During the multimodal attention joint coding process, the text modality is used as the primary modality, and the audio modality and video image modality are modulated and coded. The Transformer model includes three joint coding units, each of which includes a multi-head attention module, a first residual connection normalization module, a feed-forward module, a second residual connection normalization module, a soft-attention module, and a third residual connection normalization module connected in sequence. Based on the Transformer model, multimodal attention joint encoding is performed on the text features, audio features, and video image features to obtain text representation features, audio representation features, and video image representation features, including: The text features are input into the first joint coding unit to obtain the text representation features; The text features and audio features are input into the second joint coding unit to obtain audio representation features; The text features and video image features are input into the third joint coding unit to obtain video image representation features.

2. The method of claim 1, wherein, The text features are extracted based on a pre-trained Chinese-BERT-wmm model; The Chinese-BERT-wmm model is loaded using a Hugging face Transformer; The Chinese-BERT-wmm model employs a full-word masking mechanism during training.

3. The method of claim 1, wherein, The R(2+1)D model was used to extract video frame features from the video to be analyzed. Extracting video frame features from the video to be analyzed includes: Extract the three-dimensional sequence features from the video to be analyzed; Flatten the three-dimensional sequence features into two-dimensional sequence features; The two-dimensional sequence features are downsampled, and one frame is selected from every 16 frames to extract features. The network output features of the R(2+1)D model in the spatiotemporal pooling layer are selected as the extracted video frame features.

4. The method of claim 1, wherein, The soft attention module includes multiple soft attention layers; The soft attention module performs operations on the input features, including: Soft attention operations are performed on the input features in each soft attention layer; The results obtained from the operations of each soft attention layer are summed to obtain the output of the soft attention module, which is expressed by the following formula: ； where S M is the output of the soft attention module, stack denotes the stacking computation, and m1 denotes the one-dimensional vector obtained by the soft attention operation.

5. The method according to claim 1 or 4, characterized in that, Based on a multilayer perceptron classification model, the text representation features, audio representation features, and video image representation features are weighted to obtain the classification result of the video to be analyzed, including: The text representation features, audio representation features, and video image representation features are respectively input into the first fully connected layer, the ReLU activation function layer, and the second fully connected layer, and a random deactivation Dropout mechanism is introduced to obtain intermediate text representation features, intermediate audio representation features, and intermediate video image representation features. Based on the soft-attention mechanism, soft attention vectors are calculated according to the intermediate text representation features, intermediate audio representation features, and intermediate video frame representation features, respectively. Based on the soft attention vector, the text representation features, audio representation features, and video image representation features are weighted respectively to obtain the final text representation features, final audio representation features, and final video image representation features. Based on the pre-obtained fusion weight matrix, the final text representation features, final audio representation features, and final video image representation features are subjected to layer normalization weighted fusion and input into the third fully connected layer to obtain the classification result of the video to be analyzed.

6. The method of claim 5, wherein, Based on the soft-attention mechanism, soft attention vectors are calculated according to the intermediate text representation features, intermediate audio representation features, and intermediate video image representation features, respectively. The text representation features, audio representation features, and video image representation features are then weighted based on these soft attention vectors, as expressed by the following formula: ； ； ； ； ； ； in, This represents the final text representation features obtained through weighting. This represents the final audio representation features obtained through weighting. This represents the final video image representation features obtained through weighted calculation. The soft attention vector representing the text representation features. The soft attention vector representing the audio representation features. This represents the soft attention vector that represents the features of the video frame, where Softmax represents the normalized exponential function. This represents the characteristics of the intermediate text. This represents the intermediate audio characteristics. This indicates the representational features of the intermediate video frame. This represents the features of the i-th sentence in the text. This represents the feature of the i-th audio segment. This represents the feature of the i-th video frame. This represents the weighted hyperparameter of the i-th text. This represents the i-th audio weighting hyperparameter. This represents the weighted hyperparameter of the i-th video frame.

7. The method of claim 5, wherein, Based on the pre-obtained fusion weight matrix, the final text representation features, final audio representation features, and final video image representation features are subjected to layer-normalized weighted fusion, as expressed by the following formula: ； Among them, predicted y To predict the classification results, LayerNorm represents the layer normalization operation, W L W A W V The weights corresponding to the text modality, audio modality, and video modality satisfy the following conditions: , where d z It is a hyperparameter representing the dimension of the vector after weighted fusion.

8. A Transformer-based intermodal joint coding device, characterized in that, include: The video acquisition module is used to acquire videos containing multimodal information to be analyzed. The text feature extraction module is used to extract the text features of the video to be analyzed; An audio feature extraction module is used to extract the audio features of the video to be analyzed; The visual feature extraction module is used to extract video frame features from the video to be analyzed. The dimension unification module is used to unify the text features, audio features, and video image features into the same dimension based on the fully connected layer and LSTM layer; The representation feature calculation module is used to perform multimodal attention joint encoding on the text features, audio features and video image features based on the Transformer model to obtain text representation features, audio representation features and video image representation features; The classification result calculation module is used to weight the text representation features, audio representation features, and video image representation features based on the multilayer perceptron classification model to obtain the classification result of the video to be analyzed. During the multimodal attention joint coding process, the text modality is used as the primary modality, and the audio modality and video image modality are modulated and coded. The Transformer model includes three joint coding units, each of which includes a multi-head attention module, a first residual connection normalization module, a feed-forward module, a second residual connection normalization module, a soft-attention module, and a third residual connection normalization module connected in sequence. Based on the Transformer model, multimodal attention joint encoding is performed on the text features, audio features, and video image features to obtain text representation features, audio representation features, and video image representation features, including: The text features are input into the first joint coding unit to obtain the text representation features; The text features and audio features are input into the second joint coding unit to obtain audio representation features; The text features and video image features are input into the third joint coding unit to obtain video image representation features.

9. An electronic device comprising a processor and a storage device, characterized in that The storage device contains a plurality of instructions, and the processor is used to read the plurality of instructions in the storage device and execute the method as described in any one of claims 1-7.

Images