A video classification method and device based on multi-modal feature fusion and a medium

Abstract

The application belongs to the technical field of video classification, and relates to a video classification method and device based on multi-modal feature fusion and a medium; an image feature matrix and an audio feature matrix of a video to be classified are spliced and input into a first multi-layer perception machine to output an initial fusion feature matrix; the image feature matrix, the audio feature matrix and the initial fusion feature matrix are input into L self-alignment fusion layers connected in sequence along a positive propagation direction for mid-term fusion; after cls token vectors in a target image feature matrix output by the Lth self-alignment fusion layer, cls token vectors in a target audio feature matrix and a fusion feature matrix are spliced, post-term fusion is performed, and the fusion feature matrix is input into a third multi-layer perception machine, so that the video to be classified is classified; and the video classification method provided in the application fully fuses feature information of different modes through multi-stage feature fusion, and improves the accuracy of a video classification result.

Description

Technical Field

[0001] This invention relates to the field of video classification technology, and in particular to a video classification method, apparatus, and computer-readable storage medium based on multimodal feature fusion. Background Technology

[0002] Video classification is one of the important tasks in the fields of machine learning and computer vision. Its goal is to classify videos into corresponding categories based on their content. Video classification plays an important role in video recommendation systems, content filtering, monitoring systems, and video retrieval. With the explosive growth of internet video data, the ability to accurately and efficiently classify videos is becoming increasingly important.

[0003] Traditional video classification methods mainly rely on visual information. They typically use convolutional neural networks (CNNs) to extract image frame features from videos, and then combine them with recurrent neural networks (RNNs) or long short-term memory networks (LSTMs) to process the time-series information composed of the features of each image frame. Although this method fully extracts the image information contained in the video, it ignores the rich audio information contained in the video, resulting in low accuracy of the final classification results.

[0004] To address this issue, methods that integrate multimodal information for video classification have gradually become a research hotspot. Current video classification methods, besides relying on visual information, also integrate audio information. The methods for integrating visual and audio information include: 1. Classifying videos separately based on image features and audio features, and then combining the classification results based on image features and audio features at the model's classification decision stage. This method is simple and easy to implement, but it does not consider the complementarity and correlation between different modal features, cannot fully extract the video's feature information, and its classification results are greatly affected by single-modal features. If the classification result based on a certain modality has a large error, it will reduce the accuracy of the video classification results; 2. Directly concatenating image features and audio features at the model's input layer to obtain a high-dimensional fused feature, and then normalizing and deepening this high-dimensional fused feature. Operations such as feature extraction are performed to obtain the target features of the video for classification. However, direct splicing cannot capture the complementary information between different modalities, reducing the accuracy of video classification results. Furthermore, the features of the two modalities may contain some duplicate information, and direct splicing will also lead to information redundancy, increase the computational burden of the model, and reduce the efficiency of video classification. 3. Introducing attention mechanisms or gating mechanisms into the intermediate layers of the model to dynamically adjust the importance of image features and modal features, automatically focusing on modal features that are more helpful for video classification, thereby capturing the interaction and complementarity between the two modalities. However, since attention mechanisms or gating mechanisms often only perform feature weighting in a local range, it is difficult to capture the global relationship between different modalities. This causes some important cross-modal interaction information to be ignored, resulting in the fused feature information not containing all the information of the two modalities, thus reducing the accuracy of video classification results.

[0005] In summary, existing video classification methods based on the fusion of image and audio features cannot fully integrate and utilize features from different modalities, resulting in low accuracy and reliability of video classification results. Summary of the Invention

[0006] Therefore, the technical problem to be solved by the present invention is to overcome the problem that existing video classification methods cannot fully integrate and utilize the features of different modalities, resulting in low accuracy and reliability of video classification results.

[0007] To address the aforementioned technical problems, this invention provides a video classification method based on multimodal feature fusion, comprising:

[0008] The image feature matrix and audio feature matrix of the video to be classified are concatenated and input into the first multilayer perceptron, and the initial fused feature matrix is ​​output.

[0009] The image feature matrix, the audio feature matrix, and the initial fused feature matrix are input into L self-aligned fusion layers connected in series along the forward propagation direction for fusion; wherein, the step of the l-th self-aligned fusion layer fusing the input feature matrix includes:

[0010] The fusion feature matrix output from the (l-1)th self-aligned fusion layer is input into the second multilayer perceptron to obtain the target fusion feature matrix of the (l-1)th self-aligned fusion layer.

[0011] The image feature matrix and the target image feature matrix output by the (l-1)th self-alignment fusion layer are input into the first self-alignment module for self-alignment, so as to update the target image feature matrix output by the (l-1)th self-alignment fusion layer.

[0012] The updated target image feature matrix output from the (l-1)th self-aligned fusion layer and the target fusion feature matrix of the (l-1)th self-aligned fusion layer are input into the first encoder, and the target image feature matrix of the l-th self-aligned fusion layer and the first fusion feature matrix are output.

[0013] The audio feature matrix and the target audio feature matrix output by the (l-1)th self-aligned fusion layer are input into the second self-alignment module for self-alignment, so as to update the target audio feature matrix output by the (l-1)th self-aligned fusion layer.

[0014] The updated target audio feature matrix output from the (l-1)th self-aligned fusion layer and the target fusion feature matrix of the (l-1)th self-aligned fusion layer are input into the second encoder, and the target audio feature matrix of the l-th self-aligned fusion layer and the second fusion feature matrix are output.

[0015] The first fusion feature matrix and the second fusion feature matrix are averaged to obtain the fusion feature matrix of the l-th self-aligned fusion layer; where l∈[1,L], when l=1, the target image feature matrix output by the (l-1)-th self-aligned fusion layer is the image feature matrix, the target audio feature matrix output by the (l-1)-th self-aligned fusion layer is the audio feature matrix, and the fusion feature matrix output by the (l-1)-th self-aligned fusion layer is the initial fusion feature matrix;

[0016] The cls token vector in the target image feature matrix, the cls token vector in the target audio feature matrix, and the fused feature matrix output from the Lth self-aligned fusion layer are concatenated and input into the third multilayer perceptron to output the classification result of the video to be classified.

[0017] Preferably, the target fusion feature matrix of the (l-1)th self-aligned fusion layer is represented as:

[0018]

[0019] in, Let MLP2 represent the target fusion feature matrix of the (l-1)th self-aligned fusion layer, and MLP2 represent the second multilayer perceptron. This represents the fusion feature matrix output by the (l-1)th self-aligned fusion layer.

[0020] Preferably, the target image feature matrix output by the updated (l-1)th self-aligned fusion layer is represented as:

[0021]

[0022] in, This represents the target image feature matrix output by the updated (l-1)th self-alignment fusion layer, where Self-Alignment1 represents the first self-alignment module. Represents the image feature matrix. This represents the target image feature matrix output by the (l-1)th self-aligned fusion layer;

[0023] The updated target audio feature matrix output by the (l-1)th self-aligned fusion layer is represented as follows:

[0024]

[0025] in, Self-Alignment2 represents the target audio feature matrix output by the updated (l-1)th self-aligned fusion layer, and Self-Alignment2 represents the second self-alignment module. Represents the audio feature matrix. This represents the target audio feature matrix output by the (l-1)th self-aligned fusion layer.

[0026] Preferably, the target image feature matrix and the first fusion feature matrix of the l-th self-aligned fusion layer are represented as follows:

[0027]

[0028] in, This represents the target image feature matrix of the l-th self-aligned fusion layer. Fusion1 represents the first fusion feature matrix of the l-th self-aligned fusion layer, and Fusion1 represents the first encoder. This represents the target fusion feature matrix of the (l-1)th self-aligned fusion layer;

[0029] The target audio feature matrix and the second fused feature matrix of the l-th self-aligned fusion layer are represented as follows:

[0030]

[0031] in, This represents the target audio feature matrix of the l-th self-aligned fusion layer. Fusion2 represents the second fusion feature matrix of the l-th self-aligned fusion layer;

[0032] The fusion feature matrix of the l-th self-aligned fusion layer is represented as:

[0033]

[0034] in, Let represent the fusion feature matrix of the l-th self-aligned fusion layer, and Avg denotes the averaging.

[0035] Preferably, the initial fusion feature matrix is ​​represented as:

[0036]

[0037] in, denoted as the initial fused feature matrix, MLP1 represents the first multilayer perceptron, and stack represents the concatenation operation. Represents the image feature matrix. This represents the audio feature matrix.

[0038] Preferably, the classification result of the video to be classified is represented as follows:

[0039]

[0040] in, This represents the classification result of the video to be classified, where MLP3 represents the third multilayer perceptron. This represents the cls token vector in the target image feature matrix output by the Lth self-aligned fusion layer. This represents the fusion feature matrix output by the Lth self-aligned fusion layer. Let cls be the cls token vector in the target audio feature matrix output by the Lth self-aligned fusion layer.

[0041] Preferably, the process of obtaining the image feature matrix of the video to be classified includes:

[0042] Image frames in the video to be classified are sampled to obtain the target image frame sequence;

[0043] The target image frame sequence is normalized, and the normalized target image frame sequence is then augmented.

[0044] The augmented target image frame sequence is input into the visual converter, which outputs the image feature matrix of the video to be classified.

[0045] Preferably, the process of obtaining the audio feature matrix of the video to be classified includes:

[0046] The audio signal in the video to be classified is converted to obtain a Mel spectrogram, and the Mel spectrogram is then standardized.

[0047] The standardized Mel spectrogram is input into the audio spectrogram transformer, which outputs the audio feature matrix of the video to be classified.

[0048] The present invention also provides a video classification device based on multimodal feature fusion, comprising:

[0049] The first feature fusion module is used to concatenate the image feature matrix and audio feature matrix of the video to be classified and input them into the first multilayer perceptron, and output the initial fused feature matrix.

[0050] The second feature fusion module is used to input the image feature matrix, the audio feature matrix, and the initial fused feature matrix into L self-aligned fusion layers connected in series along the forward propagation direction for fusion; wherein, the step of the l-th self-aligned fusion layer fusing the input feature matrix includes:

[0051] The matrix dimension-upgrading submodule is used to input the fusion feature matrix output by the (l-1)th self-aligned fusion layer into the second multilayer perceptron to obtain the target fusion feature matrix of the (l-1)th self-aligned fusion layer.

[0052] The image feature self-alignment submodule is used to input the image feature matrix and the target image feature matrix output by the (l-1)th self-alignment fusion layer into the first self-alignment module for self-alignment, so as to update the target image feature matrix output by the (l-1)th self-alignment fusion layer.

[0053] The image feature fusion submodule is used to input the updated target image feature matrix output by the (l-1)th self-aligned fusion layer and the target fusion feature matrix of the (l-1)th self-aligned fusion layer into the first encoder, and output the target image feature matrix of the l-th self-aligned fusion layer and the first fusion feature matrix.

[0054] The audio feature self-alignment submodule is used to input the audio feature matrix and the target audio feature matrix output by the (l-1)th self-alignment fusion layer into the second self-alignment module for self-alignment, so as to update the target audio feature matrix output by the (l-1)th self-alignment fusion layer.

[0055] The audio feature fusion submodule is used to input the updated target audio feature matrix output from the (l-1)th self-aligned fusion layer and the target fusion feature matrix of the (l-1)th self-aligned fusion layer into the second encoder, and output the target audio feature matrix of the l-th self-aligned fusion layer and the second fusion feature matrix.

[0056] The fusion feature matrix acquisition submodule is used to average the first fusion feature matrix and the second fusion feature matrix to obtain the fusion feature matrix of the l-th self-aligned fusion layer; where l∈[1,L], when l=1, the target image feature matrix output by the (l-1)-th self-aligned fusion layer is the image feature matrix, the target audio feature matrix output by the (l-1)-th self-aligned fusion layer is the audio feature matrix, and the fusion feature matrix output by the (l-1)-th self-aligned fusion layer is the initial fusion feature matrix;

[0057] The third feature fusion and video classification module is used to concatenate the cls token vector in the target image feature matrix, the cls token vector in the target audio feature matrix, and the fused feature matrix output by the Lth self-aligned fusion layer, and input them into the third multilayer perceptron to output the classification result of the video to be classified.

[0058] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the video classification method based on multimodal feature fusion described above.

[0059] The video classification method based on multimodal feature fusion provided in this application has the following beneficial effects:

[0060] 1. A multi-stage feature fusion strategy is adopted to fully integrate the image and audio features of the video to be classified: In the first stage, the image feature matrix and the audio feature matrix are concatenated and passed through a multilayer perceptron to obtain an initial fusion feature matrix containing features of both modalities. Early fusion preserves the rich initial information of the two modalities and effectively removes redundant and repetitive information in each modality. In the second stage, the image feature matrix, the audio feature matrix, and the initial fusion feature matrix are input into multiple cascaded self-aligned fusion layers for feature fusion, thereby enhancing the richness of data features, solving the coordination problem between different modal information, and fully capturing the correlation between different modal information. Since the fusion feature matrix output by the last self-aligned fusion layer condenses the feature information of different modalities, in the third stage of feature fusion, the target image feature matrix, the target audio feature matrix, and the fusion feature matrix output by the last self-aligned fusion layer are concatenated to classify the video to be classified. This allows the concatenated features to more comprehensively express the complexity and diversity of multimodal data, avoid the loss of feature information, and thus improve the accuracy of the classification results.

[0061] 2. In the second stage of fusion, each self-aligned fusion layer first uses a multilayer perceptron to upscale the fusion features output by the previous self-aligned fusion layer to obtain the target fusion feature matrix. Simultaneously, the self-alignment module is used to align the features of each modality individually to ensure that the feature information of each modality is not weakened or lost during the fusion process, maintaining the diversity and integrity of multimodal data. Then, the self-aligned features of each modality are concatenated with the target fusion feature matrix and input into the corresponding encoder for fusion. Since the target fusion feature matrix contains features of two modalities, it plays a role in information aggregation and information transmission during the fusion process, thereby outputting a target image feature matrix fused with audio feature information, a target audio feature matrix fused with image feature information, and a new fusion feature matrix condensed with image and audio feature information. This fully integrates the image and audio features of the video to be classified, ensuring information sharing between different modal features, thereby improving the accuracy of video classification results. Attached Figure Description

[0062] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings, wherein:

[0063] Figure 1 The flowchart of the video classification method based on multimodal feature fusion provided in this application;

[0064] Figure 2 A schematic diagram of the video classification model structure provided in this application;

[0065] Figure 3 for Figure 2 A schematic diagram of the structure of the self-aligned fusion layer;

[0066] Figure 4 A schematic diagram of the video classification device based on multimodal feature fusion provided in this application. Detailed Implementation

[0067] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0068] Please see Figure 1 , Figure 1 The diagram shown is a flowchart of a video classification method based on multimodal feature fusion provided in an embodiment of this application. The method specifically includes:

[0069] S10: Concatenate the image feature matrix and audio feature matrix of the video to be classified and input them into the first multilayer perceptron, and output the initial fused feature matrix;

[0070] S20: Input the image feature matrix, audio feature matrix, and initial fused feature matrix into L self-aligned fusion layers sequentially connected along the forward propagation direction for fusion; wherein, the steps of the l-th self-aligned fusion layer fusing the input feature matrix include:

[0071] The fusion feature matrix output from the (l-1)th self-aligned fusion layer is input into the second multilayer perceptron to obtain the target fusion feature matrix of the (l-1)th self-aligned fusion layer.

[0072] The image feature matrix and the target image feature matrix output by the (l-1)th self-aligned fusion layer are input into the first self-alignment module for self-alignment, so as to update the target image feature matrix output by the (l-1)th self-aligned fusion layer.

[0073] The updated target image feature matrix output from the (l-1)th self-aligned fusion layer and the target fusion feature matrix of the (l-1)th self-aligned fusion layer are input into the first encoder, and the target image feature matrix of the l-th self-aligned fusion layer and the first fusion feature matrix are output.

[0074] The audio feature matrix and the target audio feature matrix output by the (l-1)th self-aligned fusion layer are input into the second self-alignment module for self-alignment, so as to update the target audio feature matrix output by the (l-1)th self-aligned fusion layer.

[0075] The updated target audio feature matrix output from the (l-1)th self-aligned fusion layer and the target fusion feature matrix of the (l-1)th self-aligned fusion layer are input into the second encoder, and the target audio feature matrix of the l-th self-aligned fusion layer and the second fusion feature matrix are output.

[0076] The fusion feature matrix of the l-th self-aligned fusion layer is obtained by averaging the first and second fusion feature matrices; where l∈[1,L], when l=1, the target image feature matrix output by the (l-1)-th self-aligned fusion layer is the image feature matrix, and the target audio feature matrix output by the (l-1)-th self-aligned fusion layer is the audio feature matrix.

[0077] The resulting fusion feature matrix is ​​the initial fusion feature matrix;

[0078] S30: Concatenate the cls token vector in the target image feature matrix, the cls token vector in the target audio feature matrix, and the fused feature matrix output by the Lth self-aligned fusion layer, and input them into the third multilayer perceptron to output the classification result of the video to be classified.

[0079] The video classification method based on multimodal feature fusion provided in this application adopts a multi-stage feature fusion strategy to fully integrate the image and audio features of the video to be classified: In the first stage, the image feature matrix and the audio feature matrix are concatenated and passed through a multilayer perceptron to obtain an initial fusion feature matrix containing features of both modalities, so as to retain the rich initial information of multimodal data, improve the representativeness and effectiveness of the initial fusion feature matrix, and, through preliminary screening and integration of features of different modalities, can effectively remove duplicate and redundant information in various modalities, so that the initial fusion features are already concise and efficient feature representations in the initial stage, containing the most valuable information of each modality;

[0080] In the second stage, multiple self-aligned fusion layers are used to fuse single-modal features and fused features. Data from different modalities have different features and information. Feature fusion can integrate this information to form a more comprehensive and accurate feature representation. At the same time, data from different modalities also have the same features and information. Using the repetitive information in multiple modalities to verify each other can also improve the reliability and robustness of the data. However, when fusion is excessive, the fused features will gradually concentrate on the common features of multiple modalities, resulting in the loss of single-modal information. Furthermore, since each self-aligned fusion layer in this application has multiple inputs corresponding to multiple outputs, rather than multiple inputs finally outputting a fused feature, it is necessary to consider whether the output single-modal features can retain the information of the original modality. Based on this, each self-aligned fusion layer in this application first uses a multilayer perceptron to increase the dimensionality of the fused features output by the previous self-aligned fusion layer to obtain the target fused feature matrix. At the same time, a self-alignment module is used to align the features of each modality separately to ensure that the feature information of each modality is not weakened or lost during the fusion process, and to maintain the diversity and integrity of multimodal data. Then, the self-aligned features of each modality are fused together. The features are concatenated with the target fusion feature matrix and input into the corresponding encoder for fusion. Since the target fusion feature matrix contains features from two modalities, it plays a role in information aggregation and transmission during the fusion process. Specifically, the target fusion feature pays attention to the feature information of a single modality, and the information flow is fused from the single modality feature to the fusion feature. At the same time, since the dimension of the target fusion feature is generally much smaller than the dimension of the single modality feature, the fusion feature only aggregates the necessary single modality feature. Similarly, the single modality feature pays attention to the information of another modality contained in the target fusion feature and updates it according to the attention weight, fusing information from multiple modalities. This results in the output of a target image feature matrix fused with audio feature information, a target audio feature matrix fused with image feature information, and a new fusion feature matrix condensed with both image and audio feature information. This fully integrates the image and audio features of the video to be classified, ensuring information sharing between different modal features and thus improving the accuracy of video classification results. Furthermore, since the target fusion feature matrix has a small dimension, this application uses it as an information fusion unit to transmit information, which can also effectively reduce computational complexity.

[0081] Since the fusion feature matrix output by the last self-aligned fusion layer condenses information from different modalities, in the third stage of feature fusion, the target image feature matrix, target audio feature matrix, and fusion feature matrix output by the last self-aligned fusion layer are concatenated to classify the video to be classified. This allows the concatenated features to more comprehensively express the complexity and diversity of multimodal data, avoid the loss of feature information, and thus improve the accuracy of the classification results.

[0082] Specifically, in some embodiments of this application, the initial fusion feature matrix in step S10 is represented as:

[0083]

[0084] in, denoted as the initial fused feature matrix, MLP1 represents the first multilayer perceptron, and stack represents the concatenation operation. Represents the image feature matrix. This represents the audio feature matrix.

[0085] Specifically, in step S20, the image feature matrix, audio feature matrix, and initial fusion feature matrix are input into the self-aligned fusion layer for the second stage of feature fusion to address the coordination problem between multimodal data and improve the accuracy of feature integration. In this stage, the fusion feature matrix plays the role of information aggregation and information transmission. Since the fusion feature matrix condenses information from two modalities, during the attention process, the current modality feature matrix can obtain information from the other modality feature matrix from the fusion feature matrix (information transmission), and the fusion feature matrix can condense information from the current modality feature matrix (information aggregation), thereby enhancing the richness of data features and solving the coordination problem between different modalities.

[0086] Specifically, the target fusion feature matrix of the (l-1)th self-aligned fusion layer is represented as:

[0087]

[0088] in, Let MLP2 represent the target fusion feature matrix of the (l-1)th self-aligned fusion layer, and MLP2 represent the second multilayer perceptron. This represents the fusion feature matrix output by the (l-1)th self-aligned fusion layer.

[0089] The updated target image feature matrix output by the (l-1)th self-aligned fusion layer is represented as:

[0090]

[0091] in, This represents the target image feature matrix output by the updated (l-1)th self-alignment fusion layer, where Self-Alignment1 represents the first self-alignment module. Represents the image feature matrix. This represents the target image feature matrix output by the (l-1)th self-aligned fusion layer;

[0092] The updated target audio feature matrix output by the (l-1)th self-aligned fusion layer is represented as follows:

[0093]

[0094] in, Self-Alignment2 represents the target audio feature matrix output by the updated (l-1)th self-aligned fusion layer, and Self-Alignment2 represents the second self-alignment module. Represents the audio feature matrix. This represents the target audio feature matrix output by the (l-1)th self-aligned fusion layer.

[0095] Furthermore, in some embodiments of this application, the target image feature matrix and the first fusion feature matrix of the l-th self-aligned fusion layer are represented as follows:

[0096]

[0097] in, This represents the target image feature matrix of the l-th self-aligned fusion layer. Fusion1 represents the first fusion feature matrix of the l-th self-aligned fusion layer, and Fusion1 represents the first encoder.

[0098] The target audio feature matrix and the second fused feature matrix of the l-th self-aligned fusion layer are represented as follows:

[0099]

[0100] in, This represents the target audio feature matrix of the l-th self-aligned fusion layer. Fusion2 represents the second fusion feature matrix of the l-th self-aligned fusion layer;

[0101] The fusion feature matrix of the l-th self-aligned fusion layer is represented as:

[0102]

[0103] in, Let represent the fusion feature matrix of the l-th self-aligned fusion layer, and Avg denotes the averaging.

[0104] Furthermore, the classification result of the video to be classified in step S30 can be expressed as:

[0105]

[0106] in, This represents the classification result of the video to be classified, where MLP3 represents the third multilayer perceptron. This represents the cls token vector in the target image feature matrix output by the Lth self-aligned fusion layer. This represents the fusion feature matrix output by the Lth self-aligned fusion layer. Let cls be the cls token vector in the target audio feature matrix output by the Lth self-aligned fusion layer.

[0107] Optionally, in some embodiments of this application, a convolutional neural network (CNN) can be used to extract features from each image frame in the video, and then the features of each image frame can be input into a recurrent neural network (RNN) or a long short-term memory network (LSTM) to output the image feature matrix of the video to be classified; a log-Mel spectrum sequence can be generated using short-time Fourier transform, and then the Mel spectrum sequence can be input into a CNN to output the audio feature matrix of the video to be classified.

[0108] Optionally, in some other embodiments of this application, a VisionTransformer (ViT) can be used to extract the image feature matrix of the video to be classified, which specifically includes:

[0109] Image frames in the video to be classified are sampled to obtain the target image frame sequence;

[0110] The target image frame sequence is normalized, and then data augmentation is performed on the normalized target image frame sequence.

[0111] Specifically, image frame standardization refers to scaling all image frames to a uniform size and normalizing the pixel values ​​of all image frames; image frame data augmentation refers to processing image frames by rotating, cropping, etc.

[0112] The augmented target image frame sequence is input into the visual converter, which outputs the image feature matrix of the video to be classified.

[0113] For example, in a specific example of this application, 8 keyframes are selected from the video to be classified, and each of the 8 keyframes is scaled to 224*224 pixels.

[0114] Visual converters can effectively reduce computational redundancy and improve the efficiency of feature extraction by decomposing spatial and temporal dimensions. Furthermore, they can learn deep-level features in videos through multi-layer nonlinear transformations, thereby improving the effectiveness of feature extraction.

[0115] Optionally, in some other embodiments of this application, an Audio Spectrogram Transformer (AST) can be used to extract the audio feature matrix of the video to be classified, which specifically includes:

[0116] The audio signals in the videos to be classified are converted to obtain Mel spectrograms, and then the Mel spectrograms are standardized.

[0117] Specifically, the standardization process for the Mel spectrogram includes mean centering and standard deviation scaling;

[0118] The standardized Mel spectrogram is input into the audio spectrogram transformer, which outputs the audio feature matrix of the video to be classified.

[0119] The audio spectrogram transformer, through the self-attention mechanism in the Transformer architecture, can capture long-range global context information in the audio spectrogram and also capture spatial result information in the two-dimensional audio spectrogram, thus improving the accuracy and robustness of audio feature extraction.

[0120] Optionally, in some embodiments of this application, after obtaining the image feature matrix and the audio feature matrix, the dimensions of the image feature matrix and the audio feature matrix are adjusted to ensure that the feature matrices of the two modalities have the same dimensions, and the embedded content of cls (classification tag) is added to the feature matrix.

[0121] like Figure 2 The diagram shown is a schematic of the video classification model structure provided in this application. Figure 3 As shown Figure 2 The diagram shows the structure of the self-aligned fusion layer. First, ViT and AST are used to extract image and audio features of the video to be classified. The extracted image and audio features are then input into the video classification model for three-stage feature fusion. The features obtained after the third stage fusion are then input into the multilayer perceptron for classification to obtain the video classification result.

[0122] Based on the video classification method based on multimodal feature fusion provided in the above embodiments, this application also provides a video classification device based on multimodal feature fusion, such as... Figure 4 As shown, the device specifically includes:

[0123] The first feature fusion module 10 is used to concatenate the image feature matrix and audio feature matrix of the video to be classified and input them into the first multilayer perceptron, and output the initial fused feature matrix.

[0124] The second feature fusion module 20 is used to input the image feature matrix, the audio feature matrix, and the initial fused feature matrix into L self-aligned fusion layers connected in series along the forward propagation direction for fusion; wherein, the step of the l-th self-aligned fusion layer fusing the input feature matrix includes:

[0125] The matrix dimension-upgrading submodule is used to input the fusion feature matrix output by the (l-1)th self-aligned fusion layer into the second multilayer perceptron to obtain the target fusion feature matrix of the (l-1)th self-aligned fusion layer.

[0126] The image feature self-alignment submodule is used to input the image feature matrix and the target image feature matrix output by the (l-1)th self-alignment fusion layer into the first self-alignment module for self-alignment, so as to update the target image feature matrix output by the (l-1)th self-alignment fusion layer.

[0127] The image feature fusion submodule is used to input the updated target image feature matrix output by the (l-1)th self-aligned fusion layer and the target fusion feature matrix of the (l-1)th self-aligned fusion layer into the first encoder, and output the target image feature matrix of the l-th self-aligned fusion layer and the first fusion feature matrix.

[0128] The audio feature self-alignment submodule is used to input the audio feature matrix and the target audio feature matrix output by the (l-1)th self-alignment fusion layer into the second self-alignment module for self-alignment, so as to update the target audio feature matrix output by the (l-1)th self-alignment fusion layer.

[0129] The audio feature fusion submodule is used to input the updated target audio feature matrix output from the (l-1)th self-aligned fusion layer and the target fusion feature matrix of the (l-1)th self-aligned fusion layer into the second encoder, and output the target audio feature matrix of the l-th self-aligned fusion layer and the second fusion feature matrix.

[0130] The fusion feature matrix acquisition submodule is used to average the first and second fusion feature matrices to obtain the fusion feature matrix of the l-th self-aligned fusion layer; where l∈[1,L], when l=1, the target image feature matrix output by the (l-1)-th self-aligned fusion layer is the image feature matrix, and the target audio feature matrix output by the (l-1)-th self-aligned fusion layer is the audio feature matrix.

[0131] The fusion feature matrix output by the (l-1)th self-aligned fusion layer is the initial fusion feature matrix;

[0132] The third feature fusion and video classification module is used to concatenate the cls token vector in the target image feature matrix, the cls token vector in the target audio feature matrix, and the fused feature matrix output by the Lth self-aligned fusion layer, and input them into the third multilayer perceptron to output the classification result of the video to be classified.

[0133] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the video classification method based on multimodal feature fusion described above.

[0134] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0135] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0136] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0137] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0138] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A video classification method based on multimodal feature fusion, characterized in that, include: The image feature matrix and audio feature matrix of the video to be classified are concatenated and input into the first multilayer perceptron, and the initial fused feature matrix is ​​output. The image feature matrix, the audio feature matrix, and the initial fused feature matrix are input into L self-aligned fusion layers connected in series along the forward propagation direction for fusion; wherein, the step of the l-th self-aligned fusion layer fusing the input feature matrix includes: The fusion feature matrix output from the (l-1)th self-aligned fusion layer is input into the second multilayer perceptron to obtain the target fusion feature matrix of the (l-1)th self-aligned fusion layer. The image feature matrix and the target image feature matrix output by the (l-1)th self-alignment fusion layer are input into the first self-alignment module for self-alignment, so as to update the target image feature matrix output by the (l-1)th self-alignment fusion layer. The updated target image feature matrix output from the (l-1)th self-aligned fusion layer and the target fusion feature matrix of the (l-1)th self-aligned fusion layer are input into the first encoder, and the target image feature matrix of the l-th self-aligned fusion layer and the first fusion feature matrix are output. The audio feature matrix and the target audio feature matrix output by the (l-1)th self-aligned fusion layer are input into the second self-alignment module for self-alignment, so as to update the target audio feature matrix output by the (l-1)th self-aligned fusion layer. The updated target audio feature matrix output from the (l-1)th self-aligned fusion layer and the target fusion feature matrix of the (l-1)th self-aligned fusion layer are input into the second encoder, and the target audio feature matrix of the l-th self-aligned fusion layer and the second fusion feature matrix are output. The first fusion feature matrix and the second fusion feature matrix are averaged to obtain the fusion feature matrix of the l-th self-aligned fusion layer; where l∈[1,L], when l=1, the target image feature matrix output by the (l-1)-th self-aligned fusion layer is the image feature matrix, the target audio feature matrix output by the (l-1)-th self-aligned fusion layer is the audio feature matrix, and the fusion feature matrix output by the (l-1)-th self-aligned fusion layer is the initial fusion feature matrix; The cls token vector in the target image feature matrix, the cls token vector in the target audio feature matrix, and the fused feature matrix output from the Lth self-aligned fusion layer are concatenated and input into the third multilayer perceptron to output the classification result of the video to be classified.

2. The video classification method based on multimodal feature fusion according to claim 1, characterized in that, The target fusion feature matrix of the (l-1)th self-aligned fusion layer is represented as: in, Let MLP2 represent the target fusion feature matrix of the (l-1)th self-aligned fusion layer, and MLP2 represent the second multilayer perceptron. This represents the fusion feature matrix output by the (l-1)th self-aligned fusion layer.

3. The video classification method based on multimodal feature fusion according to claim 1, characterized in that, The updated target image feature matrix output by the (l-1)th self-aligned fusion layer is represented as: in, This represents the target image feature matrix output by the updated (l-1)th self-alignment fusion layer, where Self-Alignment1 represents the first self-alignment module. Represents the image feature matrix. This represents the target image feature matrix output by the (l-1)th self-aligned fusion layer; The updated target audio feature matrix output by the (l-1)th self-aligned fusion layer is represented as follows: in, Self-Alignment2 represents the target audio feature matrix output by the updated (l-1)th self-aligned fusion layer, and Self-Alignment2 represents the second self-alignment module. Represents the audio feature matrix. This represents the target audio feature matrix output by the (l-1)th self-aligned fusion layer.

4. The video classification method based on multimodal feature fusion according to claim 3, characterized in that, The target image feature matrix and the first fusion feature matrix of the l-th self-aligned fusion layer are represented as follows: in, This represents the target image feature matrix of the l-th self-aligned fusion layer. Fusion1 represents the first fusion feature matrix of the l-th self-aligned fusion layer, and Fusion1 represents the first encoder. This represents the target fusion feature matrix of the (l-1)th self-aligned fusion layer; The target audio feature matrix and the second fused feature matrix of the l-th self-aligned fusion layer are represented as follows: in, This represents the target audio feature matrix of the l-th self-aligned fusion layer. Fusion2 represents the second fusion feature matrix of the l-th self-aligned fusion layer; The fusion feature matrix of the l-th self-aligned fusion layer is represented as: in, Let represent the fusion feature matrix of the l-th self-aligned fusion layer, and Avg denotes the averaging.

5. The video classification method based on multimodal feature fusion according to claim 1, characterized in that, The initial fusion feature matrix is ​​represented as follows: in, denoted as the initial fused feature matrix, MLP1 represents the first multilayer perceptron, and stack represents the concatenation operation. Represents the image feature matrix. This represents the audio feature matrix.

6. The video classification method based on multimodal feature fusion according to claim 1, characterized in that, The classification result of the video to be classified is represented as follows: in, This represents the classification result of the video to be classified, where MLP3 represents the third multilayer perceptron. This represents the cls token vector in the target image feature matrix output by the Lth self-aligned fusion layer. This represents the fusion feature matrix output by the Lth self-aligned fusion layer. denoted as clstoken vector in the target audio feature matrix output by the Lth self-aligned fusion layer.

7. The video classification method based on multimodal feature fusion according to claim 1, characterized in that, The process of obtaining the image feature matrix of the video to be classified includes: Image frames in the video to be classified are sampled to obtain the target image frame sequence; The target image frame sequence is normalized, and the normalized target image frame sequence is then augmented. The augmented target image frame sequence is input into the visual converter, which outputs the image feature matrix of the video to be classified.

8. The video classification method based on multimodal feature fusion according to claim 1, characterized in that, The process of obtaining the audio feature matrix of the video to be classified includes: The audio signal in the video to be classified is converted to obtain a Mel spectrogram, and the Mel spectrogram is then standardized. The standardized Mel spectrogram is input into the audio spectrogram transformer, which outputs the audio feature matrix of the video to be classified.

9. A video classification device based on multimodal feature fusion, characterized in that, include: The first feature fusion module is used to concatenate the image feature matrix and audio feature matrix of the video to be classified and input them into the first multilayer perceptron, and output the initial fused feature matrix. The second feature fusion module is used to input the image feature matrix, the audio feature matrix, and the initial fused feature matrix into L self-aligned fusion layers connected in series along the forward propagation direction for fusion; wherein, the step of the l-th self-aligned fusion layer fusing the input feature matrix includes: The matrix dimension-upgrading submodule is used to input the fusion feature matrix output by the (l-1)th self-aligned fusion layer into the second multilayer perceptron to obtain the target fusion feature matrix of the (l-1)th self-aligned fusion layer. The image feature self-alignment submodule is used to input the image feature matrix and the target image feature matrix output by the (l-1)th self-alignment fusion layer into the first self-alignment module for self-alignment, so as to update the target image feature matrix output by the (l-1)th self-alignment fusion layer. The image feature fusion submodule is used to input the updated target image feature matrix output by the (l-1)th self-aligned fusion layer and the target fusion feature matrix of the (l-1)th self-aligned fusion layer into the first encoder, and output the target image feature matrix of the l-th self-aligned fusion layer and the first fusion feature matrix. The audio feature self-alignment submodule is used to input the audio feature matrix and the target audio feature matrix output by the (l-1)th self-alignment fusion layer into the second self-alignment module for self-alignment, so as to update the target audio feature matrix output by the (l-1)th self-alignment fusion layer. The audio feature fusion submodule is used to input the updated target audio feature matrix output from the (l-1)th self-aligned fusion layer and the target fusion feature matrix of the (l-1)th self-aligned fusion layer into the second encoder, and output the target audio feature matrix of the l-th self-aligned fusion layer and the second fusion feature matrix. The fusion feature matrix acquisition submodule is used to average the first fusion feature matrix and the second fusion feature matrix to obtain the fusion feature matrix of the l-th self-aligned fusion layer; where l∈[1,L], when l=1, the target image feature matrix output by the (l-1)-th self-aligned fusion layer is the image feature matrix, the target audio feature matrix output by the (l-1)-th self-aligned fusion layer is the audio feature matrix, and the fusion feature matrix output by the (l-1)-th self-aligned fusion layer is the initial fusion feature matrix; The third feature fusion and video classification module is used to concatenate the cls token vector in the target image feature matrix, the cls token vector in the target audio feature matrix, and the fused feature matrix output by the Lth self-aligned fusion layer, and input them into the third multilayer perceptron to output the classification result of the video to be classified.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the video classification method based on multimodal feature fusion as described in any one of claims 1-8.

Images