Video classification method and device based on multi-modal feature fusion, and medium
Through the design of multi-stage feature fusion strategy and self-aligned fusion layer, the problem that video classification methods in the existing technology cannot fully integrate image and audio features is solved, the accuracy and reliability of video classification are improved, and more comprehensive multi-dimensional Modal data expression.
Patent Information
- Application Number
- CN202411103411.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-13
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2044-08-13
AI Technical Summary
The existing video classification methods cannot fully integrate and utilize image features and audio features, resulting in low accuracy and reliability of video classification results.
Using a multi-stage feature fusion strategy, the image feature matrix and audio feature matrix are initially fused through a multi-layer perceptron, and feature fusion is performed through multiple self-alignment fusion layers. The self-alignment module and encoder are used to capture the complementarity of different modal features. and correlation, and finally generate classification results through multi-layer perceptrons.
It improves the accuracy and reliability of video classification results, fully integrates image features and audio features, avoids loss and redundancy of feature information, and enhances the complexity and diversity expression of multimodal data.
Smart Images

Figure CN119048814A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of video classification, and in particular to a video classification method, device and computer-readable storage medium based on multimodal feature fusion. Background Art
[0002] Video classification is one of the important tasks in the field of machine learning and computer vision. Its goal is to classify videos into corresponding categories based on their content. Video classification plays an important role in video recommendation systems, content filtering, monitoring systems, and video retrieval. With the explosive growth of Internet video data, how to accurately and efficiently classify videos is becoming increasingly important.
[0003] Traditional video classification methods mainly rely on visual information. They usually use convolutional neural networks (CNNs) to extract image frame features in videos, and then combine recurrent neural networks (RNNs) or long short-term memory networks (LSTMs) to process the time series information composed of the features of each image frame. Although this method fully extracts the image information contained in the video, it ignores the rich audio information contained in the video, resulting in low accuracy of the final classification results.
[0004] In order to solve this problem, the method of integrating multimodal information for video classification has gradually become a research hotspot. In addition to relying on the visual information of the video, the current video classification method also integrates the audio information of the video. The fusion methods of visual information and audio information include the following: 1. Video classification is based on image features and audio features respectively, and finally the classification results based on image features and the classification results based on audio are combined in the classification decision stage of the model. This method is simple and easy to implement, but it does not consider the complementarity and correlation between different modal features, and cannot fully extract the feature information of the video. In addition, its classification results are greatly affected by single-modal features. If the classification result based on a certain modality has a large error, it will reduce the accuracy of the video classification result; 2. Directly splice the image features and audio features at the input layer of the model to obtain high-dimensional fusion features, and then normalize and deep transform the high-dimensional fusion features. Operations such as feature extraction are performed to obtain the target features of the video to complete the classification of the video. However, the direct splicing method cannot capture the complementary information between different modalities, which reduces the accuracy of the video classification results. In addition, the features of the two modalities may contain some repeated information. Direct splicing will also lead to information redundancy, increase the calculation burden of the model, and reduce the efficiency of video classification; 3. Introducing an attention mechanism or a gating mechanism in the middle layer of the model to dynamically adjust the importance of image features and modal features, and automatically pay attention to the modal features that are more helpful for video classification, thereby capturing the interaction and complementarity between the two modalities. However, since the attention mechanism or the gating mechanism often only performs feature weighting in a local range, it is difficult to capture the global relationship between different modalities, which makes some important trans-membrane interaction information ignored, resulting in the obtained fusion feature information not containing all the information of the two modalities, thereby reducing the accuracy of the video classification results.
[0005] In summary, the existing video classification methods based on the fusion of image features and audio features are unable to fully integrate and utilize the features of different modalities, resulting in low accuracy and reliability of video classification results. Summary of the invention
[0006] To this end, the technical problem to be solved by the present invention is to overcome the problem that the video classification method in the prior art cannot fully integrate and utilize the features of different modalities, resulting in low accuracy and reliability of video classification results.
[0007] In order to solve the above technical problems, the present invention provides a video classification method based on multimodal feature fusion, comprising:
[0008] The image feature matrix and the audio feature matrix of the video to be classified are concatenated and input into the first multi-layer perceptron, and the initial fused feature matrix is output;
[0009] The image feature matrix, the audio feature matrix and the initial fusion feature matrix are input to L self-aligned fusion layers connected in series in a forward propagation direction for fusion; wherein the step of fusing the input feature matrix by the lth self-aligned fusion layer includes:
[0010] Input the fusion feature matrix output by the l-1th self-aligned fusion layer into the second multi-layer perceptron to obtain the target fusion feature matrix of the l-1th self-aligned fusion layer;
[0011] Inputting the image feature matrix and the target image feature matrix output by the l-1th self-alignment fusion layer into the first self-alignment module for self-alignment, so as to update the target image feature matrix output by the l-1th self-alignment fusion layer;
[0012] Input the updated target image feature matrix output by the l-1th self-aligned fusion layer and the target fusion feature matrix of the l-1th self-aligned fusion layer to the first encoder, and output the target image feature matrix of the lth self-aligned fusion layer and the first fusion feature matrix;
[0013] Inputting the audio feature matrix and the target audio feature matrix output by the l-1th self-aligned fusion layer into the second self-aligned module for self-alignment, so as to update the target audio feature matrix output by the l-1th self-aligned fusion layer;
[0014] Input the updated target audio feature matrix output by the l-1th self-aligned fusion layer and the target fusion feature matrix of the l-1th self-aligned fusion layer to the second encoder, and output the target audio feature matrix of the lth self-aligned fusion layer and the second fusion feature matrix;
[0015] Averaging the first fusion feature matrix and the second fusion feature matrix to obtain a fusion feature matrix of the l-th self-aligned fusion layer; wherein, l∈[1,L], when l=1, the target image feature matrix output by the l-1-th self-aligned fusion layer is the image feature matrix, the target audio feature matrix output by the l-1-th self-aligned fusion layer is the audio feature matrix, and the fusion feature matrix output by the l-1-th self-aligned fusion layer is the initial fusion feature matrix;
[0016] The cls token vector in the target image feature matrix output by the Lth self-aligned fusion layer, the cls token vector in the target audio feature matrix and the fusion feature matrix are concatenated and input into the third multi-layer perceptron to output the classification result of the video to be classified.
[0017] Preferably, the target fusion feature matrix of the l-1th self-aligned fusion layer is expressed as:
[0018]
[0019] in, represents the target fusion feature matrix of the l-1th self-aligned fusion layer, MLP2 represents the second multi-layer perceptron, Represents the fused feature matrix output by the l-1th self-aligned fusion layer.
[0020] Preferably, the updated target image feature matrix output by the l-1th self-aligned fusion layer is expressed as:
[0021]
[0022] in, represents the updated target image feature matrix output by the l-1th self-aligned fusion layer, Self-Alignment1 represents the first self-alignment module, represents the image feature matrix, Represents the target image feature matrix output by the l-1th self-aligned fusion layer;
[0023] The updated target audio feature matrix output by the l-1th self-aligned fusion layer is expressed as:
[0024]
[0025] in, represents the updated target audio feature matrix output by the l-1th self-aligned fusion layer, Self-Alignment2 represents the second self-alignment module, represents the audio feature matrix, Represents the target audio feature matrix output by the l-1th self-aligned fusion layer.
[0026] Preferably, the target image feature matrix and the first fusion feature matrix of the lth self-aligned fusion layer are expressed as:
[0027]
[0028] in, represents the target image feature matrix of the lth self-aligned fusion layer, represents the first fusion feature matrix of the lth self-aligned fusion layer, and Fusion1 represents the first encoder; represents the target fusion feature matrix of the l-1th self-aligned fusion layer;
[0029] The target audio feature matrix and the second fusion feature matrix of the lth self-aligned fusion layer are expressed as:
[0030]
[0031] in, represents the target audio feature matrix of the lth self-aligned fusion layer, represents the second fusion feature matrix of the lth self-aligned fusion layer, and Fusion2 represents the second encoder;
[0032] The fusion feature matrix of the lth self-aligned fusion layer is expressed as:
[0033]
[0034] in, It represents the fusion feature matrix of the lth self-aligned fusion layer, and Avg represents the average.
[0035] Preferably, the initial fusion feature matrix is expressed as:
[0036]
[0037] in, represents the initial fusion feature matrix, MLP1 represents the first multi-layer perceptron, stack represents the concatenation operation, represents the image feature matrix, Represents the audio feature matrix.
[0038] Preferably, the classification result of the video to be classified is expressed as:
[0039]
[0040] in, represents the classification result of the video to be classified, MLP3 represents the third multi-layer perceptron, Represents the cls token vector in the target image feature matrix output by the Lth self-aligned fusion layer, represents the fusion feature matrix output by the Lth self-aligned fusion layer, Represents the cls token vector in the target audio feature matrix output by the Lth self-aligned fusion layer.
[0041] Preferably, the process of obtaining the image feature matrix of the video to be classified includes:
[0042] Sampling the image frames in the video to be classified to obtain a target image frame sequence;
[0043] Performing standardization processing on the target image frame sequence, and performing data enhancement on the target image frame sequence after the standardization processing;
[0044] The target image frame sequence after data enhancement is input into the visual converter, and the image feature matrix of the video to be classified is output.
[0045] Preferably, the process of obtaining the audio feature matrix of the video to be classified includes:
[0046] Converting the audio signal in the video to be classified to obtain a Mel-spectrogram, and performing standardization processing on the Mel-spectrogram;
[0047] The standardized Mel-spectrogram is input into the audio spectrogram converter, and the audio feature matrix of the video to be classified is output.
[0048] The present invention also provides a video classification device based on multimodal feature fusion, comprising:
[0049] A first feature fusion module is used to splice the image feature matrix and the audio feature matrix of the video to be classified and input them into the first multi-layer perceptron, and output an initial fused feature matrix;
[0050] The second feature fusion module is used to input the image feature matrix, the audio feature matrix and the initial fusion feature matrix into L self-aligned fusion layers connected in series in a forward propagation direction for fusion; wherein the step of fusing the input feature matrix by the lth self-aligned fusion layer includes:
[0051] The matrix dimension-increasing submodule is used to input the fusion feature matrix output by the l-1th self-aligned fusion layer into the second multi-layer perceptron to obtain the target fusion feature matrix of the l-1th self-aligned fusion layer;
[0052] An image feature self-alignment submodule, used for inputting the image feature matrix and the target image feature matrix output by the l-1th self-alignment fusion layer into the first self-alignment module for self-alignment, so as to update the target image feature matrix output by the l-1th self-alignment fusion layer;
[0053] An image feature fusion submodule, used to input the updated target image feature matrix output by the l-1th self-aligned fusion layer and the target fusion feature matrix of the l-1th self-aligned fusion layer into the first encoder, and output the target image feature matrix of the lth self-aligned fusion layer and the first fusion feature matrix;
[0054] An audio feature self-alignment submodule, used for inputting the audio feature matrix and the target audio feature matrix output by the l-1th self-alignment fusion layer into a second self-alignment module for self-alignment, so as to update the target audio feature matrix output by the l-1th self-alignment fusion layer;
[0055] An audio feature fusion submodule, used to input the updated target audio feature matrix output by the l-1th self-aligned fusion layer and the target fusion feature matrix of the l-1th self-aligned fusion layer into the second encoder, and output the target audio feature matrix of the lth self-aligned fusion layer and the second fusion feature matrix;
[0056] A fusion feature matrix acquisition submodule is used to average the first fusion feature matrix and the second fusion feature matrix to obtain a fusion feature matrix of the l-th self-aligned fusion layer; wherein, l∈[1,L], when l=1, the target image feature matrix output by the l-1-th self-aligned fusion layer is the image feature matrix, the target audio feature matrix output by the l-1-th self-aligned fusion layer is the audio feature matrix, and the fusion feature matrix output by the l-1-th self-aligned fusion layer is the initial fusion feature matrix;
[0057] The third feature fusion and video classification module is used to splice the cls token vector in the target image feature matrix output by the Lth self-aligned fusion layer, the cls token vector in the target audio feature matrix and the fusion feature matrix and input them into the third multi-layer perceptron to output the classification result of the video to be classified.
[0058] The present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned video classification method based on multimodal feature fusion are implemented.
[0059] The video classification method based on multimodal feature fusion provided by this application has the following beneficial effects:
[0060] 1. A multi-stage feature fusion strategy is adopted to fully integrate the image features and audio features of the video to be classified: In the first stage, the image feature matrix and the audio feature matrix are spliced and passed through a multi-layer perceptron to obtain an initial fusion feature matrix containing two modal features. The early fusion retains the rich initial information of the two modal data and effectively removes the repeated and redundant information in each modality; In the second stage, the image feature matrix, the audio feature matrix and the initial fusion feature matrix are input into multiple series-connected self-aligned fusion layers for feature fusion, thereby enhancing the richness of data features, solving the synergy problem between different modal information, and fully capturing the correlation between different modal information; Since the fusion feature matrix output by the last self-aligned fusion layer concentrates the feature information of different modalities, in the feature fusion of the third stage, the target image feature matrix, the target audio feature matrix and the fusion feature matrix output by the last self-aligned fusion layer are spliced to classify the video to be classified, so that the spliced features can more comprehensively express the complexity and diversity of multimodal data, avoid feature information loss, and thus improve the accuracy of the classification results;
[0061] 2. In the fusion process of the second stage, each self-aligned fusion layer first uses a multi-layer perceptron to upgrade the fusion features output by the previous self-aligned fusion layer to obtain a target fusion feature matrix. At the same time, the self-aligned module is used to align the features of each modality separately to ensure that the feature information of each modality is not weakened or lost during the fusion process, and to maintain the diversity and integrity of the multimodal data. Then, the self-aligned features of each modality are spliced with the target fusion feature matrix and input into the corresponding encoder for fusion. Since the target fusion feature matrix contains the features of two modalities, it plays a role in information aggregation and information transmission in the fusion process, thereby outputting a target image feature matrix fused with audio feature information, a target audio feature matrix fused with image feature information, and a new fusion feature matrix condensed with image feature information and audio feature information, which fully integrates the image features and audio features of the video to be classified, ensures information sharing between different modal features, and thus improves the accuracy of video classification results. BRIEF DESCRIPTION OF THE DRAWINGS
[0062] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below according to specific embodiments of the present invention in conjunction with the accompanying drawings, wherein:
[0063] Figure 1 A flow chart of the video classification method based on multimodal feature fusion provided in this application;
[0064] Figure 2 Schematic diagram of the video classification model structure provided for this application;
[0065] Figure 3 for Figure 2 Schematic diagram of the structure of the self-aligned fusion layer;
[0066] Figure 4 A schematic diagram of the structure of a video classification device based on multimodal feature fusion provided in this application. DETAILED DESCRIPTION
[0067] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.
[0068] See also Figure 1 , Figure 1 The flowchart of the video classification method based on multimodal feature fusion provided by the embodiment of the present application is shown, and the method specifically includes:
[0069] S10: splicing the image feature matrix and the audio feature matrix of the video to be classified and inputting them into the first multi-layer perceptron, and outputting an initial fusion feature matrix;
[0070] S20: Inputting the image feature matrix, the audio feature matrix and the initial fusion feature matrix into L self-aligned fusion layers connected in series in the forward propagation direction for fusion; wherein the step of fusing the input feature matrix by the lth self-aligned fusion layer includes:
[0071] Input the fusion feature matrix output by the l-1th self-aligned fusion layer into the second multi-layer perceptron to obtain the target fusion feature matrix of the l-1th self-aligned fusion layer;
[0072] The image feature matrix and the target image feature matrix output by the l-1th self-aligned fusion layer are input into the first self-aligned module for self-alignment, so as to update the target image feature matrix output by the l-1th self-aligned fusion layer;
[0073] Input the updated target image feature matrix output by the l-1th self-aligned fusion layer and the target fusion feature matrix of the l-1th self-aligned fusion layer to the first encoder, and output the target image feature matrix of the lth self-aligned fusion layer and the first fusion feature matrix;
[0074] Input the audio feature matrix and the target audio feature matrix output by the l-1th self-aligned fusion layer into the second self-aligned module for self-alignment, so as to update the target audio feature matrix output by the l-1th self-aligned fusion layer;
[0075] Input the updated target audio feature matrix output by the l-1th self-aligned fusion layer and the target fusion feature matrix of the l-1th self-aligned fusion layer to the second encoder, and output the target audio feature matrix of the lth self-aligned fusion layer and the second fusion feature matrix;
[0076] The first fusion feature matrix and the second fusion feature matrix are averaged to obtain the fusion feature matrix of the l-th self-aligned fusion layer; where l∈[1,L], when l=1, the target image feature matrix output by the l-1-th self-aligned fusion layer is the image feature matrix, the target audio feature matrix output by the l-1-th self-aligned fusion layer is the audio feature matrix, and the target audio feature matrix output by the l-1-th self-aligned fusion layer is the audio feature matrix.
[0077] The fused feature matrix obtained is the initial fused feature matrix;
[0078] S30: splice the cls token vector in the target image feature matrix output by the Lth self-aligned fusion layer, the cls token vector in the target audio feature matrix, and the fusion feature matrix and input them into the third multi-layer perceptron, and output the classification result of the video to be classified.
[0079] The video classification method based on multimodal feature fusion provided by the present application adopts a multi-stage feature fusion strategy to fully fuse the image features and audio features of the video to be classified: in the first stage, the image feature matrix and the audio feature matrix are spliced and passed through a multi-layer perceptron to obtain an initial fusion feature matrix containing two modal features, so as to retain the rich initial information of the multimodal data and improve the representativeness and effectiveness of the initial fusion feature matrix. In addition, by preliminarily screening and integrating the features of different modalities, the repeated information and redundant information in various modalities can be effectively removed, so that the initial fusion feature is already a streamlined and efficient feature representation in the initial stage, containing the most valuable information of each modality;
[0080] In the second stage, multiple self-aligned fusion layers are used to fuse the single modal features and fusion features. Data of different modalities have different features and information. Through feature fusion, this information can be integrated to form a more comprehensive and accurate feature representation. At the same time, data of different modalities also have the same features and information. Using repeated information in multiple modalities to verify each other can also improve the reliability and robustness of the data. However, when the fusion is excessive, the fused features will gradually concentrate on the Western Sydney shared by multiple modalities, resulting in the loss of single modal information. And because each self-aligned fusion layer in this application has multiple inputs corresponding to multiple outputs, rather than multiple inputs and finally outputting a fusion feature, it is necessary to consider whether the output single modal features can retain the information of the original modality. Based on this, each self-aligned fusion layer in this application first uses a multi-layer perceptron to increase the dimension of the fusion features output by the previous self-aligned fusion layer to obtain the target fusion feature matrix. At the same time, the self-aligned module is used to align the features of each modality separately to ensure that the feature information of each modality is not weakened or lost during the fusion process, and the diversity and integrity of the multimodal data are maintained; then, each modality is self-aligned. The features are spliced with the target fusion feature matrix respectively and input into the corresponding encoder for fusion. Since the target fusion feature matrix contains the features of two modes, it plays the role of information aggregation and information transmission in the fusion process. Specifically, the target fusion feature will pay attention to the feature information of the single mode, and the information flow is fused from the single mode feature to the fusion feature. At the same time, since the dimension of the target fusion feature is generally much smaller than the dimension of the single mode feature, the fusion feature will only aggregate the necessary single mode features; similarly, the single mode feature will pay attention to the information of another mode contained in the target fusion feature, and update it according to the attention weight, and fuse the information of multiple modes, so as to output the target image feature matrix fused with audio feature information, the target audio feature matrix fused with image feature information, and the new fusion feature matrix condensed with image feature information and audio feature information, which fully integrates the image features and audio features of the video to be classified, ensures the information sharing between different modal features, and thus improves the accuracy of the video classification result; at the same time, since the dimension of the target fusion feature matrix is small, the present application uses it as an information fusion unit to transmit information, which can also effectively reduce the computational complexity;
[0081] Since the fused feature matrix output by the last self-aligned fusion layer condenses information of different modalities, in the feature fusion of the third stage, the target image feature matrix, target audio feature matrix and fused feature matrix output by the last self-aligned fusion layer are spliced to classify the video to be classified, so that the spliced features can more comprehensively express the complexity and diversity of multimodal data, avoid feature information loss, and thus improve the accuracy of the classification results.
[0082] Specifically, in some embodiments of the present application, the initial fusion feature matrix in step S10 is expressed as:
[0083]
[0084] in, represents the initial fusion feature matrix, MLP1 represents the first multi-layer perceptron, stack represents the concatenation operation, represents the image feature matrix, Represents the audio feature matrix.
[0085] Specifically, step S20 inputs the image feature matrix, the audio feature matrix and the initial fused feature matrix into the self-aligned fusion layer for the second stage of feature fusion to solve the synergy problem between multimodal data and improve the accuracy of feature integration; in this stage, the fused feature matrix plays the role of information aggregation and information transmission. Since the fused feature matrix condenses the information of two modalities, during the attention process, the current modality feature matrix can obtain the information of the other modality feature matrix from the fused feature matrix (information transmission), and the fused feature matrix can condense the information of the current modality feature matrix (information aggregation), thereby enhancing the richness of data features and solving the synergy problem between different modalities;
[0086] Specifically, the target fusion feature matrix of the l-1th self-aligned fusion layer is expressed as:
[0087]
[0088] in, represents the target fusion feature matrix of the l-1th self-aligned fusion layer, MLP2 represents the second multi-layer perceptron, Represents the fused feature matrix output by the l-1th self-aligned fusion layer.
[0089] The updated target image feature matrix output by the l-1th self-aligned fusion layer is expressed as:
[0090]
[0091] in, represents the updated target image feature matrix output by the l-1th self-aligned fusion layer, Self-Alignment1 represents the first self-alignment module, represents the image feature matrix, Represents the target image feature matrix output by the l-1th self-aligned fusion layer;
[0092] The updated target audio feature matrix output by the l-1th self-aligned fusion layer is expressed as:
[0093]
[0094] in, represents the updated target audio feature matrix output by the l-1th self-aligned fusion layer, Self-Alignment2 represents the second self-alignment module, represents the audio feature matrix, Represents the target audio feature matrix output by the l-1th self-aligned fusion layer.
[0095] Further, in some embodiments of the present application, the target image feature matrix and the first fusion feature matrix of the lth self-aligned fusion layer are expressed as:
[0096]
[0097] in, represents the target image feature matrix of the lth self-aligned fusion layer, represents the first fusion feature matrix of the lth self-aligned fusion layer, and Fusion1 represents the first encoder;
[0098] The target audio feature matrix and the second fusion feature matrix of the lth self-aligned fusion layer are expressed as:
[0099]
[0100] in, represents the target audio feature matrix of the lth self-aligned fusion layer, represents the second fusion feature matrix of the lth self-aligned fusion layer, and Fusion2 represents the second encoder;
[0101] The fusion feature matrix of the lth self-aligned fusion layer is expressed as:
[0102]
[0103] in, It represents the fusion feature matrix of the lth self-aligned fusion layer, and Avg represents the average.
[0104] Furthermore, the classification result of the video to be classified in step S30 can be expressed as:
[0105]
[0106] in, represents the classification result of the video to be classified, MLP3 represents the third multi-layer perceptron, Represents the cls token vector in the target image feature matrix output by the Lth self-aligned fusion layer, represents the fusion feature matrix output by the Lth self-aligned fusion layer, Represents the cls token vector in the target audio feature matrix output by the Lth self-aligned fusion layer.
[0107] Optionally, in some embodiments of the present application, a convolutional neural network (CNN) may be used to extract features of each image frame in a video, and then the features of each image frame may be input into a recurrent neural network (RNN) or a long short-term memory network (LSTM), and an image feature matrix of the video to be classified may be output; a short-time Fourier transform may be used to generate a logarithmic Mel-spectrogram sequence, and then the Mel-spectrogram sequence may be input into a CNN, and an audio feature matrix of the video to be classified may be output;
[0108] Optionally, in some other embodiments of the present application, a vision transformer (ViT) may be used to extract an image feature matrix of the video to be classified, which specifically includes:
[0109] Sampling the image frames in the video to be classified to obtain a target image frame sequence;
[0110] Performing standardization processing on the target image frame sequence, and performing data enhancement on the target image frame sequence after the standardization processing;
[0111] Specifically, standardizing the image frames means scaling all the image frames to a uniform size and normalizing the pixel values of all the image frames; data enhancement of the image frames means rotating, cropping, etc. the image frames;
[0112] The target image frame sequence after data enhancement is input into the visual converter, and the image feature matrix of the video to be classified is output.
[0113] For example, in a specific example of the present application, 8 key frames are selected from the video to be classified, and the 8 key frames are scaled to 224*224 pixels respectively.
[0114] Visual converters can effectively reduce computational redundancy and improve the efficiency of feature extraction by decomposing spatial and temporal dimensions. They can also learn deep features in videos through multi-layer nonlinear transformations, thereby improving the effectiveness of feature extraction.
[0115] Optionally, in some other embodiments of the present application, an audio spectrogram transformer (AST) may be used to extract an audio feature matrix of the video to be classified, which specifically includes:
[0116] The audio signal in the video to be classified is converted into a Mel-spectrogram, and the Mel-spectrogram is standardized;
[0117] Specifically, the normalization of the mel-spectrogram includes mean centering and standard deviation scaling;
[0118] The standardized Mel-spectrogram is input into the audio spectrogram converter, and the audio feature matrix of the video to be classified is output.
[0119] The audio spectrogram transformer can capture long-distance global context information in the audio spectrogram through the self-attention mechanism in the Transformer architecture, and can also capture the spatial result information of the two-dimensional audio spectrogram, thereby improving the accuracy and robustness of audio feature extraction.
[0120] Optionally, in some embodiments of the present application, after obtaining the image feature matrix and the audio feature matrix, the dimension of the image feature matrix and the audio feature matrix is also adjusted to ensure that the feature matrices of the two modalities have the same dimension, and the embedded content of cls (classification label) is added to the feature matrix.
[0121] like Figure 2 The figure shows a schematic diagram of the video classification model structure provided by this application. Figure 3 Shown Figure 2 The schematic diagram of the structure of the self-aligned fusion layer in , first use ViT and AST to extract the image features and audio features of the video to be classified, the extracted image features and audio features are input into the video classification model for three-stage feature fusion, and the features obtained after the third stage of fusion are input into the multi-layer perceptron for classification to obtain the video classification result.
[0122] Based on the video classification method based on multimodal feature fusion provided in the above embodiment, the embodiment of the present application also provides a video classification device based on multimodal feature fusion, such as Figure 4 As shown, the device specifically includes:
[0123] A first feature fusion module 10 is used to splice the image feature matrix and the audio feature matrix of the video to be classified and input them into a first multi-layer perceptron, and output an initial fused feature matrix;
[0124] The second feature fusion module 20 is used to input the image feature matrix, the audio feature matrix and the initial fusion feature matrix into L self-aligned fusion layers connected in series in the forward propagation direction for fusion; wherein the step of fusing the input feature matrix by the lth self-aligned fusion layer includes:
[0125] The matrix dimension-increasing submodule is used to input the fusion feature matrix output by the l-1th self-aligned fusion layer into the second multi-layer perceptron to obtain the target fusion feature matrix of the l-1th self-aligned fusion layer;
[0126] An image feature self-alignment submodule, used for inputting the image feature matrix and the target image feature matrix output by the l-1th self-alignment fusion layer into the first self-alignment module for self-alignment, so as to update the target image feature matrix output by the l-1th self-alignment fusion layer;
[0127] An image feature fusion submodule, used to input the updated target image feature matrix output by the l-1th self-aligned fusion layer and the target fusion feature matrix of the l-1th self-aligned fusion layer into the first encoder, and output the target image feature matrix of the lth self-aligned fusion layer and the first fusion feature matrix;
[0128] An audio feature self-alignment submodule, used for inputting the audio feature matrix and the target audio feature matrix output by the l-1th self-alignment fusion layer into the second self-alignment module for self-alignment, so as to update the target audio feature matrix output by the l-1th self-alignment fusion layer;
[0129] An audio feature fusion submodule, used to input the updated target audio feature matrix output by the l-1th self-aligned fusion layer and the target fusion feature matrix of the l-1th self-aligned fusion layer into the second encoder, and output the target audio feature matrix of the lth self-aligned fusion layer and the second fusion feature matrix;
[0130] The fusion feature matrix acquisition submodule is used to average the first fusion feature matrix and the second fusion feature matrix to obtain the fusion feature matrix of the l-th self-aligned fusion layer; wherein, l∈[1,L], when l=1, the target image feature matrix output by the l-1-th self-aligned fusion layer is the image feature matrix, and the target audio feature matrix output by the l-1-th self-aligned fusion layer is the audio feature matrix.
[0131] The fused feature matrix output by the l-1th self-aligned fusion layer is the initial fused feature matrix;
[0132] The third feature fusion and video classification module is used to splice the cls token vector in the target image feature matrix output by the Lth self-aligned fusion layer, the cls token vector in the target audio feature matrix and the fusion feature matrix and input them into the third multi-layer perceptron to output the classification result of the video to be classified.
[0133] An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned video classification method based on multimodal feature fusion are implemented.
[0134] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.
[0135] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0136] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0137] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0138] Obviously, the above embodiments are merely examples for the purpose of clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived therefrom are still within the scope of protection of the present invention.
Claims
1. A video classification method based on multimodal feature fusion, characterized in that: include: The image feature matrix and the audio feature matrix of the video to be classified are concatenated and input into the first multi-layer perceptron, and the initial fused feature matrix is output; The image feature matrix, the audio feature matrix and the initial fusion feature matrix are input to L self-aligned fusion layers connected in series in a forward propagation direction for fusion; wherein the step of fusing the input feature matrix by the lth self-aligned fusion layer includes: Input the fusion feature matrix output by the l-1th self-aligned fusion layer into the second multi-layer perceptron to obtain the target fusion feature matrix of the l-1th self-aligned fusion layer; Inputting the image feature matrix and the target image feature matrix output by the l-1th self-alignment fusion layer into the first self-alignment module for self-alignment, so as to update the target image feature matrix output by the l-1th self-alignment fusion layer; Input the updated target image feature matrix output by the l-1th self-aligned fusion layer and the target fusion feature matrix of the l-1th self-aligned fusion layer to the first encoder, and output the target image feature matrix of the lth self-aligned fusion layer and the first fusion feature matrix; Inputting the audio feature matrix and the target audio feature matrix output by the l-1th self-aligned fusion layer into the second self-aligned module for self-alignment, so as to update the target audio feature matrix output by the l-1th self-aligned fusion layer; Input the updated target audio feature matrix output by the l-1th self-aligned fusion layer and the target fusion feature matrix of the l-1th self-aligned fusion layer to the second encoder, and output the target audio feature matrix of the lth self-aligned fusion layer and the second fusion feature matrix; Averaging the first fusion feature matrix and the second fusion feature matrix to obtain a fusion feature matrix of the l-th self-aligned fusion layer; wherein, l∈[1,L], when l=1, the target image feature matrix output by the l-1-th self-aligned fusion layer is the image feature matrix, the target audio feature matrix output by the l-1-th self-aligned fusion layer is the audio feature matrix, and the fusion feature matrix output by the l-1-th self-aligned fusion layer is the initial fusion feature matrix; The cls token vector in the target image feature matrix output by the Lth self-aligned fusion layer, the cls token vector in the target audio feature matrix and the fusion feature matrix are concatenated and input into the third multi-layer perceptron to output the classification result of the video to be classified.
2. The video classification method based on multimodal feature fusion according to claim 1, characterized in that: The target fusion feature matrix of the l-1th self-aligned fusion layer is expressed as: in, represents the target fusion feature matrix of the l-1th self-aligned fusion layer, MLP2 represents the second multi-layer perceptron, Represents the fused feature matrix output by the l-1th self-aligned fusion layer.
3. The video classification method based on multimodal feature fusion according to claim 1, characterized in that: The updated target image feature matrix output by the l-1th self-aligned fusion layer is expressed as: in, represents the updated target image feature matrix output by the l-1th self-aligned fusion layer, Self-Alignment1 represents the first self-alignment module, represents the image feature matrix, Represents the target image feature matrix output by the l-1th self-aligned fusion layer; The updated target audio feature matrix output by the l-1th self-aligned fusion layer is expressed as: in, represents the updated target audio feature matrix output by the l-1th self-aligned fusion layer, Self-Alignment2 represents the second self-alignment module, represents the audio feature matrix, Represents the target audio feature matrix output by the l-1th self-aligned fusion layer.
4. The video classification method based on multimodal feature fusion according to claim 3 is characterized in that: The target image feature matrix and the first fusion feature matrix of the lth self-aligned fusion layer are expressed as: in, represents the target image feature matrix of the lth self-aligned fusion layer, represents the first fusion feature matrix of the lth self-aligned fusion layer, and Fusion1 represents the first encoder; represents the target fusion feature matrix of the l-1th self-aligned fusion layer; The target audio feature matrix and the second fusion feature matrix of the lth self-aligned fusion layer are expressed as: in, represents the target audio feature matrix of the lth self-aligned fusion layer, represents the second fusion feature matrix of the lth self-aligned fusion layer, and Fusion2 represents the second encoder; The fusion feature matrix of the lth self-aligned fusion layer is expressed as: in, It represents the fusion feature matrix of the lth self-aligned fusion layer, and Avg represents the average.
5. The video classification method based on multimodal feature fusion according to claim 1, characterized in that: The initial fusion feature matrix is expressed as: in, represents the initial fusion feature matrix, MLP1 represents the first multi-layer perceptron, stack represents the concatenation operation, represents the image feature matrix, Represents the audio feature matrix.
6. The video classification method based on multimodal feature fusion according to claim 1, characterized in that: The classification result of the video to be classified is expressed as: in, represents the classification result of the video to be classified, MLP3 represents the third multi-layer perceptron, Represents the cls token vector in the target image feature matrix output by the Lth self-aligned fusion layer, represents the fusion feature matrix output by the Lth self-aligned fusion layer, Represents the clstoken vector in the target audio feature matrix output by the Lth self-aligned fusion layer.
7. The video classification method based on multimodal feature fusion according to claim 1, characterized in that: The process of obtaining the image feature matrix of the video to be classified includes: Sampling the image frames in the video to be classified to obtain a target image frame sequence; Performing standardization processing on the target image frame sequence, and performing data enhancement on the target image frame sequence after the standardization processing; The target image frame sequence after data enhancement is input into the visual converter, and the image feature matrix of the video to be classified is output.
8. The video classification method based on multimodal feature fusion according to claim 1, characterized in that: The process of obtaining the audio feature matrix of the video to be classified includes: Converting the audio signal in the video to be classified to obtain a Mel-spectrogram, and performing standardization processing on the Mel-spectrogram; The standardized Mel-spectrogram is input into the audio spectrogram converter, and the audio feature matrix of the video to be classified is output.
9. A video classification device based on multimodal feature fusion, characterized in that: include: A first feature fusion module is used to splice the image feature matrix and the audio feature matrix of the video to be classified and input them into the first multi-layer perceptron, and output an initial fused feature matrix; The second feature fusion module is used to input the image feature matrix, the audio feature matrix and the initial fusion feature matrix into L self-aligned fusion layers connected in series in a forward propagation direction for fusion; wherein the step of fusing the input feature matrix by the lth self-aligned fusion layer includes: The matrix dimension-increasing submodule is used to input the fusion feature matrix output by the l-1th self-aligned fusion layer into the second multi-layer perceptron to obtain the target fusion feature matrix of the l-1th self-aligned fusion layer; An image feature self-alignment submodule, used for inputting the image feature matrix and the target image feature matrix output by the l-1th self-alignment fusion layer into the first self-alignment module for self-alignment, so as to update the target image feature matrix output by the l-1th self-alignment fusion layer; An image feature fusion submodule, used to input the updated target image feature matrix output by the l-1th self-aligned fusion layer and the target fusion feature matrix of the l-1th self-aligned fusion layer into the first encoder, and output the target image feature matrix of the lth self-aligned fusion layer and the first fusion feature matrix; An audio feature self-alignment submodule, used for inputting the audio feature matrix and the target audio feature matrix output by the l-1th self-alignment fusion layer into a second self-alignment module for self-alignment, so as to update the target audio feature matrix output by the l-1th self-alignment fusion layer; An audio feature fusion submodule, used to input the updated target audio feature matrix output by the l-1th self-aligned fusion layer and the target fusion feature matrix of the l-1th self-aligned fusion layer into the second encoder, and output the target audio feature matrix of the lth self-aligned fusion layer and the second fusion feature matrix; A fusion feature matrix acquisition submodule is used to average the first fusion feature matrix and the second fusion feature matrix to obtain a fusion feature matrix of the l-th self-aligned fusion layer; wherein, l∈[1,L], when l=1, the target image feature matrix output by the l-1-th self-aligned fusion layer is the image feature matrix, the target audio feature matrix output by the l-1-th self-aligned fusion layer is the audio feature matrix, and the fusion feature matrix output by the l-1-th self-aligned fusion layer is the initial fusion feature matrix; The third feature fusion and video classification module is used to splice the cls token vector in the target image feature matrix output by the Lth self-aligned fusion layer, the cls token vector in the target audio feature matrix and the fusion feature matrix and input them into the third multi-layer perceptron to output the classification result of the video to be classified.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the video classification method based on multimodal feature fusion according to any one of claims 1 to 8 are implemented.
Citation Information
Patent Citations
Short video classification method based on multi-modal information aggregation
CN116189047A
Video behavior recognition method based on cross-modal fusion
CN116311525A
Video classification method and device, electronic equipment and storage medium
CN117132923A
Method and apparatus for classifying video
US20210019531A1
