A multi-modal video classification method based on a dynamic fusion gate mechanism

Abstract

This invention relates to a multimodal video classification method based on a dynamic fusion gate mechanism, belonging to the field of video classification technology. To overcome the shortcomings of existing technologies, this invention aims to provide a multimodal video classification method based on a dynamic fusion gate mechanism. The method includes extracting multi-level visual and textual features of the target video through a single-modal feature extraction module; using a dynamic fusion gate module to obtain multimodal fusion features based on visual data and text data based on the multi-level visual features of video frames and the multi-level textual features of video subtitles; and using a video classification module to perform video classification based on the visual and textual multimodal fusion features. This invention uses a dynamic fusion gate mechanism to allow the model to learn more suitable single-modal feature usage quantities and multimodal fusion positions, thereby accelerating the training efficiency of the video classification model and improving its classification performance.

Description

TECHNICAL FIELD

[0001] The application relates to a multi-modal video classification method based on a dynamic fusion gate mechanism and belongs to the technical field of video classification. BACKGROUND

[0002] In a common multi-modal video classification model, a multi-modal fusion strategy is extremely important for the classification effect of the final model, and a general multi-modal fusion strategy needs to perform the following steps: firstly, extracting features of each single mode, then determining the number of each single mode feature (low-level feature or middle-level feature or high-level feature), and finally selecting a specific position (low-level network or middle-level network or high-level network) of the model for multi-modal fusion.

[0003] From the above description, it can be known that the common fusion strategy needs to manually determine two influence factors: the number of single mode features and the multi-modal fusion position. The two influence factors directly affect the multi-modal fusion effect and further affect the classification effect of the final model. However, manual determination of the influence factors will greatly increase the training cost (multiple attempts to find the optimal factors) and the manually determined influence factors may not be optimal for the classification effect of the final model. SUMMARY

[0004] In order to overcome the defects in the prior art, the application aims to provide a multi-modal video classification method based on a dynamic fusion gate mechanism.

[0005] The technical solution provided by the application to solve the above technical problems is: a multi-modal video classification method based on a dynamic fusion gate mechanism, comprising:

[0006] Step S1, extracting multi-level visual features of video frames and multi-level text features of video subtitles of a target video through a single mode feature extraction module;

[0007] Step S2, based on the multi-level visual features of the video frames and the multi-level text features of the video subtitles, using a dynamic fusion gate module to obtain multi-modal fusion features with vision as the main body and multi-modal fusion features with text as the main body;

[0008] Step S3, based on the multi-modal fusion features with vision as the main body and the multi-modal fusion features with text as the main body, using a video classification module to perform video classification.

[0009] Further, the single mode feature extraction module uses a ViT model and a Bert model to extract multi-level feature information of the video frames and the video subtitles respectively, and obtains multi-level visual features of the video frames and multi-level text features of the video subtitles respectively.

[0010] Further, the dynamic fusion gate module adopts a dynamic fusion gate mechanism, specifically an attention operation, and is divided into an information-dynamic fusion gate mechanism and a position-dynamic fusion gate mechanism according to different applications, wherein the information-dynamic gate mechanism determines the number of single-modal features used by the model, and the position-dynamic gate mechanism determines the position of multi-modal fusion of the model.

[0011] Further, the information-dynamic fusion gate mechanism is used to automatically configure the number of each single-modal feature, to obtain new visual multi-level features and new multi-level text features, and then the position-dynamic fusion gate mechanism is used to automatically select the position of multi-modal fusion, to obtain multi-modal fusion features with vision as the main body and multi-modal fusion features with text as the main body.

[0012] Further, the specific process of using the information-dynamic fusion gate mechanism to automatically configure the number of each single-modal feature is as follows:

[0013] The visual multi-level features are input into an information-dynamic fusion gate mechanism, which first multiplies the multi-level visual features to obtain the correlation scores between the multi-level visual features, then sets a gate threshold, sets the values lower than the gate threshold in the correlation scores to 0, and retains the original values of the other values to obtain new visual correlation scores, and finally weights the visual multi-level features based on the multi-level visual correlation scores to obtain new visual multi-level features, which are obtained by selecting useful visual multi-level features through the information-dynamic fusion gate mechanism;

[0014] Similarly, new multi-level text features corresponding to the text modal can also be obtained.

[0015] Further, the specific process of automatically selecting the position of multi-modal fusion through the position-dynamic fusion gate mechanism is as follows:

[0016] First, cross-attention operation is performed on the new multi-level text features and each level of the multi-level visual features to obtain fusion features with each visual level, a gate threshold is set, and the values of the fusion features less than the gate threshold are set to 0, and finally the fusion features passing the gate threshold are added to obtain the final fusion features, which are multi-modal fusion features with vision as the main body;

[0017] Similarly, fusion features in which visual features are fused into text can also be obtained, which are multi-modal fusion features with text as the main body.

[0018] Further, the video classification module concatenates the two features and inputs them into a fully connected layer, and then performs video classification.

[0019] The application has the following beneficial effects: the application uses two algorithm mechanisms of an information-dynamic fusion gate and a position-dynamic fusion gate to automatically confirm two influence factors of a single-modal feature use quantity and a multi-modal fusion position in a training process, thereby completely discarding a manual participation process, and enabling a final video classification model to have better classification effect. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 An extraction process diagram for a single-modal feature extraction module;

[0021] Figure 2 A fusion flowchart of an information-dynamic fusion gate;

[0022] Figure 3 A fusion flowchart of a position-dynamic fusion gate;

[0023] Figure 4 A multi-modal video classification method flowchart based on a dynamic fusion gate mechanism. DETAILED DESCRIPTION

[0024] The technical solutions of the application will be described clearly and completely below with reference to the drawings. Obviously, the described embodiments are part of the embodiments of the application, rather than all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the application.

[0025] A multi-modal video classification method based on a dynamic fusion gate mechanism of the application mainly uses three modules:

[0026] 1. A single-modal feature extraction module, which uses a common single-modal model to extract features of different single modes in a video, so as to perform a multi-modal fusion operation subsequently;

[0027] 2. A dynamic fusion gate module, which is divided into an information-dynamic fusion gate mechanism and a position-dynamic fusion gate mechanism, and respectively determines two influence factors of a single-modal feature use quantity and a multi-modal fusion position;

[0028] 3. A video classification module, which performs video classification based on multi-modal fusion to obtain multi-modal video feature information.

[0029] Specifically, the following steps are included:

[0030] Step S1, extract multi-level visual features of video frames of a target video and multi-level text features of a video subtitle through a single-modal feature extraction module;

[0031] The single-modal feature extraction module specifically adopts a ViT model and a Bert model to extract multi-level feature information of video frames and video subtitles respectively, to obtain multi-level visual features V = {v low , v middle , v high} of the video frames and multi-level text features T = {t low , t middle , t high} of the video subtitles respectively.

[0032] In step S2, based on the multi-level visual features of the video frames and the multi-level text features of the video subtitles, a dynamic fusion gate module is used to obtain multi-modal fusion features with visual as the main body and multi-modal fusion features with text as the main body;

[0033] The dynamic fusion gate module adopts a dynamic fusion gate mechanism, which is essentially an attention operation, and is divided into an information-dynamic fusion gate mechanism and a position-dynamic fusion gate mechanism according to different applications. The information-dynamic gate mechanism determines the number of single-modal features used by the model, and the position-dynamic gate mechanism determines the multi-modal fusion position of the model.

[0034] In step S21, the multi-level visual features V are input into an information-dynamic fusion gate mechanism. The mechanism will first multiply the multi-level visual features to obtain the correlation scores V M0 between the multi-level visual features, then set a gate threshold v thresh , set the values in the correlation scores V M that are lower than the gate threshold v thresh to 0, and retain the original values of the others to obtain new visual correlation scores V M , and finally weight the multi-level visual features V based on the multi-level visual correlation scores to obtain new multi-level visual features V1. The features are obtained by selecting useful multi-level visual features through the information-dynamic fusion gate mechanism.

[0035] Similarly, the corresponding new multi-level text features T1 of the text modal can also be obtained.

[0036] In step S22, the new multi-level text features T1 and the multi-level visual features V are input into a position-dynamic fusion gate mechanism. First, cross-attention operations are performed on each level of the new multi-level text features T1 and the multi-level visual features V to obtain fusion features VT0 = {vt low , vt middle , vt high} for each visual level. A gate threshold vt thresh is set, and the values in the fusion features VT0 that are less than vt threshThe value of the fusion feature that has passed the gate threshold is 0, and finally, the fusion features that have passed the gate threshold are added to obtain the final fusion feature, that is, the multi-modal fusion feature V Cls ;

[0037] Similarly, the new multi-level visual feature V1 fused into the fusion feature in the text can also be obtained, that is, the multi-modal fusion feature T based on the text Cls ;

[0038] Step S3, based on the multi-modal fusion feature based on vision and the multi-modal fusion feature based on text, video classification is performed by using a video classification module.

[0039] The above multi-modal fusion features based on vision V Cls and the multi-modal fusion features based on text T Cls have been obtained through the dynamic fusion gate mechanism, and the video classification module splices the two features and then transmits them into a full connection layer to perform video classification.

[0040] The present application enables the model to learn the number of single-modal features and the multi-modal fusion position suitable for itself through the dynamic fusion gate mechanism, thereby accelerating the training efficiency of the video classification model and improving the classification effect of the model.

[0041] The above description does not limit the present application in any form, although the present application has been disclosed through the above examples, however, it is not intended to limit the present application, any person skilled in the art, without departing from the technical solution of the present application, can make some changes or modifications to the above disclosed technical content to obtain equivalent examples of equivalent changes, but as long as it does not deviate from the technical solution of the present application, any simple modification, equivalent change and modification of the above examples according to the technical essence of the present application, still belongs to the scope of the technical solution of the present application.

Claims

1. A multimodal video classification method based on a dynamic fusion gate mechanism, characterized in that, include: Step S1: Extract multi-level visual features of video frames and multi-level text features of video subtitles from the target video using the single-modal feature extraction module; Step S2: Based on the multi-level visual features of video frames and the multi-level text features of video subtitles, a dynamic fusion gate module is used to obtain multi-modal fusion features with vision as the main body and multi-modal fusion features with text as the main body. The dynamic fusion gate module adopts a dynamic fusion gate mechanism, an attention operation, and is divided into information-dynamic fusion gate mechanism and position-dynamic fusion gate mechanism according to different applications. The information-dynamic gate mechanism determines the number of single-modal features used in the model, while the position-dynamic gate mechanism determines the multimodal fusion position of the model. The dynamic fusion gate module first uses an information-dynamic fusion gate mechanism to automatically configure the number of each single-modal feature used to obtain new visual multi-level features and new multi-level text features; then, it uses a position-dynamic fusion gate mechanism to automatically select the position of multimodal fusion to obtain multimodal fusion features with vision as the main body and multimodal fusion features with text as the main body. Step S3: Based on the multimodal fusion features with vision as the main body and the multimodal fusion features with text as the main body, the video classification module is used to classify the video.

2. The multimodal video classification method based on a dynamic fusion gate mechanism according to claim 1, characterized in that, The single-modal feature extraction module extracts multi-level feature information of video frames and video subtitles using the ViT model and the Bert model, respectively, to obtain multi-level visual features of video frames and multi-level text features of video subtitles.

3. The multimodal video classification method based on a dynamic fusion gate mechanism according to claim 1, characterized in that, The specific process of automatically configuring the amount of each monomodal feature used using the information-dynamic fusion gate mechanism is as follows: The visual multi-level features are fed into an information-dynamic fusion gate mechanism. This mechanism first multiplies the multi-level visual features to obtain the correlation scores between the multi-level visual features. Then, a gate threshold is set, and all values ​​with correlation scores below the gate threshold are set to 0, while the others are retained to obtain new visual correlation scores. Finally, the visual multi-level features are weighted based on the multi-level visual correlation scores to obtain new visual multi-level features. These features are obtained by selecting useful visual multi-level features through the information-dynamic fusion gate mechanism. Similarly, new multi-level text features corresponding to text modalities can also be obtained.

4. The multimodal video classification method based on a dynamic fusion gate mechanism according to claim 1, characterized in that, The specific process of automatically selecting the location for multimodal fusion using the location-dynamic fusion gate mechanism is as follows: First, cross-attention operation is performed on the new multi-level text features and each level of the multi-level visual features to obtain the fusion features with each visual level. A threshold is set, and the values ​​of the fusion features that are less than the threshold are set to 0. Finally, the fusion features that have passed the threshold are added together to obtain the final fusion feature, which is the multimodal fusion feature with vision as the main body. Similarly, we can also obtain fusion features that integrate visual features into text, which are multimodal fusion features with text as the main body.

5. The multimodal video classification method based on a dynamic fusion gate mechanism according to claim 1, characterized in that, The video classification module concatenates the two features and feeds them into a fully connected layer for video classification.

Images