Video classification method and device based on multiple modes, electronic equipment and medium

Abstract

The embodiment of the invention provides a video classification method and device based on multiple modes, electronic equipment and a medium, belongs to the technical field of artificial intelligence, and is applied to financial scenes and medical scenes. The method comprises the following steps: acquiring a target video containing multi-modal information, and performing feature extraction on the multi-modal information; performing binary mask activation processing on modal pairs between every two constructed visual semantic features, text semantic features and audio semantic features, and performing modal pair screening on candidate modal pairs of binary mask pairs according to the modal pairs; and extracting and fusing each modal context feature of the multi-modal semantic feature to perform video classification on the target video according to the fused multi-modal feature. According to the embodiment of the invention, the binary mask activation processing is carried out on the multi-modal candidate modal pairs, the most relevant modal pairs can be adaptively screened for interaction, and the video classification is carried out on the target video according to the fused context multi-modal features, so that the accuracy of video classification is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology and is applied to financial and medical scenarios. In particular, it relates to a multimodal video classification method and device, electronic device and medium. Background Technology

[0002] Traditional video classification methods typically extract features from various modalities in a video using gated recurrent units (GRUs), and then concatenate these features using fully connected layers to achieve video classification. For example, in a financial scenario, for a video of a customer applying for a bank loan, GRU technology extracts visual features such as facial expressions and gestures from the image modality, audio features such as the customer's financial situation and loan purpose from the audio modality, and text features such as the loan contract and the customer's signature from the text modality. These visual, audio, and text features are then concatenated to obtain a fused feature, which is used to classify the video as a loan application. However, this method, when handling multimodal information, often processes each modality independently through gating, making it difficult to capture the semantic relationships between images, audio, and text. Furthermore, the simple concatenation of features using fully connected layers ignores the complex interactions between modalities, making it difficult for the model to capture the true meaning of the video, resulting in low accuracy in video classification. Therefore, improving the accuracy of video classification has become an urgent problem to be solved. Summary of the Invention

[0003] The main objective of this application is to propose a multimodal video classification method, apparatus, electronic device, and medium, aiming to improve the accuracy of video classification.

[0004] To achieve the above objectives, a first aspect of this application proposes a multimodal video classification method, the method comprising:

[0005] Acquire a target video; wherein the target video contains multiple modal information;

[0006] Semantic features are extracted from the multimodal information using a pre-trained video classification model to obtain multimodal semantic features; wherein, the multimodal semantic features include at least two of visual semantic features, textual semantic features, and audio semantic features;

[0007] The visual semantic features, the text semantic features, and the audio semantic features are paired to construct candidate modal pairs;

[0008] The candidate mode pairs are subjected to binary mask activation processing to obtain a mode pair binary mask, and the candidate mode pairs are filtered according to the mode pair binary mask to obtain the target mode pairs;

[0009] Based on the target modality, context features are extracted from each modality in the multimodal semantic features to obtain context features for each modality;

[0010] The context features of each modality are fused to obtain fused multimodal features;

[0011] The target video is classified based on the fused multimodal features to obtain target video classification data.

[0012] In some embodiments, performing binary mask activation processing on the candidate mode pairs to obtain a binary mask for the mode pairs includes:

[0013] The candidate mode pairs are activated to obtain the mode pair activation probabilities.

[0014] Activation detection is performed based on a preset indicator function and the activation probability of the modality pair to obtain a binary mask of the modality pair; the binary mask of the modality pair is used to characterize whether the candidate modality pair is activated or not.

[0015] In some embodiments, the step of extracting context features from each modality in the multimodal semantic features based on the target modality to obtain context features for each modality includes:

[0016] For one target mode in the target mode pair, attention weights are calculated for the target mode pair based on the binary mask of the mode pair to obtain the mode pair weights of the target mode;

[0017] For the target modality, the other modality in the target modality pair is weighted and summed according to the modality pair weights to obtain the target modality context features, and the target modality context features are determined as the context features of each modality.

[0018] In some embodiments, the step of calculating attention weights for the target mode pair based on the binary mask of the mode pair for one target mode pair to obtain the mode pair weights of the target mode includes:

[0019] For the target mode, preliminary attention weights are calculated for the target mode pair based on the binary mask of the mode pair to obtain preliminary mode pair weights;

[0020] Obtain all interaction modalities of the target modality, and perform binary masking operation on the target modality and all interaction modalities to obtain the interaction pair binary mask;

[0021] The attention weights of the interaction pair are calculated by performing attention weight calculation on the target modality based on the binary mask of the interaction pair;

[0022] The modality pair weights of the target modality are obtained by quotienting the initial modality pair weights with the interaction pair attention weights.

[0023] In some embodiments, the modal context features include visual semantic context features, text semantic context features, and audio semantic context features;

[0024] The step of fusing the context features of each modality to obtain fused multimodal features includes:

[0025] The visual semantic context features, the text semantic context features, and the audio semantic context features are concatenated to obtain concatenated multimodal features;

[0026] The spliced ​​multimodal features are rotated and position encoded to obtain the fused multimodal features.

[0027] In some embodiments, before extracting semantic features from the multiple modal information using a pre-trained video classification model to obtain multimodal semantic features, the method further includes:

[0028] Obtain the original video classification model and training videos, and label the training videos with the real video classification labels;

[0029] The training video is used to extract features through the original video classification model to obtain training multimodal semantic features; the training multimodal semantic features include at least two of the following: training visual semantic features, training text semantic features, and training audio semantic features.

[0030] The training visual semantic features, the training text semantic features, and the training audio semantic features are paired to construct training candidate modal pairs;

[0031] The training candidate mode pairs are activated to obtain the activation probability of the training mode pairs. The activation probability of the training mode pairs is then converted into a binary mask of the training mode pairs. The training candidate mode pairs are then filtered according to the binary mask of the training mode pairs to obtain the training target mode pairs.

[0032] Based on the training target modality, context features are extracted from each modality in the multimodal semantic features to obtain the training context features for each modality;

[0033] The training context features of each modality are fused to obtain the training fused multimodal features;

[0034] Based on the training fusion multimodal features, video classification prediction is performed on the training video to obtain the predicted video classification label;

[0035] The activation probability of the training mode pair, the predicted video classification label, and the target loss value of the video classification label are calculated according to the preset loss function.

[0036] The model parameters of the original video classification model are updated based on the target loss value.

[0037] In some embodiments, calculating the target loss value of the training mode pair activation probability, the predicted video classification label, and the real video classification label according to a preset loss function includes:

[0038] The activation loss value is obtained by calculating the activation probability of the training mode pair according to the loss function.

[0039] The video classification loss is calculated based on the predicted video classification labels and the actual video classification labels to obtain the video classification loss value;

[0040] The target loss value is obtained by summing the activation loss value and the video classification loss value.

[0041] To achieve the above objectives, a second aspect of this application provides a multimodal video classification device, the device comprising:

[0042] A target video acquisition module is used to acquire a target video; wherein the target video contains multiple modal information;

[0043] The semantic feature extraction module is used to extract semantic features from the multiple modal information using a pre-trained video classification model to obtain multimodal semantic features; wherein, the multimodal semantic features include at least two of visual semantic features, textual semantic features, and audio semantic features;

[0044] The modality pair construction module is used to construct candidate modality pairs by performing pairwise modality pair construction on the visual semantic features, the text semantic features, and the audio semantic features;

[0045] The modality pair filtering module is used to perform binary mask activation processing on the candidate modality pairs to obtain a modality pair binary mask, and to filter the candidate modality pairs according to the modality pair binary mask to obtain the target modality pair;

[0046] The context feature extraction module is used to extract context features of each modality in the multimodal semantic features according to the target modality, so as to obtain context features of each modality;

[0047] The feature fusion module is used to fuse the context features of each modality to obtain fused multimodal features;

[0048] The video classification module is used to classify the target video based on the fused multimodal features to obtain target video classification data.

[0049] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect.

[0050] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method of the first aspect described above.

[0051] This application proposes a multimodal video classification method, apparatus, electronic device, and medium. First, it acquires a target video containing multimodal information and extracts features from this information, enabling the extraction of features from different modalities. Second, it constructs modal pairs between visual semantic features, textual semantic features, and audio semantic features, and performs binary mask activation processing on candidate modal pairs. Based on the binary mask, it filters candidate modal pairs, adaptively selecting the most relevant modal pairs for interaction, avoiding computational redundancy caused by fully connected concatenation, and preserving key video information. Furthermore, it extracts contextual features from each modality within the multimodal semantic features based on the target modality, enabling cross-modal contextual joint processing and further strengthening intermodal interaction to accurately capture key modal information in the video. Finally, it fuses the contextual features of each modality, preserving higher-order interactions between modalities, and classifies the target video based on the fused multimodal features, effectively integrating the semantic interactions of each modality and improving the accuracy of video classification. Attached Figure Description

[0052] Figure 1 This is a flowchart of the multimodal video classification method provided in the embodiments of this application;

[0053] Figure 2 This is another flowchart of the multimodal video classification method provided in the embodiments of this application;

[0054] Figure 3 yes Figure 2 The flowchart of step S208 in the text;

[0055] Figure 4 yes Figure 1The flowchart of step S104 in the process;

[0056] Figure 5 yes Figure 1 The flowchart of step S105 in the process;

[0057] Figure 6 yes Figure 5 The flowchart of step S501 in the process;

[0058] Figure 7 yes Figure 1 The flowchart of step S106 in the process;

[0059] Figure 8 This is a schematic diagram of the structure of the multimodal video classification device provided in the embodiments of this application;

[0060] Figure 9 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0062] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0063] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0064] First, let's analyze some of the terms used in this application:

[0065] Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0066] This application provides a multimodal video classification method, apparatus, electronic device, and medium, aiming to improve the accuracy of video classification.

[0067] The video classification method, apparatus, electronic device, and medium based on multimodal characteristics provided in this application are specifically described through the following embodiments. First, the video classification method based on multimodal characteristics in this application is described.

[0068] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0069] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0070] The multimodal video classification method provided in this application relates to the field of artificial intelligence technology. This method can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the multimodal video classification method, but is not limited to the above forms.

[0071] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0072] Figure 1 This is an optional flowchart of a multimodal video classification method provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps S101 to S107.

[0073] Step S101: Obtain the target video; wherein the target video contains multiple modal information.

[0074] Step S102: Extract semantic features from multiple modal information using a pre-trained video classification model to obtain multimodal semantic features; wherein, the multimodal semantic features include at least two of visual semantic features, textual semantic features, and audio semantic features.

[0075] Step S103: Construct candidate modal pairs by pairwise constructing visual semantic features, text semantic features and audio semantic features.

[0076] Step S104: Perform binary mask activation processing on the candidate mode pairs to obtain the mode pair binary mask, and filter the candidate mode pairs according to the mode pair binary mask to obtain the target mode pair.

[0077] Step S105: Extract context features from each modality in the multimodal semantic features according to the target modality to obtain context features for each modality.

[0078] Step S106: Perform feature fusion on the context features of each modality to obtain fused multimodal features.

[0079] Step S107: Perform video classification on the target video based on the fused multimodal features to obtain target video classification data.

[0080] Steps S101 to S107 of this embodiment first acquire a target video containing multimodal information and extract features from the multimodal information, enabling the extraction of features from different modalities. Second, modal pairs are constructed between visual semantic features, textual semantic features, and audio semantic features. Candidate modal pairs are then subjected to binary mask activation processing. Modal pair filtering is performed based on the modal pair binary mask, adaptively selecting the most relevant modal pairs for interaction, avoiding computational redundancy caused by fully connected concatenation, and preserving key video information. Furthermore, contextual features are extracted from each modality in the multimodal semantic features based on the target modality, enabling cross-modal contextual joint processing, further strengthening intermodal interaction, and accurately capturing key modal information in the video. Finally, the contextual features of each modality are fused, preserving high-order interactions between modalities. The fused multimodal features are then used to classify the target video, effectively fusing the semantic interactions of each modality and improving the accuracy of video classification.

[0081] In step S101 of some embodiments, specifically, a complete video can be acquired by a video acquisition device (such as a camera), which contains multiple modal information such as images, audio and text.

[0082] For example, in financial applications, the target video could be a video of a customer applying for a loan, which could include multiple modal information such as the customer's facial expressions, voice introduction, and text information of the loan contract; in medical applications, the target video could be a video of a patient receiving treatment, which could include multiple modal information such as the patient's posture, the doctor's explanation, and medical record text.

[0083] Please see Figure 2 In some embodiments, prior to step S102, the multimodal video classification method further includes, but is not limited to, steps S201 to S205:

[0084] Step S201: Obtain the original video classification model and training videos, and label the training videos with the real video classification labels.

[0085] Step S202: Extract features from the training video using the original video classification model to obtain training multimodal semantic features; the training multimodal semantic features include at least two of the following: training visual semantic features, training text semantic features, and training audio semantic features.

[0086] Step S203: Construct pairwise modal pairs by training visual semantic features, training text semantic features, and training audio semantic features to obtain training candidate modal pairs.

[0087] Step S204: Activate the training candidate mode pairs to obtain the activation probability of the training mode pairs, convert the activation probability of the training mode pairs into a binary mask of the training mode pairs, and filter the training candidate mode pairs according to the binary mask of the training mode pairs to obtain the training target mode pairs.

[0088] Step S205: Extract context features from each modality in the multimodal semantic features according to the training target modality to obtain the training context features of each modality.

[0089] Step S206: The training context features of each modality are fused to obtain the training fused multimodal features.

[0090] Step S207: Based on the training fusion multimodal features, perform video classification prediction on the training video to obtain the predicted video classification label.

[0091] Step S208: Calculate the activation probability of the training mode pair, the predicted video classification label, and the target loss value of the video classification label according to the preset loss function.

[0092] Step S209: Update the model parameters of the original video classification model based on the target loss value.

[0093] In step S201 of some embodiments, specifically, the original video classification model includes a feature extraction layer, a dynamic sparse gating layer, a sparse cross-modal attention layer, a feature fusion layer, and an output layer; wherein, the input layer is used to extract multimodal features of the training video and perform linear projection; the dynamic sparse gating layer is used to activate modal pairs with binary masks and filter out target modal pairs according to the binary masks output by the gating layer; the sparse cross-modal attention layer is used to extract contextual features of each modality; the feature fusion layer is used to fuse contextual features of each modality; and the output layer is used to predict the category to which the video belongs.

[0094] For example, in financial applications, training videos could be videos of customers conducting banking transactions, and the actual classification labels could be loan application categories; in medical applications, training videos could be videos of patients receiving medical treatment, and the actual classification labels could be disease diagnosis categories.

[0095] In step S202 of some embodiments, for example, in a financial application scenario, training multimodal semantic features can extract facial expression features of customers from the image modality of the training video through a feature extraction layer, extract key terms features of the loan contract from the text modality, and extract voice features of customers introducing their financial situation from the audio modality; in a medical application scenario, training multimodal semantic features can extract patient posture features from the image modality, extract disease description features from the medical record from the text modality, and extract doctor's explanation of the patient's condition from the audio modality through a feature extraction layer.

[0096] In step S203 of some embodiments, specifically, semantic features of different modalities can be combined through a dynamic sparse gating layer to form possible modality pairs.

[0097] For example, in financial applications, visual semantic features (such as facial expression features) can be combined with textual semantic features (such as contract terms features) to form a modality pair; or visual semantic features (such as facial expression features) can be combined with audio semantic features (such as voice introduction features) to form another modality pair. In medical applications, visual semantic features (such as patient posture features) can be combined with textual semantic features (such as medical record description features) to form a modality pair; or visual semantic features (such as patient posture features) can be combined with audio semantic features (such as doctor's explanation features) to form another modality pair.

[0098] In step S204 of some embodiments, specifically, the training candidate mode pairs can be further activated by a dynamic sparse gating layer to obtain the activation probability of the training mode pairs, and the activation probability of the training mode pairs can be converted into a binary mask of the training mode pairs. The training candidate mode pairs are then filtered according to the binary mask of the training mode pairs to determine the training target mode pairs.

[0099] For example, in financial scenarios, if there is a high correlation between visual semantic features (such as facial expression features) and audio semantic features (such as voice introduction features), the corresponding modality pair will be retained; if there is a low correlation between visual semantic features (such as facial expression features) and textual semantic features (such as contract terms features), the corresponding modality pair will be discarded. In medical scenarios, if there is a high correlation between visual semantic features (such as patient body posture features) and audio semantic features (such as doctor's explanation features), the corresponding modality pair will be retained; if there is a low correlation between visual semantic features (such as patient body posture features) and textual semantic features (such as medical record description features), the corresponding modality pair will be discarded.

[0100] In step S205 of some embodiments, specifically, the training context features of each modality can be extracted through a sparse cross-modal attention layer.

[0101] For example, in financial scenarios, for the retained visual semantic features (such as facial expression features) and audio semantic features (such as voice introduction features) modal pairs, further extract the features of facial expressions in the context of voice introduction, and the features of voice introduction in the context of facial expressions; in medical scenarios, for the retained visual semantic features (such as patient posture features) and audio semantic features (such as doctor explanation features) modal pairs, further extract the features of patient posture in the context of doctor explanation, and the features of doctor explanation in the context of patient posture.

[0102] In step S206 of some embodiments, specifically, the training context features of each modality can be fused through a feature fusion layer.

[0103] For example, in financial scenarios, contextual features of visual semantic features (such as facial expression features) and audio semantic features (such as voice introduction features) are fused to obtain fused multimodal features; in medical scenarios, contextual features of visual semantic features (such as patient posture features) and audio semantic features (such as doctor explanation features) are fused to obtain fused multimodal features.

[0104] In step S207 of some embodiments, specifically, the predicted video classification labels of the training video can be output through the output layer.

[0105] For example, in financial scenarios, training videos are predicted as loan application video category labels based on the fusion of multimodal features; in medical scenarios, training videos are predicted as disease diagnosis video category labels based on the fusion of multimodal features.

[0106] Please see Figure 3 In some embodiments, step S208 includes, but is not limited to, steps S301 to S303:

[0107] Step S301: Calculate the activation loss value by performing activation loss calculation on the activation probability of the training mode pair according to the loss function.

[0108] Step S302: Calculate the video classification loss based on the predicted video classification label and the actual video classification label to obtain the video classification loss value.

[0109] Step S303: Sum the activation loss value and the video classification loss value to obtain the target loss value.

[0110] In step S301 of some embodiments, specifically, the loss function includes an activation loss function and a video classification loss function; wherein, the activation loss function can be a sparse regularization loss function; and the video classification loss function can be a classification cross-entropy loss function.

[0111] Specifically, the activation loss value can be calculated using the following formula:

[0112] L sparse =λ·(p (m′,n′) -0.5) 2

[0113] Among them, L sparse Let λ represent the activation loss value, λ represent the hyperparameter controlling the sparsity intensity, and p represent the other values. (m′,n′) Let m' represent the activation probability of the training mode pair between mode m' and mode n'.

[0114] Furthermore, the regularization term in the above formula is used to encourage activation probabilities to tend toward a binary distribution (0 or 1), thereby promoting sparser cross-modal interactions.

[0115] In step S302 of some embodiments, for example, in a financial scenario, if the video classification model predicts that a video with a banking application is categorized as "loan application category label", and the actual video is categorized as "loan application category label", then the video classification loss value is low; in a medical scenario, if the video classification model predicts that a video of a patient's condition is categorized as "diabetes diagnosis label", but the actual video is categorized as "hypertension diagnosis label", then the video classification loss value will be high.

[0116] In step S303 of some embodiments, the target loss value can be calculated using the following formula:

[0117] L = L task +L sparse

[0118] Where L represents the target loss value, L task L represents the video classification loss value. sparse This represents the activation loss value.

[0119] Through steps S301 to S303, by calculating the activation loss value and the video classification loss value separately and merging them into the target loss value, the activation loss value can focus on the rationality of the modality pair activation, and the video classification loss value can focus on the accuracy of the classification prediction. This ensures that the final target loss value can comprehensively evaluate the performance of the video classification model in the multimodal video classification task, improve the video classification model's ability to consider both the modality pair activation and video classification accuracy during the optimization process, and thus more effectively adjust the model's own parameters to improve the accuracy of multimodal video classification.

[0120] In step S209 of some embodiments, specifically, the parameters of the model can be adjusted using the backpropagation algorithm based on the calculated target loss value to optimize the performance of the video classification model.

[0121] Through steps S201 to S209, a series of steps from feature extraction to model parameter update can systematically optimize the performance of the pre-trained video classification model, effectively improve the model's ability to process multimodal information, and, combined with the feedback mechanism of the loss function, gradually adjust the model parameters so that the model can achieve better video classification results on the training data. This allows the optimized model to process multimodal video data more accurately, thereby improving the accuracy of video classification.

[0122] In step S102 of some embodiments, specifically, features can be extracted from each frame of the target video using a ResNet network in the feature extraction layer to obtain a visual semantic feature vector for each frame. Then, the text of the target video can be word-embedded using a word embedding network to extract text semantic feature vectors. Furthermore, the audio signal in the target video can be converted into audio text using speech recognition technology (such as DeepSpeech), and then the audio text can be word-embedded using a word embedding network to extract audio semantic feature vectors.

[0123] For example, in financial scenarios, a ResNet network can be used to extract facial expression features of customers from image modalities, a word embedding network can be used to extract key clause features from loan contracts from text modalities, and a word embedding network can be used to extract voice features of customers introducing their financial situation from audio modalities. In medical scenarios, a ResNet network can be used to extract patient posture features from image modalities, a word embedding network can be used to extract disease description features from medical records from text modalities, and a word embedding network can be used to extract doctor's explanation of the patient's condition from audio modalities.

[0124] Furthermore, after obtaining the multimodal semantic features, it is necessary to perform linear spatial projection on the multimodal semantic features to obtain the projected multimodal semantic features as the final multimodal semantic features. Specifically, this includes: projecting the dialogue image features into linear space to obtain projected image features; projecting the dialogue text features into linear space to obtain projected text features; and projecting the dialogue audio features into linear space to obtain projected audio features. The projected image features, projected text features, and projected audio features reside in a unified vector dimension space; and the projected image features, projected text features, and projected audio features are used as the final visual semantic features, respectively.

[0125] In this embodiment, by performing linear spatial projection on the multimodal semantic features, it is possible to ensure that the features of different modalities have a consistent vector dimension space representation, which helps to realize subsequent cross-modal interaction.

[0126] Specifically, linear spatial projection can be achieved using the following formula:

[0127] z m =LayerNorm(W m ·h m +b m )

[0128] Among them, z m ∈R 512 W represents the feature vector projected from the semantic features of modality m. m ∈R 512×dm The projection matrix (d) represents the semantic features of modality m. m h represents the original feature dimension of the semantic features of modality m. m The original feature vector representing modality m (generally referring to any multimodal semantic feature in set M, such as h) v Visual semantic features, h a Audio semantic features, h t Textual semantic features, etc.), b m ∈R 512 represents the bias term, LayerNorm represents the layer normalization operation, used to standardize the feature distribution, and · represents the dot product.

[0129] In step S103 of some embodiments, specifically, the semantic features of different modalities can be spliced ​​and combined through a dynamic sparse gating layer to form possible modal pairs.

[0130] Specifically, visual semantic features can be combined with textual semantic features to form a candidate modality pair; alternatively, visual semantic features (such as facial expression features) can be combined with audio semantic features to form another candidate modality pair.

[0131] Please see Figure 4 In some embodiments, step S104 includes, but is not limited to, steps S401 to S402:

[0132] Step S401: Activate the candidate mode pairs to obtain the activation probabilities of the mode pairs.

[0133] Step S402: Activation detection is performed according to the preset indicator function and mode pair activation probability to obtain a binary mask of the mode pair; the binary mask of the mode pair is used to characterize whether the candidate mode pair is activated or not.

[0134] In step S401 of some embodiments, specifically, the modality pair activation probability is used to reflect the importance of the candidate modality pair in the current video classification task, as determined by the video classification model.

[0135] Specifically, the activation probability of mode pairs can be determined using the following formula for dynamic sparse gating layers:

[0136] p (m,n) =σ(9v T ·GELU(U·[z m ;z n ))-τ) / β)

[0137] Among them, z n z represents the feature index of mode n. m The feature index of mode m is represented by [z]. m ;z n ] represents a candidate mode pair obtained by concatenating the features of mode n and mode m, where U∈R (64×1024) Let v ∈ R be a low-rank projection matrix. 64 v represents a learnable parameter vector (used to reduce high-dimensional features to low dimensions to reduce computational cost). T denoted as the transpose of the learnable parameter vector v, GELU represents the Gaussian error linear unit activation function (used to enhance the nonlinear expressiveness of the model), σ represents the Sigmoid activation function (used to compress the output to the 0-1 interval), τ represents the trainable threshold parameter (used to control the activation threshold), β represents the temperature parameter (used to control the gradient smoothness), and · represents the dot product.

[0138] In step S402 of some embodiments, specifically, the modality pair binary mask is a binary mask used to characterize whether the candidate modality pair is activated or not, that is, whether the video classification model considers the candidate modality pair to contribute to the current video classification task.

[0139] Specifically, the activation probabilities of continuous mode pairs can be converted into discrete binary masks using a pass-through estimator in a dynamic sparse gating layer. This can be achieved using the following formula:

[0140] G (m,n) =I(p) (m,n) >0.5)

[0141] Among them, G (m,n) Let I(·) represent the binary mask for the mode pair of mode m and mode n, and let p represent the indicator function. (m,n) Let p represent the activation probability of the training mode pair between mode m and mode n, when p (m,n) Output 1 if the condition is greater than 0.5, otherwise output 0.

[0142] Through steps S401 to S402, by calculating the activation probability of modal pairs and generating binary masks, the video classification model can adaptively identify and filter modal pairs that make significant contributions to the video classification task. This significantly reduces the computational redundancy of the model, not only improving the model's processing efficiency for multimodal information but also enhancing the model's understanding of the interaction relationships between different modalities. This makes the modal pairs perform more accurately in subsequent multimodal video classification tasks, which helps to improve the accuracy of video classification in the future.

[0143] Please see Figure 5 In some embodiments, step S105 includes, but is not limited to, steps S501 to S502:

[0144] Step S501: For one target mode in the target mode pair, calculate the attention weights of the target mode pair based on the binary mask of the mode pair to obtain the mode pair weights of the target mode.

[0145] Step S502: For the target modality, perform a weighted summation on another modality in the target modality pair according to the modality pair weights to obtain the target modality context features, and determine the target modality context features as the context features of each modality.

[0146] Please see Figure 6 In some embodiments, step S501 includes, but is not limited to, steps S601 to S604:

[0147] Step S601: For the target mode, perform preliminary attention weight calculation on the target mode pair based on the mode pair binary mask to obtain the preliminary mode pair weight.

[0148] Step S602: Obtain all interaction modes of the target mode, and perform binary masking operation on the target mode and all interaction modes to obtain the interaction pair binary mask.

[0149] Step S603: Calculate the attention weights of the target modality based on the binary mask of the interaction pair to obtain the attention weights of the interaction pair.

[0150] Step S604: The initial modality pair weights and the interaction pair attention weights are divided to obtain the modality pair weights of the target modality.

[0151] In step S601 of some embodiments, specifically, the preliminary mode pair weights refer to the initial weights obtained based on the binary mask of the mode pair, which are used to reflect the preliminary correlation between the two modes in the target mode pair.

[0152] Specifically, the initial attention weights between modalities can be selectively calculated using a sparse cross-modal attention layer based on the modal pair binary mask. The initial modal pair weights can be determined by combining the exponential function, the learnable alignment matrix, the scaling factor, and the calculation between the modal pair binary masks.

[0153] In step S602 of some embodiments, specifically, the interaction pair binary mask refers to the activation mask value of the mode pair formed by the target mode and all its interaction modes, which is used to indicate whether the mode pair formed by the target mode and all its interaction modes is activated.

[0154] In step S603 of some embodiments, specifically, the interaction pair attention weights are weights calculated based on the interaction pair binary mask, used to reflect the correlation between the target modality and all its interaction modalities. Specifically, the interaction pair attention weights can be determined by combining an exponential function, a learnable alignment matrix, a scaling factor, and the calculation of the interaction pair binary mask.

[0155] In step S604 of some embodiments, specifically, the mode pair weights of the target mode can be determined by the following formula:

[0156]

[0157] Where, α (m,n) Let A ∈ R be the weights of the mode pair between mode m and mode n. (512×5120 G represents the learnable alignment matrix (used to achieve inter-modal semantic alignment). (m,k) This represents a binary mask of the interaction pairs between modal m and modal k, where k represents the index variable for traversing all modalities in the target video, and z... k The semantic feature vector representing mode k. This represents the semantic feature transpose vector of modality m. The scaling factor is represented by d = 512 to prevent the gradient from vanishing due to excessively large dot product values. exp represents the exponential function, and the denominator is the normalization factor used to characterize the attention weights of the interaction pair.

[0158] Through steps S601 to S604, by calculating the initial modality pair weights, operating the binary mask of the interaction pair, calculating the attention weights of the interaction pair, and comprehensively adjusting the final modality pair weights, it can be ensured that only modality pairs activated by the binary mask will participate in the attention calculation. This achieves dynamic adjustment of modality pair weights, considering not only the initial correlation between target modality pairs but also the relationship between target modalities and other interaction modalities. This helps the model to more accurately evaluate the importance of each modality pair, thereby improving the accuracy of subsequent video classification.

[0159] In step S502 of some embodiments, specifically, each modal context feature refers to incorporating information from one modality into another modality to obtain a feature representation containing context information.

[0160] Specifically, the contextual features of each modality can be determined using the following formula in the sparse cross-modal attention layer:

[0161] c m =Σ n α (m,n) ·z n

[0162] Among them, c m ∈R 512 Σ represents the contextual features of modality m that incorporate cross-modal information n. n This represents summing over all possible modes n, where α (m,n) z represents the mode pair weights of mode m and mode n. n The feature index represents mode n.

[0163] Through steps S501 to S502, by calculating the weights of the modalities and performing weighted summation, information from one modality can be dynamically integrated into another modality, achieving cross-modal context joint processing and generating feature representations of each modality containing contextual information. This enhances the interaction between different modalities, accurately capturing key modal information in the video, and effectively solves the problem in traditional methods that ignore the complex interaction relationships between modalities, making it difficult for the model to capture the true meaning that the video is trying to express.

[0164] Please see Figure 7 In some embodiments, the modal context features include visual semantic context features, text semantic context features, and audio semantic context features. Step S106 includes, but is not limited to, steps S701 to S702:

[0165] Step S701: The visual semantic context features, text semantic context features and audio semantic context features are concatenated to obtain concatenated multimodal features.

[0166] Step S702: Rotate and encode the spliced ​​multimodal features to obtain fused multimodal features.

[0167] In step S701 of some embodiments, specifically, splicing multimodal features refers to combining contextual features of different modalities together to form a comprehensive feature representation, which is used to represent high-order multimodal interaction features.

[0168] Specifically, a feature fusion layer can be used to concatenate the visual semantic context features, text semantic context features, and audio semantic context features into feature vectors to obtain a high-order concatenated multimodal feature C = [c1; c2; ...; c M ]; where M represents the number of modes (e.g., 3), c M The semantic context features of modality M are represented.

[0169] In step S702 of some embodiments, specifically, the rotational position encoding of the concatenated multimodal features can be performed by the Transformer encoder of the feature fusion layer:

[0170] f = TransformerEncoder(C,P) RoPE )

[0171] Where, f∈R (M×512) P represents the fusion of multimodal features. RoPE (Rotary Position Encodings) refers to rotational position encoding (used to capture the positional relationships between modalities). The TransformerEncoder consists of a multi-head self-attention mechanism and a feedforward neural network, which can effectively capture the semantic interactions between modalities.

[0172] In this embodiment, by performing rotational position encoding on the spliced ​​multimodal features, the contextual features of different modalities can be effectively integrated, enabling the video classification model to better learn the sequential relationships between different modal features, thereby forming a more comprehensive and accurate multimodal feature representation and improving the accuracy of video classification.

[0173] Through steps S701 to S702, the processing of feature concatenation and rotation position encoding can preserve high-order interactions between modalities and learn the sequential relationships between features. This reduces model computational overhead while ensuring expressive power, which helps improve the efficiency and accuracy of multimodal feature fusion.

[0174] In step S107 of some embodiments, specifically, the target video classification data refers to the video classification result obtained based on the fusion of multimodal features, which is usually represented in the form of category labels.

[0175] For example, in a financial scenario, the target video classification data could be categorized under the label "loan application"; in a medical scenario, the target video classification data could be categorized under the label "diabetes diagnosis".

[0176] Specifically, the activation function of the output layer can be used to output the probability of the video category corresponding to the fused multimodal data, and the video category with the highest probability can be used as the target video classification data.

[0177] This application first acquires a target video containing multimodal information and extracts features from the multimodal information, enabling the extraction of features from different modalities. Second, it constructs modal pairs between visual semantic features, textual semantic features, and audio semantic features, and performs binary mask activation processing on candidate modal pairs. Based on the modal pair binary mask, it filters candidate modal pairs, adaptively selecting the most relevant modal pairs for interaction, avoiding computational redundancy caused by fully connected concatenation, and preserving key video information. Furthermore, it extracts contextual features from each modality in the multimodal semantic features based on the target modality, enabling cross-modal contextual joint processing, further strengthening intermodal interaction, and accurately capturing key modal information in the video. Finally, it fuses the contextual features of each modality, preserving high-order interactions between modalities, and classifies the target video based on the fused multimodal features, effectively fusing the semantic interactions of each modality and improving the accuracy of video classification.

[0178] Please see Figure 8 This application also provides a multimodal video classification device that can implement the above-described multimodal video classification method. The device includes:

[0179] The target video acquisition module is used to acquire target videos; the target videos contain multiple modal information.

[0180] The semantic feature extraction module is used to extract semantic features from multiple modal information using a pre-trained video classification model to obtain multimodal semantic features; wherein, the multimodal semantic features include at least two of the following: visual semantic features, textual semantic features, and audio semantic features;

[0181] The modality pair construction module is used to construct candidate modality pairs by combining visual semantic features, text semantic features, and audio semantic features into pairs.

[0182] The modality pair filtering module is used to perform binary mask activation processing on candidate modality pairs to obtain a modality pair binary mask, and to filter candidate modality pairs based on the modality pair binary mask to obtain the target modality pair;

[0183] The context feature extraction module is used to extract context features of each modality in the multimodal semantic features based on the target modality, so as to obtain the context features of each modality;

[0184] The feature fusion module is used to fuse the context features of each modality to obtain fused multimodal features;

[0185] The video classification module is used to classify target videos based on fused multimodal features to obtain target video classification data.

[0186] The specific implementation of this multimodal video classification device is basically the same as the specific implementation of the multimodal video classification method described above, and will not be repeated here.

[0187] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described multimodal video classification method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0188] Please see Figure 9 , Figure 9 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0189] The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0190] The memory 902 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 902 can store the processing system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called and executed by the processor 901 using the multimodal video classification method of the embodiments of this application.

[0191] The input / output interface 903 is used to implement information input and output;

[0192] The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0193] Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904);

[0194] The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.

[0195] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described multimodal video classification method.

[0196] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0197] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0198] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0199] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0200] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0201] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0202] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0203] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. The coupling or direct coupling or communication connection between the shown or discussed units may be through some interfaces, or indirect coupling or communication connection between the apparatus or units, and may be electrical, mechanical, or other forms.

[0204] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0205] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0206] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0207] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A video classification method based on multimodal modes, characterized in that, The method includes: Acquire a target video; wherein the target video contains multiple modal information; Semantic features are extracted from the multimodal information using a pre-trained video classification model to obtain multimodal semantic features; wherein, the multimodal semantic features include at least two of visual semantic features, textual semantic features, and audio semantic features; The visual semantic features, the text semantic features, and the audio semantic features are paired to construct candidate modal pairs; The candidate mode pairs are subjected to binary mask activation processing to obtain a mode pair binary mask, and the candidate mode pairs are filtered according to the mode pair binary mask to obtain the target mode pairs; Based on the target modality, context features are extracted from each modality in the multimodal semantic features to obtain context features for each modality; The context features of each modality are fused to obtain fused multimodal features; The target video is classified based on the fused multimodal features to obtain target video classification data.

2. The method according to claim 1, characterized in that, The step of performing binary mask activation processing on the candidate mode pairs to obtain a binary mask for the mode pairs includes: The candidate mode pairs are activated to obtain the mode pair activation probabilities. Activation detection is performed based on a preset indicator function and the activation probability of the modality pair to obtain a binary mask of the modality pair; the binary mask of the modality pair is used to characterize whether the candidate modality pair is activated or not.

3. The method according to claim 1, characterized in that, The step of extracting context features for each modality from the multimodal semantic features based on the target modality to obtain context features for each modality includes: For one target mode in the target mode pair, attention weights are calculated for the target mode pair based on the binary mask of the mode pair to obtain the mode pair weights of the target mode; For the target modality, the other modality in the target modality pair is weighted and summed according to the modality pair weights to obtain the target modality context features, and the target modality context features are determined as the context features of each modality.

4. The method according to claim 3, characterized in that, The step of calculating the mode pair weights for one target mode in the target mode pair based on the binary mask of the mode pair includes: For the target mode, preliminary attention weights are calculated for the target mode pair based on the binary mask of the mode pair to obtain preliminary mode pair weights; Obtain all interaction modalities of the target modality, and perform binary masking operation on the target modality and all interaction modalities to obtain the interaction pair binary mask; The attention weights of the interaction pair are calculated by performing attention weight calculation on the target modality based on the binary mask of the interaction pair; The modality pair weights of the target modality are obtained by quotienting the initial modality pair weights with the interaction pair attention weights.

5. The method according to claim 1, characterized in that, The modal context features include visual semantic context features, text semantic context features, and audio semantic context features; The step of fusing the context features of each modality to obtain fused multimodal features includes: The visual semantic context features, the text semantic context features, and the audio semantic context features are concatenated to obtain concatenated multimodal features; The spliced ​​multimodal features are rotated and position encoded to obtain the fused multimodal features.

6. The method according to claim 1, characterized in that, Before extracting semantic features from the multimodal information using a pre-trained video classification model to obtain multimodal semantic features, the method further includes: Obtain the original video classification model and training videos, and label the training videos with the real video classification labels; The training video is used to extract features through the original video classification model to obtain training multimodal semantic features; the training multimodal semantic features include at least two of the following: training visual semantic features, training text semantic features, and training audio semantic features. The training visual semantic features, the training text semantic features, and the training audio semantic features are paired to construct training candidate modal pairs; The training candidate mode pairs are activated to obtain the activation probability of the training mode pairs. The activation probability of the training mode pairs is then converted into a binary mask of the training mode pairs. The training candidate mode pairs are then filtered according to the binary mask of the training mode pairs to obtain the training target mode pairs. Based on the training target modality, context features are extracted from each modality in the multimodal semantic features to obtain the training context features for each modality; The training context features of each modality are fused to obtain the training fused multimodal features; Based on the training fusion multimodal features, video classification prediction is performed on the training video to obtain the predicted video classification label; The activation probability of the training mode pair, the predicted video classification label, and the target loss value of the video classification label are calculated according to the preset loss function. The model parameters of the original video classification model are updated based on the target loss value.

7. The method according to claim 6, characterized in that, The step of calculating the target loss value of the training mode pair activation probability, the predicted video classification label, and the real video classification label according to the preset loss function includes: The activation loss value is obtained by calculating the activation probability of the training mode pair according to the loss function. The video classification loss is calculated based on the predicted video classification labels and the actual video classification labels to obtain the video classification loss value; The target loss value is obtained by summing the activation loss value and the video classification loss value.

8. A video classification device based on multimodal modes, characterized in that, The device includes: A target video acquisition module is used to acquire a target video; wherein the target video contains multiple modal information; The semantic feature extraction module is used to extract semantic features from the multiple modal information using a pre-trained video classification model to obtain multimodal semantic features; wherein, the multimodal semantic features include at least two of visual semantic features, textual semantic features, and audio semantic features; The modality pair construction module is used to construct candidate modality pairs by performing pairwise modality pair construction on the visual semantic features, the text semantic features, and the audio semantic features; The modality pair filtering module is used to perform binary mask activation processing on the candidate modality pairs to obtain a modality pair binary mask, and to filter the candidate modality pairs according to the modality pair binary mask to obtain the target modality pair; The context feature extraction module is used to extract context features of each modality in the multimodal semantic features according to the target modality, so as to obtain context features of each modality; The feature fusion module is used to fuse the context features of each modality to obtain fused multimodal features; The video classification module is used to classify the target video based on the fused multimodal features to obtain target video classification data.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the multimodal video classification method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the multimodal video classification method according to any one of claims 1 to 7.

Images