Video dialogue style identification method and device based on multi-level and multi-mode alignment features, and medium

Abstract

The invention discloses a video dialogue style recognition method and device based on multi-level multi-mode alignment features and a medium, and the method carries out the prediction and recognition of the dialogue style of a character in a video: firstly, dividing the video into a plurality of shots, extracting the aligned visual, auditory and text features of each shot through a feature extraction model, then, different time sequence modeling strategies are utilized to introduce a time sequence dependency relationship between the lenses into a lens-level multi-modal feature; and then, aggregating the shot-level auditory features and the text features into video-level auditory features and text features, fusing the video-level auditory features and text features with the shot-level visual features to obtain shot-level multi-modal fusion features, and inputting the shot-level multi-modal fusion features into a classification network to obtain a dialogue style prediction result of a corresponding shot. And finally, selecting the most prominent shot prediction result as a final video dialogue style prediction result from the shot prediction results. According to the method, the problems of modal isomerism and modal distinguishing in an existing multi-modal dialogue style recognition method are solved, and the recognition performance is improved.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology and relates to the extraction of aligned multimodal features in videos, the selection and fusion of multimodal features at different levels, and specifically a video dialogue style recognition method based on multi-level multimodal alignment features. Background Technology

[0002] Existing human-centered video analytics focuses on recognizing human actions, such as motion recognition tasks and interaction recognition tasks. However, besides physical interaction, dialogue is also a crucial form of human interaction. Video dialogue style recognition refers to classifying the style of a video containing human dialogue, such as discussion, instruction, explanation, confrontation, or intimidation. Dialogue style is a complex judgment that requires a comprehensive evaluation of multiple modalities, including the content of the dialogue, the state of the characters, and their interactions. Dialogue style recognition facilitates deeper research into human dialogue, leading to a more comprehensive understanding of human behavior.

[0003] Previous work on video dialogue style recognition has largely focused on the visual modality. However, in addition to visual content, dialogue style is also naturally related to audio and dialogue. Therefore, existing technologies have explored several multimodal methods for dialogue style recognition. Some of these methods utilize multimodal pre-training to enhance feature semantics, while others leverage multimodal emotional information to enhance feature discriminative power. These methods have also demonstrated that multimodal approaches are more advantageous than unimodal approaches for dialogue style recognition. However, current multimodal dialogue style recognition methods still face many challenges.

[0004] First, there's the issue of modal heterogeneity. Different models have different structures and training processes, resulting in significant differences in the visual, auditory, and textual features derived from models using features from different modalities, creating a gap in their distribution. Directly fusing such multimodal models leads to features that are difficult to understand each other, ultimately introducing noise instead of the ideal modal complementarity, thus causing prediction errors.

[0005] Secondly, there is the issue of modality differentiation. Different modalities contribute differently to dialogue style recognition tasks; strong modalities contain more representative information, while weak modalities are more prone to introducing confusion. If all modalities are processed and fused in the same way, weak modalities will negatively impact strong modalities, preventing either from performing at its best and ultimately limiting the accuracy of the recognition method. Addressing modal heterogeneity and modality differentiation are two major challenges that urgently need to be solved in multimodal dialogue style recognition. Summary of the Invention

[0006] The problem this invention aims to solve is how to overcome the heterogeneity gap between different modal features and how to distinguish the roles, i.e., the degree of contribution, of video, audio, and text modalities in dialogue style recognition tasks. Multimodal alignment is considered one of the effective means to alleviate modal heterogeneity; therefore, this invention derives and fuses aligned multimodal features to analyze dialogue style. Furthermore, compared to visual modalities, which always present very similar content such as "one person talking to another face-to-face," audio and dialogue are more representative and discriminative modalities for dialogue style recognition tasks. Therefore, for auditory and text features, this invention retains all content; for visual features, only partial content is retained for the final multimodal fusion.

[0007] The technical solution of this invention is as follows: a video dialogue style recognition method based on multi-level multimodal alignment features. A recognition network is constructed to perform dialogue style recognition on videos containing dialogue. The recognition network is implemented as follows: First, the video is divided into several shots. The ImageBind multimodal feature extraction model is used to extract aligned multimodal features for each shot, including visual features, auditory features, and text features. Then, a corresponding temporal modeling method is selected based on different modal features to introduce the temporal dependencies between shots into shot-level multimodal features, resulting in shot-level multimodal features with temporal information. Next, the shot-level auditory and text features with temporal information are aggregated into video-level auditory and text features, which are then fused with the shot-level visual features of each shot to obtain shot-level multimodal fusion features. The shot-level multimodal fusion features are input into a classification network to obtain the dialogue style prediction results for the corresponding shots. Finally, the most prominent prediction result is selected from the shot prediction results as the final video dialogue style prediction result.

[0008] Furthermore, the specific methods for obtaining shot-level multimodal features with temporal information using temporal modeling are as follows:

[0009] The lens-level visual feature sequence V = {v1, v2, ..., v N Inputting a one-dimensional convolutional layer yields a sequence of temporally relevant visual features, V' = {v'1, v'2, ..., v'}. N}:

[0010] V′=Conv1(V)⊕Conv2(V)…⊕Conv d (V)

[0011] Where N represents the number of shots, ⊕ represents the stitching operation, Conv represents a one-dimensional convolution kernel, and d represents the feature size;

[0012] Similarly, from the lens-level auditory feature sequence A = {a1, a2, ..., a...} NThe sequence of auditory features with temporal information, A' = {a'1, a'2, ..., a'}, is obtained through a one-dimensional convolutional layer. N};

[0013] The scene-level text feature sequence T = {t1, t2, ..., t} N Input a Transformer network to obtain a sequence of shot-level visual features with temporal information, T' = {t'1, t'2, ..., t'}. N}:

[0014]

[0015] in Represents the Transformer network structure, θ Trans This represents the learnable Transformer parameters.

[0016] Furthermore, the shot-level auditory feature sequence A' with temporal information is input into an average pooling layer to obtain video-level auditory features. Similarly, video-level text features are obtained from the shot-level text feature sequence T' with temporal information.

[0017] Video-level auditory features and video-level text features Dimensionality reduction is performed by inputting each of the following linear layers: inputting the lens-level visual feature sequence V' with temporal information into a linear layer for dimensionality reduction.

[0018] The dimensionality-reduced video-level auditory features and video-level text features are added to each dimensionality-reduced shot-level visual feature to obtain a shot-level multimodal fusion feature sequence {f1, f2, ..., f N}

[0019] Furthermore, the image-level multimodal feature extraction model ImageBind is used to extract lens-level multimodal features. Specifically, aligned lens-level multimodal features are obtained based on the ImageBind model with fixed parameters.

[0020] The lens i For i∈{1,…,N}, input the ImageBind model to extract visual features, and obtain the lens-level visual feature sequence {v1,v2,…,v...}. N};

[0021] Using ffmpeg to extract from the lens s i The audio is extracted, and then input into the ImageBind model to extract auditory features, resulting in a shot-level auditory feature sequence {a1, a2, ..., a N};

[0022] Using a Whisper model with fixed parameters from lens s i The audio dialogue is extracted, and then the dialogue segment is divided into several sentences {l1, l2, ..., l...} based on sentence breaks, including periods, exclamation marks, and question marks. K The ImageBind model is used to extract text features, resulting in a shot-level text feature sequence {t1,t2,…,t}. N}

[0023] The present invention also provides an electronic device, the electronic device including a processor and a memory, wherein the memory stores at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by the processor to implement the above-mentioned video dialogue style recognition method based on multi-level multimodal alignment features, and to predict the dialogue style of the input video containing human dialogue.

[0024] The present invention also provides a computer-readable storage medium, characterized in that the computer-readable storage medium stores at least one instruction or at least one program, wherein when the at least one instruction or at least one program is executed, the above-described video dialogue style recognition method based on multi-level multimodal alignment features is implemented.

[0025] The beneficial effects of this invention are as follows: This invention provides a scheme for identifying video dialogue styles, addressing two major problems in current multimodal dialogue style recognition methods: modal heterogeneity and modal differentiation. This invention derives multimodal alignment features to mitigate the negative impact of the heterogeneity gap during multimodal fusion. While the ImageBind multimodal feature extraction model is designed for very short videos, this invention, through shot segmentation and subsequent temporal modeling and multi-level multimodal fusion design, transfers this model to the field of long-term video analysis. This invention creatively introduces this alignment feature into dialogue style recognition tasks, improving the robustness of using multimodal features to identify dialogue styles.

[0026] Furthermore, this invention employs hierarchical fusion of multimodal features to differentiate the contributions of different modalities, maximizing utility regardless of whether the modality is strong or weak, thus further improving the method's performance. First, this invention segments the video into shots, as the transitions between shots are significant and the differences are substantial; therefore, dividing long videos into shots is reasonable. Based on a unified multimodal feature extractor, aligned multimodal features are extracted from the shots, providing an ideal foundation for multimodal fusion. Additionally, this invention constructs temporal dependencies between shot-level multimodal features based on different temporal modeling strategies, incorporating global information into the shot-level multimodal features and improving the accuracy of predicting the final video style from shot-level features. Thus, the processing of multimodal features is consistent. Then, hierarchical fusion is performed. This invention's hierarchical fusion is based on multi-level features achieved through multi-shot segmentation. While the visual content of each shot differs significantly, visual features retain shot-level features. However, audio and text have strong overall consistency; therefore, multiple shot-level audio and text features are aggregated to obtain video-level auditory and text features. These video-level auditory and text features are then fused with shot-level visual features. This hierarchical fusion distinguishes the importance of auditory, textual, and visual modalities, clarifies the emphasis of different modal features in video dialogue style recognition, and solves the modality differentiation problem. Finally, this invention classifies based on shot-level fused features to obtain shot-level prediction results, and selects the most prominent one as the final prediction result, thereby achieving the goal of selecting the most prominent visual content and maximizing the role of each modality in dialogue style recognition. Attached Figure Description

[0027] Figure 1 The input / output definitions and architecture of the recognition network constructed by the method of this invention.

[0028] Figure 2 This is a comparison of the recognition performance of the present invention with other existing technical methods.

[0029] Figure 3 This is a comparison of the results of using different modal combinations and whether or not the features are aligned, in accordance with the present invention.

[0030] Figure 4 This invention compares the results of using hierarchical multimodal features and different hierarchical settings. Detailed Implementation

[0031] This invention proposes a video dialogue style recognition method based on multi-level multimodal alignment features. First, it derives aligned multimodal features to mitigate the impact of multimodal heterogeneity. Second, it fuses multimodal features at different levels to distinguish the contributions of different modalities to the dialogue style recognition task. Firstly, the video is divided into several shots, and a unified feature extraction model is used to extract aligned visual, auditory, and textual features for each shot. Then, different temporal modeling strategies are used to introduce the temporal dependencies between shots into shot-level multimodal features. Next, all shot-level auditory features are aggregated into video-level auditory features, and all shot-level textual features are aggregated into video-level textual features, which are then fused with several shot-level visual features to obtain multiple shot-level multimodal fused features. These are then input into a classification network to obtain the dialogue style prediction results for the corresponding shots. Finally, the most prominent prediction result from several shots is selected as the final video dialogue style prediction result.

[0032] The implementation of the present invention will be described in detail below.

[0033] like Figure 1 As shown, the input of the identification network in this invention is a video to be identified, and the output is the probability of various dialogue style categories. The implementation of the video dialogue style recognition network based on multi-level multimodal alignment features specifically includes the following steps.

[0034] 1) Using the TransNet V2 shot detection model, the input video is divided into several shots, resulting in a shot sequence {s1, s2, ..., s...}. N}; Specifically as follows:

[0035] 1.1) Perform lens detection using a TransNet V2 model with fixed parameters as input to the video;

[0036] 1.2) If a shot is longer than 30 seconds, it is considered a long shot and is further divided into 7-second segments.

[0037] 1.3) Based on the detection and segmentation results, use MoviePy to export the video footage.

[0038] The TransNetV2 video shot detection model is an efficient and fast shot boundary detector, as shown in the paper "Transnet v2: An effective deep network architecture for fast shot transition detection" (arXiv:2008.04838v1[cs.CL]11Aug 2020).

[0039] 2) Position the lens s iThe input i∈{1,…,N} is a unified multimodal feature extractor ImageBind, which extracts visual features to obtain a lens-level visual feature sequence {v1,v2,…,v...}. N}:

[0040]

[0041] in Represents the ImageBind model structure, θ IB Represents the pre-trained parameters of the frozen ImageBind model.

[0042] ImageBind, a unified multimodal feature deriver, aligns features from six modalities, including video, audio, and text, into an embedding space. See the paper "ImageBind: One Embedding Space To Bind Them All" (arXiv:2305.05665v2[cs.CL]31May 2023).

[0043] 3) Use ffmpeg to extract the image from the camera. i Extract the audio, then input ImageBind to extract auditory features a i :

[0044]

[0045] in Represents the ImageBind model structure, θ IB This represents the frozen pre-trained parameters of the ImageBind model, and ffmpeg represents a video-to-audio conversion tool.

[0046] 4) Extract a complete dialogue segment from the audio of the scene using a Whisper model with fixed parameters. Then, based on the sentence breaks (periods, exclamation marks, and question marks), divide the dialogue segment into several sentences {l1, l2, ..., l...}. K Each sentence is input into the ImageBind model to extract sentence features. Finally, all sentence features are input into an average pooling layer to obtain the text features t. i :

[0047]

[0048] in Represents the ImageBind model structure, θ IB This represents the pre-trained parameters of the frozen ImageBind model, and MeanP represents the average pooling layer.

[0049] The Whisper speech recognition model uses a large number of training samples for weak supervision to enhance the robustness of speech recognition, see the paper "Robust speech recognition via large-scale weak supervision" (arXiv:2212.04356v1[cs.CL]6Dec 2022).

[0050] 5) The lens-level visual feature sequence {v1, v2, ..., v N Input a temporal modeling network to model the temporal relationships between shots, thereby obtaining a shot-level visual feature sequence {v'1, v'2, ..., v'} with temporal information. N Similarly, the sequence of auditory features at the scene level with temporal information, {a'1,a'2,…,a'}, is obtained. N} and the time-series-informed, shot-level text feature sequence {t'1,t'2,…,t' N}

[0051] Specifically as follows:

[0052] 5.1) The lens-level visual feature sequence V = {v1, v2, ..., v N Input a one-dimensional convolutional layer. This network layer consists of multiple one-dimensional convolutional kernels, with the number of kernels and the feature size remaining consistent. The updated results of each convolutional kernel are then concatenated to obtain a sequence of lens-level visual features with temporal information, V' = {v'1, v'2, ..., v'}. N}:

[0053] V′=Conv1(V)⊕Conv2(V)…⊕Conv d (V)

[0054] Where ⊕ represents the splicing operation, Conv represents the one-dimensional convolution kernel, and d represents the size of the lens-level visual feature.

[0055] Similarly, for the lens-level auditory feature sequence A = {a1, a2, ..., a...} N The image-level auditory feature sequence {a'1,a'2,…,a'} is obtained from a one-dimensional convolutional layer. n}

[0056] 5.2) The scene-level text feature sequence T = {t1, t2, ..., t} N Input a Transformer network. This network has only one attention layer, and each attention layer consists of 8 attention heads. The result is a sequence of temporally relevant, shot-level visual features T' = {t′1, t′2, ..., t′}. N}:

[0057]

[0058] in Represents the Transformer network structure, θ Trans This represents the learnable Transformer parameters.

[0059] 6) The sequence of auditory features at the scene level with temporal information {a'1,a'2,…,a'} N Input an average pooling layer to obtain video-level auditory features. Similarly, video-level text features are obtained by average pooling the temporal-information-based shot-level text feature sequence T'.

[0060] 7) Video-level auditory features Input a linear layer for dimensionality reduction to extract video-level text features. Input a linear layer for dimensionality reduction, and extract the temporal-series visual feature sequence {v'1,v'2,…,v'...}. N Input a linear layer for dimensionality reduction.

[0061] 8) Add the dimensionality-reduced video-level auditory features and video-level text features to each dimensionality-reduced shot-level visual feature, and fuse them to obtain the shot-level multimodal fusion feature sequence {f1, f2, ..., f N}

[0062] 9) Fuse the lens-level multimodal feature sequence {f1,f2,…,f N Input a classifier to obtain shot-level dialogue style prediction results {p1, p2, ..., p N}

[0063] 10) Finally, the camera-level dialogue style prediction results {p1, p2, ..., p N Input a max pooling layer to obtain the final video-level dialogue style prediction result. That is, the most prominent result among the prediction results of each shot, which serves as the video dialogue style of the video described in that shot.

[0064] This invention includes shot segmentation technology, multimodal feature extraction technology, temporal modeling technology, multimodal feature fusion technology, and video classification technology. Shot segmentation technology divides the video into multiple shots (step 1). Multimodal feature extraction technology extracts aligned visual, auditory, and textual features from the shots (steps 2)-4). Temporal modeling technology introduces the temporal dependencies between shots into shot-level multimodal features (step 5). Multimodal feature fusion technology aggregates all shot-level auditory and textual features to form video-level auditory and textual features, which are then fused with shot visual features (steps 6)-8). Video classification technology predicts based on the fused shot-level features and selects the most prominent result as the final video dialogue style recognition result (steps 9)-10).

[0065] The dataset is obtained to train the recognition network designed in steps 1)-10) above. This network learns to perceive various dialogue styles, such as discussion, teaching, explanation, confrontation, and intimidation. The trained recognition model then performs dialogue style recognition on new videos, identifying the dialogue style type of the current video based on the predicted classification results. Notably, during multimodal feature extraction, the ImageBind model with fixed parameters is used; the ImageBind model does not need to participate in the training of the recognition network in this invention.

[0066] This invention is implemented on the LVU dataset, which includes five dialogue style categories: discussion, confrontation, instruction, threat, and explanation. The dataset contains 1339 samples: 937 training samples, 203 validation samples, and 199 test samples. To verify the effectiveness of this invention, we compared it with other methods. Furthermore, to verify the effectiveness of fusing multimodal features and incorporating emotion features, we compared different combinations of features. The metric used is Top-1 Accuracy, a commonly used metric for video classification tasks.

[0067] Figure 2 Rows 1-7 of the table show the results of video dialogue style recognition using Object Transformer, ViS4mer, STAN, S5, LF-VILA, Movie2Scenes, and MMSF, respectively. Row 8, ASMF, shows the results of video dialogue style recognition using this invention. It can be seen that the performance of this invention on the video dialogue style recognition task is far superior to other methods, demonstrating the effectiveness of the proposed video dialogue style recognition method based on multi-level multimodal alignment features.

[0068] Figure 3In this context, v, a, and t represent the visual modality, auditory modality, and text modality, respectively. `aligned` represents the aligned multimodal features derived in this invention, and `unaligned` represents the unaligned multimodal features derived using different unimodal models. Specifically, in this embodiment, for unaligned features, VideoMAE is used to derive visual features, wav2vec2.0 to derive auditory features, and Roberta to derive text features. Figure 3 The first column shows that, when using aligned multimodal features, using multiple modalities for dialogue style recognition performs better than using a single modality, especially when visual, auditory, and textual features are combined, resulting in the best recognition performance. Figure 3 The second column shows that when using unaligned multimodal features, the effect of fusing multimodal features actually deteriorates due to modal heterogeneity. (Comparison) Figure 3 The first and second columns demonstrate that aligned multimodal features have a positive impact on dialogue style recognition tasks.

[0069] This invention addresses the task of dialogue style recognition, where visual and auditory elements directly influence the dialogue, thus requiring the preservation of all content—that is, video-level features. However, in dialogue videos, visual content is prone to confusion because most visual presentations are relatively consistent; for example, the camera may switch back and forth between two people, with only a few key shots revealing a clear connection between the characters. Therefore, visual content needs to be selected based on the shot, identifying the most representative shots. Ultimately, shot-level features are used for the visual features. Figure 4 In this diagram, v, a, and t represent the visual, auditory, and textual modalities, respectively. Modalities enclosed by "<>" retain shot-level features, while those without "<>" retain video-level features. Comparing the first and second rows, it's clear that the hierarchical fusion of multimodal features adopted in this invention is effective. Furthermore, comparing the first to fourth rows shows that retaining shot-level visual features while maintaining auditory and textual features at the video level is the most reasonable hierarchical fusion setting. This further illustrates that in dialogue style recognition tasks, audio and dialogue contain more and more important information than visual content; therefore, maintaining auditory and textual modalities at the video level and visual modalities at the shot level yields the best results.

Claims

1. A video dialogue style recognition method based on multi-level multimodal alignment features, characterized by: A recognition network is constructed to perform dialogue style recognition on videos containing dialogue between characters. The recognition network is implemented as follows: First, the video is divided into several shots. The multimodal feature extraction model ImageBind is used to extract aligned multimodal features for each shot, including visual features, auditory features, and text features. Then, according to different modal features, the corresponding temporal modeling method is selected to introduce the temporal dependency between shots into the shot-level multimodal features, resulting in shot-level multimodal features with temporal information. Then, the shot-level auditory features and text features with temporal information are aggregated into video-level auditory features and text features, which are then fused with the shot-level visual features of each shot to obtain shot-level multimodal fusion features. The multimodal fusion features at the shot level are input into the classification network to obtain the dialogue style prediction results for the corresponding shot. Finally, the most prominent prediction results are selected from the shot prediction results as the final video dialogue style prediction results.

2. The video dialogue style recognition method based on multi-level multimodal alignment features according to claim 1, characterized in that: Specifically, obtaining shot-level multimodal features with temporal information using temporal modeling is as follows: The lens-level visual feature sequence V = {v1, v2, ..., v...} is used to define the visual feature sequence V = {v1, v2, ..., v...}. N Inputting a one-dimensional convolutional layer yields a sequence of shot-level visual features V′={v′1,v′2,...,v N }: Where N represents the number of shots. d represents the concatenation operation, Conv represents a one-dimensional convolution kernel, and d represents the feature dimension size. Similarly, from the lens-level auditory feature sequence A = {a1, a2, ..., a...} N The sequence of auditory features with temporal information, A′={a′1,a′2,...,a′1,a′2,a′3,a′4,a′4,a′5,a′4,a′5,a′6,a′7 ...7,a′6,a′7,a′7,a′7,a N }; The scene-level text feature sequence T = {t1, t2, ..., t} is used to define the scene-level text feature sequence T = {t1, t2, ..., t}. N Inputting the data into a Transformer network yields a sequence of temporally relevant visual features, T′ = {t′1, t′2, ..., t′}. N }: in Represents the Transformer network structure, θ Trans This represents the learnable Transformer parameters.

3. The video dialogue style recognition method based on multi-level multimodal alignment features according to claim 1, characterized in that: The temporal-information-based shot-level auditory feature sequence A′ is input into an average pooling layer to obtain video-level auditory features. Similarly, video-level text features are obtained from the shot-level text feature sequence T′ with temporal information. Video-level auditory features and video-level text features Dimensionality reduction is performed by inputting each of the following linear layers: inputting the lens-level visual feature sequence V′ with temporal information into a linear layer for dimensionality reduction. The dimensionality-reduced video-level auditory features and video-level text features are added to each dimensionality-reduced shot-level visual feature to obtain a shot-level multimodal fusion feature sequence {f1, f2, ..., f...}. N } 4. The video dialogue style recognition method based on multi-level multimodal alignment features according to claim 1, characterized in that: The ImageBind multimodal feature extraction model is used to extract shot-level multimodal features. Specifically, aligned shot-level multimodal features are obtained based on the ImageBind model with fixed parameters. The lens i Let i ∈ {1, ..., N}. Input the ImageBind model to extract visual features, resulting in a lens-level visual feature sequence {v1, v2, ..., v...}. N }; Using ffmpeg to extract from the lens s i The audio is extracted, and then the audio is input into the ImageBind model to extract auditory features, resulting in a shot-level auditory feature sequence {a1, a2, ..., a...}. N }; Using a Whisper model with fixed parameters from lens s i The audio dialogue is extracted, and then the dialogue segment is divided into several sentences {l1, l2, ..., l...} based on sentence breaks, including periods, exclamation marks, and question marks. K The ImageBind model is used to extract text features, resulting in a shot-level text feature sequence {t1, t2, ..., t}. N } 5. The video dialogue style recognition method based on multi-level multimodal alignment features according to claim 1, characterized in that: Using the TransNet V2 shot detection model, the input video is divided into several shots, resulting in a shot sequence {s1, s2, ..., s...}. N } 6. The video dialogue style recognition method based on multi-level multimodal alignment features according to claim 5, characterized in that: If a shot is longer than 30 seconds, it is considered a long shot and is further divided into 7-second segments. Finally, based on the detection and segmentation results, the video is exported using MoviePy.

7. The video dialogue style recognition method based on multi-level multimodal alignment features according to claim 1, characterized in that: The lens-level multimodal fusion feature sequence {f1, f2, ..., f N Input a classifier to obtain shot-level dialogue style prediction results {p1, p2, ..., p} N Finally, the camera-level dialogue style prediction results {p1, p2, ..., p} are used to predict the dialogue style. N Input a max pooling layer to obtain the final video-level dialogue style prediction result.

8. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the processor to implement the video dialogue style recognition method based on multi-level multimodal alignment features as described in any one of claims 1-7, and to output dialogue style prediction for the input video containing human dialogue.

9. A computer-readable storage medium, characterized in that... The computer-readable storage medium stores at least one instruction or at least one program, which, when executed, implements the video dialogue style recognition method based on multi-level multimodal alignment features as described in any one of claims 1-7.

Images