Character viewpoint extraction method based on adaptive multi-modal information fusion

Abstract

The invention relates to a character viewpoint extraction method based on self-adaptive multi-modal information fusion, belongs to the technical field of video character viewpoint extraction, and completes identification of speaker identities based on self-adaptive multi-modal information feature fusion. Visual information and audio information are fused and are fused by using adaptive weights, the weights of different modalities can be adaptively adjusted according to the importance of actual tasks, and the flexibility and robustness of the model are enhanced; according to the method, viewpoints are extracted by using the graph neural network GNN, the inference ability across multiple nodes is realized through a multi-layer message passing mechanism, a system is helped to identify more complex viewpoint relationships and implicit conclusions, and even if some spokesmen do not clearly express the viewpoints of the spokesmen, the spokesmen do not clearly express the viewpoints of the spokesmen. The GNN can deduce the potential attitude of the spokesman through the propagation information of the neighbor nodes.

Description

Technical Field

[0001] The present invention relates to the technical field of video character viewpoint extraction, and in particular to a character viewpoint extraction method based on adaptive multimodal information fusion. Background Art

[0002] Video character opinion extraction is a technology that automatically extracts character opinions, emotions, and positions from video content. With the explosive growth of video content, the application demand for this technology is also increasing. Whether in social media, news media, or user-generated content, video has become one of the main media for people to express their opinions. This makes it very important to automatically extract characters' emotional attitudes, value judgments, and opinions from videos.

[0003] In practical applications, video character opinion extraction can be used in multiple scenarios. For example, in the analysis of social media and news reports, it can help automatically detect the emotions and positions of characters, providing a basis for public opinion analysis. In the field of customer service, by analyzing customer feedback in videos, customers' implicit needs and attitudes can be identified. In terms of content review and supervision, video opinion extraction technology can also help identify the spread of sensitive content, hate speech, or false information.

[0004] Existing speaker recognition technology generally uses information from one modality for recognition, such as voiceprint recognition models, such as GMM-UBM models, etc. These models are all based on a single modality. In a noisy environment, using only sound to recognize a person's identity will result in a decrease in accuracy.

[0005] Moreover, existing character opinion extraction algorithms all extract existing clear character opinions and emotions, but cannot reason based on the interactions between different characters and the relationship between opinions. Summary of the invention

[0006] In view of the above problems, the present invention provides a method for extracting character opinions based on adaptive multimodal information fusion. The present invention is based on adaptive multimodal feature fusion technology, fuses the modalities of visual information and audio information, and uses adaptive weights for fusion. Different modalities can adaptively adjust weights according to the importance of actual tasks, thereby enhancing the flexibility and robustness of the model. The present invention uses a graph neural network (GNN) to extract speaking characters and character opinions in video information, and can achieve reasoning capabilities across multiple nodes through its multi-layer message passing mechanism, helping the system predict character opinions and corresponding attitudes in video texts, and further infer complex character opinion relationships and implicit conclusions.

[0007] The present invention provides a method for extracting character viewpoints based on adaptive multimodal information fusion, comprising:

[0008] Step S1, let n=n+1, when n=1, it represents the first target video text;

[0009] Step S2, obtaining the nth target video text; preprocessing the nth target video text to obtain the nth video data and the nth audio data;

[0010] Step S3: Input the nth video data and the nth audio data into the visual feature extraction module A respectively. n And audio feature extraction module B n , get the nth visual feature and the nth audio feature respectively;

[0011] Preferably, the specific steps of obtaining the visual features in step S3 include:

[0012] Input the video data into a visual feature extraction module, and detect facial features of a person in the video data through a multi-task convolutional neural network MTCNN; characterize the facial features of the person as visual features;

[0013] It is understandable that the facial features include the facial features of the person in the video data;

[0014] Preferably, the specific steps of obtaining the audio features in step S3 include:

[0015] Input the facial features of the face into the Transformer-based visual attention model to extract the visual features v of the face image;

[0016] Extract the basic features of audio data based on Mel-frequency cepstral coefficients (MFCC) to obtain a low-dimensional audio spectrum.

[0017] Input the low-frequency audio spectrum into the pre-trained model SpeakerNet, and output the audio feature a of the character;

[0018] The visual feature extraction module in the present invention receives the video frame data, first detects the face area in the video frame through the multi-task convolutional neural network MTCNN, and intercepts the face part to ensure the concentrated extraction of facial features. The ViT network is used to extract the features of the face image, and the features of the face area are separated to obtain clearer visual information as the visual feature vector v.

[0019] Step S4: Input the nth visual feature and the nth audio feature into the multimodal adaptive attention fusion module C n , based on the adaptive attention mechanism, output the nth fused multimodal features;

[0020] Preferably, the specific steps of the nth fused multimodal feature in step S4 include:

[0021] Step S41, inputting the visual features and the audio features into a multimodal adaptive attention fusion module;

[0022] Step S42: Use a fully connected layer to map the visual features and audio features to the same vector space, and obtain the visual feature modality v′ and the audio feature modality a′, respectively, expressed as:

[0023] v′=W v v,a′=W a a

[0024] Among them, W v is the mapping matrix of visual features, W a is the mapping matrix of audio features.

[0025] Step S43, adaptively assigning weights to the visual feature modality v′ and the audio feature modality a′, to obtain the weight of the visual feature modality v′ and the weight of the audio feature modality a′ respectively;

[0026] Step S44: Use the additive attention mechanism to calculate the weight of the visual feature modality v′ and the weight of the audio feature modality a′, respectively, to obtain the visual feature modality attention weight and the audio feature modality attention weight, respectively, and the expressions are:

[0027] e v =tanh(W vh v′+W ah a′)·w h

[0028]

[0029] Among them, W vh is the learnable parameter 1, W ah is the second learnable parameter, W ah ∈R d×d , α v is the attention weight of the visual feature modality, α a is the attention weight of the audio feature modality, e v represents the intermediate calculated value of the visual modality feature weight, e a represents the intermediate calculated value of the audio modal feature weight, tanh(·) is the hyperbolic tangent function, and w h are learnable parameters.

[0030] Step S45: Based on the visual feature modality attention weight and the audio feature modality attention weight, the visual feature modality and the audio feature modality are weighted and summed to obtain the fused multimodal feature, which is expressed as:

[0031] f=α v v′+αa a′

[0032] Among them, f is the fused multimodal feature.

[0033] The present invention uses the graph neural network GNN to extract opinions, and realizes reasoning ability across multiple nodes through its multi-layer message passing mechanism, helping the system to identify more complex opinion relationships and implicit conclusions. Even if some speakers do not clearly express their opinions, the graph neural network GNN can infer the speaker's potential attitude through the propagation information of neighboring nodes. For example, speaker A and speaker B have a strong connection in the graph neural network, and speaker B has clearly expressed the character's opinion. The graph neural network GNN can use the character's opinion that speaker B has clearly expressed to infer the possible character's opinion of speaker A.

[0034] Step S5: input the nth fused multimodal feature into the speaker recognition module D n , output multiple speaker IDs in the nth target video text;

[0035] Step S6, recognizing the voice information in the nth target video text and converting it into the nth text information; preprocessing the nth text information to obtain the nth updated text information;

[0036] It can be understood that the specific steps of the preprocessing in step S5 include word segmentation of the text information and removal of stop words.

[0037] Exemplarily, the Jieba word segmentation algorithm is used to segment text information.

[0038] Step S7: Input the nth updated text information into the BERT model E n , extract multiple text features; use k-means clustering algorithm to cluster multiple text features and obtain multiple topics;

[0039] The LDA algorithm is used to extract the subject words corresponding to each topic; it can be understood that the BERT model is a bidirectional encoder representation from Transformer model;

[0040] It can be understood that dividing a text into different paragraphs and using the themes described in step S7 is helpful for analyzing the viewpoints of characters in different paragraphs, that is, the viewpoints of characters are all centered around a certain theme; the theme words are further determined by calculating the probability distribution of each word in the paragraph on each theme;

[0041] Step S8: Input the nth group of multiple topics and the nth group of multiple speaker identity IDs into the BERT model E n , and obtain the viewpoints and corresponding attitudes of different people under each topic in the nth group;

[0042] Exemplarily, the attitude of the viewpoint is the emotional attitude of the viewpoint, including positive, negative, and neutral;

[0043] Preferably, the expression of the attitude corresponding to the viewpoints of different speakers in step 8 is:

[0044]

[0045] where, represents the attitude vector output by the person node i in the (k + 1)-th convolutional layer, represents the attitude vector output by the person node i in the k-th convolutional layer, represents the neighbor set of the person node i, and W k represents the weight matrix of the k-th convolutional layer, and σ(·) is the activation function, represents the attitude vector output by the person node j in the k-th convolutional layer, and b k represents the bias term of the k-th convolution.

[0046] Step S9: Input the multiple topics in the nth group, the topic words of each topic, the viewpoints of the people corresponding to each topic and their corresponding attitudes into the graph neural network F n , and based on the message passing mechanism, divide the people with the same attitude, update the graph neural network F n , obtain multiple groups of people in the nth video file and update the graph neural network F n , and record; use the updated graph neural network F n as the graph neural network F n+1 ;

[0047] Exemplarily, the graph neural network is a graph neural network of topic-person-viewpoint;

[0048] Each group of people includes multiple people with the same attitude;

[0049] Preferably, the graph neural network in step S9 includes multiple nodes and multiple edges; the nodes include the people, viewpoints, and topics in the video file, where the topic is marked as a special type of node representing the topic discussed by the people;

[0050] The edges include the edges formed by connecting people and viewpoints, the edges formed by connecting people and people, and the edges formed by connecting viewpoints and topics; the connection between the viewpoint and the topic by an edge indicates the viewpoint of the person on the topic;

[0051] The edge connecting the characters includes the intersection of the opinions between two characters, and they are connected by an edge; for example, if two characters have expressed related or opposing opinions on the same topic, they can be connected by an edge;

[0052] The edge connecting the character and the viewpoint indicates that the viewpoint belongs to the topic;

[0053] It can be understood that the specific steps of dividing the characters holding the same attitude based on the message transmission mechanism in step S9 include:

[0054] Obtain the relationship between the characters' opinions based on their opinions and their corresponding attitudes;

[0055] It is understood that the relationship between the characters' opinions includes support or refutation; the attitudes of the characters' opinions include positive, negative and neutral;

[0056] Based on the relationship, determining whether the attitudes corresponding to the character's viewpoints are consistent, and dividing the characters with the consistent attitudes into the same character group to obtain multiple character groups;

[0057] It can be understood that if multiple characters express similar opinions on the same topic, and each character's opinion has a consistent attitude (such as all positive), the attitude will propagate along the edges between characters, identifying a group of characters with the same attitude;

[0058] Step S10, determine whether n is greater than or equal to N, where N represents the total number of target video texts. If so, obtain the final graph neural network. If not, set n=n+1 and return to step S2;

[0059] Exemplarily, an emotional dialogue MELD dataset is used as the target video text, wherein the emotional dialogue MELD dataset is a multimodal emotional dataset, including video dialogue text and its emotional attitude annotations;

[0060] Step S11: input the video data to be predicted into the final graph neural network and output the predicted character’s point of view.

[0061] For example, in the video file, character B did not express any opinions in topic A. Through the final graph neural network, transfer inference can be performed to obtain the predicted opinions and attitudes of character B.

[0062] The method also includes step S12, inputting the video data to be predicted into the final graph neural network, and outputting the attitude corresponding to the predicted character's point of view.

[0063] Compared with the prior art, the present invention has at least the following beneficial effects:

[0064] (1) The present invention realizes the identification of the speaker based on adaptive multi-modal information feature fusion;

[0065] By fusing visual information and audio information and using adaptive weights for fusion, the weights of different modalities can be adaptively adjusted according to the importance of the actual task, enhancing the flexibility and robustness of the model;

[0066] (2) In the case of noise or unclear signal of a certain modality, the attention mechanism of the present invention can reduce the negative impact of the modality on the final recognition result by reducing the weight of the modality. It combines visual and audio features, overcomes the limitations of single modality recognition, and greatly improves the accuracy of speaker recognition in complex scenarios.

[0067] (3) The present invention utilizes graph neural network (GNN) to extract opinions and realizes reasoning capabilities across multiple nodes through its multi-layer message passing mechanism, helping the system to identify more complex opinion relationships and implicit conclusions. Even if some speakers do not explicitly express their opinions, the graph neural network (GNN) can infer the speaker’s potential attitude through the propagation information of neighboring nodes. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] The drawings are only for the purpose of illustrating particular embodiments and are not to be construed as limiting the invention.

[0069] Figure 1 is a schematic diagram of a process of person recognition based on adaptive multimodal information fusion in an embodiment of the present invention;

[0070] Figure 2 Schematic diagram of the process of opinion extraction and reasoning based on graph neural network in an embodiment of the present invention. DETAILED DESCRIPTION

[0071] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other. In addition, the present invention can also be implemented in other ways different from those described herein, and therefore, the protection scope of the present invention is not limited by the specific embodiments disclosed below.

[0072] A specific embodiment of the present invention, as Figure 1-2 , discloses a method for extracting character viewpoints based on adaptive multimodal information fusion. In order to illustrate the effectiveness of the method proposed in the present invention, the above technical solution of the present invention is described in detail through a specific embodiment below. The specific implementation steps are as follows:

[0073] The present invention provides a method for extracting character viewpoints based on adaptive multimodal information fusion, comprising:

[0074] Step S1, let n=n+1, when n=1, it represents the first target video text;

[0075] Step S2, obtaining the nth target video text; preprocessing the nth target video text to obtain the nth video data and the nth audio data;

[0076] Step S3: Input the nth video data and the nth audio data into the visual feature extraction module A respectively. n And audio feature extraction module B n , get the nth visual feature and the nth audio feature respectively;

[0077] Preferably, the specific steps of obtaining the visual features in step S3 include:

[0078] Input the video data into a visual feature extraction module, and detect facial features of a person in the video data through a multi-task convolutional neural network MTCNN; characterize the facial features of the person as visual features;

[0079] It is understandable that the facial features include the facial features of the person in the video data;

[0080] Preferably, the specific steps of obtaining the audio features in step S3 include:

[0081] Input the facial features of the face into the Transformer-based visual attention model to extract the visual features v of the face image;

[0082] Extract the basic features of audio data based on Mel-frequency cepstral coefficients (MFCC) to obtain a low-dimensional audio spectrum.

[0083] Input the low-frequency audio spectrum into the pre-trained model SpeakerNet, and output the audio feature a of the character;

[0084] The visual feature extraction module in the present invention receives the video frame data, first detects the face area in the video frame through the multi-task convolutional neural network MTCNN, and intercepts the face part to ensure the concentrated extraction of facial features. The ViT network is used to extract the features of the face image, and the features of the face area are separated to obtain clearer visual information as the visual feature vector v.

[0085] Step S4: Input the nth visual feature and the nth audio feature into the multimodal adaptive attention fusion module C n , based on the adaptive attention mechanism, output the nth fused multimodal features;

[0086] Preferably, the specific steps of the nth fused multimodal feature in step S4 include:

[0087] Step S41, inputting the visual features and the audio features into a multimodal adaptive attention fusion module;

[0088] Step S42: Use a fully connected layer to map the visual features and audio features to the same vector space, and obtain the visual feature modality v′ and the audio feature modality a′, respectively, expressed as:

[0089] v′=W v v,a′=W a a

[0090] Among them, W v is the mapping matrix of visual features, W a is the mapping matrix of audio features.

[0091] Step S43, adaptively assigning weights to the visual feature modality v′ and the audio feature modality a′, to obtain the weight of the visual feature modality v′ and the weight of the audio feature modality a′ respectively;

[0092] Step S44: Use the additive attention mechanism to calculate the weight of the visual feature modality v′ and the weight of the audio feature modality a′, respectively, to obtain the visual feature modality attention weight and the audio feature modality attention weight, respectively, and the expressions are:

[0093] e v =tanh(W vh v′+W ah a′)·w h

[0094]

[0095] Among them, W vh is the learnable parameter 1, W ah is the second learnable parameter, W ah ∈R d×d , α v is the attention weight of the visual feature modality, α a is the attention weight of the audio feature modality, e v represents the intermediate calculated value of the visual modality feature weight, e a represents the intermediate calculated value of the audio modal feature weight, tanh(·) is the hyperbolic tangent function, and w h are learnable parameters.

[0096] Step S45: Based on the visual feature modality attention weight and the audio feature modality attention weight, the visual feature modality and the audio feature modality are weighted and summed to obtain the fused multimodal feature, which is expressed as:

[0097] f=α v v′+α a a′

[0098] Among them, f is the fused multimodal feature.

[0099] The present invention uses the graph neural network GNN to extract opinions, and realizes reasoning ability across multiple nodes through its multi-layer message passing mechanism, helping the system to identify more complex opinion relationships and implicit conclusions. Even if some speakers do not clearly express their opinions, the graph neural network GNN can infer the speaker's potential attitude through the propagation information of neighboring nodes. For example, speaker A and speaker B have a strong connection in the graph neural network, and speaker B has clearly expressed the character's opinion. The graph neural network GNN can use the character's opinion that speaker B has clearly expressed to infer the possible character's opinion of speaker A.

[0100] Step S5: input the nth fused multimodal feature into the speaker recognition module E n , output multiple speaker IDs in the nth target video text;

[0101] Step S6, recognizing the voice information in the nth target video text and converting it into the nth text information; preprocessing the nth text information to obtain the nth updated text information;

[0102] It can be understood that the specific steps of the preprocessing in step S5 include word segmentation of the text information and removal of stop words.

[0103] Exemplarily, the Jieba word segmentation algorithm is used to segment text information.

[0104] Step S7: Input the nth updated text information into the BERT model E n , extract multiple text features; use k-means clustering algorithm to cluster multiple text features and obtain multiple topics;

[0105] The LDA algorithm is used to extract the subject words corresponding to each topic; it can be understood that the BERT model is a bidirectional encoder representation from Transformer model;

[0106] It can be understood that dividing a text into different paragraphs and using the themes described in step S7 is helpful for analyzing the viewpoints of characters in different paragraphs, that is, the viewpoints of characters are all centered around a certain theme; the theme words are further determined by calculating the probability distribution of each word in the paragraph on each theme;

[0107] Step S8: Input the nth group of multiple topics and the nth group of multiple speaker IDs into the BERT model E n , obtain the perspectives and corresponding attitudes of different characters under each theme of the nth group;

[0108] Exemplarily, the attitude of the opinion is the emotional attitude of the opinion, including positive, negative and neutral;

[0109] Preferably, the expression of the attitude corresponding to the character viewpoints of different speakers in step 8 is:

[0110]

[0111] in, represents the attitude vector output by the character node i at the k+1th convolutional layer, represents the attitude vector output by the character node i at the kth convolutional layer, represents the neighbor set of character node i, W k represents the weight matrix of the kth convolutional layer, σ(·) is the activation function, represents the attitude vector output by the character node j at the kth convolutional layer, b k Represents the bias term of the k-th convolution layer.

[0112] Step S9: Input the multiple topics of the nth group, the keywords of each topic, the viewpoints of the characters corresponding to each topic and their corresponding attitudes into the graph neural network F n Based on the message passing mechanism, people with the same attitude are divided into groups and the graph neural network F n Update to obtain multiple character groups of the nth video file and update the graph neural network F n , and record; update the graph neural network F n As a graph neural network F n+1 ;

[0113] Exemplarily, the graph neural network is a graph neural network of topic-character-character opinion;

[0114] Each character group includes a plurality of characters with the same attitude;

[0115] Preferably, the graph neural network in step S9 includes a plurality of nodes and a plurality of edges; the nodes include characters, character opinions, and topics in the video file, wherein the topic is marked as a special type of node, representing the topic discussed by the characters;

[0116] The edges include edges formed by connecting characters with character viewpoints, edges formed by connecting characters with characters, and edges formed by connecting character viewpoints with topics; character viewpoints and topics are connected by edges to represent the character's viewpoint on the topic;

[0117] The edge connecting the characters includes the intersection of the opinions between two characters, and they are connected by an edge; for example, if two characters have expressed related or opposing opinions on the same topic, they can be connected by an edge;

[0118] The edge connecting the character and the viewpoint indicates that the viewpoint belongs to the topic;

[0119] It can be understood that the specific steps of dividing the characters holding the same attitude based on the message transmission mechanism in step S9 include:

[0120] Obtain the relationship between the characters' opinions based on their opinions and their corresponding attitudes;

[0121] It is understood that the relationship between the characters' opinions includes support or refutation; the attitudes of the characters' opinions include positive, negative and neutral;

[0122] Based on the relationship, determining whether the attitudes corresponding to the character's viewpoints are consistent, and dividing the characters with the consistent attitudes into the same character group to obtain multiple character groups;

[0123] It can be understood that if multiple characters express similar opinions on the same topic, and each character's opinion has a consistent attitude (such as all positive), the attitude will propagate along the edges between characters, identifying a group of characters with the same attitude;

[0124] Step S10, determine whether n is greater than or equal to N, where N represents the total number of target video texts. If so, obtain the final graph neural network. If not, set n=n+1 and return to step S2;

[0125] Exemplarily, an emotional dialogue MELD dataset is used as the target video text, wherein the emotional dialogue MELD dataset is a multimodal emotional dataset, including video dialogue text and its emotional attitude annotations;

[0126] Step S11: input the video data to be predicted into the final graph neural network and output the predicted character’s point of view.

[0127] For example, in the video file, character B did not express any opinions in topic A. Through the final graph neural network, transfer inference can be performed to obtain the predicted opinions and attitudes of character B.

[0128] The method also includes step S12, inputting the video data to be predicted into the final graph neural network, and outputting the attitude corresponding to the predicted character's point of view.

[0129] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A method for extracting character opinions based on adaptive multimodal information fusion, characterized in that: include: Step S1, let n=1, when n=1, it represents the first target video text; Step S2, obtaining the nth target video text; Preprocessing the nth target video text to obtain nth video data and nth audio data; Step S3: Input the nth video data and the nth audio data into the visual feature extraction module A respectively. n And audio feature extraction module B n , get the nth visual feature and the nth audio feature respectively; Step S4: Input the nth visual feature and the nth audio feature into the multimodal adaptive attention fusion module C n , based on the adaptive attention mechanism, output the nth fused multimodal features; Step S5: input the nth fused multimodal feature into the speaker recognition module D n , output multiple speaker IDs in the nth target video text; Step S6, recognizing the voice information in the nth target video text and converting it into the nth text information; Preprocessing the nth text information to obtain nth updated text information; Step S7: Input the nth updated text information into the BERT model E n , get multiple topics; Use the LDA algorithm to extract the subject words corresponding to each topic; Step S8: Input the nth group of multiple topics and the nth group of multiple speaker IDs into the BERT model E n , obtain the opinions of different characters under each theme of the nth group; Step S9: Input the multiple topics of the nth group, the keywords of each topic, and the corresponding opinions of the characters of each topic into the graph neural network F n Based on the message passing mechanism, multiple character groups of the nth video file are obtained and the graph neural network F is updated n , and record; the graph neural network E will be updated n As a graph neural network F n+1 ; Step S10, determine whether n is greater than or equal to N, where N represents the total number of target video texts. If so, obtain the final graph neural network. If not, set n=n+1 and return to step S2; Step S11: input the video data to be predicted into the final graph neural network and output the predicted character’s point of view.

2. The method for extracting character opinions based on adaptive multimodal information fusion according to claim 1, characterized in that: The specific steps of obtaining the visual features in step S3 include: The video data is input into a visual feature extraction module, and the facial features of the characters in the video data are detected by a multi-task convolutional neural network MTCNN; the facial features of the characters are represented as visual features.

3. The method for extracting character viewpoints based on adaptive multimodal information fusion according to claim 1, characterized in that: The specific steps of obtaining the audio features in step S3 include: Input facial features into the Transformer-based visual attention model to extract visual features of facial images; Extract the basic features of audio data based on Mel-frequency cepstral coefficients (MFCC) to obtain a low-dimensional audio spectrum. The audio features of the person are extracted based on the low-dimensional audio spectrum.

4. The method for extracting character viewpoints based on adaptive multimodal information fusion according to claim 1, characterized in that: The specific steps of the nth fused multimodal feature in step S4 include: Step S41, inputting the visual features and the audio features into a multimodal adaptive attention fusion module; Step S42: Use a fully connected layer to map the visual features and the audio features to the same vector space to obtain a visual feature modality and an audio feature modality, respectively; Step S43, adaptively assigning weights to the visual feature modality and the audio feature modality to obtain visual feature modality weights and audio feature modality weights respectively; Step S44: using an additive attention mechanism to calculate the visual feature modality weight and the audio feature modality weight respectively, to obtain the visual feature modality attention weight and the audio feature modality attention weight respectively; Step S45: Based on the visual feature modality attention weight and the audio feature modality attention weight, perform weighted summation on the visual feature modality and the audio feature modality to obtain a fused multimodal feature.

5. The method for extracting character viewpoints based on adaptive multimodal information fusion according to claim 4, characterized in that: The expressions of the visual feature modality attention weight and the audio feature modality attention weight are respectively: e v =tanh(W vh v′+W ah a′)·w h Among them, W vh is the learnable parameter 1, W ah is a learnable parameter; v′ is the visual feature mode, a′ is the audio feature mode, α v is the attention weight of the visual feature modality, α a is the attention weight of the audio feature modality, e v represents the intermediate calculated value of the visual modality feature weight, e a represents the intermediate calculated value of the audio modal feature weight, tanh(·) is the hyperbolic tangent function, and w h are learnable parameters.

6. The method for extracting character viewpoints based on adaptive multimodal information fusion according to claim 5, characterized in that: The expression of the fused multimodal feature is: f=a v v′+a a a′ Among them, f is the fused multimodal feature.

7. The method for extracting character viewpoints based on adaptive multimodal information fusion according to claim 1, characterized in that: The specific steps of obtaining the multiple topics of the nth group in step S7 include: Input the nth updated text information into the BERT model E n , extract multiple text features; use the k-means clustering algorithm to cluster multiple text features to obtain multiple topics.

8. The method for extracting character viewpoints based on adaptive multimodal information fusion according to claim 1, characterized in that: Step S9: The graph neural network includes a plurality of nodes and a plurality of edges; the nodes include characters, character viewpoints, and themes in the video file; The edges include edges formed by connecting characters with character viewpoints, edges formed by connecting characters with characters, and edges formed by connecting character viewpoints with themes.

Images