A Multimodal Sentiment Analysis Method Based on Modality Fusion and Spatial Location Encoding

Abstract

This invention relates to a multimodal sentiment analysis method based on modality fusion and spatial location coding, belonging to the field of natural language processing. The method comprises: 1) processing the original data samples to extract text, speech, and video modalities; 2) constructing a spatial location coding module to reduce the loss of location feature information; 3) designing a fusion embedding module to fuse text as the basic modality with speech and video modalities; 4) designing a sentiment classifier and a loss function, with the loss function used for updating model training parameters; and 5) verifying the effectiveness of the proposed method through comparison with various other methods. This invention addresses the problems of existing fusion methods lacking the ability to retain key modal features and ignoring location feature information, thereby increasing the generalization ability of the multimodal sentiment analysis model.

Description

Technical Field

[0001] This invention belongs to the field of multimodal sentiment analysis and relates to a multimodal sentiment analysis method based on modality fusion and spatial location coding. Background Technology

[0002] With advancements in communication technology and the widespread adoption of social media platforms like Facebook and YouTube, people are increasingly accustomed to expressing their emotional opinions through various data on these platforms. Early sentiment analysis could only extract and analyze people's emotions, attitudes, and stances towards a particular subject or behavior from single-modal data such as text, failing to consider the emotional information contained in other modalities like voice and video. Therefore, in most situations, it couldn't effectively extract and analyze the true emotional opinions expressed in content. Multimodal sentiment analysis, therefore, focuses on extending text-based sentiment analysis to include multimodal information such as text, visual, voice, or other forms. Multimodal sentiment analysis has achieved significant improvements in handling complex multimodal data and has received increasing attention. Because it can extract people's most authentic emotions and opinions from this multimodal data, sentiment analysis can be applied to practical fields such as social recommendation, public opinion early warning, trust management, and mental health.

[0003] Beyond its applications at the individual level, sentiment analysis also has significant implications at the national level. In government decision-making and governance, sentiment analysis can provide governments with accurate public opinion feedback, helping to formulate more targeted and feasible policies. Furthermore, for national image management and crisis response, sentiment analysis can monitor sentiment feedback through channels such as social media, helping countries respond promptly to public concerns and effectively prevent negative emotions from damaging their national image. In international relations, by analyzing the emotional tone in international social media and news reports, sentiment analysis helps countries better understand the international community's reactions to their policies and behaviors, contributing to enhanced national image, strengthened international communication and cooperation, and ultimately promoting a country's reputation and influence on the global stage.

[0004] However, early sentiment analysis could only extract and analyze people's emotions, attitudes, and stances towards a particular subject or behavior from single-modal data such as text, without considering the emotional information contained in other modal data such as voice and video. In most situations, it could not effectively extract and analyze the true emotional opinions of people's published content. Therefore, sentiment analysis targeting multimodal data has gradually gained more favor among scholars.

[0005] In fact, current multimodal sentiment analysis encompasses many different sub-tasks, largely due to the diversification of data types resulting from advancements in sensor technology. By definition, multimodal data is already a massive dataset. Among the numerous multimodal sentiment analysis tasks, video multimodal data, which combines text, speech, and video data, appears more frequently on various social media platforms than other types of multimodal data. Therefore, research on video multimodal sentiment analysis that combines multiple modalities is of significant importance.

[0006] Unlike traditional unimodal sentiment analysis, video multimodal sentiment analysis requires extracting and fusing sentiment information from more complex multimodal video data to determine sentiment trends. Initially, many researchers simply extracted sentiment information from each modality separately and fused it directly to achieve multimodal sentiment analysis. While the model's sentiment analysis capabilities can be improved by refining the methods for extracting sentiment information from different modalities, these methods neglect intramodal and intermodal correlations, making it difficult to effectively combine sentiment information from different modalities and failing to fully realize the advantages of video multimodal sentiment analysis.

[0007] To address the aforementioned issues, current multimodal sentiment analysis methods, based on modality fusion, are more effective at preserving key features than those based on a single modality. However, existing methods do not fully consider the spatial location information loss that occurs during the propagation of modal data. Therefore, it is necessary to spatially encode the modal data to reduce this loss. Summary of the Invention

[0008] In view of this, the purpose of this invention is to provide a multimodal sentiment analysis method based on modality fusion and spatial location coding.

[0009] To achieve the above objectives, the present invention provides the following technical solution:

[0010] A multimodal sentiment analysis method based on modality fusion and spatial location coding includes the following steps:

[0011] Step 1: Process the original data sample to extract the text modality, speech modality, and video modality;

[0012] Step 2: Construct a spatial location coding module to reduce the loss of location feature information;

[0013] Step 3: Design the fusion embedding module to fuse text as the basic modality with speech and video modalities;

[0014] Step 4: Design the sentiment classifier and loss function;

[0015] Step 5: Verify the effectiveness of the proposed method by comparing it with a variety of different methods.

[0016] Optionally, step one specifically includes: considering that the original data contains noise unrelated to modal information, we preprocess the original data to extract text modality, speech modality, and video modality information from the original data samples. T X A and X V These represent text modality, speech modality, and video modality, respectively. The definition of a modality pair is as follows: T = (X... T ), V' = (X T ,F V ),A'=(X T ,F A ).

[0017] Optionally, step two specifically includes: a spatial location encoding module: constructing an L×N matrix for the input modal data, where L represents rows and N represents columns, and encoding the row and column position information respectively. To ensure the integrity of the position information, a specific correspondence rule is followed: odd-numbered positions correspond to odd-numbered position information, and even-numbered positions correspond to even-numbered position information. Where PE... R Represents row position encoding, PE C The column position encoding is represented by k and j; k and j represent the positions of the tokens in the sequence; q represents odd-numbered dimensions, and 2q represents even-numbered dimensions; d model This indicates that the token has 512 dimensions.

[0018]

[0019] After spatial location encoding, we obtained the row position vector and the column position vector respectively:

[0020] R = [R0, R1, R2, ..., R] k ] T C = [C0, C1, C2, ..., C j ] T

[0021] At this point, we can obtain the encoded matrix O':

[0022] O'=R⊙C T

[0023] After spatial location encoding, the obtained modal information needs to be added to the original information. This step aims to minimize the loss of modal feature information and ensure that the encoded data maintains consistency with the original data in key features. Since the spatially encoded information and the original information may differ in content and structure, direct addition may not achieve the desired effect. Therefore, we introduce a weight parameter to adjust the weight ratio of these two in the final fused information. This allows for the effective fusion of encoded and original information, preserving the features of the original data while fully utilizing the advantages of spatial location encoding. The final output is:

[0024] F = (1-α)·O + αO', F∈{A,V}

[0025] Where O represents the original information, O' represents the encoded information, and α represents the weight parameter in the fused information.

[0026] Optionally, step three specifically includes: a fusion embedding module: fusion embedding integrates text information embedding and a fusion gate mechanism. The text embedding part includes marker embedding and position embedding, which respectively embed the text X... T The word tokens are converted into real numerical information, providing positional information for the text. Text embedding uses ordinary BERT token embedding and positional embedding, with the outputs of text embedding and spatial location encoding serving as input to the fusion gate.

[0027] BERT uses three pairs of processed data—T, V', and A'—as input. The spatially encoded data (F) is fused with the embedded text data to generate V' and A'. The text modality serves as the base modality and is fused with the other two modalities, such as... Figure 4 As shown, this fusion process first uses a linear layer to perform dimensionality matching on the text and other modal data, ensuring they reside in the same dimensional space. Next, the data from these two modalities are concatenated to form a unified representation that incorporates information from both text and other modalities. Finally, the concatenated modal data undergoes layer normalization to eliminate scale differences between modalities, further promoting information fusion. The corresponding formula is as follows:

[0028]

[0029] in It is defined as the fusion of T and F, where T represents the text modality and F represents the speech modality or video modality.

[0030] Because these modalities are processed differently, they have different dimensions. We set the text modality as the basic modality, so the dimensions of the other two modalities will vary with the size of the text modality. The specific process is as follows: F is projected onto the text modality T through a linear layer in the formula. The corresponding formula is as follows:

[0031] F' = Linear(F)

[0032] Then, based on the sequence length operation combination T and F in the following formula, the corresponding formula is as follows:

[0033]

[0034] Finally, to eliminate scale differences between different modalities and enhance model stability, we employed a regularization dimension d. T The LayerNorm layer. At this point, X has undergone fusion gate processing. V' and X A' The dimensions and length are R T×2L The length of the text modality T remains L. In BERT, a Pooler layer is used to complete the downstream tasks. The Pooler layer uses a fully connected layer and the tanh activation function. After the three pairs T, A', and V' are processed by the Pooler layer, we can obtain the following output:

[0035] W=tanh(Linear(Z)),Z∈{T,A',V'}

[0036] Optionally, in step four, regarding the sentiment classifier and loss function: we introduce a self-attention layer for W. T W V and W A The processing is then performed. Finally, we perform a concatenation operation on the fused representations of each multimodal group and feed them into a fully connected layer for result prediction. The corresponding formula is as follows:

[0037]

[0038] Due to fusion loss Loss f The calculation is performed on these three pairs of modal data, therefore the Loss f There are three types: Loss T Loss V' and Loss A' The same method is used to calculate the loss value for each type. Meanwhile, Loss task It is a task-dependent loss; since sentiment analysis is a regression task, the mean squared error loss function is used. Finally, the loss is calculated. task and Loss fThe sum of the averages of the three fusion losses is used to perform the overall learning of the model by minimizing the sum of these values. The corresponding formula is as follows:

[0039]

[0040] The beneficial effects of this invention are: it proposes a multimodal sentiment analysis method based on modality fusion and spatial location coding.

[0041] This includes: 1) processing the original data samples to extract text modalities, speech modalities, and video modalities; 2) constructing a spatial location coding module to reduce the loss of location feature information; 3) designing a fusion embedding module to fuse text as the basic modality with speech and video modalities; and 4) designing an emotion classifier and a loss function, with the loss function used for updating model training parameters.

[0042] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0043] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein:

[0044] Figure 1 Overall process;

[0045] Figure 2 Overall framework diagram of a multimodal sentiment analysis method based on modal fusion and spatial location coding;

[0046] Figure 3 Spatial location coding module structure diagram;

[0047] Figure 4 Fusion gate module structure diagram;

[0048] Figure 5 Self-attention layer structure diagram. Detailed Implementation

[0049] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0050] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0051] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0052] See Figures 1-5 This invention provides a multimodal sentiment analysis method based on modality fusion and spatial location coding.

[0053] Figure 1 The flowchart is for the specific implementation.

[0054] Figure 2 This is a diagram of the overall structure of the present invention. Figure 3 , Figure 4 and Figure 5 They are Figure 2 The detailed structure of the module is described below with reference to the accompanying drawings, including the following steps:

[0055] Optionally, step one specifically includes: preprocessing the raw data to extract text modality, speech modality, and video modality information. In this study, sentiment analysis used three modalities for experiments: text, speech, and video. These three input modalities are defined as follows: X T ∈R dT×L ,XV ∈R dV×L ,X A ∈R dA×L X T ,X V and X A These represent text, video, and speech modalities, respectively. These modal vectors have a length of L and dimensions d. T ,d V and d A Because L is a fixed length we set, when some of our inputs are smaller than L, we will use zero padding to ensure that the input lengths are consistent. Our model also uses (X... T ,F A ) and (X T ,F V The modality pair is defined as follows: T = (X...). The text modality is treated as the basic modality in the model. The modality pair is defined as follows: T = (X...). T ), V' = (X T ,F V ),A'=(X T ,F A ).

[0056] Optionally, step two specifically includes: Figure 2 This is an overall structural diagram of the present invention, which mainly includes a spatial location encoding module. Figure 3 Fusion gate module Figure 4 and the self-attention layer in the classifier Figure 5 The proposed modality fusion and spatial location coding network structure is as follows: Figure 2 As shown in the figure. The main structure consists of a spatial location encoding module, a fusion gate module, and a self-attention layer.

[0057] Spatial Location Encoding: Due to various factors, speech and video modalities often inevitably suffer information loss during modal data encoding. To minimize the loss of original modal data, we performed spatial location encoding on these two modalities. This step is crucial because it ensures that the key features of each modality are effectively preserved and fully utilized in subsequent data fusion. This paper establishes a spatial location encoding mechanism... Figure 3 The spatial location encoding module shown.

[0058] An L×N matrix is ​​constructed from the input modal data, where L represents rows and N represents columns. Row and column position information are encoded separately. To ensure the integrity of the position information, a specific correspondence rule is followed: odd-numbered positions correspond to odd-numbered position information, and even-numbered positions correspond to even-numbered position information. Here, PE... R Represents row position encoding, PE CThe column position encoding is represented by k and j; k and j represent the positions of the tokens in the sequence; q represents odd-numbered dimensions, and 2q represents even-numbered dimensions; d model This indicates that the token has 512 dimensions.

[0059]

[0060] After spatial location encoding, we obtained the row position vector and the column position vector respectively:

[0061] R = [R0, R1, R2, ..., R] k ] T C = [C0, C1, C2, ..., C j ] T

[0062] Where R represents the row position vector and C represents the column position vector, we can now obtain the encoded matrix O':

[0063] O'=R⊙C T

[0064] After spatial location encoding, the obtained modal information needs to be added to the original information. This step aims to minimize the loss of modal feature information and ensure that the encoded data maintains consistency with the original data in key features. Since the spatially encoded information and the original information may differ in content and structure, direct addition may not achieve the desired effect. Therefore, we introduce a weight parameter to adjust the weight ratio of these two in the final fused information. This allows for the effective fusion of encoded and original information, preserving the features of the original data while fully utilizing the advantages of spatial location encoding. The final output is:

[0065] F = (1-α)·O + αO', F∈{A,V}

[0066] Where O represents the original information, O' represents the encoded information, and α represents the weight parameter in the fused information.

[0067] The fusion gate module integrates text information embedding and the fusion gate mechanism. The text embedding part includes token embedding and position embedding, which respectively embed text X... T The word tokens are converted into real numerical information, providing positional information for the text. Text embedding uses ordinary BERT token embedding and positional embedding, with the outputs of text embedding and spatial location encoding serving as input to the fusion gate.

[0068] BERT uses three pairs of processed data—T, V', and A'—as input. The spatially encoded data (F) is fused with the embedded text data to generate V' and A'. The text modality serves as the base modality and is fused with the other two modalities, such as... Figure 4 As shown, this fusion process first uses a linear layer to perform dimensionality matching on the text and other modal data, ensuring they reside in the same dimensional space. Next, the data from these two modalities are concatenated to form a unified representation that incorporates information from both text and other modalities. Finally, the concatenated modal data undergoes layer normalization to eliminate scale differences between modalities, further promoting information fusion. The corresponding formula is as follows:

[0069]

[0070] in It is defined as the fusion of T and F, where T represents the text modality and F represents the speech modality or video modality.

[0071] Because these modalities are processed differently, they have different dimensions. We set the text modality as the basic modality, so the dimensions of the other two modalities will vary with the size of the text modality. The specific process is as follows: F is projected onto the text modality T through a linear layer in the formula. The corresponding formula is as follows:

[0072] F' = Linear(F)

[0073] Then, based on the sequence length operation combination T and F in the following formula, the corresponding formula is as follows:

[0074]

[0075] Finally, to eliminate scale differences between different modalities and enhance model stability, we employed a regularization dimension d. T The LayerNorm layer. At this point, X has undergone fusion gate processing. V' and X A' The dimensions and length are R T×2L The length of the text modality T remains L. In BERT, a Pooler layer is used to complete the downstream tasks. The Pooler layer uses a fully connected layer and the tanh activation function. After the three pairs T, V', and A' are processed by the Pooler layer, we can obtain the following output:

[0076] W=tanh(Linear(Z)),Z∈{T,A',V'}

[0077] Sentiment classifier and loss function: To obtain meaningful fusion performance, we introduce a self-attention layer on W.T W V and W A The processing is then performed. Finally, we perform a concatenation operation on the fused representations of each multimodal group and feed them into a fully connected layer for result prediction. The corresponding formula is as follows:

[0078]

[0079] Due to fusion loss Loss f The calculation is performed on these three pairs of modal data, therefore the Loss f There are three types: Loss T Loss V' and Loss A' The same method is used to calculate the loss value for each type. Meanwhile, Loss task It is a task-dependent loss; since sentiment analysis is a regression task, the mean squared error loss function is used. Finally, the loss is calculated. task and Loss f The sum of the averages of the three fusion losses is used to perform the overall learning of the model by minimizing the sum of these values. The corresponding formula is as follows:

[0080]

Claims

1. A multimodal sentiment analysis method based on modality fusion and spatial location coding, characterized in that: The method includes the following steps: Step 1: Process the original data sample to extract the text modality, speech modality, and video modality; Step 2: Construct a spatial location coding module to reduce the loss of location feature information; Step 3: Design the fusion embedding module to fuse text as the basic modality with speech and video modalities; Step 4: Design the sentiment classifier and loss function; Step 5: Verify the effectiveness of the proposed method by comparing it with various different methods; In step one, considering that the original data contains noise unrelated to modal information, we preprocess the original data to extract text modality, speech modality, and video modality information from the original data samples. , and These represent text modality, speech modality, and video modality, respectively. The definitions of modality pairs are as follows: , , ,in This represents the video modal features after spatial location coding. This represents the speech modal features after spatial location coding; In step two, the spatial location encoding module: constructs a [database] from the input modal data. The matrix, where Representative line, The columns represent the row and column position information, respectively. To ensure the integrity of the position information, we follow a specific correspondence rule: odd-numbered positions correspond to odd-numbered position information, and even-numbered positions correspond to even-numbered position information. Represents the row position code. The column position encoding is represented by k and j; k and j represent the position of the token in the sequence; q indicates odd-numbered dimensions, and 2q indicates even-numbered dimensions. This indicates that the token has 512 dimensions. ， , After spatial location encoding, we obtained the row position vector and the column position vector respectively: , At this point, we can obtain the encoded matrix. : , After spatial location encoding, the obtained modal information still needs to be added to the original information. We introduce a weight parameter to adjust the weight ratio of these two in the final fused information. This allows for the effective fusion of encoded and original information, preserving the features of the original data while fully utilizing the advantages of spatial location encoding. Here we use... Indicates the modality type currently undergoing spatial location coding fusion processing and ,in, Indicates video modality, Represents the speech modality; the final output is: , in Represents the original information. Represents the encoded information. This represents the weighting parameter in the fused information.

2. The multimodal sentiment analysis method based on modal fusion and spatial location coding according to claim 1, characterized in that: In step three, the fusion embedding module integrates text information embedding and a fusion gate mechanism. Text embedding uses ordinary BERT tag embedding and positional embedding, with the outputs of text embedding and spatial positional encoding serving as input to the fusion gate. BERT uses three pairs of processed data as input, with the text modality as the base modality fused with the other two modalities. This fusion process first uses a linear layer to perform dimensionality matching on the text and other modal data, ensuring they reside in the same dimensional space. Next, the data from these two modalities are concatenated for a unified representation, which simultaneously includes information from the text and other modalities. Finally, the concatenated modal data undergoes layer normalization, as shown in the following formula: , in Defined as and To integrate, among which This represents the text modality, while This refers to the speech or video modality after processing by the spatial location coding module; Because these modalities are processed differently and have different dimensions, we set the text modality as the basic modality. The sizes of the other two modalities will vary depending on the size of the text modality. The specific process is as follows: Projecting the linear layer in the formula to the text modality To make the connection, the corresponding formula is as follows: , Then combine the sequence length operations according to the following formula. and The corresponding formula is as follows: , In BERT, the Pooler layer is used to complete downstream tasks. The Pooler layer uses a fully connected layer and the tanh activation function. , and After processing by the Pooler layer, we can obtain the following output: 。 3. The multimodal sentiment analysis method based on modality fusion and spatial location coding according to claim 2, characterized in that: In step four, regarding the sentiment classifier and loss function: we introduce a self-attention layer... , and To process this, we perform a concatenation operation on the fused representation of each multimodal group and feed it into a fully connected layer for result prediction. Due to fusion loss The calculations are performed on these three pairs of modal data, so There are three types: , and The same method is used to calculate the loss value for each type; at the same time It is a task-dependent loss; since sentiment analysis is a regression task, the mean squared error loss function is used. The final calculation... and The sum of the averages of the three fusion losses, and by minimizing and To perform the overall learning of the model.

4. The multimodal sentiment analysis method based on modality fusion and spatial location coding according to claim 3, characterized in that: In step five, the effectiveness of the proposed method is verified by comparing it with a variety of different methods. We conduct experiments on the CMU-MOSI and CMU-MOSEI datasets to verify its effectiveness by comparing it with the state-of-the-art methods.

Images