A dynamic topological hierarchical sentiment analysis method based on multimodal information fusion

Abstract

The present invention belongs to the field of sentiment analysis technology, and specifically relates to a dynamic topological hierarchical sentiment analysis method of multimodal information fusion, which extracts image features and text features; aligns image features and text features through a multi-head attention mechanism to obtain an aligned image-text feature vector; maps the image-text feature vector to a dynamic topological hierarchy, and uses a high-dimensional expansion coding method to fuse and high-dimensionally expand the multimodal features of the image and text to obtain high-dimensional expansion coding features of the image features and text features, and inputs them into a sentiment classifier to obtain three types of sentiment classification results: positive, neutral, and negative. The present invention combines the multimodal feature alignment and fusion strategy of deep learning, can improve the accuracy and applicability of sentiment classification on an imbalanced data set, and provides an effective technical solution for multimodal sentiment analysis.

Description

Technical Field

[0001] The present invention belongs to the technical field of sentiment analysis combining natural language processing with computer vision, and specifically relates to a dynamic topological hierarchical structure sentiment analysis method of multimodal information fusion. Background Art

[0002] As an important direction in the field of natural language processing, sentiment analysis mainly studies how to identify and extract emotional tendencies and attitudes in text, images or audio. By comprehensively analyzing the user's expressions in different modalities (such as text, images, etc.), the emotional polarity can be identified more accurately. In recent years, researchers have continuously explored multimodal information processing technology, committed to giving machines the ability to understand complex emotional information so that they can have the same emotional recognition ability as humans. Multimodal fusion sentiment analysis has broad application prospects in various fields, such as user comment value mining on e-commerce platforms, customer relationship management, auxiliary marketing decision-making, and agricultural product evaluation and analysis in the agricultural field. It has also gradually expanded to emerging application scenarios such as drug activity prediction, and dynamic fusion is achieved by integrating information from different modalities. This cross-modal information fusion requirement has become a core technical requirement in modern data-driven applications.

[0003] However, current multimodal sentiment analysis methods face many challenges. First, the significant semantic differences between different modalities make it very difficult to effectively align information between modalities. Second, the frequent feature redundancy problem in multimodal data leads to a doubling of computational complexity, which not only increases the computational overhead of the model, but also limits the applicability of the model to a certain extent. In addition, the diversity of user opinions and their informal expressions (such as emoticons, sarcastic language, and abbreviations) further increase the difficulty of identifying true emotions. These factors often lead to the inability of unimodal analysis to accurately capture the user's true emotions, highlighting the importance of multimodal sentiment analysis in complex data scenarios, and at the same time showing that there is a broad prospect for promoting research on multimodal feature fusion, semantic alignment, and model generalization in this field. Summary of the invention

[0004] In order to solve the problem of feature redundancy and overfitting in existing fusion methods while maintaining the semantic integrity of cross-modal information, the purpose of the present invention is to propose a dynamic topological hierarchical sentiment analysis method for multimodal information fusion. By quantifying multimodal feature processing layer by layer on user opinion image and text datasets, information loss is reduced and classification accuracy is improved. In the process of iterative fusion, the non-Euclidean distance information of image and text modalities is effectively aligned to generate a robust high-dimensional feature representation, thereby realizing sentiment classification of multimodal data.

[0005] The present invention is achieved through the following technical solutions.

[0006] A dynamic topological hierarchical sentiment analysis method based on multimodal information fusion, the steps are as follows:

[0007] Step 1: Preprocess the user opinion image and text dataset of the social media platform to obtain the image and text dataset for multimodal sentiment analysis;

[0008] Step 2: Use the residual network model to extract image features from the image and text dataset; use the bidirectional encoder representation model to extract text features from the image and text dataset;

[0009] Step 3: Align image features and text features through a multi-head attention mechanism, extract the interactive information between modalities, and obtain the aligned image-text feature vector;

[0010] Step 4: Map the aligned image and text feature vectors to a dynamic topological hierarchy, construct a multi-level non-Euclidean space to capture the complex correlation between different modalities, and use a high-dimensional extended coding method to fuse and expand the multimodal features of the image and text to obtain high-dimensional extended coding features of the image and text features;

[0011] Step 5: Input the high-dimensional extended encoding features of image features and text features into the sentiment classifier to obtain the three types of sentiment classification results: positive, neutral, and negative.

[0012] Further preferably, in step 2, the image data is enhanced by combining a random data enhancement method, and then the residual network model is used to extract image features in the image and text data set.

[0013] Further preferably, in step three, a multi-head attention mechanism is applied to each pair of image features and text features, and a query vector, a key vector, and a value vector of the image and text are obtained through linear transformation; the image features and the text features are aligned through the multi-head attention mechanism to obtain the attention representation of the image features on the text features. And the attention representation of text features on image features , the attention of text features and image features to text features is represented by A 1 Connect and get the text-image connection vector containing text features and image-to-text attention information , connect the image features with the attention representation of the text features on the image features, and obtain the image-text connection vector containing the image features and the text's attention information on the image .

[0014] Further preferably, in step five, the high-dimensional extended encoding feature vectors of all image features and text features are concatenated and input into the Softmax classifier to obtain the sentiment prediction probability of each image-text pair.

[0015] The present invention also provides a computer-readable storage medium on which computer instructions are stored. When the computer instructions are executed by a processor, the dynamic topological hierarchical structure sentiment analysis method of multimodal information fusion is implemented.

[0016] The present invention adopts a research method that quantifies multimodal features layer by layer, maps feature information between modalities into a dynamic topological hierarchy, and combines a high-dimensional encoder to reduce information loss and improve classification accuracy. By quantifying the features of image and text multimodal information layer by layer, a dynamic topological hierarchy is constructed to accurately capture the association between features of different modalities, and then achieve efficient alignment in non-Euclidean distance space. The high-dimensional encoder is combined to generate robust high-dimensional feature encoding, enhance the stability of features and classification accuracy, improve the tolerance of classification models to noise data, and promote comprehensive sentiment prediction. It not only improves the prediction performance of multimodal information, but also significantly enhances the generalization ability, providing strong technical support for large-scale, multimodal sentiment analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a flow chart of a dynamic topological hierarchical structure sentiment analysis method for multimodal information fusion provided by the present invention;

[0018] Figure 2 It is a schematic diagram of the overall structure of the dynamic topological hierarchical structure sentiment analysis method of multimodal information fusion;

[0019] Figure 3 This is a graph showing the label distribution of user opinion image and text datasets on four social media platforms;

[0020] Figure 4 It is a schematic diagram of the confusion matrix effect on two types of user opinion image and text datasets. DETAILED DESCRIPTION

[0021] The present invention is further described in detail below with reference to the accompanying drawings and embodiments.

[0022] Reference Figure 1 and Figure 2 , a dynamic topological hierarchical sentiment analysis method based on multimodal information fusion, the steps are as follows:

[0023] Step 1: Preprocess the user opinion image and text datasets of social media platforms. There are four types of datasets (Twitter15, Twitter17, MVSA-Single, MVSA-Multiple). The Twitter15 and Twitter17 datasets are composed of user comment image and text pairs with sentiment annotations. Each dataset is randomly divided into three subsets: training set, validation set, and test set. Each tweet is labeled as positive, negative, or neutral. The three subsets are merged respectively, and finally 5337 Twitter15 data and 5971 Twitter17 data image and text datasets are obtained.

[0024] Both the MVSA-Single and MVSA-Multiple datasets are derived from image-text pairs on social media platforms, and are manually annotated with three sentiment polarity labels: positive, neutral, and negative. For MVSA-Single, one annotator annotated each sample with a single sentiment, totaling 4,869 image-text pairs. For MVSA-Multiple, three annotators annotated each sample with three sentiments, totaling 19,600 image-text pairs. For MVSA-Single, the image-text pairs with conflicting positive and negative polarity labels in the dataset are deleted. For the remaining image-text pairs, if one party is neutral, the positive or negative label of the other party is selected as the emotional label of the image-text pair, and finally 4511 image-text datasets of MVSA-Single data are obtained; for MVSA-Multiple, a voting mechanism is used to count the most frequent occurrences of the three emotional labels as the true label of a single modality. When at least two of the three annotators give the same exact label, the image or text is considered a valid sample; if the text and visual labels are consistent, they are retained; if two labels each account for one-third, they are deleted; if one label is positive (negative) and the other is neutral, the emotional polarity of the sample is considered to be positive (negative), and finally 16779 image-text datasets of MVSA-Multiple data are obtained. The image-text dataset samples of the four types of datasets finally obtained are shown in Table 1:

[0025] Table 1

[0026]

[0027] The emotional label distribution of the four types of data sets is as follows Figure 3As shown in the figure, it can be observed that the distribution of each category in the four data sets is unbalanced. The four types of image and text data sets are uniformly labeled, and the text labels "positive", "negative" and "neutral" in the data sets are converted into numerical labels "2", "0" and "1", thereby constructing an image and text data set containing the "image-text-emotional label" format, and the four types of data sets are divided into training set, validation set and test set in a ratio of 8:1:1.

[0028] Step 2: Combine the random data enhancement method to enhance the image data, use the residual network model to extract the image features in the image and text dataset; use the bidirectional encoder representation model to extract the text features in the image and text dataset. The original image data is known to be ,in Respectively represent the image data in the 1st, 2nd, …, Mth image and text data set, perform random data enhancement operations on each image data, and use the grid search method to perform incremental simple grid search. The calculation process is shown in formula (1):

[0029] (1);

[0030] in Represents the image The result after the jth data enhancement, i is the image number, represents the data enhancement function, represents a set of enhanced parameters, where represents the 1st, 2nd, ..., Nth enhancement parameters, represents the jth enhancement parameter, each of which defines a specific enhancement method and its corresponding parameter configuration (such as rotation angle, scaling ratio, etc.). A collection of enhanced images after data enhancement under different enhancement parameters ,in Respectively represent images The data enhancement results under the 1st, 2nd, ..., Nth enhancement parameters. Finally, the enhanced image set for each image is merged through formula (2) .

[0031] (2);

[0032] in represents the augmented image dataset, Represents the set union operation, which is the enhanced image set of the 1st, 2nd, .., Mth images. Merge to get a complete enhanced image dataset .

[0033] Get the enhanced image dataset After that, It represents the N enhanced images obtained by different enhancement parameters for the i-th image data. The residual network model is used to extract features for each enhanced image to obtain a high-dimensional response graph representation. The calculation formula is shown in formula (3):

[0034] (3);

[0035] in Represents the i-th image The jth enhanced image The feature response map extracted by the residual network model, is the feature extraction function of the residual network model. Converted to sequence form, it is known that ,in Indicates the number of images processed each time, represents the image dimension, represents the dimension of the intermediate hidden space (the depth of the output feature map of the residual network model), represents the real number domain. Finally, the feature encoding tensor converted into sequence form is obtained, as shown in formula (4):

[0036] (4);

[0037] in, denote the 1st, 2nd, …, feature vectors, Represents the shape Finally, the feature coding tensors of all enhanced images are combined to obtain the image features of the entire enhanced image dataset, as shown in formula (5):

[0038] (5);

[0039] in Represents the image features of the entire enhanced image dataset, which is a set of image features extracted after data enhancement and residual network model. ,in Respectively represent the image features of the 1st, 2nd, …, Mth image data.

[0040] For input text data ,in Represent the text data in the 1st, 2nd, ..., Mth image and text dataset respectively, and use the bidirectional encoder representation model to extract the text hidden state features of the last layer of the text data ,in Respectively represent the hidden state features of the 1st, 2nd, ..., Mth texts, Indicates the amount of text processed each time. Indicates the length of the text sequence, represents the output hidden dimension of the bidirectional encoder representation model. In order to map the text feature dimension to the same intermediate hidden dimension as the image feature , the text hidden state feature H is transformed using a linear layer and ReLu activation function, as shown in formula (6):

[0041] (6);

[0042] (7);

[0043] in Represents the transformed hidden state tensor, with a shape size of , Represents the weight matrix of the linear layer, with a size of , maps H to the target hidden dimension d t , b 1 Represents the bias vector of the linear layer, with a size of d t Finally, we get the text features in the image and text dataset ,in Represent the text features of the 1st, 2nd, …, Mth text data respectively.

[0044] Step 3: Align image features and text features through a multi-head attention mechanism, extract the interactive information between modalities, and obtain the aligned image-text feature vector. and text features Apply the multi-head attention mechanism to obtain the query vectors of images and texts through linear transformation , , the key vector , , value vector , , specifically obtained through formulas (8)-(9):

[0045] (8);

[0046] (9);

[0047] in Represent the query vector, key vector, value vector of the image and the learnable weight matrix of the query vector, key vector, value vector of the text. Align the image features and text features through the multi-head attention mechanism to obtain the attention representation of the image features on the text features. And the attention representation of text features on image features , as shown in formula (10)-formula (11):

[0048] (10);

[0049] (11);

[0050] Among them A 1 A represents the attention representation of image features to text features. 2 represents the attention representation of text features to image features, Respectively represent K T and K G The transpose of the matrix, Represent the scaling factors of text features and image features respectively to prevent the values ​​from being too large. The image features of the entire enhanced image dataset are known. , text features in the image and text dataset , the attention of text features and image features to text features is represented by A 1 Connect the image features and the attention representation of the text features to the image features. The specific calculation process is shown in formula (12):

[0051] (12);

[0052] in represents the text-image concatenation vector containing text features and image-to-text attention information, Represents the image-text connection vector containing image features and text-to-image attention information, and cat() represents the connection operation.

[0053] Step 4: Map the aligned image and text feature vectors into a dynamic topological hierarchy, construct a multi-level non-Euclidean space to capture the complex correlation between different modalities, and use high-dimensional extended coding methods to fuse and expand the multimodal features of images and texts to obtain high-dimensional extended coding features of image features and text features.

[0054] Known ,in Respectively represent the 1st, 2nd, ..., Mth text-image connection vectors, ,in Represent the 1st, 2nd, …, Mth image-text connection vectors respectively. and Each element in the prime number sequence is obtained by formula (13)-formula (16). The following maps to dynamic topology hierarchy features:

[0055] (13);

[0056] (14);

[0057] (15);

[0058] (16);

[0059] in Represents the dynamic topological hierarchical structure features of the i-th element in the text-image connection vector under the specified prime number p, Represents the dynamic topological hierarchical structure features of the jth element in the image-text connection vector under the specified prime number p, and They correspond to the coefficients of different powers of the specified prime number p, where Represents the text-image connection vector The kth coefficient of the ith element in the dynamic topological hierarchy feature, Represents the image-text connection vector The jth element in the dynamic topological hierarchy feature Bit coefficient, It represents the weight corresponding to the expansion of the k-th coefficient of the i-th element in the text-image connection vector under the specified prime number p. represents the weight corresponding to the expansion of the lth coefficient of the jth element in the image-text vector under the specified prime number p. The two are used to represent the decomposition expression of text features and image features in the multimodal dynamic topological hierarchy. Since the sequence lengths obtained by mapping operations under different text or image modalities may be inconsistent, in order to make the feature sequence of the dynamic topological hierarchy reach a consistent length, the padding operation is defined as follows:

[0060] (17);

[0061] in and Represents the dynamic topological hierarchical structure of text and image modalities, Represents the concatenated text-image concatenation vector Dynamic topology hierarchy The text features after the padding operation, , Represent the text features of the dynamic topological hierarchy of the 1st, 2nd, ..., Mth text-image connection vectors, respectively. Represents the concatenated image-text concatenation vector Dynamic topology hierarchy The image features after padding operation are: , Represent the image features of the dynamic topological hierarchy of the 1st, 2nd, ..., Mth image-text connection vectors, is a unified variable name, It is a function to calculate the length of feature sequence.

[0062] The high-dimensional expansion coding method is used to fuse and expand the multimodal features of images and texts in a high-dimensional manner to improve the robustness and generalization ability of the classification model. and Convert to polynomial form. The specific process is shown in formula (18)-formula (19):

[0063] (18);

[0064] (19);

[0065] in The polynomial expansion of the dynamic topological hierarchy representing the i-th text feature, ,in The polynomial expansions of the 0th, 1st, …, kth coefficients of the dynamic topological hierarchy of the i-th text feature, v, v 2 ,…,v m-1 The expanded variables of the 1st, 2nd, ..., m-1th positions of the dynamic topological hierarchical polynomial expansion of the text features are used to capture the characteristic information of the dynamic topological hierarchical structure. The coefficient b 0 ,b 1 ,…,b m-1 Respectively The complexity of text feature encoding of the 0th, 1st, ..., m-1th order polynomial in the dynamic topological hierarchy; Represents the polynomial expansion of the dynamic topological hierarchy of the j-th image feature, ,in The polynomial expansions of the 0th, 1st, …, lth coefficients of the dynamic topological hierarchy of the jth image feature, v, v 2 ,…,v n-1 The dynamic topological hierarchical polynomial expansions of the text features represent the expanded variables of the 1st, 2nd, ..., n-1th positions, and the coefficients c 0 ,c 1 ,…,c n-1 Respectively The dynamic topological hierarchy in the above example expands the weights of the 0th, 1st, ..., n-1th order polynomials, where n represents the polynomial order of the image feature, also known as The number of items controls the complexity of image feature encoding.

[0066] The polynomial forms of the acquired text features and image features are respectively used to generate corresponding extended polynomials, which are constructed using coefficients in a finite field to ensure the robustness and error correction capability of the encoding, as shown in formula (20)-formula (21):

[0067] (20);

[0068] (twenty one);

[0069] in represents the generating polynomial of the text feature polynomial expansion, nm represents the order, v,v 2 ,…,v n -m They represent the expansion variables of the 1st, 2nd, …, nmth positions of the generator polynomial of the text feature polynomial expansion respectively; represents the generating polynomial used for the polynomial expansion of image features, and o represents the generating polynomial The maximum order of is used to generate redundant symbols to ensure the robustness of image feature encoding. on represents its order, v,v 2 ,…,v o-n Respectively represent the expansion variables of the 1st, 2nd, …, onth positions of the generating polynomial of the image feature polynomial expansion; coefficient g 0 ,g 1 ,…g n-m and h 0 ,h 1 ,…h o-n is the value defined in the finite field in which the polynomial structure is constructed, where g 0 ,g 1 ,…,g n-m Respectively represent the weights of the text generating polynomials of order 0, 1, …, nm, h 0 ,h 1 ,…,h o-n Respectively represent the weights of the image generating polynomial of order 0, 1,…, on.

[0070] Through the above-mentioned expansion polynomial representation operation, the dynamic topological hierarchical polynomial expansion of the text features is obtained respectively. The corresponding generating polynomial , Dynamic topological hierarchical polynomial expansion of image features The corresponding generating polynomial , a high-dimensional extended coding method is adopted to perform a remainder operation on the polynomial and the generating polynomial to generate redundant information. By expanding the text and image features into a polynomial form and adding redundant symbols, the correlation between the multimodal fusion features is ensured to be more robust. The specific remainder operation is shown in formula (22) and formula (23):

[0071] (twenty two);

[0072] (twenty three);

[0073] in Dynamic topological hierarchical polynomial expansion for text features Divide by the generator polynomial of business, Represents the remainder after division, ,in Respectively represent the remainder of the 0th, 1st, ..., kth coefficient of the dynamic topological hierarchy of the i-th text feature; Dynamic topological hierarchical polynomial expansion for representing image features Divide by the generator polynomial of business, Represents the remainder after division. ,in Respectively represent the remainder of the 0th, 1st, ..., lth coefficient of the dynamic topological hierarchy of the jth image feature. These remainders and It represents the redundant symbols after encoding, which enhances the robustness and error correction capability of the encoding features.

[0074] In order to obtain high-dimensional extended coding features, high-dimensional expansion is performed by combining the original polynomial f and the remainder (redundant information) R of the text and image modalities, as shown in formula (24)-formula (25):

[0075] (twenty four);

[0076] (25);

[0077] in High-dimensional expanded encoding features representing text features, The high-dimensional extended encoding features representing image features, including the original polynomial and remainder information, will be further used in the classification task to ensure that the complementary information between different modal features can be fused.

[0078] Step 5: Input the high-dimensional extended encoding features of image features and text features into the sentiment classifier to obtain the three types of sentiment classification results: positive, neutral, and negative. and text features (j=1,2,…,M;i=1,2,…,M), the corresponding high-dimensional extended coding features are and , the high-dimensional extended encoding feature vectors of all image features and text features are concatenated and input into the Softmax classifier to obtain the sentiment prediction probability of each image-text pair, as shown in formula (26):

[0079] (26);

[0080] in Represents the concatenation operation, which combines the high-dimensional encoded feature vectors of each pair of images and texts into an input vector and passes it to the classifier, and finally outputs the sentiment classification probability result ,in Indicates the probability of sentiment classification of the 1st, 2nd, ..., Mth pairs of images and texts, covering three types of sentiment results: positive, neutral, and negative. In order to predict and compare the performance of the model constructed by the present invention, the currently popular multimodal model and Twitter and MVSA datasets were selected for performance evaluation, and the accuracy and weighted-F1 value were selected as model evaluation indicators. The specific results of the Twitter dataset comparison experiment are shown in Table 2:

[0081] Table 2

[0082]

[0083] It can be seen from the data in Table 2 that the multimodal information fusion dynamic topological hierarchical sentiment analysis method proposed in the present invention is superior to other models proposed in recent years, and has achieved the best results in accuracy and weighted-F1 value. Compared with the best VauLT method, the accuracy and weighted-F1 value of the model proposed in the present invention on the Twitter15 dataset are improved by 0.35% and 5.24%, respectively. On the Twitter17 dataset, the accuracy and weighted-F1 value are improved by 1.89% and 3.37%, respectively. This shows that the model of the present invention effectively improves the accuracy and stability of sentiment classification by optimizing the multimodal feature fusion method.

[0084] The specific results of the MVSA dataset comparison experiment are shown in Table 3:

[0085] Table 3

[0086]

[0087] It can be seen from the data in Table 3 that the model proposed in the present invention performs better than the model methods proposed in recent years in the study of multimodal fusion, and has achieved the best results in both accuracy and weighted-F1 value. Specifically, for the MITN method with the best effect, the accuracy and weighted-F1 value of the present invention on the MVSA-Single dataset were improved by 0.50% and 1.12% respectively; the accuracy and weighted-F1 value on the MVSA-Multiple dataset were improved by 1.21% and 0.91% respectively. These results show that the model of the present invention has achieved a significant improvement in the accuracy and stability of sentiment analysis by optimizing the multimodal feature fusion strategy.

[0088] To further verify the effectiveness of each module of the model proposed in this invention, this invention conducts ablation experiments on the MVSA dataset. Three ablation experiments are designed for the three core strategies proposed in this invention to evaluate the contribution of each strategy to multimodal information fusion. The details are shown in Table 4.

[0089] Table 4

[0090]

[0091] At the single modality level, we designed a dataset prediction experiment using only a single text modality and a single image modality. Specifically, on the one hand, sentiment analysis prediction is performed based only on the text modality on the preprocessed MVSA-Single and MVSA-Multiple datasets; on the other hand, sentiment analysis prediction is performed based only on the image modality. Combining the above strategies 1-3, we conducted an ablation experiment on the MVSA dataset. The experimental results show that removing any strategy will lead to a decrease in model performance. The specific results are shown in Table 5.

[0092] Table 5

[0093]

[0094] As shown in Table 5, the model that only applies Strategy 1 has an accuracy of 0.6785 and a weighted-F1 value of 0.6797 on the MVSA-Single dataset; on the MVSA-Multiple dataset, the accuracy is 0.6501 and the weighted-F1 value is 0.6574, indicating that the method of simply relying on multimodal feature quantification is not enough to capture the deep associations between multimodal data. After Strategy 2 removes the dynamic topological hierarchy, the weighted-F1 values ​​on the MVSA-Single and MVSA-Multiple datasets are 0.7056 and 0.7054, respectively, indicating the important role of the dynamic topological hierarchy of the present invention in enhancing the flow of information between modalities. In addition, the weighted-F1 value of the model that removes Strategy 3 on the two datasets is also significantly reduced, reflecting the necessity of high-dimensional space feature encoding strategies in multimodal fusion and classification tasks. With all modules working together, the multimodal fusion model proposed in this paper achieved an accuracy of 0.7654 and a weighted-F1 value of 0.7643 on the MVSA-Single dataset; and an accuracy of 0.7536 and a weighted-F1 value of 0.7488 on the MVSA-Multiple dataset, which is significantly better than other strategy combinations. Figure 4 The confusion matrix effect diagram shows that the model classification effect shows that the model proposed in the present invention has a high classification accuracy in positive and negative emotion recognition, especially in the negative emotion category, which shows the superiority of the multimodal sentiment analysis method in the recognition of specific emotion polarity. This shows that the dynamic topological hierarchical structure sentiment analysis method of multimodal information fusion proposed in the present invention can effectively integrate image and text features in the sentiment analysis task, achieve more accurate sentiment classification prediction, and help improve the performance and robustness of the multimodal sentiment analysis system.

[0095] This embodiment provides a computer-readable storage medium on which computer instructions are stored. When the computer instructions are executed by a processor, the dynamic topological hierarchical structure sentiment analysis method for multimodal information fusion is implemented.

[0096] The invention described above only expresses the implementation methods of the embodiments of the present invention, and cannot be understood as limiting the scope of the invention patent, nor does it impose any form of limitation on the structure of the embodiments of the present invention. It should be pointed out that for ordinary technicians in this field, several changes and improvements can be made without departing from the concept of the embodiments of the present invention, which all belong to the protection scope of the embodiments of the present invention.

Claims

1. A dynamic topological hierarchical structure sentiment analysis method based on multimodal information fusion, characterized in that: Here are the steps: Step 1: Preprocess the user opinion image and text dataset of the social media platform to obtain the image and text dataset for multimodal sentiment analysis; Step 2: Use the residual network model to extract image features from the image and text dataset; use the bidirectional encoder representation model to extract text features from the image and text dataset; Step 3: Align image features and text features through a multi-head attention mechanism, extract the interactive information between modalities, and obtain the aligned image-text feature vector; Step 4: Map the aligned image and text feature vectors to a dynamic topological hierarchy, construct a multi-level non-Euclidean space to capture the complex correlation between different modalities, and use a high-dimensional extended coding method to fuse and expand the multimodal features of the image and text to obtain high-dimensional extended coding features of the image and text features; Step 5: Input the high-dimensional extended encoding features of image features and text features into the sentiment classifier to obtain the three types of sentiment classification results: positive, neutral, and negative; In step 4, the aligned image-text feature vector is mapped to a dynamic topological hierarchy to construct a multi-level non-Euclidean space to capture the complex association relationship between different modalities. The process is as follows: Given a text-image connection vector ,in Respectively represent the 1st, 2nd, …, Mth text-image connection vector, image-text connection vector ,in Represent the 1st, 2nd, …, Mth image-text connection vectors respectively. and Each element in the prime number sequence is obtained by formula (13)-formula (16). The following maps to dynamic topology hierarchy features: (13); (14); (15); (16); in Represents the dynamic topological hierarchical structure features of the i-th element in the text-image connection vector under the specified prime number p, Represents the dynamic topological hierarchical structure features of the jth element in the image-text connection vector under the specified prime number p, and They correspond to the coefficients of different powers of the specified prime number p, where Represents the text-image connection vector The kth coefficient of the i-th element in the dynamic topological hierarchy feature, Represents the image-text connection vector The jth element in the dynamic topological hierarchy feature Bit coefficient, Represents the weight corresponding to the expansion of the k-th coefficient of the i-th element in the text-image connection vector under the specified prime number p, represents the weight corresponding to the expansion of the lth coefficient of the jth element in the image-text vector under the specified prime number p. The two are used to represent the decomposition expression of text features and image features in the multimodal dynamic topological hierarchy. In order to make the feature sequence of the dynamic topological hierarchy reach a consistent length, the padding operation is defined as follows: (17); in and Represents the dynamic topological hierarchical structure of text and image modalities, Represents the concatenated text-image concatenation vector Dynamic topology hierarchy The text features after the padding operation, , The text features representing the dynamic topological hierarchy of the 1st, 2nd, ..., Mth text-image connection vectors, Represents the concatenated image-text concatenation vector Dynamic topology hierarchy The image features after padding operation are: , The image features representing the dynamic topological hierarchy of the 1st, 2nd, ..., Mth image-text connection vectors, is a unified variable name, It is a function to calculate the length of feature sequence.

2. The dynamic topological hierarchical structure sentiment analysis method of multimodal information fusion according to claim 1 is characterized in that: In step 2, the image data is enhanced by combining the random data enhancement method, and then the residual network model is used to extract image features in the image and text dataset.

3. The dynamic topological hierarchical structure sentiment analysis method of multimodal information fusion according to claim 1 is characterized in that: In step 3, a multi-head attention mechanism is applied to each pair of image features and text features, and the query vector, key vector, and value vector of the image and text are obtained through linear transformation; the image features and text features are aligned through the multi-head attention mechanism to obtain the attention representation of the image features on the text features. And the attention representation of text features on image features , connect the text features with the image features’ attention representation A1 to obtain the text-image connection vector containing the text features and the image’s attention information to the text , connect the image features with the attention representation of the text features on the image features, and obtain the image-text connection vector containing the image features and the text's attention information on the image .

4. The method for dynamic topological hierarchical structure sentiment analysis based on multimodal information fusion according to claim 1, characterized in that: In step 4, the multimodal features of the image and text are fused and expanded in high dimension using the high dimensional expansion coding method to obtain the high dimensional expansion coding features of the image features and text features. The process is as follows: and Convert to polynomial form. The specific process is shown in formula (18)-formula (19): (18); (19); in The polynomial expansion of the dynamic topological hierarchy representing the i-th text feature, ,in The polynomial expansions of the 0th, 1st, …, kth coefficients of the dynamic topological hierarchy of the i-th text feature, v, v 2 ,…,v m-1 The expanded variables of the 1st, 2nd, ..., m-1th positions of the dynamic topological hierarchical polynomial expansion of the text features are used to capture the characteristic information of the dynamic topological hierarchical structure. The coefficients b0, b1, ... b m-1 express The complexity of text feature encoding of the 0th, 1st, ..., m-1th order polynomial in the dynamic topological hierarchy; Represents the polynomial expansion of the dynamic topological hierarchy of the j-th image feature, ,in The polynomial expansions of the 0th, 1st, …, lth coefficients of the dynamic topological hierarchy of the jth image feature, v, v 2 ,…,v n-1 The dynamic topological hierarchical polynomial expansions of the text features represent the expanded variables of the 1st, 2nd, ..., n-1th positions, and the coefficients c0, c1, ..., c n-1 Respectively The dynamic topological hierarchy in the above example expands the weights of the 0th, 1st, ..., n-1th order polynomials, where n represents the polynomial order of the image feature, also known as The number of items controls the complexity of image feature encoding; The polynomial forms of the acquired text features and image features are respectively used to generate corresponding extended polynomials, which are constructed using coefficients in a finite field to ensure the robustness and error correction capability of the encoding, as shown in formula (20)-formula (21): (20); (21); in represents the generating polynomial of the text feature polynomial expansion, nm represents the order, v,v 2 ,…,v n-m They represent the expansion variables of the 1st, 2nd, …, nmth positions of the generator polynomial of the text feature polynomial expansion respectively; represents the generating polynomial used for the polynomial expansion of image features, and o represents the generating polynomial The maximum order of is used to generate redundant symbols to ensure the robustness of image feature encoding. on represents its order, v,v 2 ,…,v o-n The coefficients g0, g1, …, g represent the expansion variables of the 1st, 2nd, …, onth positions of the generator polynomial of the image feature polynomial expansion respectively; n-m and h0,h1,…,h o-n are the values ​​defined in the finite field used to construct the polynomial structure, where g0, g1, …, g n-m Respectively represent the weights of the text generating polynomials of order 0, 1, ..., nm, h0, h1, ..., h o-n Respectively represent the weights of the image generating polynomials of order 0, 1, …, on; By expanding the polynomial representation operation, the dynamic topological hierarchical polynomial expansion of the text features is obtained. The corresponding generating polynomial , Dynamic topological hierarchical polynomial expansion of image features The corresponding generating polynomial , a high-dimensional extended coding method is adopted to perform a remainder operation on the polynomial and the generating polynomial to generate redundant information. By expanding the text and image features into a polynomial form and adding redundant symbols, the correlation between the multimodal fusion features is ensured to be more robust. The specific remainder operation is shown in formula (22) and formula (23): (22); (23); in Dynamic topological hierarchical polynomial expansion for text features Divide by the generator polynomial of business, Represents the remainder after division, ,in Respectively represent the remainder of the 0th, 1st, ..., kth coefficient of the dynamic topological hierarchy of the i-th text feature; Dynamic topological hierarchical polynomial expansion for representing image features Divide by the generator polynomial of business, Represents the remainder after division. ,in Respectively represent the remainder of the 0th, 1st, ..., lth coefficient of the dynamic topological hierarchy of the jth image feature; In order to obtain high-dimensional extended coding features, high-dimensional expansion is performed by combining the original polynomial f and remainder R of the text and image modalities, as shown in formula (24)-formula (25): (24); (25); in High-dimensional expanded encoding features representing text features, High-dimensional expanded encoding features representing image features.

5. The method for dynamic topological hierarchical structure sentiment analysis based on multimodal information fusion according to claim 1, characterized in that: In step 5, the high-dimensional extended encoding feature vectors of all image features and text features are concatenated and input into the Softmax classifier to obtain the sentiment prediction probability of each image-text pair.

6. The method for dynamic topological hierarchical structure sentiment analysis based on multimodal information fusion according to claim 2, characterized in that: In step 2, the original image data is known to be ,in Respectively represent the image data in the 1st, 2nd, …, Mth image and text data set, perform random data enhancement operations on each image data, and use the grid search method to perform incremental simple grid search. The calculation process is shown in formula (1): （1）； in Represents the image The result after the jth data enhancement, i is the image number, represents the data enhancement function, represents a set of enhanced parameters, where represents the 1st, 2nd, ..., Nth enhancement parameters, Represents the jth enhancement parameter, and finally obtains the image A collection of enhanced images after data enhancement under different enhancement parameters ,in Representing images The data enhancement results under the 1st, 2nd, ..., Nth enhancement parameters; finally, the enhanced image set for each image is merged through formula (2) ; （2）； in represents the augmented image dataset, Represents the set union operation, which is the enhanced image set of the 1st, 2nd, .., Mth images. Merge to get a complete enhanced image dataset ; Get the enhanced image dataset After that, It represents the N enhanced images obtained by different enhancement parameters for the i-th image data. The residual network model is used to extract features for each enhanced image to obtain a high-dimensional response map representation. The calculation formula is shown in formula (3): （3）； in Represents the i-th image The jth enhanced image The feature response map extracted by the residual network model, is the feature extraction function of the residual network model; each feature response map Converted to sequence form, it is known that ,in Indicates the number of images processed each time, represents the image dimension, represents the dimension of the intermediate latent space, represents the real number domain; finally, the feature encoding tensor converted into sequence form is obtained, as shown in formula (4): （4）； in, denote the 1st, 2nd, …, feature vectors, Represents the shape Finally, the feature coding tensors of all enhanced images are combined to obtain the image features of the entire enhanced image dataset, as shown in formula (5): （5）; in Represents the image features of the entire enhanced image dataset, which is a set of image features extracted after data enhancement and residual network model. ,in Respectively represent the image features of the 1st, 2nd, …, Mth image data.

7. The method for dynamic topological hierarchical structure sentiment analysis based on multimodal information fusion according to claim 6, characterized in that: In step 2, for the input text data ,in Represent the text data in the 1st, 2nd, ..., Mth image and text dataset respectively, and use the bidirectional encoder representation model to extract the text hidden state features of the last layer of the text data ,in Respectively represent the hidden state features of the 1st, 2nd, ..., Mth texts, Indicates the amount of text processed each time. Indicates the length of the text sequence, represents the output hidden dimension of the bidirectional encoder representation model. The linear layer and ReLu activation function are used to transform the text hidden state feature H, as shown in formula (6): (6); (7); in Represents the transformed hidden state tensor, with a shape size of , Represents the weight matrix of the linear layer, with a size of , maps H to the target hidden dimension d t , b1 represents the bias vector of the linear layer, with a size of d t ; Finally, we get the text features in the image and text dataset ,in Represent the text features of the 1st, 2nd, …, Mth text data respectively.

8. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the computer instructions are executed by the processor, the dynamic topological hierarchical structure sentiment analysis method of multimodal information fusion according to any one of claims 1 to 7 is implemented.

Images