Chinese sentiment classification method based on multi-source bilingual and cross-modal feature fusion and enhancement

Abstract

The application discloses a Chinese sentiment classification method based on multi-source bilingual and cross-modal feature fusion and enhancement, and belongs to the technical field of sentiment computing, and specifically comprises the following steps: obtaining a Chinese text to be classified and translating the Chinese text; obtaining an image Chinese text of an image to be classified and translating the image Chinese text; constructing a bilingual feature extraction model, and extracting bilingual fusion features of the Chinese text to be classified and the image Chinese text by using the bilingual feature extraction model; extracting a global visual sentiment semantic representation of the image to be classified; constructing a sentiment knowledge base, and searching and processing the bilingual fusion features of the Chinese text to be classified to obtain an external sentiment representation; enhancing the bilingual fusion features of the Chinese text to be classified through cross-modal attention fusion of the image to be classified, and predicting a sentiment classification result based on the obtained cross-modal sentiment representation. The application fully excavates the features of the Chinese-English mixed text and the image published by a user, effectively captures strong sentiment expression words of different sources, significantly improves the accuracy of the Chinese sentiment classification task, and takes into account the model calculation efficiency.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of natural language processing, computer vision and affective computing, and particularly relates to a Chinese sentiment classification method based on multi-source bilingual and cross-modal feature fusion and enhancement. BACKGROUND

[0002] With the rapid development of various social media platforms, the massive amount of user-generated text and image data provides valuable resources for understanding public sentiment. However, current sentiment classification methods face many challenges when dealing with mixed language texts, especially in the case of Chinese texts frequently interspersed with English words and mixed expressions. In addition, current sentiment classification methods do not take into account image data with obvious emotional information. These images with emotional information often contain text with concise and explicit emotional characteristics. Nowadays, the text in images often appears in a mixed language context, expressing strong emotions. For example, the mixed language text "If the hair is not to your liking, then the whole day will be low in mood", and the accompanying image as shown, these mixed language words often contain rich emotional information, such as the English word "low" with a clear negative emotional tendency, and the accompanying image also contains the emotional word "emo" and the corresponding visual elements of emotional characteristics. Figure 1

[0003] Sentiment classification technology has evolved from early machine learning methods such as support vector machines (SVM), naive Bayes, and classification regression trees (CRT) to deep learning methods. The emergence of Transformer networks has greatly promoted the progress of natural language processing. Pre-trained language models based on Transformer, such as BERT (Bidirectional Encoder Representations from Transformers) and its variants such as RoBERTa, XLNet, Nezha, ELECTRA, ERNIE, have become the mainstream models for sentiment analysis tasks. However, when faced with mixed language texts, existing models often fail to fully utilize the emotional information contained therein.

[0004] Current multi-modal sentiment classification methods attempt to integrate text, speech, and visual signals, but they do not adequately consider the increasingly common language mixing phenomenon in the text channel, and fail to effectively utilize the multi-language modeling capabilities of pre-trained models. In addition, these multi-modal methods only consider the emotional visual elements expressed by the image itself, often ignoring the text information contained in the image. This text information in the image often contains a lot of explicit and concise emotional information. Furthermore, with the development of the Internet, the mixed language scenario in images has also gradually become popular. ​

[0005] Therefore, there is an urgent need for an effective sentiment classification method for multi-source mixed language text. This method should make full use of the multilingual representation capability of pre-trained language models, extract and fuse bilingual features, thereby significantly improving the performance of sentiment analysis for Chinese-English mixed text. In addition, this method can also extract the corresponding significant emotional features from the image information carried by the user, as well as the mixed language text information in the image. The emotional elements in the image are combined with the emotional elements in the text, and the significant emotional characteristics from different languages are fused with the significant emotional characteristics from different modalities, so as to better meet the actual application requirements in the current Chinese social media context. SUMMARY

[0006] In view of the problems existing in the prior art, the present application provides a Chinese sentiment classification method based on multi-source bilingual and cross-modal feature fusion and enhancement, which can significantly improve the accuracy of Chinese sentiment classification task, and has strong practical value.

[0007] In order to achieve the above purpose, the technical method adopted by the present application is as follows:

[0008] The Chinese sentiment classification method based on multi-source bilingual and cross-modal feature fusion and enhancement comprises the following steps:

[0009] Step 1, obtaining a to-be-classified Chinese text and a to-be-classified image matched therewith, and translating the to-be-classified Chinese text into a to-be-classified English text with semantic alignment; the to-be-classified Chinese text is a Chinese-English mixed text containing a small amount of English;

[0010] Step 2, constructing and training a bilingual feature extraction model, including a pre-trained language model, a Chinese lightweight network module, an English lightweight network module and a bilingual cross-attention module;

[0011] Step 3, inputting the to-be-classified Chinese text and the to-be-classified English text into the trained bilingual feature extraction model to extract corresponding Chinese features and English features , and bilingual fusion features ;

[0012] Step 4, using a computer vision model to detect the text in the to-be-classified image and extract the text region contained in the to-be-classified image; then using a text recognition module (OCR) to recognize the text region as a readable string as an image Chinese text, and translating it into an image English text with semantic alignment;

[0013] Step 5, inputting the image Chinese text and the image English text into the trained bilingual feature extraction model to extract corresponding image Chinese features and image English features and bilingual fusion features of images ;

[0014] Step 5, based on the multilingual corpus containing multiple text segments and known emotion classification, the text segment in Chinese is translated into a semantically aligned text segment in English, the corresponding Chinese features and English features of the text segment are extracted by using the trained bilingual feature extraction model, and an emotion knowledge base is constructed;

[0015] Step 6, nearest neighbor search is performed on in the emotion knowledge base to obtain multiple most relevant semantic representations, external emotion representations are obtained by weighting, weighted summation and averaging thereof; ;

[0016] Step 7, a pre-trained convolutional neural network is used to extract global features of the image to be classified to obtain a global image representation; a sentiment feature extraction module is constructed and trained, and the sentiment feature extraction module is used to extract sentiment-related visual features in the global image representation as a global visual sentiment semantic representation ;

[0017] Step 8, a cross-modal attention fusion module is constructed and trained, and the trained cross-modal attention fusion module is used to perform cross-modal attention fusion on 、 and , obtain a cross-modal fusion representation, and combine the cross-modal fusion representation with by using a residual mechanism to obtain a cross-modal sentiment representation ;

[0018] Step 9, an emotion classifier is used to predict the emotion category of to obtain the emotion classification result of the Chinese text to be classified.

[0019] Further, the bilingual feature extraction model specifically comprises:

[0020] The Chinese text and the English text are input into the bilingual feature extraction model, a pre-trained language model extracts hidden layer representations of the Chinese text and the English text; a Chinese lightweight network module and an English lightweight network module are added to each linear layer of the pre-trained language model; the Chinese text is processed by the Chinese lightweight network module and combined with the hidden layer representation of the Chinese text to obtain Chinese features; the English text is processed by the English lightweight network module and combined with the hidden layer representation of the English text to obtain English features; the Chinese features and the English features are fused by a bilingual cross-attention module to output bilingual fusion features.

[0021] Further, the Chinese lightweight network module and the English lightweight network module are designed based on Low Rank Adaptation (LoRA) technology.

[0022] Further, the bilingual feature extraction model is trained based on an emotion analysis evaluation database, and during the process of training the pre-trained language model in the bilingual feature extraction model, only the parameters in the embedding layer are trained, and other parameters of the pre-trained language model are frozen.

[0023] Further, the pre-trained language model is BERT, RoBERTa or XLNet.

[0024] Further, the bilingual fusion feature is extracted in step 3 The specific process is as follows:

[0025] The Chinese feature and the English feature are respectively linearly transformed to obtain corresponding Chinese query matrix (Q) , Chinese key matrix (K) , Chinese value matrix (V) , English query matrix , English key matrix and English value matrix .

[0026] The similarity of and is calculated, and the attention weight matrix of the similarity is obtained by using the Softmax function, and then is weighted and added, and then is combined, and after normalization, the feature enhancement from English to Chinese is obtained.

[0027] The similarity of and is calculated, and the attention weight matrix of the similarity is obtained by using the Softmax function, and then is weighted and added, and then is combined, and after normalization, the feature enhancement from Chinese to English is obtained.

[0028] Further information interaction and semantic integration are performed on and by using a feed-forward neural network (FFN) to obtain cross-hidden layer representation and cross-hidden layer representation :

[0029] ;

[0030] ;

[0031] wherein, denotes a normalization process; denotes a feedforward neural network process;

[0032] Further, the bilingual fusion feature is obtained by summing and .

[0033] Further, the bilingual fusion feature is obtained by summing the image Chinese feature and the image English feature

[0034] ;

[0035] The similarity of and is calculated, and the attention weight matrix of the similarity is obtained by using a Softmax function, and then the weighted is added, and then combined with , and after normalization, the feature enhancement from English to Chinese is obtained ;

[0036] The similarity of and is calculated, and the attention weight matrix of the similarity is obtained by using a Softmax function, and then the weighted is added, and then combined with , and after normalization, the feature enhancement from Chinese to English is obtained ;

[0037] Further information interaction and semantic integration are performed on and by using a feedforward neural network, and the cross image Chinese hidden layer representation and the cross image English hidden layer representation are obtained:

[0038] ;

[0039] ​​​​​​​​ ;

[0040] The word vectors contained in the are averaged, and the word vectors contained in the are averaged, and the two averages are summed to obtain image bilingual fusion features .

[0041] Further, in step 6, a scoring mechanism is used to assign weights to the most relevant semantic representations.

[0042] Further, in step 7, the sentiment feature extraction module is implemented based on a fully connected layer, and is trained based on an emotion analysis evaluation database, specifically taking the global image representation output by the pre-trained convolutional neural network as input, and training the parameters of the sentiment feature extraction module.

[0043] Further, in step 8, the training process of the cross-modal attention fusion module is as follows: based on the emotion analysis evaluation database, the output by the trained bilingual feature extraction model, the output by the trained sentiment feature extraction module, and the obtained in step 6 are taken as inputs of the cross-modal attention fusion module, the weights of the cross-modal attention fusion module are trained, and the parameters of the trained bilingual feature extraction model and the trained sentiment feature extraction module are kept unchanged during the training process.

[0044] The sentiment feature extraction module is implemented based on a fully connected layer.

[0045] Compared with the prior art, the present application has the following advantages:

[0046] 1. The Chinese sentiment classification method based on multi-source bilingual and cross-modal feature fusion and enhancement proposed in the present application effectively captures strong emotional expression words of different sources (including text content and text information in images) by fully mining the features of mixed Chinese-English text published by users and the cooperating images (i.e. accompanying images), especially the bilingual significant emotional text information contained in the images, thereby significantly improving the accuracy of the Chinese sentiment classification task.

[0047] 2. The present application constructs a sentiment knowledge base to find historical information for text with not obvious emotional expression, to perform feature enhancement, and to further improve the effectiveness of emotional information extraction in bilingual mixed scenarios.

[0048] 3、The Chinese emotion classification method provided by the present application is particularly prominent when processing social media text data containing mixed languages and English mixed with Chinese. Compared with existing methods, the present application not only fuses bilingual features, but also adopts a lightweight model fine-tuning strategy, which significantly improves the emotion classification performance while taking into account the model calculation efficiency, has strong practical value, and is expected to be widely used in intelligent customer service, personalized recommendation and other fields, and provides strong support for the landing of artificial intelligence technology in practical scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0050] Figure 1 is a picture with emotional words "emo";

[0051] Figure 2 is a flowchart of the Chinese emotion classification method based on multi-source bilingual and cross-modal feature fusion and enhancement proposed in embodiment 1;

[0052] Figure 3 is a schematic diagram of the principle of the bilingual feature extraction model proposed in embodiment 1;

[0053] Figure 4 is a flowchart of the lightweight model fine-tuning strategy proposed in embodiment 1. DETAILED DESCRIPTION

[0054] In order to further understand the present application, the preferred embodiments of the present application will be described below in conjunction with the embodiments, but it should be understood that these descriptions are only to further illustrate the features and advantages of the present application, and are not limitations of the claims.

[0055] Embodiment 1

[0056] The present embodiment proposes a Chinese emotion classification method based on multi-source bilingual and cross-modal feature fusion and enhancement, and the flowchart is as shown in Figure 2 The steps include:

[0057] Step 1, obtaining the Chinese text to be classified and the image to be classified, and using a machine translation model to translate the Chinese text to be classified into a semantic-aligned English text to be classified;

[0058] The machine translation model can adopt an existing online translation tool, such as Google Translate, Bing Microsoft Translator, etc., or a high-quality neural machine translation model trained by the user, such as Transformer, BART, etc.

[0059] Example: The Chinese text to be classified is "If the hair does not follow the heart, the emotion will be very low", and the translated English text to be classified is "If your hair does not follow your heart, your emotion will be very low".

[0060] Step 2, build a bilingual feature extraction model, as shown in Figure 3 The pre-trained language model is BERT.

[0061] The pre-trained language model is BERT.

[0062] The Chinese lightweight network module and the English lightweight network module are designed based on low-rank adaptation technology and added to each linear layer of the pre-trained language model. Specifically, a low-rank matrix is introduced to each linear layer , which is added to the original weight matrix to obtain the adapted weight matrix .

[0063] The Chinese text and the English text are input into the bilingual feature extraction model, the pre-trained language model extracts the hidden layer representation of the Chinese text and the hidden layer representation of the English text; the Chinese text is processed by the Chinese lightweight network module and combined with the hidden layer representation of the Chinese text to obtain Chinese features; the English text is processed by the English lightweight network module and combined with the hidden layer representation of the English text to obtain English features; the Chinese features and the English features are fused by the bilingual cross-attention module to output bilingual fusion features; wherein the Chinese features and the English features each include a corresponding global sentiment representation and a local word-level sentiment representation; in BERT, for example, the first [CLS] or the 0th word vector can be regarded as the global sentiment representation, and the subsequent word vectors are regarded as the local word-level sentiment representation; in the training process of the bilingual feature extraction model, a lightweight model fine-tuning strategy is adopted, such as Figure 4As shown, the original parameters of each layer in the fixed pre-training language model are unchanged, and only the low-rank matrices of the various embedding layers (including word type embedding encoding, position embedding encoding, and word embedding encoding), the Chinese lightweight network module, and the English lightweight network module of the pre-training language model are trained , and the bilingual cross-attention module; only the low-rank matrices need to be learned , while keeping the original weight matrices unchanged, thereby greatly reducing the amount of parameters that need to be updated. In this way, the model can be efficiently fine-tuned on downstream tasks without significantly increasing computational overhead. By training only the embedding matrices in the embedding layers, the word representations of different languages can be aligned in the semantic space, making them more suitable for the target task. Freezing the other parameters of the pre-training language model can greatly reduce the number of fine-tuning parameters while still effectively adapting to downstream tasks.

[0064] By using a lightweight model fine-tuning strategy, the computational overhead is greatly reduced while ensuring fine-tuning effectiveness, making the bilingual feature extraction model more efficient in adapting to sentiment classification tasks, significantly reducing training time and resource consumption, and having strong practical value.

[0065] Step 3, input the Chinese text to be classified and the English text to be classified into the trained bilingual feature extraction model to extract the corresponding Chinese features and English features , and bilingual fusion features .

[0066] The specific process of extracting bilingual fusion features is as follows:

[0067] Linearly transform the Chinese features and English features to obtain the corresponding Chinese query matrix , Chinese key matrix , Chinese value matrix , English query matrix , English key matrix , and English value matrix :

[0068] ;

[0069] ;

[0070] ;

[0071] ;

[0072] ;

[0073] ;

[0074] wherein, , and represent linear transformation matrix of Chinese; , and represent linear transformation matrix of English; through linear transformation, original features are mapped to the same feature space, preparing for subsequent attention calculation;

[0075] the similarity between and is calculated, and the attention weight matrix of the similarity is obtained by using a Softmax function, and then is added to the weight, and then is combined, and after normalization, the feature enhancement from English to Chinese is obtained:

[0076] ;

[0077] wherein, the superscript "T" represents transposition; d represents the length of the word vector; represents a Softmax function; represents normalization processing;

[0078] the similarity between and is calculated, and the attention weight matrix of the similarity is obtained by using a Softmax function, and then is added to the weight, and then is combined, and after normalization, the feature enhancement from Chinese to English is obtained:

[0079] ;

[0080] by using a feed-forward neural network (FFN), further information interaction and semantic integration are performed on and , and cross-after Chinese hidden layer representation and cross-after English hidden layer representation are obtained:

[0081] ;

[0082] ;

[0083] wherein, represents feed-forward neural network processing;

[0084] Further, the bilingual feature representation is obtained by summing up and ; ; including the corresponding global sentiment representation and the local word-level sentiment representation.

[0085] Step 4, using a computer vision model to perform text detection on the image to be classified and extract the text region contained in the image to be classified; then using a text recognition module to recognize the text region as a readable string as image Chinese text and translate it into semantically aligned image English text; wherein the embodiment specifically adopts DBNet (Differentiable Binarization Network) as the computer vision model and CRNN (Convolutional Recurrent Neural Network) as the text recognition module;

[0086] The image Chinese text and the image English text are input into the trained bilingual feature extraction model to extract the corresponding image Chinese feature and the image English feature , and the image bilingual fusion feature ;

[0087] The specific process of extracting the image bilingual fusion feature is as follows:

[0088] The image Chinese feature and the image English feature are respectively linearly transformed to obtain the corresponding image Chinese query matrix , the image Chinese key matrix , the image Chinese value matrix , the image English query matrix , the image English key matrix , and the image English value matrix ;

[0089] The similarity of and is calculated, and the attention weight matrix of the similarity is obtained by using the Softmax function, and then added to , and combined with , and after normalization, the feature enhancement of image English to Chinese is obtained ;

[0090] The similarity of and is calculated, and the attention weight matrix of the similarity is obtained by using the Softmax function, and then added to , and combined with The image Chinese-to-English feature enhancement is obtained in combination and after normalization ;

[0091] The image Chinese-to-English feature enhancement is obtained in combination and after normalization and Further information interaction and semantic integration are performed on and the cross-image English hidden layer representation :

[0092] ;

[0093] ;

[0094] All word vectors contained in are averaged, and all word vectors contained in are averaged, and the sum of the two averages is obtained as the image bilingual fusion feature :

[0095] ;

[0096] In the formula, represents the number of word vectors contained in ; represents the number of word vectors contained in ; represents the i-th word vector in ; represents the i-th word vector in ; and further is only a vector; The bilingual cross-attention module used in this embodiment can fully mine the correlation between Chinese and English features, complement and enhance each other, make the sentiment tendency more explicit, and provide high-quality feature representation for subsequent sentiment classification tasks.

[0097] The bilingual cross-attention module used in this embodiment can fully mine the correlation between Chinese and English features, complement and enhance each other, make the sentiment tendency more explicit, and provide high-quality feature representation for subsequent sentiment classification tasks.

[0098] Step 5: Based on the multilingual corpus containing multiple text segments and known sentiment classification, the Chinese text segment is translated into a semantically aligned English text segment, the trained bilingual feature extraction model is used to extract the corresponding Chinese features and English features of the text segment, and a sentiment knowledge base is constructed;

[0099] Step 6: Nearest neighbor search is performed on in the sentiment knowledge base, specifically by calculating ​A plurality of most relevant semantic representations are obtained according to cosine similarities of all features in the emotion knowledge base, a scoring mechanism is used to assign weights to the most relevant semantic representations, and the scores of the cosine similarities are normalized by using Softmax to obtain the external emotion representation , which is a vector.

[0100] Step 7: Global feature extraction is performed on the image to be classified by using the pre-trained convolutional neural network to obtain a global image representation.

[0101] The emotion feature extraction module is constructed and trained, and in this embodiment, a fully connected layer is used as the emotion feature extraction module, the global image representation output by the pre-trained convolutional neural network is used as the input, and the parameters of the emotion feature extraction module are trained.

[0102] The visual features related to emotions in the global image representation are extracted by using the trained emotion feature extraction module, and the visual features are used as the global visual emotion semantic representation , which is a vector.

[0103] Step 8: A cross-modal attention fusion module is constructed based on a cross-attention mechanism, the output by the trained bilingual feature extraction model, the output by the trained emotion feature extraction module, and the obtained in step 6 are used as the inputs of the cross-modal attention fusion module, the weights of the cross-modal attention fusion module are trained, and the parameters of the trained bilingual feature extraction model and the trained emotion feature extraction module are kept unchanged during the training process.

[0104] The cross-modal attention fusion module is used to perform cross-modal attention fusion on the , and , specifically, the , and are spliced to obtain a matrix , and the query weight , the key weight and the value weight are used to map the to a query matrix , a key matrix and a value matrix :

[0105] ;

[0106] ;

[0107] ;

[0108] By attention operation, update :

[0109] ;

[0110] Updated is spliced by three new vectors , and , that is ;

[0111] Further, the cross-modal fusion representation is calculated:

[0112] ;

[0113] By adaptively assigning the importance of each modal feature, sufficient complement and enhancement can be performed between vision and text, bilingual and external sentiment knowledge.

[0114] Then, the cross-modal fusion representation is combined with by using the residual mechanism, to obtain the unified and enhanced cross-modal sentiment representation ;

[0115] Step 9, using the sentiment classifier to predict the sentiment category of , to obtain the sentiment classification result of the Chinese text to be classified, specifically:

[0116] A linear classification layer is used to map the cross-modal sentiment representation to the probability distribution of the sentiment category :

[0117] ;

[0118] In the formula, denotes the weight matrix of the linear classification layer; denotes the bias vector of the linear classification layer; the Softmax function is used to normalize the output of the linear transformation into a probability form, so as to perform a multi-classification task;

[0119] contains global sentiment category probability and local word-level sentiment category probability, and the final probability vector is obtained by using weighted average, and the final sentiment category is determined according to the element with the highest score in it.

[0120] = 0.8 x + 0.2 x ;

[0121] wherein, denotes the number of local lexical-level sentiment category probabilities contained in the ith local lexical-level sentiment category probability. denotes the ith local lexical-level sentiment category probability.

[0122] To comprehensively evaluate the performance of the Chinese sentiment classification method based on multi-source bilingual and cross-modal feature fusion and enhancement, a large number of experiments were conducted on the SMP2020-EWECT dataset. The SMP2020-EWECT dataset contains social media text data and general domain text data in a certain special period, covers different sentiment polarities and topics, and is one of the largest and most comprehensive benchmark datasets in Chinese sentiment analysis tasks. To further enrich the dataset, the Stable Diffusion model was used to automatically generate corresponding images based on the text content of SMP2020-EWECT and design specific instructions to guide Stable Diffusion to generate images with obvious emotional tendencies to supplement the missing visual information in the original dataset. In addition, the bilingual feature extraction model trained was used to further process the SMP2020-EWECT dataset, and a comprehensive sentiment knowledge base was constructed.

[0123] In the experiment, the embodiment is compared with a plurality of existing pre-training language models, including BERT, RoBERTa, ERNIE, Nezha and BSF, and data results as shown in Table 1 are obtained. As can be seen, the Chinese sentiment classification method based on multi-source bilingual and cross-modal feature fusion and enhancement proposed in the embodiment significantly exceeds the existing pre-training language models in two key indicators of accuracy and macro-F1 score, proving the effectiveness and robustness of the method in processing mixed language text sentiment classification tasks. Compared with ERNIE (Ernie 3.0: Large-scale knowledge enhanced pre-training for language understanding and generation), the method of the embodiment improves the accuracy by 4.53% and the macro-F1 score by 10.21%, fully embodying the significant advantage of the method of the embodiment in processing mixed language text sentiment classification tasks. By fusing bilingual features, especially using the more rich and explicit sentiment expression of English part, the method of the embodiment can more accurately capture the sentiment information contained in the text, thereby greatly improving the performance of sentiment classification. Compared with BSF (Improving Chinese Emotion Classification based on Bilingual Feature Fusion), the method of the embodiment improves the accuracy by 3.1% and the macro-F1 score by 3.22%, indicating the effectiveness of the method of the embodiment for multi-source bilingual information, image sentiment features and sentiment database construction and application in sentiment recognition, thereby improving the overall performance of sentiment classification.

[0124] Table 1

[0125]

[0126] In addition to the SMP2020-EWECT dataset, the Chinese sentiment classification method based on multi-source bilingual and cross-modal feature fusion and enhancement proposed in this embodiment also performs well on general datasets. General datasets cover a wider range of topics and fields, and have higher requirements for the generalization ability of the model. Table 2 shows the experimental results of the method of this embodiment on general datasets, and is compared with other existing pre-trained language models. As can be seen, the method of this embodiment also achieves the best performance on general datasets, ranking first in both accuracy and macro F1 value. Compared with ERNIE, the method of this embodiment improves the accuracy by 1.63% and the macro F1 value by 1.96%, again proving the significant effect of fusing bilingual features, multi-source bilingual features, image sentiment elements, and sentiment feature library support on improving sentiment classification performance. It is worth noting that compared with the SMP2020-EWECT dataset, the general dataset has more diverse fields, including different types of text such as e-commerce reviews, social media, and news comments. Despite this, the method of this embodiment still stably outperforms each existing model, demonstrating strong generalization ability and robustness.

[0127] Table 2

[0128]

[0129] In summary, the Chinese sentiment classification method based on multi-source bilingual and cross-modal feature fusion and enhancement proposed in this embodiment has the following advantages compared with existing pre-trained language models:

[0130] 1. Make full use of the multilingual modeling ability of pre-trained language models, effectively capture bilingual information by inputting Chinese text and its English translation at the same time, and make up for the shortcomings of monolingual models. Moreover, this embodiment also extracts bilingual text information contained in the image, further enriching the input of sentiment analysis;

[0131] 2. Introduce bilingual cross-attention mechanism, realize the deep fusion and interaction of text bilingual features and image bilingual features by calculating the similarity and information transmission between Chinese and English features, so that the model can more comprehensively understand the text sentiment from different languages and modalities;

[0132] 3. Construct a sentiment knowledge base to store sentiment words and their semantic features in different languages, use the knowledge base for retrieval and fusion, provide external knowledge enhancement for text with unclear sentiment expression, and strengthen the effective extraction of sentiment information in mixed language scenarios;

[0133] 4. A cross-modal attention fusion mechanism is designed to adaptively adjust the importance of text bilingual features, image visual sentiment features and knowledge base retrieval features, so as to mutually enhance them, obtain unified cross-modal sentiment representation, and through a residual connection, fuse the cross-modal features into the original bilingual features, further enhancing the sentiment representation capability;

[0134] 5. A lightweight model fine-tuning strategy is adopted, combining low-rank adaptation and embedding layer fine-tuning technology, to realize efficient adaptation of the model on downstream tasks without significantly increasing computational overhead, having strong practical value;

[0135] 6. Comprehensive experimental evaluation is carried out on multiple benchmark datasets, proving the significant advantages of the method in the mixed language text sentiment classification task, providing a new idea and reference for the development of Chinese sentiment analysis technology.

[0136] The above embodiments are intended to better further understand the present application and are not limited to the best mode of the present application, and do not limit the content and scope of protection of the present application. Any person who obtains any product identical or similar to the present application under the inspiration of the present application or by combining the present application with other existing technical features is within the scope of protection of the present application.

Claims

1. A Chinese sentiment classification method based on multi-source bilingual and cross-modal feature fusion and enhancement, characterized in that, Includes the following steps: Step 1: Obtain the Chinese text to be classified and the corresponding image to be classified, and translate the Chinese text to be classified into semantically aligned English text to be classified; the Chinese text to be classified is a mixed Chinese-English text containing a small amount of English. Step 2: Construct and train a bilingual feature extraction model, including a pre-trained language model, a lightweight Chinese network module, a lightweight English network module, and a bilingual cross-attention module; Step 3: Input the Chinese text and English text to be classified into the trained bilingual feature extraction model to extract the corresponding Chinese features. and English characteristics and bilingual integration features ; Step 4: Using a computer vision model, perform text detection on the image to be classified and extract the text regions contained in the image; then use the text recognition module to recognize the text regions as strings, which are used as Chinese text in the image, and translate them into semantically aligned English text in the image. The Chinese and English text of the image are input into the trained bilingual feature extraction model to extract the corresponding Chinese features of the image. and image English features and image bilingual fusion features ; Step 5: Based on a multilingual corpus containing multiple text fragments with known sentiment classifications, translate the Chinese text fragments into semantically aligned English text fragments. Then, using the trained bilingual feature extraction model, extract the corresponding Chinese and English features of the text fragments to construct a sentiment knowledge base. Step 6: In the sentiment knowledge base... Nearest neighbor retrieval is performed to obtain multiple most relevant semantic representations. These representations are then weighted, summed, and averaged to obtain the external sentiment representation. ; Step 7: Use a pre-trained convolutional neural network to extract global features from the image to be classified, obtaining a global image representation; construct and train an emotion feature extraction module, and use the trained emotion feature extraction module to extract emotion-related visual features from the global image representation as a global visual emotion semantic representation. ; Step 8: Construct and train a cross-modal attention fusion module, and use the trained cross-modal attention fusion module to... , and Cross-modal attention fusion is performed to obtain a cross-modal fused representation, and a residual mechanism is used to combine the cross-modal fused representation with... By combining these, we obtain cross-modal emotion representations. ; Step 9: Use a sentiment classifier to classify... Sentiment category prediction is performed to obtain the sentiment classification results of the Chinese text to be classified.

2. The Chinese sentiment classification method based on multi-source bilingual and cross-modal feature fusion and enhancement according to claim 1, characterized in that, The bilingual feature extraction model is specifically as follows: Using Chinese and English text as inputs to the bilingual feature extraction model, a pre-trained language model extracts the hidden layer representations of the Chinese and English texts. Lightweight Chinese and English network modules are added to each linear layer of the pre-trained language model. After processing by the lightweight Chinese network module, the Chinese text is combined with its hidden layer representation to obtain Chinese features. After processing by the lightweight English network module, the English text is combined with its hidden layer representation to obtain English features. The Chinese and English features are then fused through a bilingual cross-attention module to output bilingual fused features.

3. The Chinese sentiment classification method based on multi-source bilingual and cross-modal feature fusion and enhancement according to claim 2, characterized in that, Step 3: Extract bilingual fusion features The specific process is as follows: Chinese features and English characteristics Perform linear transformations respectively to obtain the corresponding Chinese query matrix. Chinese key matrix Chinese value matrix English query matrix English Key Matrix and English value matrix ; calculate and The similarity is calculated, and the attention weight matrix for this similarity is obtained using the Softmax function. After weighting, then add to By combining and normalizing, we obtain the feature enhancement from English to Chinese. ; calculate and The similarity is calculated, and the attention weight matrix for this similarity is obtained using the Softmax function. After weighting, then add to By combining and normalizing, we obtain the feature enhancement from Chinese to English. ; This leads to the Chinese hidden layer representation after crossover. The hidden layer after the intersection represents the English text. : ； ； In the formula, This indicates normalization processing; This indicates feedforward neural network processing; Furthermore, regarding and Summation yields the bilingual integration features. .

4. The Chinese sentiment classification method based on multi-source bilingual and cross-modal feature fusion and enhancement according to claim 3, characterized in that, Step 4: Extract bilingual fusion features from the image The specific process is as follows: Chinese features of images and image English features Perform linear transformations respectively to obtain the corresponding Chinese query matrix for the image. Image Chinese Key Matrix Image Chinese value matrix Image English Query Matrix Image English Key Matrix and image English value matrix ; calculate and The similarity is calculated, and the attention weight matrix for this similarity is obtained using the Softmax function. After weighting, then add to By combining and normalizing the data, we obtain the feature enhancement method for converting English to Chinese in the image. ; calculate and The similarity is calculated, and the attention weight matrix for this similarity is obtained using the Softmax function. After weighting, then add to By combining and normalizing the data, we obtain the feature enhancement method for converting Chinese to English in the image. ; This leads to the Chinese hidden layer representation of the cross-image. The hidden layer representation of the image after crossing : ； ； right The average of all word vectors contained therein is calculated, and the average of all word vectors contained therein is calculated. The average of all word vectors contained in the image is calculated, and the two averages are summed to obtain the bilingual fusion feature of the image. .

5. The Chinese sentiment classification method based on multi-source bilingual and cross-modal feature fusion and enhancement according to claim 2, characterized in that, The Chinese lightweight network module and the English lightweight network module are designed based on low-rank adaptive technology.

6. The Chinese sentiment classification method based on multi-source bilingual and cross-modal feature fusion and enhancement according to claim 2, characterized in that, In step 2, the bilingual feature extraction model is trained based on the sentiment analysis evaluation database. During the training of the pre-trained language model in the bilingual feature extraction model, only the parameters in the embedding layer are trained, while the other parameters of the pre-trained language model are frozen.

7. The Chinese sentiment classification method based on multi-source bilingual and cross-modal feature fusion and enhancement according to claim 2, characterized in that, The pre-trained language model is BERT, RoBERTa, or XLNet.

8. The Chinese sentiment classification method based on multi-source bilingual and cross-modal feature fusion and enhancement according to claim 2, characterized in that, In step 6, a scoring mechanism is used to assign weights to the most relevant semantic representations.

9. The Chinese sentiment classification method based on multi-source bilingual and cross-modal feature fusion and enhancement according to claim 2, characterized in that, In step 7, the emotion feature extraction module is implemented based on a fully connected layer and trained on an emotion analysis and evaluation database. Specifically, the global image representation output by the pre-trained convolutional neural network is used as input to train the parameters of the emotion feature extraction module.

10. The Chinese sentiment classification method based on multi-source bilingual and cross-modal feature fusion and enhancement according to claim 9, characterized in that, In step 8, the training process of the cross-modal attention fusion module is as follows: training is performed based on a sentiment analysis evaluation database, specifically using the output of the trained bilingual feature extraction model. The output of the trained sentiment feature extraction module and the results obtained in step 6 The weights of the cross-modal attention fusion module are trained as input to the cross-modal attention fusion module, and the parameters of the bilingual feature extraction model and the sentiment feature extraction module remain unchanged during the training process.

Images