A method for mongolian and han cross-language sentiment analysis based on knowledge sharing and adaptive learning

Abstract

A Mongolian-Chinese cross-language sentiment analysis method based on knowledge sharing and adaptive learning, preprocessing Mongolian sentiment corpus and Chinese sentiment corpus, respectively obtaining initial word vectors of two languages; learning the personalized offset of Mongolian-Chinese words respectively by using a personalized adapter, and combining with the respective initial word vectors, obtaining the calibrated Mongolian word embedding vector and Chinese word embedding vector; using Householder projection to orthogonally map the calibrated Mongolian word embedding vector space and Chinese word embedding vector space to a shared hidden space, and keeping the orthogonality of the mapping in model optimization; extracting the global features of the word vectors in the hidden space, and giving different sentiment weights to different words; performing sentiment classification in the classification layer. The present application can solve the problem of cross-language sentiment analysis difficulty caused by the lack of Mongolian resources to a certain extent.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, and relates to sentiment analysis in natural language processing, and particularly to a cross-language sentiment analysis method between Mongolian and Chinese based on knowledge sharing and adaptive learning. Background Technology

[0002] Online platform comment data contains a wealth of valuable information. Sentiment analysis of this data can help us understand people's views on certain events and things, as well as their related value orientations.

[0003] Supervised sentiment analysis research often requires a large amount of sentiment-labeled data. However, most existing sentiment analysis research is based on Chinese. Although there are some sentiment analysis studies in other languages, they are generally far behind sentiment classification research in Chinese. Mongolian sentiment data is relatively scarce, and manual annotation is extremely time-consuming and labor-intensive. Therefore, a common approach is to leverage existing labeled corpora to facilitate research. Cross-linguistic sentiment analysis uses source languages ​​rich in sentiment resources to help target languages ​​complete sentiment analysis, thus promoting sentiment analysis research in languages ​​with scarce sentiment resources.

[0004] In terms of implementation, cross-linguistic sentiment analysis uses abundant and high-quality source language data and relatively scarce and low-quality target language data. Its aim is to leverage source language resources to assist target language resources in sentiment analysis. Due to structural differences between languages, the challenges of cross-linguistic research lie in two aspects: first, how to achieve spatial transformation or knowledge transfer between two different languages, i.e., how to better preserve the original meaning of the languages ​​while conducting transfer learning; second, based on cross-linguistic capabilities, how to more fully acquire sentiment information and conduct better sentiment analysis. For example, for Mongolian, due to the relatively late start of sentiment analysis research on Mongolian, the complexity of Mongolian morphological changes compared to languages ​​like English and Chinese, and the relative scarcity of Mongolian corpora, traditional cross-linguistic sentiment analysis methods are difficult to directly apply to Mongolian. Summary of the Invention

[0005] To overcome the shortcomings of the existing technologies, the present invention aims to provide a Mongolian-Chinese cross-language sentiment analysis method based on knowledge sharing and adaptive learning. By leveraging the rich corpus resources of Chinese, it constructs knowledge connections between Mongolian and Chinese, realizes the sharing of Mongolian and Chinese feature resources, and thus solves the problem of cross-language sentiment analysis difficulties caused by the scarcity of Mongolian language resources to a certain extent.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] A cross-linguistic sentiment analysis method between Mongolian and Chinese based on knowledge sharing and adaptive learning includes the following steps:

[0008] Step 1: Preprocess the Mongolian sentiment corpus and the Chinese sentiment prediction to obtain the initial word vectors for the two languages ​​respectively;

[0009] Step 2: Use the personalized adapter to learn the personalized offsets of Mongolian and Chinese words respectively, and combine them with their respective initial word vectors to obtain the calibrated Mongolian word embedding vectors and Chinese word embedding vectors;

[0010] Step 3: Use Householder projection to orthogonally map the calibrated Mongolian word embedding vector space and the Chinese word embedding vector space to a shared latent space, and maintain the orthogonality of the mapping during model optimization;

[0011] Step 4: Extract the global features of word vectors in the latent space and assign different sentiment weights to different words;

[0012] Step 5: Perform sentiment classification at the classification level.

[0013] In one embodiment, step 2, given the word vector x of a word, first considers the word vectors x of its neighboring words. j Perform mean aggregation to obtain a vector that contains contextual semantic information.

[0014] Then, using vectors As input to the personalization adapter, it enables the personalization adapter to learn its personalization offset A. s (x);

[0015] Finally, the initial word vector x and the personalized offset A are combined. s Adding (x) together yields the calibrated word embedding vector.

[0016] In one embodiment, the Householder projection parameters of the source language and the target language are not shared; that is, the Mongolian word embedding vector space and the Chinese word embedding vector space each have a separate Householder projection module, and the two projection modules have different parameters.

[0017] In one embodiment, step 4 involves processing the word vectors in the shared latent space using a Bi-LSTM layer and an attention layer, and extracting global features using LSTM networks in both forward and backward directions.

[0018] In one embodiment, step 4 utilizes an attention mechanism to focus on emotional information in the text, assigning different emotional weights to different words.

[0019] In one embodiment, the classification layer is a SoftMax layer.

[0020] Compared with the prior art, the beneficial effects of the present invention are:

[0021] This invention aligns Mongolian and Chinese word features, employing an adaptive learning-based method to construct cross-lingual word vectors. It utilizes a learnable personalized adapter, based on monolingual word vectors and a bilingual dictionary, to generate personalized offsets for each word using contextual semantic information, thus calibrating the original word vectors to a more appropriate position. The core of cross-lingual word embedding lies in designing an ideal mapping function to map them into a shared latent space. After obtaining the calibrated word vectors, a Householder matrix is ​​introduced, and a Householder projection that strictly maintains the orthogonal mapping property during optimization is constructed based on the Householder matrix, better ensuring that the structural information of the word vector space is not destroyed. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the overall process of the present invention.

[0023] Figure 2 This is a diagram of a shared space.

[0024] Figure 3 This is a diagram of the BiLSTM model.

[0025] Figure 4 This is a diagram of the attention mechanism structure. Detailed Implementation

[0026] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings and examples.

[0027] Sentiment analysis involves analyzing text data generated on the internet and determining sentiment orientation. This process often requires a large amount of labeled text resources. However, due to differences in data resources across different languages, the development of sentiment analysis is uneven across languages. This situation has spurred the emergence of cross-linguistic sentiment analysis. For Mongolian, on the one hand, its linguistic resources are scarce, and on the other hand, the morphological changes in Mongolian vocabulary and grammar are quite unique and complex. Therefore, the effectiveness of traditional cross-linguistic sentiment analysis in Mongolian applications still needs improvement.

[0028] This invention utilizes Chinese to conduct sentiment analysis research on Mongolian. In terms of specific principles, this invention uses cross-linguistic word embedding based on adaptive learning to better achieve spatial transformation, and combines bidirectional BiLSTM and attention mechanisms to conduct cross-linguistic sentiment analysis between Mongolian and Chinese.

[0029] like Figure 1 As shown, the Mongolian-Chinese cross-language sentiment analysis method based on knowledge sharing and adaptive learning in the present invention mainly consists of two parts. The first part: shared space. First, the Personalized adapter generates personalized offsets for each word based on the monolingual word vectors and the seed dictionary, using the contextual semantic information of the words, so that the original word vectors are calibrated to a more appropriate position; then, the Householder projection orthogonally maps the two calibrated word vector spaces to a shared latent space, and the orthogonality of the mapping is maintained during model optimization. The second part: sentiment classification. Use the Bi-LSTM layer and the attention layer to process the word vectors, extract global features using the LSTM networks in the forward and backward directions, and then use the attention mechanism to focus on the sentiment information in the text, assign different sentiment weights to different words, and finally enter the SoftMax layer for sentiment classification to obtain the final accuracy rate.

[0030] Refer again to Figure 1 , the method of the present invention specifically includes the following steps:

[0031] Step 1, preprocess the Mongolian sentiment corpus and the Chinese sentiment corpus, and perform word vector representation on the preprocessed corpus.

[0032] First, perform data cleaning operations on the Mongolian sentiment corpus and the Chinese sentiment corpus, and then perform word segmentation operations. For the word segmentation operation, separate each Mongolian word as the smallest unit. Taking the Chinese sentence "The product quality is okay, but the service attitude is poor" as an example, its corresponding Mongolian representation is The result after word segmentation is The result after using Jieba word segmentation for Chinese is "product", "quality", "okay", "selection", "service", "attitude", "poor". Then use the Word2vec pre-trained model to obtain the word vector representations of the Mongolian and Chinese languages respectively, and use them as their initial word vectors.

[0033] Step 2, use the Personalized adapter to learn the personalized offsets of Mongolian words and Chinese words respectively, and combine them with their respective initial word vectors to obtain the calibrated Mongolian word embedding vectors and Chinese word embedding vectors.

[0034] Mongolian and Chinese have their own unique Personalized adapters. Taking the source language (Chinese) as an example. Given the word vector x of a word in the source language. First, by aggregating the word vectors x j of its neighboring words by mean, a vector containing contextual semantic information is obtained For example, when dealing with "quality," it aggregates neighboring words like "product" and "good," while for "attitude," it aggregates "service" and "poor." Compared to individual words, such sets of similar words provide richer and more accurate information, helping to learn personalized word shifts. Contextual semantic vectors The calculation formula is as follows:

[0035]

[0036] M s (x)={x h | <x j ,x>>τ s}

[0037] Where m s It is M s The magnitude of (x) is given by <>, where τ represents the dot product. s This is a hyperparameter representing the similarity threshold. Compared to the initial word vector x, the context semantic vector... Provide more information by including richer semantics.

[0038] Subsequently, the personalized adapter can be based on contextual semantic vectors. The unique offset for each word is learned, and this offset can be effectively optimized through the training objective. Considering that semantically similar words have stronger isomorphic structures in the word vector spaces of different languages, it is assumed that words with similar contextual semantics also tend to have similar personalized offsets. In the case study, "goods" and... "Difference" and "difference" are words with similar meanings, and their offsets are more similar when learning offsets. Therefore, with... As input to the personalization adapter, it enables the personalization adapter to learn its personalized offset A. s (x). Specifically, a single-layer feed-forward network (FFN) is used as the structure of the personalization adapter. It calculates the personalized offset based on the context semantic vector of the initial word vector, and finally uses the calculated personalized offset A. s (x) is added to the initial word vector x to obtain the calibrated word embedding vector. The formula is shown below:

[0039]

[0040]

[0041] Where σ represents the activation function, which can be linear or nonlinear. W s∈R d*d represent learnable parameters. According to Figure 2 shown, after offset calibration of the initial word vectors of Mongolian and Chinese using personalized adapters respectively, "goods" and "bad" are both used to express the same meaning respectively, so they may be mapped to similar positions in the shared space.

[0042] Step 3, use Householder projection to orthogonally map the calibrated Mongolian word embedding vector space and Chinese word embedding vector space to a shared latent space, and maintain the orthogonality of the mapping in model optimization.

[0043] According to Figure 2 shown, after calibrating the initial word vectors, an ideal mapping function is also required to map them to the shared latent space for facilitating downstream sentiment analysis tasks. Previous work has demonstrated that the orthogonality of the mapping function is crucial for model performance. A common approach is to add additional constraints to the objective function to force the mapping matrix to be orthogonal. However, this constraint can only achieve an approximate orthogonal matrix and cannot achieve a strict orthogonal matrix. The present invention is prepared to use Householder projection. Because the Householder matrix is an orthogonal matrix. Based on such a Householder matrix, a mapping function Householder projection is used to ensure that the mapping process is strictly orthogonal.

[0044] The Householder matrix represents a reflection about a hyperplane containing the origin. Given a unit vector The d×d Householder matrix H defined with v as a parameter is defined as H(v), and the formula is as follows:

[0045] H(v) = I - 2vv T <​​​​​​​​​​​​Represents the i-th unit vector. n is the number of unit vectors, and the Householder projection is as follows:

[0049]

[0050] H(v i ) indicates that v i The Householder matrix with parameters of d×d, as mentioned above, has the following formula:

[0051] H(v i )=I-2v i v i T

[0052] Where ||v2||=1, I is a d×d identity matrix, given the input vector z, H(v i The z-axis is transformed by reflection about a hyperplane orthogonal to the normal vector v. Referring to the foregoing, the formula is as follows:

[0053]

[0054] Specifically, in sentiment analysis, Householder projection does not share parameters between the source and target languages. When using Householder projection to orthogonally map two calibrated word embedding vector spaces to a shared latent space, parameter non-sharing is necessary to maintain the orthogonality of the mapping. If parameters were shared, the mapping might lose orthogonality, leading to information confusion and loss. By using a parameter-non-sharing Householder projection module, Mongolian and Chinese each have their own mapping functions, ensuring that the mapped word vectors can adequately represent any orthogonal matrix in the shared space. This ensures that the orthogonality of the mapping is not compromised during model optimization, thus improving the sentiment analysis task. Therefore, parameter non-sharing is crucial for maintaining the orthogonality and accuracy of the mapping. In Householder projection, two calibrated word embedding vector spaces are orthogonally mapped to a shared latent space. Parameter non-sharing means that the Mongolian and Chinese word embedding vector spaces each have a separate Householder projection module with different parameters. Taking Mongolian as an example, its Householder projection module uses the parameter n″, where the value of n″ is d (the dimension of the word vector) to ensure that it can fully represent any orthogonal matrix. This parameter n″ limits the dimension of the Householder projection module, enabling it to adapt to word vectors of different dimensions.

[0055] Specifically, for the source language, i.e., Chinese, given a calibrated source word embedding vector Use V s ′={v1′,..,v n The source Householder projection of '} is used as the mapping function, and the resulting mapping vector is shown below:

[0056]

[0057] For the target language, namely Mongolian, given the calibrated target word embedding vector Use V s "={v1″,..,v n The target Householder projection of "} is used as the mapping function, and the resulting mapping vector is shown below:

[0058]

[0059] Among them, V s ' is the set of unit vectors obtained by aggregating the initial word vectors x of the source language words using the mean, v i ' represents its i-th unit vector, and n' is the number of its unit vectors; V s "" is the set of unit vectors obtained by aggregating the initial word vectors x of the target language words using the mean, v i " represents its i-th unit vector, and n" represents the number of its unit vectors.

[0060] Regardless of the optimized values ​​of the Householder projection parameters during gradient descent, the entire Householder projection process remains strictly orthogonal. Furthermore, through mathematical simplification and derivation, it is ensured that the entire computation process still has the same time complexity as a simple linear transformation.

[0061] When using Householder projection to map source and target languages ​​into the same space, the degree of matching between two vectors can be determined by calculating their cosine similarity. Cosine similarity is a commonly used metric for measuring the similarity between vectors.

[0062] First, the two vectors need to be normalized, that is, transformed into unit vectors. Normalization can be achieved by dividing each vector by its norm. Then, the cosine similarity between the two vectors is calculated using the cosine formula:

[0063]

[0064] in, Representing vectors sum vector dot product, and Representing vectors respectively sum vector The norm of cosine similarity. The value of cosine similarity ranges from [-1, 1]. The closer the value is to 1, the more similar the two vectors are, while the closer the value is to -1, the less similar the two vectors are.

[0065] Therefore, when the calculated cosine similarity is close to 1, the two mapped vectors can be considered to be more matched. Conversely, if the cosine similarity is close to -1, it indicates that the two vectors are less matched. That is, the closer the calculated cosine similarity is to 1, the better the match is considered to be. and The better the match.

[0066] Step 4: Process the word vectors in the shared latent space using Bi-LSTM layers and attention layers. Specifically, global features are extracted using LSTM networks in both forward and backward directions, and then an attention mechanism is used to focus on the sentiment information in the text, assigning different sentiment weights to different words.

[0067] After mapping Mongolian and Chinese to a shared representation space, words in the two languages ​​will exhibit a high degree of similarity as long as they are semantically similar. For example, "goods" and... The similarity is low in the monolingual space, but a high similarity is achieved in the mapping space after cross-lingual word embedding. This allows a model trained on the emotionally rich Chinese language to be applied to the emotionally limited Mongolian language. In other words, by embedding Mongolian and Chinese words into the same space, the Chinese classification model can be used to classify the polarity of Mongolian words. Therefore, in downstream sentiment analysis tasks, this invention uses BiLSTM with an attention mechanism for feature extraction and sentiment classification.

[0068] LSTM is a network structure using Long Short-Term Memory (LSTM) cells as hidden units. It considers the sequential dependencies between word sequences, thus capturing both long-range and short-range dependencies. LSTM is suitable for extracting information from long texts, but basic LSTM scanning only occurs in one direction of the sequence. Bi-LSTM is a further development. Bi-LSTM is a bidirectional LSTM network composed of forward and backward LSTMs, allowing processing of text data from both directions, thus solving the problem that LSTM networks cannot encode from back to front. Bi-LSTM utilizes two-directional LSTMs to realize the dependencies of contextual information in the sequence, making better use of contextual information. This invention uses BiLSTM for modeling, such as... Figure 3As shown. After the Bi-LSTM layer extracts the features of the text, it cannot distinguish the importance of the information. Given that different parts of a sentence contribute differently to the sentiment polarity of the entire corpus, some words in a sentence may have a significant impact on sentiment polarity, while others may be insignificant. For example, in this case, "service attitude" and "poor" have a significant impact. Therefore, an attention layer is added to the BiLSTM to weight the sentiment information, highlighting the parts that have a greater impact on the sentence's sentiment. Figure 4 As shown in the diagram, the output feature vector contains the sentiment information of the word and the calculated weights, which are used to determine the importance of the word.

[0069] Step 5: Perform sentiment classification in the SoftMax layer to obtain the final accuracy.

[0070] After passing through a BiLSTM and attention layer, the output feature vector contains the sentiment information of each word and its calculated weights, used to determine the importance of the word. Finally, sentiment classification can be performed. The sentiment classification layer determines the sentiment polarity of the document, using a linear function to transform the feature representation of the text into a real-valued vector, mapping this vector to a vector space with C classes. Finally, a softmax layer is added to classify the sentiment of the text information processed by the model, outputting a classification probability p. c :

[0071]

[0072] Where W represents the weight coefficient matrix to be trained from the Attention mechanism layer to the output layer, b c This represents the bias to be trained. In the case of this invention, after softmax sentiment classification, the output vector is [0.2, 0.8], where 0.8 indicates a higher probability of negative sentiment and 0.2 indicates a lower probability of positive sentiment. Therefore, the sentiment in this case is negative, which is consistent with reality, indicating that the analysis results of this invention are accurate.

Claims

1. A Mongolian-Chinese cross-language sentiment analysis method based on knowledge sharing and adaptive learning, characterized in that, The method comprises the following steps: Step 1: Preprocessing the Mongolian emotional corpus and the Chinese emotional corpus to obtain initial word vectors of the two languages respectively; Step 2: Learning the personalized bias of Mongolian and Chinese words respectively by using a personalized adapter, and combining the personalized bias with the initial word vectors to obtain calibrated Mongolian word embedding vectors and Chinese word embedding vectors; Step 3: Orthogonally mapping the calibrated Mongolian word embedding vector space and the Chinese word embedding vector space to a shared hidden space by using Householder projection, and maintaining the orthogonality of the mapping in model optimization; Step 4: Extracting global features of the word vectors in the hidden space, and assigning different emotional weights to different words; Step 5: Performing emotional classification in the classification layer; Wherein: The step 2, given a word's word vector , first mean-pooling its neighbor words' word vectors , to get a vector containing contextual semantic information ; Afterwards, the vector As input to the personalization adapter, the personalization adapter learns its personalization bias ; Finally, the initial word vector is added to the personalized bias to obtain the calibrated word embedding vector ; The step 3, Householder projection consists of a sequence of Householder matrices, for the unit vector set The Householder projection is as follows: wherein denotes the i-th unit vector, , is the number of unit vectors; denotes a d x d Householder matrix parametrized by with the following formula: in Let I be a d×d identity matrix, given the input vector , By about the orthogonality to the normal vector The reflection of the hyperplane will Transform into The formula is shown below: For the source language, i.e. Chinese, given calibrated source word embedding vectors , using the source Householder projection as the mapping function, the resulting mapped vectors are as follows: For the target language, i.e. Mongolian, given the calibrated target word embedding vectors , using the target Householder projection as the mapping function, the resulting mapped vectors are as follows: wherein, is an initial word vector of the source language word a set of unit vectors obtained by mean aggregation, denotes its i-th unit vector, is the number of its unit vectors; is an initial word vector of the target language word a set of unit vectors obtained by mean aggregation, denotes its i-th unit vector, is the number of its unit vectors; In step 4, the Bi-LSTM layer and the attention layer are used to process the word vectors in the shared hidden space, and the global features are extracted by using the LSTM network in the front and back directions.

2. The Mongolian-Chinese cross-language sentiment analysis method based on knowledge sharing and adaptive learning according to claim 1, characterized in that, In step 1, the initial word vectors of the two languages are obtained by using the Word2vec pre-training model.

3. The Mongolian-Chinese cross-language sentiment analysis method based on knowledge sharing and adaptive learning according to claim 1, characterized in that, The vector The calculation formula is as follows: wherein is the size of, denotes the dot product, is a hyperparameter denoting a similarity threshold; The structure of the individualization adapter is a single-layer feedforward neural network network, and the individualization offset is calculated according to the following formula: wherein denotes an activation function, represent learnable parameters.

4. The Mongolian-Chinese cross-language sentiment analysis method based on knowledge sharing and adaptive learning according to claim 1, characterized in that, In the Householder projection, the Householder projection parameters of the source language and the target language are not shared; that is, the Mongolian word embedding vector space and the Chinese word embedding vector space each have a separate Householder projection module, and the two projection modules have different parameters.

5. The Mongolian-Chinese cross-language sentiment analysis method based on knowledge sharing and adaptive learning according to claim 1, characterized in that, Through calculation and The cosine similarity between the two values ​​determines their degree of matching, and the formula is as follows: Similarity( )= wherein, denotes and the dot product of vectors and denote the norm of vector and vector respectively, the closer the calculated cosine similarity is to 1, the more and are considered to match.

6. The Mongolian-Chinese cross-language sentiment analysis method based on knowledge sharing and adaptive learning according to claim 1, characterized in that, In step 4, the attention mechanism is used to focus on the emotional information in the text, and different emotional weights are assigned to different words.

7. The method of claim 1, wherein the method further comprises: The classification layer is a SoftMax layer.

Images