Emotion analysis method and device based on BERT and xLSTM and medium

Abstract

The invention belongs to the technical field of natural language processing, and particularly relates to an emotion analysis method and device based on BERT and xLSTM and a medium, the method comprises the steps that a text feature extraction model is formed by integrating BERT and two layers of xLSTM, and the BERT uses a comparison self-supervision method for pre-training, defining a task target of a sentence level and capturing semantics of the sentence level; the two layers of xLSTM respectively pay attention to forward and backward sentence embedded representations processed by BERT, comprehensively capture deep semantic information in a text, and perform sentiment analysis by using high-quality sentence-level text representations extracted by the model, so that the accuracy of sentence-level sentiment analysis is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a sentiment analysis method, device and medium based on BERT and xLSTM. Background Art

[0002] With the continuous development of the Internet, sentiment analysis technology has been widely used in social media and online platforms. The amount of information on these platforms is huge, the speed of dissemination is fast, and the content style is diverse. Sentiment analysis technology aims to identify and extract emotional states and subjective information by analyzing text data. Sentiment analysis technology is particularly important in this context, and can help companies and researchers better understand public sentiment, user satisfaction, and market trends.

[0003] Sentiment analysis technology has evolved from the initial rule-based and dictionary-based methods to machine learning and deep learning-based methods, and then to today's pre-trained language models and methods combined with graph neural networks, which have greatly improved the accuracy and adaptability of text sentiment recognition. However, existing sentiment analysis technology still faces some challenges, such as insufficient semantic understanding, strong dependence on large amounts of annotated data, and weak ability to process long texts.

[0004] In summary, sentiment analysis methods, devices and media based on BERT and xLSTM are urgently needed to solve the problems in the existing technology. Summary of the invention

[0005] The purpose of the present invention is to provide a sentiment analysis method, device and medium based on BERT and xLSTM. The specific technical solution is as follows: The sentiment analysis method based on BERT and xLSTM includes the following steps: S100: construct a text feature extraction model, wherein the text feature extraction model includes a BERT sentence embedding extraction module and a bidirectional xLSTM deep semantic information extraction module; S200: extracting sentence-level semantic information in the text through the BERT sentence embedding extraction module to obtain a first sentence embedding representation, and using a bidirectional xLSTM deep semantic information extraction module to focus on the forward and backward sentence embedding representations respectively to obtain a second sentence embedding representation; S300: Perform a sentiment analysis task using the second sentence embedding representation.

[0006] Optionally, in S100, the BERT sentence embedding extraction module is trained using a contrastive self-supervised learning pre-training module, and the process is as follows: Prepare positive and negative pairs for pre-training; Initialize the parameters of the BERT model; Input the positive and negative pairs into the BERT model to obtain the embedded representation; Calculate the cosine similarity of each positive and negative pair; Input the cosine similarity into the contrastive loss function and calculate the loss for all positive and negative pairs; The gradient is calculated through back propagation and the BERT model parameters are updated.

[0007] Optionally, in S200, the BERT sentence embedding extraction module includes an embedding layer, a Transformer structure and a hidden layer, and the Transformer structure includes a self-attention mechanism, a feedforward neural network, a residual connection and layer normalization; in the self-attention mechanism, the attention score is calculated through a multi-head attention layer.

[0008] Optionally, in S200, sentence-level semantic information in the text is extracted by a BERT sentence embedding extraction module to obtain a first sentence embedding representation, and the process is as follows: Use embedding layer to calculate word embedding sequence; Input the word embedding sequence into the multi-head attention layer to obtain the attention score; The word embedding sequence is processed using attention scores to obtain the first sentence embedding representation.

[0009] Optionally, in S200, the bidirectional xLSTM deep semantic information extraction module includes an xLSTM structure formed by stacking a plurality of xLSTM blocks, each xLSTM block including one of an sLSTM and an mLSTM; sLSTM: Introduces an exponential gating mechanism into the long short-term memory network to change the update method of the memory unit and is used to process sequence data of different time steps; mLSTM: Achieve parallel computing and memory processing capabilities through matrix memory cell states and covariance update rules; The xLSTM structure includes at least one xLSTM block including an sLSTM and at least one xLSTM block including an mLSTM.

[0010] Optionally, in S300, the sentiment analysis task is performed using the second sentence embedding representation, and the process is as follows: The second sentence embedding represents the input random inactivation layer that randomly discards a part of neurons; The normalization layer is used to standardize the output of the random dropout layer; The mean layer is used to average the data; A fully connected layer is used to process the output of the mean layer to map the high-dimensional text representation to the dimension of the target category to achieve sentiment classification.

[0011] Optionally, the loss function calculation process for the sentiment classification task is as follows: Convert sentiment classification results into binary matrix form , where each row represents a sample , each column represents a category , the expression is as follows: ; For each example, extract the output value of the correct category , the expression is as follows: ; Calculate the final loss , the expression is as follows: ; ; in, represents the number of samples, represents the hyperparameter, Represents the loss value of each sample, and the final loss is the mean of all sample loss values. Indicates that it is The prediction scores of samples on other categories.

[0012] In addition, the present invention also discloses a computer device, including a memory and a processor; The memory is used to store a computer program executable on the processor; The processor is used to implement the steps of the above-mentioned sentiment analysis method when executing the computer program.

[0013] In addition, the present invention also discloses a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the sentiment analysis method as described above are implemented.

[0014] The application of the technical solution of the present invention has the following beneficial effects: The present invention discloses a sentiment analysis method based on BERT and xLSTM. In the method of the present invention, the method forms a text feature extraction model by integrating BERT and two layers of xLSTM, wherein BERT is pre-trained using a contrastive self-supervision method, defines a task target at the sentence level, and captures the semantics at the sentence level; the two layers of xLSTM respectively focus on the forward and backward sentence embedding representations processed by BERT, comprehensively capture the deep semantic information in the text, and use the high-quality sentence-level text representation extracted by the above model to perform sentiment analysis, thereby significantly improving the accuracy of sentence-level sentiment analysis.

[0015] In addition to the above-described purposes, features and advantages, the present invention has other purposes, features and advantages. The present invention will be further described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions of the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 It is a flowchart of the steps of the sentiment analysis method in the preferred embodiment of the present invention. DETAILED DESCRIPTION

[0018] In order to enable those skilled in the art to better understand the scheme of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0019] like Figure 1 As shown, this embodiment discloses a sentiment analysis method based on BERT and xLSTM, comprising the following steps: The sentiment analysis method based on BERT and xLSTM includes the following steps: S100: construct a text feature extraction model, wherein the text feature extraction model includes a BERT sentence embedding extraction module and a bidirectional xLSTM deep semantic information extraction module; S200: extracting sentence-level semantic information in the text through the BERT sentence embedding extraction module to obtain a first sentence embedding representation, and using a bidirectional xLSTM deep semantic information extraction module to focus on the forward and backward sentence embedding representations respectively to obtain a second sentence embedding representation; S300: Perform a sentiment analysis task using the second sentence embedding representation.

[0020] Preferably, in S100, the BERT sentence embedding extraction module is trained using a contrastive self-supervised learning pre-training module, and the process is as follows: Prepare positive and negative pairs for pre-training. Specifically, assuming the length of the dataset is n, store each sentence (without label) in the dataset into a queue S: ,in, Each sentence in the S queue is enhanced in turn. The data enhancement is replaced by synonyms. The enhanced sentences are stored in a queue in turn. middle: . S and The corresponding elements in will form n pairs: In addition, this embodiment forms a negative pair with each element in the queue S and a random element other than it, thus generating n negative pairs. ( The value range is from 1 to n. Not equal to , Randomly select a non-zero value in the range 1 to n. of ).

[0021] Initialize the parameters of the BERT model; select a BERT model denoted as Initialize the model parameters and select a BERT model as Initialize model parameters.

[0022] Input the positive and negative pairs into the BERT model to obtain the embedded representation; Calculate the cosine similarity of each positive and negative pair; Input the cosine similarity into the contrastive loss function and calculate the loss for all positive and negative pairs; The gradient is calculated through back propagation and the BERT model parameters are updated.

[0023] It should be noted that the expression of the contrast loss function is as follows: ; in, represents the contrast loss, represents the cosine similarity, represents the temperature parameter, yes , yes , represents the first element of a positive or negative pair, represents the second element of a positive or negative pair, express The weight of express The weight of Use dynamic updates.

[0024] Optionally, in S200, the BERT sentence embedding extraction module includes an embedding layer, a Transformer structure and a hidden layer, and the Transformer structure includes a self-attention mechanism, a feedforward neural network, a residual connection and layer normalization. In the self-attention mechanism, the Query, Key and Value vectors are calculated, and the attention score is calculated by the multi-head attention mechanism. Subsequently, the output of the attention mechanism is processed by residual connection and layer normalization, and then further processed by the feedforward neural network, and residual connection and layer normalization are applied again.

[0025] Furthermore, after being processed by the multi-layer Transformer structure, the hidden state matrix of the last layer is obtained, which contains the contextual embedding representation of each token. Finally, the embedding vector of the special token "[CLS]" is extracted from the hidden state matrix as the embedding representation of the entire sentence for downstream text classification tasks.

[0026] Optionally, in S200, sentence-level semantic information in the text is extracted by a BERT sentence embedding extraction module to obtain a first sentence embedding representation, and the process is as follows: The embedding layer is used to calculate the word embedding sequence. The process is as follows: the input text is converted into subwords and then into a token sequence. After word segmentation, the word block sequence T is obtained. The word block T is converted into a token ID using the BERT vocabulary. Then a mask tag is generated to mark which tokens are valid inputs and which are fillers. Let the token ID be , the mask is marked as , convert them to tensors: , , Indicates converting a matrix into a tensor representation. and Input to the BERT model, let is the word embedding matrix of the BERT model, where is the vocabulary size, is the dimension of the embedding vector. In this embodiment, the dimension of the basic version of the BERT model is 768. Mapped to the corresponding word embedding matrix, we get .in, represents the embedding vector of each token, is the word embedding sequence of the input sentence. Add position encoding to each word chunk: .in, is the position encoding matrix, is the length of the input sequence. It is the word embedding sequence after adding position encoding.

[0027] Feed the word embedding sequence into the multi-head attention layer; It will be input into the multi-head attention layer to calculate the Query vector, Key vector and Value vector, where the Query vector represents the "query" for the currently processed word or tag, which is used to calculate the similarity with the Key vector to determine which positions need to be obtained from. The Key vector represents the "features" or "descriptions" of each word or tag in the input sequence, which is used to compare with the Query vector to determine which part of the input sequence is most relevant to the current query. The Value vector contains the actual content or information of each word or tag in the input sequence. The attention mechanism ultimately extracts information from the Value and weighs the contribution of the Value at different positions based on the match (similarity) between the Query and the Key.

[0028] The output of the multi-head attention layer is processed to obtain the first sentence embedding representation. The process is as follows: Residual connections and layer normalization : ; in, is the word embedding matrix after adding position encoding, is the output of the multi-head attention layer, express and The result of layer normalization after residual connection.

[0029] Feedforward neural network for each transformer layer : ; in, represents the output of the feedforward neural network, , , , , Both represent the hidden layer dimensions of the feedforward neural network.

[0030] Residual connections and layer normalization: ; in, represents the result of residual layer normalization of the output of the self-attention sublayer and the output of the feedforward neural network. is the output of the feedforward neural network.

[0031] After being processed by L layers of transformer layers, the last hidden state is output: , is the last hidden state matrix. Next, we extract the embedding of the “[CLS]” token, ,in is the embedding representation of the entire sentence, that is, the first sentence embedding representation.

[0032] Optionally, in S200, the bidirectional xLSTM deep semantic information extraction module includes an xLSTM structure formed by stacking a plurality of xLSTM blocks, each xLSTM block including one of an sLSTM and an mLSTM; sLSTM (scalar LSTM): introduces an exponential gating mechanism into the long short-term memory network to change the update method of the memory unit and is used to process sequence data of different time steps; mLSTM (matrix LSTM): achieves parallel computing and memory processing capabilities through matrix memory cell states and covariance update rules; The xLSTM structure includes at least one xLSTM block including an sLSTM and at least one xLSTM block including an mLSTM.

[0033] It should be noted that the exponential gating mechanism of sLSTM allows the model to dynamically adjust the state of the memory unit at each time step, thereby better capturing short-term and long-term dependencies. The matrix memory structure and covariance update rule of mLSTM enable the model to handle more complex dependencies while maintaining efficient computing performance, complementing the deficiencies of the existing BERT model.

[0034] In addition, compared with the existing one-layer xLSTM structure, the stacked two-layer xLSTM structure in this embodiment can further enhance the representation learning ability of the model. The forward xLSTM structure and the backward xLSTM structure process the forward and reverse sentence embedding sequences respectively, which can better integrate the deep semantic information in the context.

[0035] Furthermore, the forward propagation process of the xLSTM block containing sLSTM is as follows: After BERT processing, the embedding representation of the entire sentence is obtained ,make Sentence Embedding , the memory state of the previous moment , the regularized state at the previous moment and the hidden state at the previous moment First, it is fed into an xLSTM block containing an sLSTM: ; ; ; ; ; ; ; in, represents part of the current state of the input gate, Another part representing the current state of the input gate; Indicates the memory status at the current moment. Indicates the normalized state at the current moment; Indicates the hidden state at the current moment; Represents the intermediate vector generated by combining the input vector and the hidden state; Indicates the current state of the output gate, which is used to control the output ratio of the hidden state; Represents the input vector The weight coefficient of Represents the gain term, which is used to adjust the impact of the previous hidden state on the current state. Represents the bias term, which is used for bias adjustment. It should be noted that: , , The lower right corner mark , , and They are used to distinguish the intermediate vector generated by combining the input vector and the hidden state, a part of the current state of the input gate, another part of the current state of the input gate, and the current state of the output gate; Exponential activation functions lead to larger values ​​through the extra state To constrain to stabilize the gating: ; ; ; in, represents a portion of the recomputed input gate state, Represents the recomputed portion of the input gate state.

[0036] Furthermore, the forward propagation process of the xLSTM block containing mLSTM is as follows: The sentence embedding is processed by the xLSTM block containing sLSTM to obtain the new memory state , the new regularized state , the new hidden state The new hidden layer state and sentence embedding Residual connections form new text representations These new states , and text representation It will be regarded as the state of the previous moment and input into the xLSTM block containing mLSTM: ; ; ; ; ; ; ; ; ; in, Indicates the scaling factor to avoid excessive numerical fluctuations; Represents input The weight parameter, Represents the bias term, which is used for bias adjustment. It should be noted that: and The lower right corner mark , , , , and They are used to distinguish the input gate, the forget gate, the output gate, a part of the current state of the input gate, another part of the current state of the input gate, and the current state of the output gate.

[0037] It should be noted that this xLSTM block outputs will be the same as its input Perform residual connection to form the output of the final forward xLSTM structure Then, the backward xLSTM structure processes the reverse sentence embedding, and the calculation process is the same as the forward xLSTM. and reverse Concatenate element by element into a vector , i.e., the second sentence embedding representation, is used to perform subsequent sentiment classification tasks.

[0038] Optionally, in S300, the sentiment analysis task is performed using the second sentence embedding representation, and the process is as follows: The second sentence embedding represents the input random inactivation layer that randomly discards a part of neurons; The normalization layer is used to standardize the output of the random dropout layer; The mean layer is used to average the data; A fully connected layer is used to process the output of the mean layer to map the high-dimensional text representation to the dimension of the target category to achieve sentiment classification.

[0039] Optionally, the loss function calculation process for the sentiment classification task is as follows: Convert sentiment classification results into binary matrix form , where each row represents a sample , each column represents a category , the expression is as follows: ; For each example, extract the output value of the correct category , the expression is as follows: ; Calculate the final loss , the expression is as follows: ; ; in, represents the number of samples, represents the hyperparameter, Represents the loss value of each sample, and the final loss is the mean of all sample loss values. Indicates that it is The prediction scores of samples on other categories.

[0040] In addition, this embodiment also discloses a computer device, including a memory and a processor; The memory is used to store a computer program executable on the processor; The processor is used to implement the steps of the above-mentioned sentiment analysis method when executing the computer program.

[0041] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in the memory and executed by the processor to implement the present invention. The one or more modules / units may be a series of computer program instruction segments capable of implementing specific functions, which are used to describe the execution process of the computer program in the computer device.

[0042] The computer device may be a computing device such as a mobile phone, a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may include, but is not limited to, a processor and a memory. For example, the computer device may also include an input / output device, a network access device, a bus, etc.

[0043] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the computer device, and uses various interfaces and lines to connect various parts of the entire computer device.

[0044] The memory can be used to store the computer program and / or module, and the processor implements the computer program by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0045] Wherein, if the module / unit integrated in the computer device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0046] In addition, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above-mentioned sentiment analysis method are implemented.

[0047] The embodiment of the present invention discloses a sentiment analysis method based on BERT and xLSTM. In the method of the present invention, the method forms a text feature extraction model by integrating BERT and two layers of xLSTM, wherein BERT is pre-trained using a contrastive self-supervision method, defines sentence-level task objectives, and captures sentence-level semantics; the two layers of xLSTM respectively focus on the forward and backward sentence embedding representations processed by BERT, comprehensively capture deep semantic information in the text, and use the high-quality sentence-level text representation extracted by the above model to perform sentiment analysis, thereby significantly improving the accuracy of sentence-level sentiment analysis.

[0048] It should be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0049] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. The sentiment analysis method based on BERT and xLSTM is characterized by: The steps include: S100: construct a text feature extraction model, wherein the text feature extraction model includes a BERT sentence embedding extraction module and a bidirectional xLSTM deep semantic information extraction module; S200: extracting sentence-level semantic information in the long text to be analyzed through the BERT sentence embedding extraction module to obtain a first sentence embedding representation, and using a bidirectional xLSTM deep semantic information extraction module to focus on the forward and backward sentence embedding representations respectively to obtain a second sentence embedding representation; S300: Using the second sentence embedding representation to perform a sentiment analysis task and output a sentiment category.

2. The sentiment analysis method according to claim 1, characterized in that: In S100, the BERT sentence embedding extraction module is trained using a contrastive self-supervised learning pre-training module, and the process is as follows: Prepare positive and negative pairs for pre-training; Initialize the parameters of the BERT model; Input the positive and negative pairs into the BERT model to obtain the embedded representation; Calculate the cosine similarity of each positive and negative pair; Input the cosine similarity into the contrastive loss function and calculate the loss for all positive and negative pairs; The gradient is calculated through back propagation and the BERT model parameters are updated.

3. The sentiment analysis method according to claim 2, characterized in that: In S200, the BERT sentence embedding extraction module includes an embedding layer, a Transformer structure and a hidden layer, and the Transformer structure includes a self-attention mechanism, a feedforward neural network, a residual connection and layer normalization; in the self-attention mechanism, the attention score is calculated through a multi-head attention layer.

4. The sentiment analysis method according to claim 3, characterized in that: In S200, the sentence-level semantic information in the text is extracted by the BERT sentence embedding extraction module to obtain the first sentence embedding representation, and the process is as follows: Use embedding layer to calculate word embedding sequence; Input the word embedding sequence into the multi-head attention layer to obtain the attention score; The word embedding sequence is processed using attention scores to obtain the first sentence embedding representation.

5. The sentiment analysis method according to claim 4, characterized in that: In S200, the bidirectional xLSTM deep semantic information extraction module includes an xLSTM structure formed by stacking a plurality of xLSTM blocks, each of which includes one of an sLSTM and an mLSTM; sLSTM: Introduces an exponential gating mechanism into the long short-term memory network to change the update method of the memory unit and is used to process sequence data of different time steps; mLSTM: Achieve parallel computing and memory processing capabilities through matrix memory cell states and covariance update rules; The xLSTM structure includes at least one xLSTM block including an sLSTM and at least one xLSTM block including an mLSTM.

6. The sentiment analysis method according to claim 5, characterized in that: In S300, the sentiment analysis task is performed using the second sentence embedding representation, and the process is as follows: The second sentence embedding represents the input random inactivation layer that randomly discards a part of neurons; The normalization layer is used to standardize the output of the random dropout layer; The mean layer is used to average the data; A fully connected layer is used to process the output of the mean layer to map the high-dimensional text representation to the dimension of the target category to achieve sentiment classification.

7. The sentiment analysis method according to claim 6, characterized in that: The loss function calculation process for the sentiment classification task is as follows: Convert sentiment classification results into binary matrix form , where each row represents a sample , each column represents a category , the expression is as follows: ； For each example, extract the output value of the correct category , the expression is as follows: ； Calculate the final loss , the expression is as follows: ； ； in, represents the number of samples, represents the hyperparameter, Represents the loss value of each sample, and the final loss is the mean of all sample loss values. Indicates that it is The prediction scores of samples on other categories.

8. A computer device, characterized in that: including memory and processor; The memory is used to store a computer program executable on the processor; The processor is used to implement the steps of the sentiment analysis method according to any one of claims 1 to 7 when executing the computer program.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the sentiment analysis method according to any one of claims 1 to 7 are implemented.

Images