Multi-mode electric power customer emotion analysis method and system based on text and voice

Abstract

The invention discloses a text and voice-based multi-modal electric power customer emotion analysis method and system, and the method comprises the steps: collecting customer service text data and customer service voice data, and carrying out the preprocessing; inputting the preprocessed customer service text data into a BERT-based text emotion recognition model to extract features of a text data modal; inputting the preprocessed customer service voice data into a voice emotion recognition model based on Transform, and extracting features of a voice data modal by using a multi-head self-attention mechanism; based on a cross-modal attention mechanism, performing multi-modal fusion on the features of the text and voice data modals to realize alignment and splicing of the features of the text and voice data modals; and marking the emotion polarity and emotion intensity output by the classification model through a conditional random field (CRF) layer according to the aligned and spliced features of the text and voice data modes. According to the method, the emotion can be effectively labeled, so that the emotion tendency can be more accurately judged.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing technology and deep learning, and in particular to a text- and voice-based multimodal power customer sentiment analysis method and system. Background Art

[0002] High-quality power supply services are the lifeline of power grid companies. The goal of State Grid Corporation of China is to provide services to all power users and meet the diverse power needs of customers. With the continuous enrichment of power demand, the number of customer demands has increased year by year, and service risk prevention and control is facing tremendous pressure. In recent years, the construction of new power systems has been deepened, and the increase in customer demands has reached an average annual rate of 20.48% in five years. The coordinated monitoring capabilities of service risks in various channels need to be improved urgently. In order to improve the service level of "getting power", State Grid Corporation of China launched nine measures in 2020 to provide a more worry-free power application process, more convenient services and more reliable power supply guarantees. These measures not only strengthen the standardization of power supply services, but also comprehensively improve customers' power consumption experience. In 2021, State Grid Corporation of China formulated the "Excellent Service Improvement Project Action Plan", which clearly requires enhancing the power supply service guarantee capabilities, continuously optimizing the power business environment, and striving to achieve the excellent service goals of "modernization, digitalization, leanness" and "high quality, high efficiency, and high quality" by 2025. In 2023, State Grid Corporation of China launched a three-year action plan to further improve the power business environment, emphasizing the acceleration of intelligent transformation and upgrading of business halls and strengthening operation management. At the same time, it explored the application of artificial intelligence technology for automatic inspections and analysis of audio and video information in business halls to assist in service anomaly monitoring and intelligent early warning, so as to better meet the needs of a wide range of power customers.

[0003] The current customer service work has problems such as the difficulty in accurately analyzing customer sentiment, which is a factor with strong risk correlation. This is the core problem facing customer service risks.

[0004] CN108319666A discloses a power supply service evaluation method based on multimodal public opinion analysis, which relates to a power supply service evaluation method. The following steps are included: speech feature emotion recognition based on the method of increasing and decreasing components and long short-term memory network; text appeal classification based on word2vec and LSTM. This technical solution extracts features from speech and combines them with text features to effectively improve the accuracy of the model's emotion recognition; after word segmentation and feature extraction of the text, the model is combined with the feature information of the speech to better mine useful information in the text appeal, and a deep recurrent neural network with an attention mechanism is used to pay attention to different keywords at different times, so that the model can better capture the semantic relationship between contexts and mine the timing information and semantic information in the target text. However, the invention only evaluates the text data, and does not evaluate the equally important speech data, and the patent only uses LSTM for classification, which is insufficient to characterize the timing capability and the information of the extracted text features combined with the context. Summary of the invention

[0005] In order to solve the deficiencies in the prior art, the present invention provides a multimodal power customer sentiment analysis method and system based on text and voice; by analyzing the tone and intensity of the voice and the vocabulary used in the text, the emotional state of the power customer can be more accurately identified, thereby providing the power customer with better personalized services.

[0006] The present invention adopts the following technical solution.

[0007] The first aspect of the present invention provides a multimodal power customer sentiment analysis method based on text and speech, comprising the following contents:

[0008] Collect customer service text data and customer service voice data and perform preprocessing; the customer service text data includes social media customer interaction information, customer service conversation texts and user comments; the customer service voice data includes online calls, voice service records and simulated conversation scenario records;

[0009] Input the preprocessed customer service text data into the BERT-based text sentiment recognition model to extract the features of the text data modality;

[0010] The pre-processed customer service voice data is input into the Transformer-based speech emotion recognition model and the attention mechanism is used to extract the features of the speech data modality.

[0011] Perform weighted fusion of features of text and speech data modalities based on the attention mechanism;

[0012] The features of the fused text and speech data modalities are used to annotate the sentiment tendency of the classification model output through the conditional random field (CRF) layer.

[0013] Preferably, the preprocessing is specifically:

[0014] Deleting or replacing sensitive fields set in customer service text data with set symbols; the sensitive fields include the identity information of the power customer;

[0015] Take the same number of samples from the customer service text data and the customer service voice data;

[0016] The emotional tendency of each sample is annotated as a label, and the annotated emotional tendency includes emotional polarity and emotional intensity. The emotional polarity includes positive emotion, negative emotion and neutral emotion, and the emotional intensity includes strong, general and objective.

[0017] Preferably, the BERT-based text emotion recognition model is specifically:

[0018] The BERT model is used to segment the preprocessed customer service text data, and each segmented character and word is converted into a vector that integrates context information; the vector is input into the CNN structure based on the attention mechanism to extract local sequence features; the local sequence features are then input into the LSTM dual-channel model to integrate the text data features from the perspectives of characters and words to obtain the final text sequence features.

[0019] Preferably, the text data features are fused from the perspectives of characters and words to obtain the final text sequence features, specifically:

[0020] The LSTM dual-channel model extracts character-level and word-level context features H through the BiLSTM network. char and H word :

[0021] H char ={h c1 ,h c2 ,...,h ct ,...,h cn}

[0022] H word ={h w1 ,h w2 ,...,h wi ,...,h wm}

[0023] Among them, h ct 、h wi They represent the features corresponding to the proposed t-th character and i-th word, respectively, and n and m are the total number of characters and words segmented respectively;

[0024] Calculate the attention weights α for characters and words respectively tand β i :

[0025]

[0026] Among them, score(h wi ,q)、score(h ct ,q) represent the h calculated by the LSTM dual-channel model ct 、h wi The attention score of q is:

[0027] The character-level and word-level features are weighted, and the final text sequence features are:

[0028]

[0029] where λ is a tunable parameter.

[0030] Preferably, the attention score is specifically:

[0031] The LSTM two-channel model calculates the score through an additive attention network:

[0032] score(h φ ,q)=v T tanh(W[h φ ;q]+b)

[0033] Among them, h φ =[h ct ,h wi ]; W represents the learnable weight matrix of the corresponding character or word; v represents the learnable weight vector of the corresponding character or word; [; ] represents the vector concatenation operation; b represents the bias term of the corresponding character or word; tanh represents the nonlinear activation function.

[0034] Preferably, the pre-processed customer service voice data is input into a Transformer-based voice emotion recognition model to extract the features of the voice data modality using an attention mechanism, specifically:

[0035] The vector input into the Transformer-based speech emotion recognition model is positionally embedded, and the position embedding is a vector that embeds the position information; then multiple linear transformations are performed through the linear layer, and the results of the linear transformations are used to calculate the attention weights and weighted sums on each head network using the attention mechanism, and finally the results of these weighted sums are concatenated and linearly transformed again through the output layer to obtain the final output.

[0036] Preferably, the attention mechanism adopts a multi-head attention mechanism, specifically:

[0037] head j =Attention(QW j Q , KW j K , VW j V )

[0038] MultiHead(Q,K,V)=Concat(head1,...,head h )

[0039] head j is the j-th single-head attention mechanism; Attention(QW j Q , KW j K , VW j V ) is the attention mechanism; QW j Q , KW j K 、VW j V are the query projection matrix, key projection matrix, and value projection matrix of the jth head respectively; MultiHead(Q, K, V) is a multi-head attention mechanism; Concat(head1, ..., head h ) is to concatenate the outputs of h heads together; h is the number of attention heads; Q, K, and V are the query matrix, key matrix, and value matrix, respectively.

[0040] Preferably, the features of the fused text and speech data modalities are labeled with the sentiment tendency output by the conditional random field CRF layer classification model; the training of the conditional random field CRF layer classification model is specifically as follows:

[0041] Training the feature function weight parameters of the conditional random field (CRF) layer annotation classification model The maximum likelihood estimation method is used for training; the features of the fused text and speech data modalities are combined into an observation sequence X, and the sentiment tendency corresponding to the observation sequence X is randomly generated as the output identification sequence Y. A group of samples is separated from the observation sequence X and the output identification sequence Y as a training set. The likelihood function of this training set is:

[0042]

[0043] Among them, x (k) is the kth sample in the training set separated from the observation sequence X, y (k)The kth sample in the training set separated by the output identification sequence Y, k = 1, 2, ..., N, N is the total number of training set data, find the corresponding value that maximizes the likelihood function.

[0044] Preferably, the obtaining of the corresponding Value, so that the likelihood function is maximized, specifically:

[0045] The iterative optimization method is used to obtain the first-order partial derivative of the likelihood function of the training data:

[0046]

[0047] Where fi(y, x) represents the characteristic function of the data in the observation sequence X and the data in the output identification sequence Y, E P(y，x) [f i (y, x)] is the empirical expected value of the feature function fi(y, x) on the entire training data set. is the model expectation of the characteristic function fi(y, x), is the derivative of the Gaussian smoothing factor, λ i is the parameter in the maximum likelihood estimation, σ 2 is the regularization parameter.

[0048] The second aspect of the present invention proposes a text- and voice-based multimodal power customer sentiment analysis system using the analysis method described in the first aspect of the present invention, comprising: an acquisition module, a text data feature extraction module, a voice data feature extraction module, a feature fusion module, and a sentiment tendency analysis module, characterized in that:

[0049] Collection module: collects customer service text data and customer service voice data and performs preprocessing; the customer service text data includes social media customer interaction information, customer service conversation texts and user comments; the customer service voice data includes online calls, voice service records and simulated conversation scenario records;

[0050] Text data feature extraction module: input the preprocessed customer service text data into the BERT-based text sentiment recognition model to extract the features of the text data modality;

[0051] Speech data feature extraction module: input the preprocessed customer service speech data into the Transformer-based speech emotion recognition model and use the attention mechanism to extract the features of the speech data modality;

[0052] Feature fusion module: weighted fusion of features of text and speech data modalities based on the attention mechanism;

[0053] Sentiment tendency analysis module: The features of the fused text and voice data modalities are used to annotate the sentiment tendency output by the classification model through the conditional random field (CRF) layer.

[0054] The beneficial effect of the present invention is that, compared with the prior art,

[0055] 1. After the preprocessing of customer service text data is completed, the present invention performs word segmentation and word feature extraction on the preprocessed customer service text data based on the pretrained model BERT, and then uses the CNN structure based on the attention mechanism to more comprehensively obtain temporal context information and learn the importance of local sequence features. In addition, using the entity recognition model LSTM of word boundary features in the text, a model structure based on LSTM dual-channel word feature fusion is proposed, and the attention score calculation of the relevance between the word and the matching word is further performed to optimize the extraction and matching capabilities of word feature information.

[0056] 2. After the preprocessing of the customer service voice data is completed, the present invention realizes voice modal feature extraction based on the Transformer's multi-head self-attention mechanism. The multi-head self-attention mechanism processes multiple attention heads in parallel, and each head can focus on different parts and different features of the input data, thereby capturing multi-dimensional information of the voice signal. This mechanism allows the model to focus on different features of the voice signal at the same time, such as pitch, rhythm, and intensity, which helps to better understand and process voice data.

[0057] 3. The present invention uses the conditional random field (CRF) layer to label and classify the sentiment polarity and sentiment intensity of the model's final output. The conditional random field can consider the contextual relationship in the sequence data, and as a sequence model, it is good at handling sequence labeling tasks and can effectively label emotions, thereby more accurately judging the sentiment tendency. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 This is a flow chart of a text- and voice-based multimodal power customer sentiment analysis system in an embodiment of the present invention;

[0059] Figure 2 It is a basic structure diagram of CNN based on attention mechanism in an embodiment of the present invention;

[0060] Figure 3 This is a structural diagram of the LSTM dual-channel word feature fusion model in an embodiment of the present invention;

[0061] Figure 4 This is a structural diagram of a Transformer-based multi-head attention mechanism in an embodiment of the present invention;

[0062] Figure 5 4 is a chain structure diagram of CRF in an embodiment of the present invention. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical scheme and advantages of the present invention clearer, the technical scheme of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. The embodiments described in this application are only embodiments of a part of the present invention, rather than all embodiments. Based on the spirit of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work belong to the protection scope of the present invention.

[0064] like Figure 1 As shown, embodiment 1 of the present invention provides a multimodal power customer sentiment analysis method based on text and voice, including the following contents:

[0065] Collect customer service text data and customer service voice data and perform preprocessing; the customer service text data includes social media customer interaction information, customer service conversation texts and user comments; the customer service voice data includes online calls, voice service records and simulated conversation scenario records;

[0066] Input the preprocessed customer service text data into the BERT-based text sentiment recognition model to extract the features of the text data modality;

[0067] The pre-processed customer service voice data is input into the Transformer-based speech emotion recognition model and the attention mechanism is used to extract the features of the speech data modality.

[0068] Perform weighted fusion of features of text and speech data modalities based on the attention mechanism;

[0069] The features of the fused text and speech data modalities are used to annotate the sentiment tendency of the classification model output through the conditional random field (CRF) layer.

[0070] Preferably, the preprocessing is specifically:

[0071] Deleting or replacing sensitive fields set in customer service text data with set symbols; the sensitive fields include the identity information of the power customer;

[0072] Take the same number of samples from the customer service text data and the customer service voice data;

[0073] The emotional tendency of each sample is annotated as a label, and the annotated emotional tendency includes emotional polarity and emotional intensity. The emotional polarity includes positive emotion, negative emotion and neutral emotion, and the emotional intensity includes strong, general and objective.

[0074] Preferably, if Figure 2As shown, the BERT-based text emotion recognition model is specifically:

[0075] The BERT model is used to segment the preprocessed customer service text data, and each segmented character and word is converted into a vector that integrates context information; the vector is input into the CNN structure based on the attention mechanism to extract local sequence features; the local sequence features are then input into the LSTM dual-channel model to integrate the text data features from the perspectives of characters and words to obtain the final text sequence features.

[0076] It should be noted that the CNN structure based on the attention mechanism is specifically based on the parallel standard CNN module and the attention mechanism module. The multi-scale input method of the standard CNN module and the attention mechanism module with sequences of different lengths as input can effectively extract richer short sequence features. The attention mechanism module designed in this embodiment has a larger input scale than the standard CNN module to expand the input receptive field, thereby more comprehensively acquiring temporal context information and learning the importance of local sequence features.

[0077] Preferably, the text data features are fused from the perspectives of characters and words to obtain the final text sequence features, specifically:

[0078] like Figure 3 As shown in Figure 2, the LSTM dual-channel model extracts character-level and word-level context features H through the BiLSTM network. char and H word :

[0079] H char ={h c1 ,h c2 ,...,h ct ,...,h cn}

[0080] H word ={h w1 ,h w2 ,...,h wi ,...,h wm}

[0081] Among them, h ct 、h wi They represent the features corresponding to the proposed t-th character and i-th word, respectively, and n and m are the total number of characters and words segmented respectively;

[0082] Calculate the attention weights α for characters and words respectively t and β i :

[0083]

[0084] Among them, score(hwi ,q)、score(h ct ,q) represent the h calculated by the LSTM dual-channel model ct 、h wi The attention score of q is:

[0085] The character-level and word-level features are weighted, and the final text sequence features are:

[0086]

[0087] where λ is a tunable parameter.

[0088] Preferably, the attention score is specifically:

[0089] The LSTM two-channel model calculates the score through an additive attention network:

[0090] score(h φ ,q)=v T tanh(W[h φ ;q]+b)

[0091] Among them, h φ =[h ct ,h wi ]; W represents the learnable weight matrix of the corresponding character or word; v represents the learnable weight vector of the corresponding character or word; [; ] represents the vector concatenation operation; b represents the bias term of the corresponding character or word; tanh represents the nonlinear activation function.

[0092] Preferably, the pre-processed customer service voice data is input into a Transformer-based voice emotion recognition model to extract the features of the voice data modality using an attention mechanism, specifically:

[0093] The vector input into the Transformer-based speech emotion recognition model is positionally embedded, and the position embedding is a vector that embeds the position information; then multiple linear transformations are performed through the linear layer, and the results of the linear transformations are used to calculate the attention weights and weighted sums on each head network using the attention mechanism, and finally the results of these weighted sums are concatenated and linearly transformed again through the output layer to obtain the final output.

[0094] Preferably, if Figure 4 As shown, the attention mechanism adopts a multi-head attention mechanism, specifically:

[0095] head j =Attention(QW j Q , KW jK , VW j V )

[0096] MultiHead(Q,K,V)=Concat(head1,...,head h )

[0097] head j is the j-th single-head attention mechanism; Attention(QW j Q , KW j K , VW j V ) is the attention mechanism; QW j Q , KW j K 、VW j V are the query projection matrix, key projection matrix, and value projection matrix of the jth head respectively; MultiHead(Q, K, V) is a multi-head attention mechanism; Concat(head1, ..., head h ) is to concatenate the outputs of h heads together; h is the number of attention heads; Q, K, and V are the query matrix, key matrix, and value matrix, respectively.

[0098] Preferably, the features of the fused text and speech data modalities are labeled with the sentiment tendency output by the conditional random field CRF layer classification model; the training of the conditional random field CRF layer classification model is specifically as follows:

[0099] Training the feature function weight parameters of the conditional random field (CRF) layer annotation classification model The maximum likelihood estimation method is used for training; the features of the fused text and speech data modalities are combined into an observation sequence X, and the sentiment tendency corresponding to the observation sequence X is randomly generated as the output identification sequence Y. A group of samples is separated from the observation sequence X and the output identification sequence Y as a training set. The likelihood function of this training set is:

[0100]

[0101] Among them, x (k) is the kth sample in the training set separated from the observation sequence X, y (k) The kth sample in the training set separated by the output identification sequence Y, k = 1, 2, ..., N, N is the total number of training set data, find the corresponding value that maximizes the likelihood function.

[0102] Preferably, the obtaining of the corresponding Value, so that the likelihood function is maximized, specifically:

[0103] The iterative optimization method is used to obtain the first-order partial derivative of the likelihood function of the training data:

[0104]

[0105] Among them, fi(y,x) represents the characteristic function of the data in the observation sequence X and the data in the output identification sequence Y, E P(y,x) [f i (y,x)] is the empirical expected value of the feature function fi(y,x) on the entire training data set. is the model expectation of the characteristic function fi(y,x), is the derivative of the Gaussian smoothing factor, λ i is the parameter in the maximum likelihood estimation, σ 2 is the regularization parameter.

[0106] It should be noted that the conditional random field CRF layer annotation classification model is specifically as follows: Figure 5 As shown in Figure 1, an undirected graph G = (V, E) is defined, where a point v∈V in the undirected graph corresponds to a random variable y∈Y. The observation sequence X and the output identification sequence Y are used for modeling, and the conditional probability P(Y|X) of the two is the description of the specific conditional random field model.

[0107] It should be noted that the total number of samples of customer service text data and customer service voice data in this embodiment, the ratio of training set, verification set and test set is 7:2:1. The number of samples of customer service text data and customer service voice data collected in this embodiment, as well as the number of sentiment polarity and sentiment intensity are shown in Table 1.

[0108] Table 1 The number of samples of customer service text data and customer service voice data and the number of sentiment polarity and sentiment intensity

[0109]

[0110] On the same data set, other feature extraction and fusion methods are used in comparison with the method of this embodiment, and the comparison on the same performance indicators is shown in Table 2.

[0111] Table 2 Accuracy and F1 score of other methods and the method of this embodiment

[0112]

[0113]

[0114] This shows that the feature extraction and fusion method adopted in this embodiment has high accuracy and effectiveness.

[0115] Embodiment 2 of the present invention proposes a text- and voice-based multimodal power customer sentiment analysis system using the analysis method described in Embodiment 1 of the present invention, comprising: a collection module, a text data feature extraction module, a voice data feature extraction module, a feature fusion module, and a sentiment tendency analysis module, characterized in that:

[0116] Collection module: collects customer service text data and customer service voice data and performs preprocessing; the customer service text data includes social media customer interaction information, customer service conversation texts and user comments; the customer service voice data includes online calls, voice service records and simulated conversation scenario records;

[0117] Text data feature extraction module: input the preprocessed customer service text data into the BERT-based text sentiment recognition model to extract the features of the text data modality;

[0118] Speech data feature extraction module: input the preprocessed customer service speech data into the Transformer-based speech emotion recognition model and use the attention mechanism to extract the features of the speech data modality;

[0119] Feature fusion module: weighted fusion of features of text and speech data modalities based on the attention mechanism;

[0120] Sentiment tendency analysis module: The features of the fused text and voice data modalities are used to annotate the sentiment tendency output by the classification model through the conditional random field (CRF) layer.

[0121] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0122] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0123] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0124] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0125] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents, and any modifications or equivalent replacements that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A multimodal power customer sentiment analysis method based on text and speech, characterized in that: The steps include: Collect customer service text data and customer service voice data and perform preprocessing; the customer service text data includes social media customer interaction information, customer service conversation texts and user comments; the customer service voice data includes online calls, voice service records and simulated conversation scenario records; Input the preprocessed customer service text data into the BERT-based text sentiment recognition model to extract the features of the text data modality; The pre-processed customer service voice data is input into the Transformer-based speech emotion recognition model and the attention mechanism is used to extract the features of the speech data modality. Perform weighted fusion of features of text and speech data modalities based on the attention mechanism; The features of the fused text and speech data modalities are used to annotate the sentiment tendency of the classification model output through the conditional random field (CRF) layer.

2. According to claim 1, a multi-modal power customer sentiment analysis method based on text and voice is characterized by: The pre-processing is specifically as follows: Deleting or replacing sensitive fields set in customer service text data with set symbols; the sensitive fields include the identity information of the power customer; Take the same number of samples from the customer service text data and the customer service voice data; The emotional tendency of each sample is annotated as a label, and the annotated emotional tendency includes emotional polarity and emotional intensity. The emotional polarity includes positive emotion, negative emotion and neutral emotion, and the emotional intensity includes strong, general and objective.

3. The method for analyzing the sentiment of electric power customers based on text and speech according to claim 1 is characterized by: The BERT-based text sentiment recognition model is specifically: The BERT model is used to segment the preprocessed customer service text data, and each segmented character and word is converted into a vector that integrates context information; the vector is input into the CNN structure based on the attention mechanism to extract local sequence features; the local sequence features are then input into the LSTM dual-channel model to integrate the text data features from the perspectives of characters and words to obtain the final text sequence features.

4. The method for analyzing the sentiment of electric power customers based on text and speech according to claim 3 is characterized in that: The text data features are integrated from the perspectives of characters and words to obtain the final text sequence features, specifically: The LSTM dual-channel model extracts character-level and word-level context features H through the BiLSTM network. char and H word : H char ={h c1 ,h c2 ,...,h ct ,...,h cn } H word ={h w1 ,h w2 ,...,h wi ,...,h wm } Among them, h ct 、h wi They represent the features corresponding to the proposed t-th character and i-th word, respectively, and n and m are the total number of characters and words segmented respectively; Calculate the attention weights α for characters and words respectively t and β i : Among them, score(h wi ,q)、score(h ct ,q) represent the h calculated by the LSTM dual-channel model ct 、h wi The attention score of q is: The character-level and word-level features are weighted, and the final text sequence features are: where λ is a tunable parameter.

5. The method for analyzing the multimodal power customer sentiment based on text and speech according to claim 4 is characterized in that: The attention score is specifically: The LSTM two-channel model calculates the score through an additive attention network: score(h φ ,q)=v T tanh(W[h φ ;q]+b) Among them, h φ =[h ct ,h wi ]; W represents the learnable weight matrix of the corresponding character or word; v represents the learnable weight vector of the corresponding character or word; [; ] represents the vector concatenation operation; b represents the bias term of the corresponding character or word; tanh represents the nonlinear activation function.

6. The method for analyzing the sentiment of electric power customers based on text and speech according to claim 1 is characterized by: The pre-processed customer service voice data is input into the Transformer-based voice emotion recognition model to extract the features of the voice data modality using the attention mechanism, specifically: The vector input into the Transformer-based speech emotion recognition model is positionally embedded, and the position embedding is a vector that embeds the position information; then multiple linear transformations are performed through the linear layer, and the results of the linear transformations are used to calculate the attention weights and weighted sums on each head network using the attention mechanism, and finally the results of these weighted sums are concatenated and linearly transformed again through the output layer to obtain the final output.

7. The method for analyzing the sentiment of electric power customers based on text and speech according to claim 6 is characterized by: The attention mechanism adopts a multi-head attention mechanism, specifically: head j =Attention(QW j Q ,KW j W ,VW j V ) MultiHead(Q,K,V)=Concat(head1,…,head h ) head j is the j-th single-head attention mechanism; Attention(QW j Q ,KW j K ,VW j V ) is the attention mechanism; QW j Q , KW j K 、VW j V are the query projection matrix, key projection matrix, and value projection matrix of the jth head respectively; MultiHead(Q,K,V) is a multi-head attention mechanism; Concat(head1,…,head h ) is to concatenate the outputs of h heads together; h is the number of attention heads; Q, K, and V are the query matrix, key matrix, and value matrix, respectively.

8. The method for analyzing the sentiment of electric power customers based on text and speech according to claim 7 is characterized by: The features of the fused text and speech data modalities are used to output the sentiment tendency of the conditional random field CRF layer annotation classification model; the training of the conditional random field CRF layer annotation classification model is specifically as follows: Training the feature function weight parameters of the conditional random field (CRF) layer annotation classification model The maximum likelihood estimation method is used for training; the features of the fused text and speech data modalities are combined into an observation sequence X, and the sentiment tendency corresponding to the observation sequence X is randomly generated as the output identification sequence Y. A group of samples is separated from the observation sequence X and the output identification sequence Y as a training set. The likelihood function of this training set is: Among them, x (k) is the kth sample in the training set separated from the observation sequence X, y (k) The kth sample in the training set separated by the output identification sequence Y, k = 1, 2, ..., N, N is the total number of training set data, find the corresponding value that maximizes the likelihood function.

9. A text- and voice-based multimodal power customer sentiment analysis method according to claim 8, characterized in that The corresponding Value, so that the likelihood function is maximized, specifically: The iterative optimization method is used to obtain the first-order partial derivative of the likelihood function of the training data: Among them, fi(y,x) represents the characteristic function of the data in the observation sequence X and the data in the output identification sequence Y, E P(y,x) [f i (y,x)] is the empirical expected value of the feature function fi(y,x) on the entire training data set. is the model expectation of the characteristic function fi(y,x), is the derivative of the Gaussian smoothing factor, λ i is the parameter in the maximum likelihood estimation, σ 2 is the regularization parameter.

10. A text- and voice-based multimodal power customer sentiment analysis system using the analysis method according to any one of claims 1 to 9, comprising: The acquisition module, the text data feature extraction module, the voice data feature extraction module, the feature fusion module, and the sentiment tendency analysis module are characterized by: Collection module: collects customer service text data and customer service voice data and performs preprocessing; the customer service text data includes social media customer interaction information, customer service conversation texts and user comments; the customer service voice data includes online calls, voice service records and simulated conversation scenario records; Text data feature extraction module: input the preprocessed customer service text data into the BERT-based text sentiment recognition model to extract the features of the text data modality; Speech data feature extraction module: input the preprocessed customer service speech data into the Transformer-based speech emotion recognition model and use the attention mechanism to extract the features of the speech data modality; Feature fusion module: weighted fusion of features of text and speech data modalities based on the attention mechanism; Sentiment tendency analysis module: The features of the fused text and voice data modalities are used to annotate the sentiment tendency output by the classification model through the conditional random field (CRF) layer.

Images