A Hierarchical Control Method for Power Customer Service Early Warning Based on Multimodal Risk Identification

Abstract

This invention discloses a graded control method for early warning of power customer service based on multimodal risk identification, belonging to the field of customer service risk early warning technology. The method includes: collecting real-time voice data from power customer service and extracting voiceprint features; determining new speakers online and generating new class center vectors based on the distance metric between voiceprint features and historical voiceprint class centers, and performing dynamic role separation; extracting voiceprint features and text semantic features from the voice segments separated by roles, calculating enhanced voiceprint features, and combining them with text semantic features to generate a multimodal fusion feature vector, which is then sparsified to generate potential risk features; pre-training the potential risk features on unlabeled voice data and fine-tuning them with labeled data from the power customer service scenario to generate a robust risk representation vector; and determining the risk type and level, outputting a graded early warning control strategy. This invention solves the problems of low accuracy in risk identification, difficulty in role differentiation, and insufficient real-time and graded control strategies.

Description

Technical Field

[0001] This invention relates to the field of customer service risk warning and control technology, and more specifically, to a power customer service early warning and hierarchical control method based on multimodal risk identification. Background Technology

[0002] With the rapid development of the power industry, power customer service systems are playing an increasingly important role in daily operation and maintenance, customer service, and emergency response. These systems not only handle a large volume of customer inquiries, fault reports, and complaints, but also face complex risk scenarios, such as customer mood swings, unauthorized operations, and system anomalies. If these risks are not identified and effectively controlled in a timely manner, they may lead to a decline in service quality, an increase in customer complaints, and even affect the stable operation of the power system.

[0003] Traditional power customer service systems rely on customer service personnel or managers for real-time monitoring and risk assessment. While manual methods offer some flexibility in handling complex emotions and abnormal events, they suffer from problems such as large response delays, limited monitoring scope, and highly subjective judgment criteria, failing to achieve efficient and real-time risk control. Furthermore, recent research has attempted to utilize machine learning models for risk prediction, such as using text sentiment analysis or voice emotion recognition to determine risk levels. While this improves the automation of risk identification, it still suffers from insufficient model robustness, weak ability to distinguish roles in multi-role dialogues, and low utilization of unlabeled data. In addition, existing methods often generate fixed policy outputs when generating control strategies, lacking real-time, hierarchical control capabilities tailored to different risk levels. To address these issues, this invention proposes a solution. Summary of the Invention

[0004] To overcome the aforementioned deficiencies of the prior art, embodiments of the present invention provide a power customer service early warning hierarchical control method based on multimodal risk identification. By extracting potential risk features based on multimodal information and generating robust risk characterizations, and combining this with the real-time issuance of hierarchical early warning control strategies, the method solves the problems of low accuracy in risk identification, difficulty in role differentiation, and insufficient real-time and hierarchical nature of control strategies.

[0005] To achieve the above objectives, the present invention provides the following technical solution: A hierarchical control method for early warning of power customer service based on multimodal risk identification includes: collecting real-time voice data of power customer service and extracting voiceprint features; determining new speakers online and generating new class center vectors through clustering based on the distance metric between the current voiceprint features and historical voiceprint class centers and dynamic thresholds, and performing dynamic role separation; extracting voiceprint features and text semantic features from the voice segments separated by roles, mapping the text semantic features as causal intermediate variables, and calculating modulation weights to weight the voiceprint features to obtain enhanced voiceprint features, which are combined with text semantic features to generate a multimodal fusion feature vector, and generating potential risk features through sparsification; pre-training the potential risk features on unlabeled voice data and fine-tuning them on labeled data from the power customer service scenario to generate a robust risk representation vector; determining the risk type and level based on the robust risk representation vector, and outputting a hierarchical early warning control strategy.

[0006] In a preferred embodiment, the voiceprint features are extracted as follows: the speech data is preprocessed and converted into a Mel spectrogram; local feature extraction and dimensionality reduction are performed on the Mel spectrogram to obtain time series features; the time series features are input into a recurrent neural network layer for time series modeling to generate high-dimensional time series features; the high-dimensional time series features are input into a fully connected mapping layer for mapping to output a voiceprint feature vector.

[0007] In a preferred embodiment, dynamic role separation is performed, specifically by: calculating the class center vector and intra-class covariance matrix for each historical speaker in the voiceprint feature database; calculating Mahalanobis distance based on the covariance matrix, class center vector, and voiceprint feature vector; dynamically setting a judgment threshold based on the Mahalanobis distance statistics; comparing the Mahalanobis distance with the judgment threshold, and determining the current voice segment as a new speaker when the distance of all historical classes is greater than the judgment threshold, and generating a new class center vector through clustering; and performing role separation based on the new class center vector to accurately distinguish between customer and customer service roles.

[0008] In a preferred embodiment, role separation is performed based on the new class center vector, specifically: constructing edges of nodes based on the class center vector and its corresponding speech segments to form a speech graph; generating a role determination feature vector for each node through a graph neural network; calculating the role probability based on the role determination feature vector, and labeling each node with a role label through weighted judgment; mapping the role label of each node back to the corresponding speech segment, and outputting according to the role label.

[0009] In a preferred embodiment, voiceprint features and text semantic features are extracted from the speech segments separated from the character. The text semantic features are mapped to causal intermediate variables. Modulation weights are generated based on the causal intermediate variables and applied to the voiceprint features to obtain enhanced voiceprint features. The enhanced voiceprint features are then fused with the text semantic features to generate a multimodal fusion feature vector. Specifically, the process involves: obtaining the marked speech segments separated from the character; extracting the voiceprint feature vector and text semantic feature vector corresponding to each speech segment; mapping the text semantic feature vector to causal intermediate variables through a multi-layer neural network; generating a modulation weight vector based on the causal intermediate variables; modulating the voiceprint feature vector element-wise with the modulation weight vector to obtain enhanced voiceprint features; and fusing the enhanced voiceprint features with the text semantic feature vector to generate a multimodal fusion feature vector.

[0010] In a preferred embodiment, the potential risk features are generated through sparsification, specifically by: evaluating the risk importance of each dimension of the multimodal fusion features and generating a risk-sensitive weight vector; multiplying the risk-sensitive weight vector element-wise with the multimodal fusion feature vector, and weighting and enhancing the high-risk dimensions based on the risk enhancement coefficient to obtain an enhanced fusion feature vector and performing preprocessing; and sparsifying the processed enhanced fusion feature vector, retaining the first few largest eigenvalues ​​to generate highly sensitive potential risk features.

[0011] In a preferred embodiment, the potential risk features are pre-trained on unlabeled speech data and fine-tuned on labeled data from the power customer service scenario to generate a robust risk representation vector. Specifically, a self-supervised learning model based on convolutional neural networks and Transformers is constructed and pre-trained using unlabeled speech data; labeled data from the power customer service scenario is input into the self-supervised learning model, keeping the convolutional neural network parameters frozen, and the last few layers of the Transformer encoder are fine-tuned under supervision; the potential risk features are input into the fine-tuned self-supervised model to obtain the robust risk representation vector.

[0012] In a preferred embodiment, a self-supervised learning model based on convolutional neural networks and Transformers is constructed, specifically as follows: Local feature sequences are extracted from unlabeled speech data using a convolutional neural network. The local feature sequence is passed through a Transformer encoder to generate a global feature representation; Self-supervised learning tasks, including mask prediction, contrastive learning, and reconstruction, are constructed based on global feature representations. By minimizing the weighted loss of the self-supervised learning task, and training the model with unlabeled speech data, a self-supervised learning model that can simultaneously capture local short-term features and global contextual dependencies is obtained.

[0013] In a preferred embodiment, a multi-task risk assessment model is constructed, which includes a risk type branch and a risk level branch. Through the risk type branch, robust risk representation vectors are classified and probability-filtered to generate dialogue risk types. Through the risk level branch, a risk level probability distribution is output for the robust risk representation vectors, and the final risk level is determined by combining historical risk level statistics with current dialogue context features. Based on the dialogue risk type and the final risk level, corresponding strategies are matched from a preset hierarchical early warning strategy library to generate control strategies. The control strategies are output to the customer service system, enabling real-time response and hierarchical control of dialogue risks through an interface.

[0014] In a preferred embodiment, the control strategy is output to the customer service system, and real-time response and hierarchical control of dialogue risks are realized through the interface. Specifically, the control strategy is encapsulated into a structured strategy instruction set; the strategy instruction set is mapped to the interface events of the customer service system; based on the mapping result, the customer service system interface is called to issue the control strategy in real time, realizing differentiated responses to different risk levels. Among them, high-risk dialogues trigger an interface for automatic transfer or interception, medium-risk dialogues trigger a log recording and reminder interface, and low-risk dialogues trigger a status archiving interface.

[0015] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages: End-to-end optimization from low-level speech signals to high-level risk decision-making is achieved: First, by fusing convolutional neural networks and recurrent neural networks for speaker modeling, and combining Mahalanobis distance and a dynamic update mechanism for class centers, highly robust speaker differentiation is achieved in complex multi-speaker scenarios, effectively reducing interference from noise, accent differences, and abnormal speech; Second, a multimodal approach is introduced that differs from the traditional Transformer. This attention-based cross-modal causal-guided fusion approach modulates voiceprint features through causal intermediate variables, shifting the fusion process from "relevance-driven" to "semantic-driven," thus enhancing feature consistency and interpretability from the source. Furthermore, risk-sensitive weight learning and sparsification adaptively highlight key risk dimensions and suppress redundant information, improving the compactness and discriminative power of feature representation. Building upon this, a combination of self-supervised learning and supervised transfer fine-tuning strategies enables adaptive modeling for different noise environments, equipment conditions, and accent differences, significantly improving the stability and generalization ability of risk representation. Finally, a multi-task learning model sharing features simultaneously outputs risk types and levels, linked to a tiered early warning strategy library, enabling differentiated real-time intervention and control for high, medium, and low-risk dialogues. The overall solution not only significantly improves feature extraction accuracy, fusion effectiveness, and model robustness but also possesses excellent real-time performance, scalability, and engineering feasibility, effectively supporting risk monitoring and service quality assurance in power customer service scenarios. Attached Figure Description

[0016] Figure 1 A schematic diagram of the power customer service early warning and hierarchical control method based on multimodal risk identification provided in this application embodiment.

[0017] Figure 2 A line graph comparing Mahalanobis distance and decision threshold provided in an embodiment of this application.

[0018] Figure 3 The diagram provided is for an embodiment of this application.

[0019] Figure 4 A topology diagram for distinguishing customer and customer service voice roles provided in the embodiments of this application. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0021] Example 1, Figure 1A flowchart illustrating the power customer service early warning and hierarchical control method based on multimodal risk identification provided in this application embodiment includes the following steps: S1. Collect real-time voice data in the power customer service scenario, extract voiceprint feature vectors from the voice data, and dynamically update the voiceprint class center vectors of different speakers in combination with the voiceprint feature library to perform role separation.

[0022] In the context of electricity customer service, voice data is collected through smart wearable devices and fixed microphones, and the collected data is stored in segments according to timestamps to ensure the integrity of the conversation sequence.

[0023] In this embodiment, voiceprint features are extracted from the speech data. The extraction is based on convolutional neural networks and recurrent neural networks, specifically as follows: The speech data is preprocessed, and each frame of the processed data is converted into a Mel spectrogram in order to extract short-time spectral features. The preprocessing includes denoising, endpoint detection and frame segmentation. Mel spectrograms are input into convolutional neural network layers. Local spectral pattern features are extracted through multiple convolution and pooling operations, while reducing the feature dimension to form time series features. The CNN (convolutional neural network layer) includes at least two convolutional layers, each with a kernel size of 3×3 or 5×5 and an activation function of ReLU. Each convolutional layer is followed by a max pooling layer to reduce the feature dimension and output a time series feature map. The time series features are input into a recurrent neural network layer, which includes 1-3 layers of long short-term memory network (LSTM) or gated recurrent unit (GRU) to capture the long-term dependence features of the speech signal in the time dimension and generate high-dimensional time series features. High-dimensional temporal features are input into a fully connected mapping layer and mapped to a fixed-dimensional voiceprint embedding vector of 128-512 dimensions. After L2 normalization, a voiceprint feature vector is generated as the voiceprint feature for subsequent voiceprint class center calculation, clustering, and role separation. The voiceprint embedding vector is used to represent the speaker features of the current speech segment.

[0024] Furthermore, based on the voiceprint feature library, the voiceprint class center vectors of different speakers are dynamically updated to perform role separation, specifically as follows: For each historical speaker class in the voiceprint feature database, calculate its class center vector and intra-class covariance matrix. The class center vector is the mean of the speaker's voiceprint embedding vector, and the covariance matrix reflects the distribution characteristics of the speaker's voiceprint embedding vector in each dimension. The Mahalanobis distance is obtained based on the voiceprint features, the center vector of each historical class, and the covariance matrix. The Mahalanobis distance is calculated using the following formula:

[0025] In the formula, The Mahalanobis distance, For voiceprint feature vectors, Let be the class center vector of the i-th historical speaker. This is the transpose of the deviation vector between the current voiceprint features and the class center vector. Let be the inverse matrix of the covariance matrix of the i-th historical speaker.

[0026] The judgment threshold is dynamically set based on the mean and standard deviation of the historical Mahalanobis distance; The specific calculation formula for the determination threshold is as follows:

[0027] In the formula, To determine the threshold, The mean of the Mahalanobis distance for historical classes. The standard deviation of the Mahalanobis distance for historical classes. This is an empirical coefficient used to adjust the sensitivity of the new speaker determination.

[0028] like Figure 2 As shown, Mahalanobis distance is compared with a decision threshold. When the distance of all historical classes is greater than the decision threshold, the current speech segment is determined to be a new speaker, and a new class center vector is generated through clustering. Roles are separated based on the new class center vector, enabling accurate differentiation between customer and customer service roles.

[0029] It should be noted that a voiceprint feature database refers to a data structure that stores the voiceprint class center vectors, covariance matrices, and corresponding labels of historical speakers. This structure can be used to calculate similarity with real-time generated voiceprint feature vectors, thereby enabling speaker identification, new speaker determination, dynamic updating of class centers, and separation of roles among multiple speakers. The class center vector refers to the "mean vector" of all voiceprint feature vectors of a historical speaker in the multi-speaker voiceprint processing or embedding space, used to represent the overall characteristics of that speaker.

[0030] The advantage of updating the corresponding class center vector using Mahalanobis distance is that it adaptively considers the variance and correlation of each speaker's voiceprint feature vector across various dimensions. This allows for the dynamic updating of class centers while preserving historical features and effectively incorporating information from the current speech segment. Compared to simple Euclidean averaging, this method more accurately reflects the true distribution of a speaker's voiceprint features, improves the robustness of matching new voiceprint feature vectors with historical class centers, reduces interference from noise, accent differences, or abnormal speech on class centers, and simultaneously supports the accuracy of subsequent speaker identification and multi-speaker clustering. Furthermore, role separation is performed based on the new class center vector to accurately distinguish between customer and customer service roles, specifically as follows: The newly generated class center vector and its corresponding speech segment are used as node inputs, and each node represents a speech segment and its speaker features. Construct edges between nodes based on Mahalanobis distance, speaking order, and call direction; Construct a speech graph based on node inputs and edges between nodes, such as Figure 3 As shown, the speech graph is input into a graph neural network (GNN). The graph neural network can adopt a two- to five-layer structure, with each layer's node feature dimension being 64 to 512 dimensions, the activation function being ReLU, and the training using the cross-entropy loss function. By propagating node features and updating edge relationships, global context information is fused to generate a role determination feature vector for each node; Based on the role determination feature vector, the role probability is calculated by the softmax of the multilayer perceptron classifier, and weighted by combining the historical class center role probability. Each node is labeled with a role tag, which is either a customer role or a customer service role. The specific calculation formula for the weighted determination is as follows:

[0031] In the formula, For the role label of node i, Output probabilities for GNN The probability of a central historical figure. This is the preset fusion coefficient.

[0032] like Figure 4 As shown, the role label of each node is mapped back to the corresponding voice segment, and the mapped voice segment is output according to the role label to achieve accurate differentiation between customer voice and customer service voice.

[0033] It should be noted that by constructing a speech graph using newly generated class center vectors and corresponding speech segments, and by using graph neural networks to fuse node features with global context information, accurate role determination for each speech segment can be achieved. By combining a multilayer perceptron classifier and historical class center role probabilities for weighted determination, it is possible to accurately distinguish between customer and customer service roles in complex, multi-speaker, cross-session, or short-segment speech scenarios. At the same time, it dynamically maps and outputs speech segments, making the role separation results both highly accurate and applicable in real time, providing reliable input data for subsequent customer service risk monitoring and anomaly warning.

[0034] S2, extract voiceprint features and text semantic features from the voice segments separated from the character, map the text semantic features as causal intermediate variables, generate modulation weights based on the causal intermediate variables and weight the voiceprint features to obtain enhanced voiceprint features, fuse the enhanced voiceprint features with the text semantic features to generate a multimodal fusion feature vector, and generate potential risk features by sparsification of the multimodal fusion features.

[0035] In this example, voiceprint features and text semantic features are extracted from the speech segments separated from the character. The text semantic features are mapped to causal intermediate variables. Modulation weights are generated based on the causal intermediate variables and weighted according to the voiceprint features to obtain enhanced voiceprint features. The enhanced voiceprint features are then fused with the text semantic features to generate a multimodal fusion feature vector, specifically: Obtain the tagged voice segments separated from the roles, the tagged voice segments including customer voice segments and customer service voice segments; Acoustic feature analysis is performed on the marked speech segments to extract the voiceprint feature vector corresponding to each speech segment. The voiceprint feature vector is used to characterize the speaker's speech attribute information and serves as the applied variable for subsequent causal modulation. The tagged speech segments are converted into text using automatic speech recognition (ASR) methods, and natural language processing methods (such as BERT and Transformer) are used to extract semantic feature vectors from the text to represent the semantic information of the speech content. Based on textual semantic feature vectors, causal intermediate variables are constructed through a trainable mapping function, wherein the trainable mapping function is a multi-layer neural network structure, including at least one linear transformation layer and a non-linear activation function, as detailed below:

[0036] In the formula, As an intermediate variable in causality, It is the Sigmoid activation function. , These are the bias parameters. , For trainable weight matrix, This is a text semantic feature vector.

[0037] The causal intermediate variable is used to characterize the underlying semantic intent or emotional state determined by the text semantics, and serves as a causal bridge connecting the text modality and the voiceprint modality.

[0038] Generate a modulation weight vector based on causal intermediate variables; The modulation weight vector is calculated using the following formula:

[0039] In the formula, This is a modulation weight vector used to characterize the strength of the influence of text semantics on voiceprint features. To preset weights, This is the bias vector.

[0040] The modulation weight vector is used to modulate the voiceprint feature vector element by element to obtain enhanced voiceprint features; The specific calculation formula for the enhanced voiceprint feature is as follows:

[0041] In the formula, To enhance voiceprint features and characterize speech expression features corrected under textual semantic causal constraints, This is the voiceprint feature vector.

[0042] The enhanced voiceprint features are fused with the text semantic feature vector to generate a multimodal fusion feature vector.

[0043] It should be noted that by introducing a cross-modal causal guidance mechanism, textual semantic features are mapped into causal intermediate variables via a multi-layer neural network, and modulation weights are further generated to perform targeted modulation on voiceprint features. This achieves causal chain modeling from "textual semantics → latent intent / emotion → speech expression." Compared to traditional Transformer-based attention fusion methods that only characterize intermodal correlations, this method can explicitly introduce semantically driven regulatory relationships, enabling voiceprint features to complete semantic consistency correction before fusion, thereby effectively reducing interference from speech noise, emotional bias, or inconsistent expression. At the same time, through adaptive weighted enhancement of voiceprint features, key information related to semantic intent can be highlighted, while redundant or irrelevant components are suppressed. Ultimately, this improves the discriminative ability, robustness, and interpretability of multimodal fusion features in tasks such as emotion recognition, intent understanding, and dialogue analysis.

[0044] Furthermore, potential risk features are generated by sparsifying the multimodal fusion features, specifically as follows: Using a feature selection model trained with historical customer service risk data, the importance of each feature dimension of the multimodal fusion feature is evaluated, and a risk-sensitive weight vector is output. The feature selection model is constructed through a pre-trained potential risk identification network and trained using supervised learning. The training data includes risk samples labeled with customer dissatisfaction, complaints, and abnormal emotions. The risk-sensitive weight vector is multiplied element-wise with the multimodal fusion feature vector, and the high-risk dimension is weighted and enhanced based on the risk enhancement coefficient to obtain the enhanced fusion feature vector. The specific calculation formula for the enhanced fusion feature vector is as follows:

[0045] In the formula, To enhance the fused feature vector, For risk-sensitive weight vectors, This represents the risk amplification factor.

[0046] The steps for obtaining the risk enhancement coefficient are as follows: First, based on the multimodal fusion features, high-risk labeled samples and ordinary samples are simultaneously input into the training model; then, during the model training process, the risk enhancement coefficient is introduced as a learnable parameter into the feature weighting stage; next, by calculating the loss function (such as weighted cross-entropy loss) between the predicted results and the true risk labels on the validation set, and using gradient descent to iteratively optimize, the value of the risk enhancement coefficient gradually converges within the range of positive real numbers; finally, the obtained risk enhancement coefficient can adaptively highlight high-risk features while suppressing irrelevant noise and low-risk features, thereby improving sensitivity and accuracy in risk identification and early warning classification.

[0047] The enhanced fusion feature vectors are normalized to keep all feature dimensions within a uniform scale. The normalized enhanced fusion feature vector is sparsified, retaining only the largest number of high-risk feature dimensions, generating highly sensitive potential risk features for customer service risk monitoring, anomaly warning and hierarchical control. The generated results highlight risk-sensitive feature dimensions while effectively suppressing low-risk noise features.

[0048] It should be noted that by progressively processing risk-sensitive dimensions through identification, weighted enhancement, normalization, and sparsification, this approach not only adaptively highlights high-risk feature dimensions strongly correlated with customer complaints, dissatisfaction, or abnormal emotions, but also effectively compresses and suppresses low-risk noise dimensions, avoiding interference from feature redundancy in model judgments. Through dynamic adjustment and sparsification selection of the risk enhancement coefficients, this scheme maintains high-risk feature sensitivity while also ensuring stability and robustness, thereby significantly improving the accuracy and real-time performance of customer service risk monitoring and providing more reliable input features for subsequent anomaly warnings and tiered control.

[0049] S3 inputs potential risk features into a self-supervised learning model, and generates robust risk representation vectors that adapt to different noise, accents and equipment conditions by combining pre-training with unlabeled speech data with fine-tuning with labeled data from power customer service scenarios.

[0050] In this embodiment, potential risk features are input into a self-supervised learning model. By combining pre-training with unlabeled speech data and fine-tuning with labeled data from power customer service scenarios, a robust risk representation vector adapted to different noise, accent, and equipment conditions is generated. Specifically: Unlabeled speech data from a large-scale electricity customer service voice database is converted into acoustic feature vectors and used as input feature matrices. A self-supervised learning model is then constructed based on convolutional neural networks and Transformers. Voice segments and text data labeled with risk levels (low, medium, and high) in power customer service scenarios are converted into labeled acoustic feature matrices and input into a self-supervised learning model. Keep the parameters of the convolutional neural network layers in the model frozen to preserve the local short-term feature representations learned by the original pre-trained model and avoid local feature loss or overfitting due to limited data scale. The last few layers of the Transformer encoder in the self-supervised learning model are identified, and the weights of these layers are fine-tuned by transfer. The input of the last few layers is the local feature output after the labeled acoustic feature matrix is ​​processed by the convolutional neural network. A supervised learning approach is adopted, with the risk level corresponding to the input speech as the label and cross-entropy loss as the optimization objective to solve the problem of imbalanced risk category samples. The gradient descent method is used to update the weights of the last few layers of the Transformer encoder to obtain a self-supervised model after transfer fine-tuning. By inputting potential risk features into the self-supervised model after transfer fine-tuning, a robust risk representation vector is obtained after context optimization and multi-scale modeling.

[0051] Furthermore, the unlabeled speech data in the large-scale power customer service voice database is converted into acoustic feature vectors and used as the input feature matrix. A self-supervised learning model is then constructed based on convolutional neural networks and Transformer, specifically as follows: The input feature matrix is ​​input into a convolutional neural network (CNN). The CNN consists of 2 to 5 one-dimensional or two-dimensional convolutional layers, with each convolutional layer having a kernel size of 3 to 7, a stride of 1 to 2, and an activation function of ReLU. Short-term pattern features of speech within a local time-frequency range are extracted by convolution and pooling operations to obtain a local feature sequence; The local feature sequence is input into the Transformer encoder, which includes 4 to 12 stacked multi-head self-attention modules and feedforward network modules. Position encoding is introduced at the input end, and the global temporal dependencies of speech segments are captured by multi-head self-attention to generate context-related global feature representations. A self-supervised learning task is constructed based on the output global feature representation, and the self-supervised learning task includes mask prediction, contrastive learning and reconstruction task; Among them, mask prediction involves randomly masking the input frame, and training the model to predict the features of the masked region based on the context. Contrastive learning involves taking different segments of the same speech segment as positive samples and different speech segments as negative samples, maximizing the similarity of positive samples and minimizing the similarity of negative samples. The reconstruction task reconstructs the input features by adding a decoder and minimizing the mean square error between the original input features and the reconstructed features. By employing the Adam optimizer to minimize the weighted sum of the losses of each task in the self-supervised learning task, and training on large-scale unlabeled speech data, a self-supervised learning model that can simultaneously capture local short-term features and global contextual dependencies is obtained.

[0052] It should be noted that by inputting potential risk features into a self-supervised learning model built on convolutional neural networks and Transformers, and combining pre-training with unlabeled speech data with transfer fine-tuning with labeled data from power customer service scenarios, the model can simultaneously capture local short-term pattern features and global context-dependent features of speech, achieving robust representations with multi-scale and context-optimized features. Furthermore, by freezing the CNN layers to retain stable local features obtained from pre-training, and fine-tuning several layers of the Transformer for specific scenarios, the model can adapt to different noise levels, accents, and equipment conditions, improving the accuracy and robustness of potential risk signal recognition. This provides high-quality and reliable feature input for risk monitoring, anomaly warning, and tiered control in power customer service.

[0053] Robust risk representation vectors refer to high-dimensional feature vectors generated by pre-training a self-supervised learning model on large-scale unlabeled speech data and then fine-tuning it using labeled data from power customer service scenarios. These vectors simultaneously contain local short-term pattern features (extracted by a CNN) and global temporal dependency features (captured by a Transformer encoder), comprehensively representing the complex information of speech segments. Through transfer fine-tuning, the vectors can adapt to the unique noise environment, accent differences, and equipment acquisition conditions of power customer service, enabling the model to stably extract risk-related features under different conditions. During training, the vectors incorporate labeled risk level information, learning sensitive representations of different risk types and levels, allowing them to distinguish between low, medium, and high-risk dialogues in subsequent risk assessment and early warning classification. By freezing the CNN layers to retain pre-trained local features, fine-tuning only the later Transformer layers, and employing data augmentation or adversarial perturbations during training, the vectors exhibit strong stability and anti-interference capabilities against input noise, accents, and abnormal variations.

[0054] S4. Based on the robust risk representation vector, determine the risk type and risk level of the dialogue, and output a hierarchical early warning and control strategy to achieve real-time risk intervention and service quality assurance for power customer service dialogues. In this embodiment, the risk type and risk level of the dialogue are determined based on the robust risk representation vector, and a graded early warning and control strategy is output, specifically as follows: Construct a multi-task risk assessment model and input the robust risk representation vector into the multi-task risk assessment model; The multi-task risk assessment model comprises two branches: a risk type branch, consisting of several fully connected layers or a hybrid convolutional-fully connected layer, outputting a probability distribution of dialogue risk types; and a risk level branch, consisting of several fully connected layers, outputting a probability distribution of risk levels, corresponding to low, medium, and high levels, respectively. Both branches share input layer and some intermediate layer feature representations to enhance feature utilization efficiency and inter-task relevance. Labeled training data is used to train the type branch, and a cross-entropy loss function is defined. During training, prior probabilities of historical risk types can be incorporated for weighted loss, enhancing the ability to determine rare or few-sample types. For the level branch, a cross-entropy loss function is used, combining historical risk level statistics with current dialogue context features to determine the final risk level through weighted fusion. The loss functions of the type and level branches are jointly optimized, and network parameters are updated through gradient descent or the Adam optimizer, enabling the model to simultaneously and efficiently determine both the risk type and risk level of a dialogue.

[0055] The specific calculation formula for the type-branch cross-entropy loss function is as follows:

[0056] The specific calculation formula for the hierarchical branch cross-entropy loss function is as follows:

[0057] In the formula, Here, is the type-branch cross-entropy loss function, used to measure the difference between the model's predicted risk type probability and the true label, where m is the total number of risk types. This is the indicator value for the actual type label. The model predicts the probability that the sample belongs to the i-th risk type (output via softmax). Here, is the loss function for the risk level branch, used to measure the difference between the model's predicted risk level probability and the true label. j is the risk level index, taking values ​​of 1, 2, and 3, corresponding to low, medium, and high risk levels, respectively. This is the indicator value for the true rating label. Predict the probability value of the sample belonging to the j-th risk level for the model (output via softmax).

[0058] By using the type branches of the multi-task risk assessment model, the robust risk representation vector is classified to generate a probability distribution of dialogue risk types. Based on the principle of maximizing probability, determine the type of risk in the final dialogue, such as complaints, grievances, or abnormal emotions; By using the hierarchical branching of the multi-task model, the probability distribution of risk level is output for the robust risk representation vector, corresponding to low, medium and high levels respectively; The final risk level is determined by combining historical risk level statistics with the characteristics of the current dialogue context through a weighted fusion method. Based on the final risk type and risk level of the dialogue, a corresponding strategy is matched from a preset tiered early warning strategy library to generate a control strategy. The control strategy includes providing real-time reminders or operational guidance to customer service personnel; automatically intercepting or transferring high-risk dialogues to senior customer service; generating reminder logs for medium-risk dialogues for subsequent analysis; and simply recording low-risk dialogues without real-time intervention. The generated control policies are output to the customer service system, and the interface enables real-time response and hierarchical control of dialogue risks, ensuring that dialogues of different risk levels are handled differently.

[0059] Furthermore, the generated control policies are output to the customer service system, enabling real-time response and tiered control of dialogue risks through an interface, specifically: The control strategy is encapsulated into a structured strategy instruction set, wherein each instruction contains a strategy identifier, triggering condition, execution action, and execution time limit; Map the policy instruction set to the corresponding interface events of the customer service system; For example, corresponding mappings are made, such as mapping "real-time reminders" to the message push interface; mapping "automatic transfer" to the customer service dispatch interface; and mapping "log recording" to the log management interface, to ensure that the policy execution actions can be directly called by the system interface; Based on the corresponding mapping results, the customer service system interface is called to issue control policies in real time, enabling differentiated responses to different risk levels. Among them, high-risk conversations trigger automatic transfer or interception interfaces, medium-risk conversations trigger log recording and alert interfaces, and low-risk conversations trigger status archiving interfaces.

[0060] It should be noted that by constructing a multi-task risk judgment model with a shared input layer and intermediate features, the model can simultaneously determine the risk type and risk level of a dialogue. It fully utilizes the local speech features and global contextual information of the robust risk representation vector to improve feature utilization efficiency and the correlation between tasks. By combining the weighted fusion of historical prior probabilities and contextual features, the model can maintain high judgment accuracy in scenarios with few samples or rare types. Finally, through a pre-set hierarchical early warning strategy library, it can achieve real-time and differentiated management and control of dialogues with different risk levels, ensuring that high-risk dialogues are intervened in a timely manner while avoiding false interference from low-risk dialogues, thereby improving the security, response efficiency, and service quality of the power customer service system.

[0061] The above formulas are all dimensionless calculations. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters in the formulas are set by those skilled in the art according to the actual situation.

[0062] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, in the form of a computer program product.

[0063] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0064] In addition, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0065] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0066] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A power customer service early warning and hierarchical control method based on multimodal risk identification, characterized in that, include: Collect real-time voice data from power customer service representatives and extract voiceprint features; Based on the distance metric between the current voiceprint features and the historical voiceprint class centers, and a dynamic threshold, new speakers are identified online, and new class center vectors are generated through clustering to perform dynamic role separation. Voiceprint features and text semantic features are extracted from the voice segments separated from the characters. The text semantic features are mapped to causal intermediate variables. Modulation weights are generated based on the causal intermediate variables and the voiceprint features are weighted to obtain enhanced voiceprint features. The enhanced voiceprint features are fused with the text semantic features to generate a multimodal fusion feature vector, and then the potential risk features are generated after sparsification. Pre-training the potential risk features on unlabeled voice data and fine-tuning them with labeled data from the power customer service scenario generates a robust risk representation vector. Based on robust risk representation vectors, risk types and levels are determined, and graded early warning and control strategies are output.

2. The power customer service early warning and hierarchical control method based on multimodal risk identification according to claim 1, characterized in that, The specific method for extracting the voiceprint features is as follows: The speech data is preprocessed and converted into a Mel spectrogram; Local feature extraction and dimensionality reduction are performed on the Mel spectrogram to obtain time series features; Time series features are input into a recurrent neural network layer for time series modeling to generate high-dimensional time series features. High-dimensional temporal features are input into a fully connected mapping layer for mapping, and the output is a voiceprint feature vector.

3. The power customer service early warning and hierarchical control method based on multimodal risk identification according to claim 1, characterized in that, The dynamic role separation is specifically performed as follows: Calculate the class center vector and intra-class covariance matrix for each historical speaker in the voiceprint feature database; Mahalanobis distance is calculated based on the covariance matrix, class center vector, and voiceprint feature vector. The judgment threshold is dynamically set based on the Mahalanobis distance statistics; The Mahalanobis distance is compared with the decision threshold. When the distance of all historical classes is greater than the decision threshold, the current speech segment is determined to be a new speaker, and a new class center vector is generated by clustering. Roles are separated based on the new class center vector, enabling accurate differentiation between customer and customer service roles.

4. The power customer service early warning and hierarchical control method based on multimodal risk identification according to claim 3, characterized in that, The role separation based on the new class center vector is specifically as follows: Edges of nodes are constructed based on class center vectors and their corresponding speech segments to form a speech graph; The speech graph is processed through a graph neural network to generate a role determination feature vector for each node; The probability of a role is calculated based on the feature vector of the role determination, and a role label is assigned to each node through weighted determination. Map the role label of each node back to the corresponding voice clip and output it according to the role label.

5. The power customer service early warning and hierarchical control method based on multimodal risk identification according to claim 1, characterized in that, The process involves extracting voiceprint features and text semantic features from the extracted speech segments, mapping the text semantic features to causal intermediate variables, generating modulation weights based on these intermediate variables, and weighting the voiceprint features to obtain enhanced voiceprint features. These enhanced voiceprint features are then fused with the text semantic features to generate a multimodal fusion feature vector. Specifically: Obtain the marked speech segments separated from the characters, and extract the voiceprint feature vector and text semantic feature vector corresponding to each speech segment; Based on textual semantic feature vectors, they are mapped to causal intermediate variables through a multi-layer neural network; Generate a modulation weight vector based on causal intermediate variables; The modulation weight vector is used to modulate the voiceprint feature vector element by element to obtain enhanced voiceprint features; The enhanced voiceprint features are fused with the text semantic feature vector to generate a multimodal fusion feature vector.

6. The power customer service early warning and hierarchical control method based on multimodal risk identification according to claim 1, characterized in that, The potential risk features generated through sparsification are specifically as follows: Assess the risk importance of each dimension of multimodal fusion features and generate a risk-sensitive weight vector; The risk-sensitive weight vector is multiplied element-wise with the multimodal fusion feature vector, and the high-risk dimension is weighted and enhanced based on the risk enhancement coefficient to obtain the enhanced fusion feature vector, which is then preprocessed. The processed enhanced fusion feature vector is sparsified, and the top few largest feature values ​​are retained to generate highly sensitive potential risk features.

7. The power customer service early warning and hierarchical control method based on multimodal risk identification according to claim 1, characterized in that, The process of pre-training potential risk features on unlabeled voice data and fine-tuning them with labeled data from the power customer service scenario to generate a robust risk representation vector is as follows: A self-supervised learning model based on convolutional neural networks and Transformers is constructed and pre-trained using unlabeled speech data; Labeled data from the power customer service scenario is input into a self-supervised learning model, the parameters of the convolutional neural network are kept frozen, and the last few layers of the Transformer encoder are fine-tuned under supervision. By inputting the potential risk characteristics into the fine-tuned self-supervised model, a robust risk representation vector is obtained.

8. The power customer service early warning and hierarchical control method based on multimodal risk identification according to claim 7, characterized in that, The construction of the self-supervised learning model based on convolutional neural networks and Transformers is specifically as follows: Local feature sequences are extracted from unlabeled speech data using a convolutional neural network. The local feature sequence is passed through a Transformer encoder to generate a global feature representation; Self-supervised learning tasks, including mask prediction, contrastive learning, and reconstruction, are constructed based on global feature representations. By minimizing the weighted loss of the self-supervised learning task, and training the model with unlabeled speech data, a self-supervised learning model that can simultaneously capture local short-term features and global contextual dependencies is obtained.

9. The power customer service early warning and hierarchical control method based on multimodal risk identification according to claim 1, characterized in that, The method for determining risk type and level based on robust risk representation vectors and outputting a tiered early warning and control strategy is as follows: Construct a multi-task risk assessment model, which includes risk type branches and risk level branches; By classifying and probabilistically filtering the robust risk representation vector through risk type branches, dialogue risk types are generated. By branching through risk levels, the probability distribution of risk levels is output from the robust risk representation vector, and the final risk level is determined by combining historical risk level statistics with the current dialogue context features. Based on the type of dialogue risk and the final risk level, the corresponding strategy is matched from the preset hierarchical early warning strategy library to generate a control strategy. The control strategy is output to the customer service system, enabling real-time response and tiered control of dialogue risks through an interface.

10. The power customer service early warning and hierarchical control method based on multimodal risk identification according to claim 9, characterized in that, The process of outputting control strategies to the customer service system, and implementing real-time response and tiered control of dialogue risks through an interface, specifically involves: The control strategy is encapsulated into a structured strategy instruction set; Map the policy instruction set to the corresponding interface events of the customer service system; Based on the mapping results, the customer service system interface is called to issue control policies in real time, enabling differentiated responses to different risk levels. Specifically, high-risk conversations trigger an interface for automatic transfer or interception, medium-risk conversations trigger a log recording and alert interface, and low-risk conversations trigger a status archiving interface.

Images