Voice classification method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a voice classification method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring target voice data and a target text of the target voice data; performing feature extraction on the target voice data to obtain emotion feature information; performing feature extraction on the semantics of the target text to obtain intermediate feature information, and converting the intermediate feature information into graph structure feature information; and classifying the target voice data according to the emotion feature information and the graph structure feature information to obtain a classification result of the target voice data. Therefore, the accuracy of voice classification can be improved.

Description

Technical Field

[0001] The present application relates to the field of natural language processing technology, and in particular to a speech classification method, device, electronic device and storage medium. Background Art

[0002] With the development of electronic technology, the application of NLP (Natural Language Processing) technology is becoming more and more extensive.

[0003] In informal conversations, users often don't use strictly grammatically correct written expressions; instead, they simply express their thoughts clearly. Furthermore, as the conversation progresses, a participant who has previously mentioned a specific piece of information may simply use a different word to refer to it for convenience, rather than repeating the entire specific information in each sentence. For other real users participating in the same conversation, understanding the specific information referred to in the context is not difficult. However, from a machine's perspective, processing speech data individually for each sentence is likely to distort semantics. Therefore, the difficulty of processing speech data using NLP technology increases significantly with the length and colloquialism of the speech data. Summary of the Invention

[0004] The embodiments of the present application provide a speech classification method, apparatus, electronic device, and storage medium, which can improve the accuracy of speech classification.

[0005] In a first aspect, an embodiment of the present application provides a speech classification method, comprising:

[0006] Acquire target voice data and target text of the target voice data;

[0007] Extracting features from the target speech data to obtain emotional feature information;

[0008] Extracting features from the semantics of the target text to obtain intermediate feature information, and converting the intermediate feature information into graph structure feature information;

[0009] The target speech data is classified according to the emotion feature information and the graph structure feature information to obtain a classification result of the target speech data.

[0010] In a second aspect, an embodiment of the present application provides a speech classification device, comprising:

[0011] An acquisition unit, configured to acquire target speech data and a target text of the target speech data;

[0012] An extraction unit, configured to extract features from the target speech data to obtain emotional feature information;

[0013] The extraction unit is further configured to extract features from the semantics of the target text to obtain intermediate feature information, and convert the intermediate feature information into graph structure feature information;

[0014] The classification unit is used to classify the target speech data according to the emotional feature information and the graph structure feature information to obtain a classification result of the target speech data.

[0015] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor; and a memory configured to store computer-executable instructions, wherein the computer-executable instructions, when executed, cause the processor to perform the speech classification method as described in the first aspect.

[0016] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium for storing computer-executable instructions, which, when executed by a processor, implement the speech classification method as described in the first aspect.

[0017] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the speech classification method as described in the first aspect.

[0018] It can be seen that in the embodiment of the present application, first, the target speech data and the target text of the target speech data are obtained; then, feature extraction is performed on the target speech data to obtain emotional feature information; then, feature extraction is performed on the semantics of the target text to obtain intermediate feature information, and the intermediate feature information is converted into graph structure feature information; finally, the target speech data is classified according to the emotional feature information and the graph structure feature information to obtain the classification result of the target speech data. Therefore, on the one hand, although the data content of the target text and the target voice data is the same, the data in voice form and the data in text form have different characteristics. Compared with text, voice can better reflect the user's emotional fluctuations and is conducive to perceiving the feelings of the user who makes the voice. Compared with voice, text is more conducive to extracting semantic features. Therefore, the comprehensive use of graph structure feature information derived from the target text and emotional feature information derived from the target voice data for classification can fully utilize the data advantages of different modalities, improve classification accuracy, and make the classification results of the target voice data more accurate; on the other hand, the longer the time of the target voice data, the higher the difficulty of understanding the semantics of the target text of the target voice data. Therefore, the classification difficulty of the target voice data may increase with the increase of the time length of the target voice data. Considering that the graph structure can better capture long-distance contextual dependencies, by extracting intermediate feature information from the target text and converting the intermediate feature information into graph structure feature information, the advantages of the graph structure can be fully utilized to more accurately express the semantics of the target text, which is conducive to improving the classification accuracy of the target voice data. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments described in this specification. Those skilled in the art can also derive other drawings based on these drawings without inventive work.

[0020] Figure 1 A schematic diagram of an implementation environment of a speech classification method provided in an embodiment of the present application;

[0021] Figure 2 A processing flow chart of a speech classification method provided in an embodiment of the present application;

[0022] Figure 3 A schematic diagram of a model structure provided in an embodiment of the present application;

[0023] Figure 4 A schematic diagram of a grouped multi-head attention mechanism provided in an embodiment of the present application;

[0024] Figure 5 A processing flow chart of another speech classification method provided in an embodiment of the present application;

[0025] Figure 6 This is an algorithm architecture diagram of another speech classification method provided in an embodiment of the present application;

[0026] Figure 7 A schematic diagram of a speech classification device provided in an embodiment of the present application;

[0027] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0029] This specification provides an embodiment of a speech classification method:

[0030] In real life, the phenomenon of malicious users exploiting phone calls to engage in anomalous behavior is increasing. To combat this behavior, the numbers of users engaging in anomalous behavior are often flagged, warned, or blocked. Typically, identifying a specific number for anomalous behavior requires observing its behavioral data for a period of time or receiving numerous reports. However, if a new number is being used for anomalous behavior, it can be difficult to detect it in a timely manner, potentially causing losses to the other party in the conversation.

[0031] If we determine whether a call is abnormal by classifying the call voice data, there is the following problem: in spoken conversations, users usually do not speak complete sentence information in every sentence, especially the content that has been mentioned previously does not need to be repeated. This can easily lead to dependencies between the earlier and later sentences in a complete call conversation. If intent recognition is performed on each individual sentence separately, the actual semantics may be misunderstood, thereby reducing classification accuracy.

[0032] Therefore, in order to solve the above problems, an embodiment of the present application provides a speech classification method.

[0033] The speech classification method provided in one or more embodiments of this specification may be applicable to an implementation environment of the speech classification method, which implementation environment at least includes a server 101 for classifying target speech data.

[0034] The server 101 may be a single server, or a server cluster consisting of several servers, or one or more cloud servers in a cloud computing platform, for classifying target voice data.

[0035] In this implementation environment, when classifying target voice data, the server 101 first obtains the target voice data and the target text of the target voice data; then, performs feature extraction on the target voice data to obtain emotional feature information; then, performs feature extraction on the semantics of the target text to obtain intermediate feature information, and converts the intermediate feature information into graph structure feature information; finally, classifies the target voice data according to the emotional feature information and the graph structure feature information to obtain the classification result of the target voice data. Therefore, on the one hand, although the data content of the target text and the target voice data is the same, the data in voice form and the data in text form have different characteristics. Compared with text, voice can better reflect the user's emotional fluctuations and is conducive to perceiving the feelings of the user who makes the voice. Compared with voice, text is more conducive to extracting semantic features. Therefore, the comprehensive use of graph structure feature information derived from the target text and emotional feature information derived from the target voice data for classification can fully utilize the data advantages of different modalities, improve classification accuracy, and make the classification results of the target voice data more accurate; on the other hand, the longer the time of the target voice data, the higher the difficulty of understanding the semantics of the target text of the target voice data. Therefore, the classification difficulty of the target voice data may increase with the increase of the time length of the target voice data. Considering that the graph structure can better capture long-distance contextual dependencies, by extracting intermediate feature information from the target text and converting the intermediate feature information into graph structure feature information, the advantages of the graph structure can be fully utilized to more accurately express the semantics of the target text, which is conducive to improving the classification accuracy of the target voice data.

[0036] Figure 2 This is a processing flow chart of a speech classification method provided in an embodiment of the present application. Figure 2 The speech classification method provided in this embodiment specifically includes steps S202 to S208:

[0037] Step S202: Acquire target voice data and target text of the target voice data.

[0038] The target voice data can be any piece of voice data, such as call voice data, conference recording data, voice command data, etc.

[0039] The target text may be text obtained by performing speech-to-text processing on the target speech data.

[0040] For example, when the target voice data is call voice data, the target text may be a conversation text. When the target voice data is voice instruction data, the target text may be an instruction text.

[0041] Acquiring the target voice data may include receiving voice data input by the user and using the voice data as the target voice data, or reading voice data from a designated storage area according to a predetermined data identifier and using the read voice data as the target voice data.

[0042] Obtaining the target text of the target voice data may involve performing speech-to-text processing on the target voice data to obtain the target text, or, after previously performing speech-to-text processing on the target voice data to obtain the target text and establishing a correspondence between the target voice data and the target text, querying the target text corresponding to the target voice data based on a data identifier of the target voice data. The data identifier may be used to uniquely identify the corresponding voice data.

[0043] The target speech data is subjected to speech-to-text processing to obtain the target text, and the target speech data may be input into a speech recognition model for speech recognition processing to obtain the target text.

[0044] In specific implementation, the target speech data can be first encoded by the speech encoder in the speech recognition model to obtain speech coding information, and then the speech coding information can be decoded by the first decoder in the speech recognition model to obtain the target text.

[0045] The ordinal numbers such as "first" and "second" appearing in this specification are used to distinguish different features and have no actual meaning, so they will not be elaborated below.

[0046] During specific implementation, the target speech data can also be input into the feature extraction module in the speech recognition model for speech feature extraction processing to obtain first speech feature data, and the first speech feature data is weighted processed by the one-dimensional convolutional layer in the speech recognition model to obtain second speech feature data. The target speech data is subjected to speech recognition processing based on the second speech feature data by the speech recognition module in the speech recognition model to obtain the target text.

[0047] Illustratively, the speech recognition model may adopt a Conformer model, an EfficientConformer model, or any other unlisted model that can be used to process ASR (Automatic Speech Recognition) tasks.

[0048] Exemplarily, the aforementioned speech encoder may be an encoding module in an Efficient Conformer model, and the aforementioned first decoder may be a decoding module in an Efficient Conformer model.

[0049] The Conformer architecture is a neural network widely used in speech recognition tasks. It combines the features of CNNs (Convolutional Neural Networks) and RNNs (Recurrent Neural Networks) to improve its ability to model speech signals. By integrating the local feature extraction capabilities of convolutional networks with the sequence modeling capabilities of recurrent networks, the Conformer model can better capture the temporal and frequency domain information in speech signals.

[0050] The Conformer structure usually includes the following main components:

[0051] Feed-Forward Embedding Layer: Converts the input speech features into embedding vectors suitable for model processing.

[0052] Multi-Head Self-Attention: used to capture long-distance dependencies within the sequence.

[0053] Convolution Module: Contains one or more convolutional layers to extract local features and enhance the model's sensitivity to local patterns.

[0054] Feed-Forward Neural Network: Performs nonlinear transformation on the features at each position to enhance the expressive power of the model.

[0055] Residual Connections and Layer Normalization: Used to stabilize the training process and prevent gradients from disappearing or exploding.

[0056] The design of the Conformer structure enables it to perform well when processing long sequences of data, especially in speech recognition tasks that require fine temporal resolution. Through this structure, the Conformer model can effectively combine and utilize different types of information, thereby achieving excellent performance in a variety of speech processing tasks.

[0057] The Conformer structure and the Conformer model are conceptually different, but they are closely related. The Conformer structure refers to the specific architectural design of the Conformer model, which is a neural network structure that combines the characteristics of CNN and RNN. This structure includes components such as a multi-head self-attention mechanism, convolutional modules, feedforward neural networks, residual connections, and layer normalization. It is used to process sequence data and performs particularly well in speech recognition tasks. The Conformer model is a complete model built based on the Conformer structure. It not only includes the Conformer structure, but also other components of the model, such as input and output layers, loss functions, optimizers, etc., as well as training and inference processes for specific tasks. The Conformer model is a complete solution for solving speech recognition or other sequence modeling problems in practical applications. In short, the Conformer structure is the core component of the Conformer model, and the Conformer model is a complete system built based on this structure.

[0058] The Efficient Conformer model is a variant of the Conformer model that can be used to improve computational efficiency while maintaining high performance.

[0059] The core module of the Efficient Conformer model is a stack of MHSA (Multi-Headed Self-Attention) and convolution modules. When using the Efficient Conformer model for speech recognition, the target speech data must first be preprocessed to convert the original audio file (for example, .wav or .flac) into a format that the model can process.

[0060] The specific implementation process of preprocessing includes steps (a1) to (a3) ​​as shown below:

[0061] (a1) Windowed slice:

[0062] The continuous speech signal is divided into a series of short time segments, usually called "frames." The duration of each frame can be set within a preset duration range of [10 milliseconds, 30 milliseconds]. For example, each frame can be configured to last 25 milliseconds. To ensure smooth transitions between frames, there is a certain amount of overlap between them. For example, if the window is moved in steps of 10ms, the time range corresponding to the first frame is [0 milliseconds, 25 milliseconds), the time range corresponding to the second frame is [10 milliseconds, 35 milliseconds), the time range corresponding to the third frame is [20 milliseconds, 45 milliseconds), and so on.

[0063] (a2) Signal conversion:

[0064] After slicing, the time-domain signal is converted to the frequency domain by applying an STFT (short-time Fourier transform) to each frame. This yields a representation of time versus frequency intensity, known as a spectrogram. A Mel filter bank is applied to the resulting spectrogram. A Mel filter bank is a set of triangular filters that covers the entire auditory spectrum, with each filter width designed according to the Mel scale, which is based on how the human ear perceives sounds of different frequencies. This process produces a Mel spectrum, a nonlinear representation of frequency.

[0065] (a3) Feature extraction:

[0066] Extract MFCC (Mel Frequency Cepstrum Coefficient) features from the Mel spectrum.

[0067] After preprocessing is completed, the MFCC features of the target speech data can be used as input to the Efficient conformer model.

[0068] The Efficient Conformer model is a deep learning model that combines features of CNN and Transformer networks. Transformer is a deep learning model architecture used for NLP and other sequence-to-sequence tasks.

[0069] A fully connected layer in the Efficient Conformer model processes the MFCC features to generate query (q), key (k), and value (v) vectors, which are used by the self-attention mechanism of the Transformer network.

[0070] The self-attention mechanism used by the Efficient Conformer model allows the model to consider information from all other frames when processing one frame. This enables the model to capture long-term dependencies in speech signals, thereby improving recognition accuracy.

[0071] Next, combine Figure 3 and Figure 4 The specific implementation process of performing text-to-speech processing on the target speech data to obtain the target text is exemplified. Figure 3A schematic diagram of a model structure provided in an embodiment of the present application exemplifies a local model structure of a speech recognition model. The speech recognition model may be an improved EfficientConformer model. Figure 4 A schematic diagram of a grouped multi-head attention mechanism provided in an embodiment of the present application.

[0072] like Figure 3 As shown, the speech recognition model includes a speech enhancement layer 302, a convolution downsampling layer 304, a linear layer 306, a random dropout layer 308, a Conformer block 310, a downsampling block 312, a Conformer block 314, a downsampling block 316, a Conformer block 318, and so on, which are connected in series.

[0073] There may be one or more Conformer blocks 310, which together with the downsampling block 312 constitute the first stage (Stage 1) of the speech recognition model. There may be one or more Conformer blocks 314, which together with the downsampling block 316 constitute the second stage (Stage 2) of the speech recognition model. There may be one or more Conformer blocks 318, which together with the downsampling block 316 constitute the third stage (Stage 3) of the speech recognition model.

[0074] The speech enhancement layer 302 may be used to perform speech enhancement processing on target speech data.

[0075] Exemplarily, the speech enhancement layer 302 may adopt the SpecAugment method.

[0076] SpecAugment is a data augmentation technique for ASR tasks. It increases the diversity of training data by performing specific transformations on the spectrogram of speech signals, thereby improving the model's generalization capabilities. SpecAugment is designed to help models better handle the various speech variations that may be encountered in real-world applications.

[0077] SpecAugment mainly includes the following transformations:

[0078] Time Masking: Randomly select a continuous region in the time dimension of the spectrogram and set all feature values ​​in the region to zero to simulate missing or noise in the speech signal.

[0079] Frequency Masking: Randomly select a continuous region in the frequency dimension of the spectrogram and set all eigenvalues ​​in the region to zero to simulate spectral distortion or filtering effects in speech signals.

[0080] Time Warping: Performs nonlinear distortion on the spectrogram in the time dimension to simulate the time stretching effect in speech signals.

[0081] These transformations can be randomly applied to the input spectrogram during training, with each training iteration potentially applying a different transformation, enabling the model to learn more robust feature representations. SpecAugment is a simple and effective data augmentation method that significantly improves the performance of speech recognition models without requiring additional data collection or annotation.

[0082] Convolution downsampling 304 is also known as convolution downsampling. Convolution downsampling is a common technique in image processing and computer vision used to reduce the spatial resolution of an image or feature map. Convolution downsampling is an important step in feature extraction and model optimization in deep learning. It helps reduce overfitting, improves model generalization, and accelerates computation.

[0083] Linear layer 306 is a linear layer. In deep learning, a linear layer generally refers to a simple linear transformation layer, also known as a fully connected layer or dense layer. The role of a linear layer in a neural network is to learn linear relationships in the input data and map these relationships to the output space.

[0084] The random dropout layer 308 is the Dropout layer. The Dropout layer is a commonly used regularization technique in deep learning. Its main purpose is to prevent the neural network from overfitting, that is, the model performs well on the training data, but performs poorly on unseen test data. The working principle of the Dropout layer is to randomly "drop out" (i.e. temporarily remove) a part of the neurons in the network during the training process. Specifically, for each neuron in the Dropout layer, there is a certain probability of being randomly set to 0 in each training iteration, which means that in this iteration, the neuron will not have any effect on the output of the previous layer. The neurons that are not dropped retain their original weights. This process of randomly dropping neurons can be regarded as a form of model averaging, because it forces the network not to rely on any single neuron, but to learn to extract features from multiple different network structures. In the test phase, all neurons are retained, but their weights are usually multiplied by the Dropout probability to maintain the consistency of the desired output between the training and test phases.

[0085] Conformer block 310, Conformer block 314 and Conformer block 318 are all Conformer Blocks. For Conformer Block, please refer to the corresponding description of the Conformer structure mentioned above.

[0086] Downsampling blocks 312 and 316 are both called downsampling blocks. A downsampling block is a structure used in deep learning models to reduce the spatial resolution of feature maps. This type of block is commonly used in CNNs, particularly when processing images or time series data, to reduce computational effort and extract higher-level features. The purpose of a downsampling block is to reduce the spatial dimensionality of the data without losing important information.

[0087] like Figure 3 As shown, the downsampling block 312 includes: a feedforward module 31202, a grouped multi-head attention module 31204, a convolutional downsampling module 31206, an effective channel attention 31208, a feedforward module 31210, and a layer normalization 31212.

[0088] Feedforward modules 31202 and 31210 are both feedforward modules. In deep learning, a feedforward module typically refers to a small neural network consisting of multiple fully connected layers (or linear layers) that perform a series of linear transformations and nonlinear activation functions on the input data. This module is a fundamental component of many neural network architectures, including CNN and Transformer models.

[0089] The grouped multi-head attention module 31204 is also known as the Grouped Multi-Head Self-Attention (MHSA) module. Grouped MHSA is an attention mechanism used in deep learning models, particularly in the Transformer architecture. While maintaining the advantages of multi-head self-attention, it reduces computational and memory requirements, offering significant advantages when processing high-dimensional feature maps.

[0090] The convolution downsampling module 31206 is the Convolution Downsampling Module, and the corresponding description of the convolution downsampling 304 can be referred to.

[0091] The effective channel attention 31208 is the ECA (Efficient Channel Attention) module.

[0092] The ECA module can perform adaptive channel calibration, strengthening channels with strong expressive capabilities and suppressing noisy channels.

[0093] By introducing the ECA module between the convolutional downsampling module 31206 and the feedforward module 31210, the features of different channels can be adaptively weighted, highlighting important channel features and suppressing unimportant channel features. This helps the network better capture key information in the input data and enhances feature representation capabilities. The ECA module can learn channel attention weights through one-dimensional convolution and apply them to each channel of the input feature map.

[0094] Layer normalization 31212 is also known as Layer Norm. Layer normalization is a technique used in deep learning models to stabilize the training process.

[0095] After introducing the structure of the speech recognition model, we will now illustrate some of the steps in the process of converting target speech data into text using the speech recognition model:

[0096] First, the target speech data is preprocessed.

[0097] Specifically, the speech sequence X of the target speech data is [x1, x2, ..., x n ] is divided into multiple segments, with one frame cut every 25ms. n is an integer greater than 1. STFT processing is performed on each frame to obtain the frequency domain features [X1, X2, ..., X n The frequency domain features are converted into Mel spectrum feature sequences [f1, f2, ..., fn ], from the Mel spectrum feature sequence [f1, f2, ..., f n ] and use the MFCC features as the input data of the speech recognition model.

[0098] Then, the MFCC features are converted into a query vector q, a key vector k, and a value vector v through the linear layer 306 in the speech recognition model, and an attention operation is performed based on q, k, and v.

[0099] In the self-attention mechanism, the query vector, key vector, and value vector are three key concepts that form the core components of attention computation. The self-attention mechanism is widely used in the Transformer model in natural language processing (NLP) and other sequence modeling tasks.

[0100] The following is a detailed explanation of these three concepts:

[0101] The query vector represents the current position or element and is used to query information about other positions or elements in the sequence. In the self-attention mechanism, each position has a query vector, which is used to calculate the similarity or correlation with the key vectors at other positions in the sequence. The query vector is usually obtained by multiplying the input features with the query weight matrix.

[0102] A key vector represents each position or element in the sequence that responds to the query vector's "query." The key vector contains information about each position in the sequence and is used to calculate similarity with the query vector. Similar to the query vector, the key vector is generated by multiplying the input features by a key weight matrix.

[0103] The value vector represents the actual information or features of each position or element in the sequence. In the attention calculation, the value vector is used to aggregate information based on the similarity between the query vector and the key vector. The value vector is obtained by multiplying the input features with the value weight matrix.

[0104] In the self-attention mechanism, the calculation of these three vectors is usually performed in parallel, and each vector is obtained by linearly transforming the input features.

[0105] q i =FC(F i ) (1)

[0106] k i =FC(F i ) (2)

[0107] v i =FC(F i ) (3)

[0108]

[0109] In the above formulas (1)-(4):

[0110] F i Represents the MFCC features of the i-th frame, FC(F i ) indicates that the MFCC features of the i-th frame are transformed by the linear layer 306. The number of linear layers 306 can be multiple, and different linear layers 306 can be used in the process of obtaining q, k, and v. i k represents the query vector of the i-th frame. i Represents the key vector of the i-th frame. v i A vector of values ​​representing the i-th frame.

[0111] Attention(F i ) represents the feature information obtained after the attention operation is performed on the MFCC features of the i-th frame. represents the vector obtained by transposing the key vector of the first frame, Similarly, transposition refers to converting the rows of a vector into columns, or converting the columns of a vector into rows. M is used to represent the number of frames included in the target speech data, and M can be an integer greater than 1. It is used to represent the vector obtained by transposing the query vector of the i-th frame. i Represents the key vector of the i-th frame. v i represents the value vector of the i-th frame. The Softmax function is a normalized exponential function that "compresses" a K-dimensional vector z containing arbitrary real numbers into another K-dimensional real vector σ(z), such that each element ranges between (0, 1) and the sum of all elements is 1. K is an integer greater than 0. The function exp represents the natural exponential function, which is an exponential function with the real number e as its base.

[0112] For example, the number of attention heads in the speech recognition model can be 4, and each attention head can be used to process the MFCC feature F of the i-th frame. i Get the corresponding Attention(F i ). That is, the MFCC features of each frame can be used to generate Attention( 1) ,...,Attention(4).

[0113] The grouped multi-head attention module 31204 can reduce the attention computation complexity from O(n 2d) Reduced to O(n 2 d / g), where n is the time dimension, d is the hidden layer dimension, and g is the group size. For example, the four attention heads can be divided into two groups to reduce the computational overhead, and then the relationship within the sequence can be calculated through grouped multi-head self-attention.

[0114] like Figure 4 As shown, the dimension information of the MFCC feature 402 is d*n, and illustratively, d=3, n=9.

[0115] In the process of grouping and attention processing of the MFCC features 402 by the grouped multi-head attention module 31204, grouping can be understood as a reshaping of the feature structure. The data content remains unchanged before and after the reshaping, but the structure changes.

[0116] The grouped multi-head attention module 31204 reshapes the MFCC features 402 to obtain the first intermediate features 404. The dimensional information of the first intermediate features 404 is changed to (d*g)*(n / g). For example, g=3.

[0117] In a specific example, the MFCC feature 402 can be divided into feature information of 1-9 dimensions according to a preset direction, and in the first intermediate feature 404, the feature information of the 1st, 4th, and 7th dimensions together constitute a row, the feature information of the 2nd, 5th, and 8th dimensions together constitute a row, and the feature information of the 3rd, 6th, and 9th dimensions together constitute a row.

[0118] The grouped multi-head attention module 31204 performs attention processing on the first intermediate feature 404 to obtain a second intermediate feature 406. The dimension information of the second intermediate feature 406 is changed to (d*g)*(n / g).

[0119] The grouped multi-head attention module 31204 reshapes the second intermediate feature 406 to obtain the attention feature 408.

[0120] Next, the specific use of effective channel attention 31208 is illustrated as follows:

[0121] The feature map X∈R output by the convolutional downsampling module 31206 CxHxW is the input data of the effective channel attention 31208. C represents the number of channels (Channels), H represents the height (Height), and W represents the width (Width). C, H, and W are all integers greater than or equal to 1.

[0122] Effective channel attention 31208 first performs global average pooling on the feature map X to obtain the channel descriptor z∈R C .

[0123]

[0124] Among them, Z c Indicates the channel descriptor of the cth channel, c represents the channel index, x c (i, j) represents the eigenvalue at position (i, j) on the cth channel. C represents the number of channels, H represents the height, and W represents the width. c, i, j, C, H, and W are all integers greater than or equal to 1.

[0125] Then, the effective channel attention 31208 uses a one-dimensional convolutional layer to learn the channel attention weight α∈R C .

[0126] α=σ(Conv1D(z,W α )) (6)

[0127] Among them, σ represents the Sigmoid activation function, Conv1D represents the one-dimensional convolution operation, z is the aforementioned channel descriptor, and W α Represents the convolution kernel parameters.

[0128] Through one-dimensional convolution, effective channel attention 31208 can adaptively adjust the weight of each channel, highlighting important channel features and suppressing unimportant channel features.

[0129] Finally, the learned channel attention weights are applied to each channel of the input feature map to obtain the calibrated feature map X':

[0130] x'{C}=α{c}·x c (7)

[0131] Among them, x'{c} represents the calibrated c-th channel feature map, α{c} represents the attention weight of the c-th channel, and x c Represents the feature information of the feature map X on the cth channel.

[0132] Compared with the traditional SE (Squeeze-and-Excitation) attention mechanism, the effective channel attention 31208 can avoid the use of fully connected layers and adopt one-dimensional convolution to learn channel attention, which reduces the number of parameters and computation, making the network more lightweight. In the process of reducing the resolution of the feature map in the downsampling block 312, the channel attention mechanism can be used to optimize the feature representation, which is beneficial to reduce computational overhead and improve the efficiency of the model while maintaining model performance.

[0133] For example, the input and output of the effective channel attention 31208 can be expressed by the following formula:

[0134] C i =ECA(Conv(Grouped MHSA(F i ))) (8)

[0135] Among them, Grouped MHSA (F i ) can be used to represent the attention features obtained after the grouped multi-head attention module 31204 performs grouping processing on the MFCC features of the i-th frame and the attention processing, which can refer to the aforementioned attention features 408. The Conv function is used to represent the downsampling processing, specifically, the convolution downsampling module 31206 performs downsampling processing on the attention features. ECA is used to represent the effective channel attention 31208 performing average pooling processing and one-dimensional convolution processing on the input data to obtain the output data. Conv(Grouped MHSA(F i )) is the input data of effective channel attention 31208, C i It is the output data of the effective channel attention 31208, that is, the aforementioned calibrated feature map.

[0136] Then, after obtaining C i Afterwards, the target speech features are obtained through residual connection and layer normalization, which can be referred to the following formula (9):

[0137]

[0138] in, is the target speech feature of the i-th frame, F i is the MFCC feature of the i-th frame, C i is the calibrated feature map of the i-th frame, Grouped MHSA (F i ) can be used to represent the attention features obtained after the grouped multi-head attention module 31204 performs grouping processing and attention processing on the MFCC features of the i-th frame, which can refer to the aforementioned attention features 408, and LayerNorm can refer to the aforementioned layer normalization 31212.

[0139] Finally, the target speech features are classified through the Softmax classification layer to obtain the speech recognition result, that is, the target text.

[0140] The above content merely illustrates some of the steps for converting target speech data into text using a neural network-based speech recognition model in the field of deep learning to obtain target text. It is only an optional implementation method for obtaining the target text and does not constitute a special limitation on the present invention. In addition, obtaining the target text can also be achieved in other ways, for example, using a statistical model to handle the sequence labeling problem in speech recognition to achieve speech recognition, or using linguistic rules and knowledge to perform speech recognition, for example, using pronunciation rules for phonemes, syllables, and words, etc. Alternatively, it can also be achieved by combining other processing processes based on the above-mentioned process of converting target speech data into text using a neural network-based speech recognition model in the field of deep learning to obtain the target text, which is not limited in this specification.

[0141] Step S204: extract features from the target speech data to obtain emotional feature information.

[0142] The emotion characteristic information may reflect the emotion category of the user who utters the target voice data.

[0143] For example, the emotion category includes but is not limited to: positive emotions such as anticipation, happiness and joy, or negative emotions such as fear, tension and worry, etc.

[0144] The target speech data is subjected to feature extraction processing to obtain emotional feature information. This can be done by first extracting speech features of one or more dimensions, inputting the speech features into a pre-trained emotion recognition model for emotion recognition processing, obtaining an emotion recognition result, and using the emotion recognition result as the emotional feature information.

[0145] Commonly used emotion recognition models include but are not limited to: SVM (Support Vector Machine), random forest, deep learning models, etc.

[0146] For example, the speech features of multiple dimensions may include one or more of the following: acoustic features, prosodic features, spectral features, non-acoustic features, and the like.

[0147] Acoustic features, such as MFCC features, LPC (linear predictive coding), etc.

[0148] Prosodic features, such as pitch, volume, speed, pauses, etc., can reflect the speaker's emotional state.

[0149] Spectral features, such as spectral centroid, spectral bandwidth, spectral tilt, etc., can capture the frequency domain characteristics of speech signals.

[0150] Non-acoustic features, such as the speaker's identity, gender, age, etc., although these features do not directly reflect emotions, may be related to emotional expression.

[0151] In a specific implementation method, feature extraction is performed on the target speech data to obtain emotional feature information, including: encoding the target speech data to obtain first encoded information of the target speech data; performing attention processing on the first encoded information to obtain an emotional feature vector; performing nonlinear transformation on the emotional feature vector to obtain an emotional feature vector after transformation; classifying the emotional feature vector after transformation to obtain probability distribution information of the target speech data in multiple preset emotional categories, and using the probability distribution information as emotional feature information.

[0152] Feature extraction is performed on the target speech data to obtain emotional feature information. The target speech data can be first encoded by a speech encoder in a speech recognition model to obtain first encoded information, and then the first encoded information is decoded by a second decoder in the speech recognition model to obtain emotional feature information.

[0153] Exemplarily, the aforementioned speech encoder may be an encoding module in an Efficient Conformer model.

[0154] The aforementioned second decoder may include an attention pooling layer, a fully connected layer, and a sentiment classification layer connected in series.

[0155] The Attention Pooling Layer is a feature aggregation technique used in deep learning models. It draws on the concept of the Attention Mechanism. Initially widely used in natural language processing, the Attention Mechanism simulates how humans allocate attention when processing information. This allows the model to dynamically focus on certain parts of the input sequence, thereby better understanding and processing the information.

[0156] In image processing and computer vision tasks, attention pooling layers are often used to extract useful information from a set of feature vectors. These feature vectors may come from different layers of a CNN or features extracted from different regions of an image. The attention pooling layer learns a weight distribution and performs a weighted summation of these feature vectors, resulting in a more compact and information-rich representation.

[0157] The attention pooling layer can be used to perform an attention operation on the first encoded information to obtain a sentiment feature vector.

[0158] The fully connected layer can be used to perform nonlinear transformation on the emotional feature vector to obtain the transformed emotional feature vector.

[0159] The Sentiment Classification Layer is a specific layer in a deep learning model used for sentiment analysis tasks, typically located at the last layer of the model. Sentiment analysis, also known as opinion mining, refers to the process of automatically determining and extracting subjective information from text. The goal is to identify and classify the emotional tendencies in the text, such as positive, negative, or neutral.

[0160] In a deep learning model, the role of the sentiment classification layer is to map the features extracted by the model from lower layers (such as convolutional layers, recurrent layers, or embedding layers) to sentiment categories.

[0161] The emotion classification layer can be used to classify the transformed emotion feature vector, obtain the probability distribution information of the target speech data in multiple preset emotion categories, and use the probability distribution information as emotion feature information.

[0162] For example, the feature sequence h=(h1, h2, ..., h T ) represents the speech coding information output by the speech encoder, where h t ∈R D Refers to the tth element in the feature sequence h, R is a real number, D is the feature dimension, and T is an integer greater than 1.

[0163] Next, the attention pooling layer can calculate the attention weight based on the feature sequence h. For details, please refer to the following formula (10):

[0164]

[0165] Where ω∈R D , and b∈R D , ω and b are two learnable parameters, R is a real number, and D is the feature dimension. t Used to represent the attention weight. h t Refers to the tth element in the feature sequence h, h i Refers to the i-th element in the feature sequence h. Both i and t are integers greater than or equal to 1 and less than or equal to T.

[0166] The attention pooling layer can also transform the feature sequence h according to the attention weight a t Perform weighted summation to obtain the sentiment feature vector. For details, please refer to the following formula (11):

[0167]

[0168] Where v∈R D, v is the sentiment feature vector, R is a real number, and D is the feature dimension. t Refers to the tth element in the feature sequence h, a t Used to represent the attention weight. t is an integer greater than or equal to 1 and less than or equal to T.

[0169] The fully connected layer can perform nonlinear transformation on the emotional feature vector v through the ReLU activation function to obtain the transformed emotional feature vector z. r Unit) is an activation function commonly used in deep learning. Its full name is rectified linear unit.

[0170] z=ReLU(W1v+b1) (12)

[0171] Among them, W1 and b1 are two learnable parameters, W1∈R H×D , b1∈R H , R is a real number, H is the number of hidden units in the fully connected layer, and D is the feature dimension.

[0172] The emotion classification layer can classify the transformed emotion feature vector z to obtain emotion feature information. For details, refer to the following formula (13):

[0173] o=W2z+b2 (13)

[0174] Among them, W2 and b2 are two learnable parameters, W2∈R C×H , b1∈R C , R is a real number, H is the number of hidden units in the fully connected layer, C is the number of emotion categories, that is, the number of multiple preset emotion categories is C, z represents the emotion feature vector after conversion, o represents the first probability information, specifically, the first probability information o can include C probability values, each probability value can reflect the probability information of the target speech data in a preset emotion category.

[0175] Finally, the Softmax function can be used for activation, as shown in the following formula (14):

[0176] p=Softmax(o) (14)

[0177] Wherein, o represents the first probability information, p can be used to represent the predicted probability value of the target speech data in multiple preset emotion categories, p∈R C , R is a real number, C is the number of emotion categories, that is, the number of multiple preset emotion categories is C.

[0178] The Softmax function can be used to convert the first probability information o into C predicted probability values ​​with values ​​between 0 and 1. The C predicted probability values ​​together constitute p. After obtaining p, p can be used as the emotional feature information of the target speech data.

[0179] It should be noted that, in this implementation, the reason why the predicted probability values ​​of the target speech data in multiple preset emotion categories are used as the emotion feature information is that when the emotion feature information is used as the classification basis for determining whether the target speech data is abnormal, compared with a single emotion category, the predicted probability values ​​of the target speech data in multiple preset emotion categories can provide richer information, can reflect the relative importance of different preset emotion categories, and is conducive to improving the classification accuracy of subsequent steps. Moreover, the emotion feature information can be a probability set represented by a vector with a fixed data length, which is convenient for subsequent processing and analysis, avoids additional parameters and calculations, and simplifies the model structure.

[0180] Step S206 , extracting features from the semantics of the target text to obtain intermediate feature information, and converting the intermediate feature information into graph structure feature information.

[0181] The semantics of the target text are subjected to feature extraction processing to obtain intermediate feature information. This can be achieved by encoding each sentence in the target text through a pre-trained language model to obtain a sentence vector representation of each sentence. The sentence vector representation contains the semantic information of the sentence, and the sentence vector representation of each sentence in the target text is used as the intermediate feature information.

[0182] Exemplarily, the pre-trained language model may be BERT or RoBERTa.

[0183] Extracting intermediate feature information representing semantic meaning from the target text using a pre-trained language model is merely an optional implementation and does not limit the present invention. Alternatively, intermediate feature information can be obtained by extracting semantic meaning from the target text using various methods, such as word embeddings, topic models, and matrix factorization.

[0184] Furthermore, after acquiring intermediate feature information and before converting it into graph-structured feature information, some data preparation can be performed based on this intermediate feature information for subsequent conversion. Specifically, a distance position encoding of each sentence can be superimposed on its intermediate feature information. This distance position encoding is used to indicate the position of the corresponding sentence in the target text. The following example illustrates the specific implementation of this data preparation process, using the extraction of intermediate feature information from the target text using a pretrained language model.

[0185] When the target text is a dialogue text and the dialogue text includes multiple rounds of speech, a dialogue context can be constructed based on the dialogue text. Specifically, it can be assumed that the multi-round dialogue sequence is X=x1, x2, ..., x n , where x i Represents the i-th round of speech. Each round of speech corresponds to a sentence in the multi-round dialogue.

[0186] In specific implementation, each round of speech can be encoded through a pre-trained language model:

[0187] h i =BERT(x i ), i=1,...,n(15)

[0188] Among them, h i Indicates the speech x in round i i The semantic features of , that is, the intermediate feature information of the i-th sentence.

[0189] In h i Superimposing a distance position code on it can get p i , indicating the position of the statement in a multi-turn dialogue. For example, a multi-turn dialogue includes 3 statements.

[0190] x1: Hello? I'm xxxxxx.

[0191] x2: Hello.

[0192] x3: I am calling you because I found that you have overseas consumption records. Was this consumption made by you personally?

[0193] After encoding, the intermediate feature information of each sentence can be represented as a fixed-length vector, for example, with a length of 768.

[0194] h1: [0.1, 0.3, 0.5, ..., 0.6] # vector of length 768

[0195] h2: [0.7, 0.2, 0.4, ..., 0.8]

[0196] h3: [0.3, 0.5, 0.1, ..., 0.9]

[0197] In h i Superimpose a distance position code on it to get p i , for example, an integer can be added after the above code to indicate the statement position:

[0198] h1: [0.1, 0.3, 0.5, ..., 0.6, 1] # plus position 1

[0199] h2: [0.7, 0.2, 0.4, ..., 0.8, 2] # plus position 2

[0200] h3: [0.3, 0.5, 0.1, ..., 0.9, 3] # plus position 3

[0201] Converting the intermediate feature information into graph structure feature information may be converting the intermediate feature information into graph structure data, performing feature extraction processing on the graph structure data, and obtaining the graph structure feature information.

[0202] In a specific implementation method, the target text includes a first subtext and a second subtext; the intermediate feature information includes semantic features of the first subtext and semantic features of the second subtext; the intermediate feature information is converted into graph structure feature information, including: converting the semantic features of the first subtext into a first node representing the first subtext; converting the semantic features of the second subtext into a second node representing the second subtext; determining a first edge connecting the first node and the second node based on the relationship between the first subtext and the second subtext; constructing graph data based on the first node, the second node and the first edge; encoding the graph data through a graph neural network to obtain graph structure feature information.

[0203] Based on the order and similarity between sentences, an undirected graph G = (V, E) can be constructed, where the node set V represents all sentences in the target text, and the edge set E represents the relationship between sentences.

[0204] The relationships between statements include but are not limited to: adjacency, coreference, dependency, etc.

[0205] This implementation uses a target text consisting of two subtexts, a first subtext and a second subtext, as an example to illustrate how to convert intermediate feature information into graph structure feature information. In practice, the target text can include two subtexts or at least three subtexts. In this case, refer to the corresponding instructions in this implementation.

[0206] For example, the first subtext may be the i-th sentence in the conversation text, and the second subtext may be the j-th sentence in the conversation text. i and j are both integers greater than 0, and i≠j. The semantic features of the first subtext can be expressed as h i Indicates that the semantic features of the second subtext can be expressed as h j express.

[0207] According to h i Construct the first node representing the first subtext, according to h j A second node representing the second subtext is constructed.

[0208] In a specific implementation, determining the first edge connecting the first node and the second node based on the relationship between the first subtext and the second subtext can be achieved by:

[0209] Calculate the similarity score s between the first subtext and the second subtext using cosine similarity ij , refer to the following formula (16):

[0210]

[0211] Among them, h i represents the semantic features of the first subtext, h j Represents the semantic features of the second subtext, ||h i || represents the norm of the semantic features of the first subtext, ||h j || represents the norm of the semantic features of the second subtext, cosine function is the cosine function, s ij Represents the similarity score between the first subtext and the second subtext.

[0212] A similarity threshold is configured. When the similarity score between the first subtext and the second subtext is greater than or equal to the similarity threshold, it is determined that the first node and the second node have a connection relationship, and a first edge connecting the first node and the second node is constructed; when the similarity score between the first subtext and the second subtext is less than the similarity threshold, it is determined that the first node and the second node do not have a connection relationship, and there is no need to construct the first edge connecting the first node and the second node.

[0213] Graph data is constructed based on the first node, the second node, and the first edge. The graph data may include at least one node and at least one edge. Each node represents a statement, and each edge represents a relationship between two statements.

[0214] By encoding graph data through graph neural networks and obtaining graph structural feature information, this can be achieved in the following ways:

[0215] First, the edge weight matrix E is calculated based on the similarity score and threshold:

[0216]

[0217] Among them, s ij represents the similarity score between the first subtext and the second subtext, τ represents the similarity threshold, and e ij Indicates the weight information of the first edge connecting the first node and the second node.

[0218] The edge weight matrix E includes the weight information of each edge in the graph data. The method for obtaining the weight information of each edge can refer to the corresponding description part of the first edge above.

[0219] Then, construct the adjacency matrix A based on the edge weight matrix E:

[0220] A=E+E T (18)

[0221] E T represents the matrix obtained by transposing the edge weight matrix E.

[0222] Next, based on the adjacency matrix A, the graph neural network is used to encode the graph data to obtain the graph structure feature information.

[0223] Exemplarily, the graph neural network can be a GCN (Graph Convolutional Network) or a GAT (Graph Attention Network).

[0224] The following formula (19) can be used to encode graph data using GAT:

[0225]

[0226] in, Represents the i-th node feature of the L-th layer, where L is an integer greater than or equal to 1. represents the i+1th node feature of the Lth layer, N(i) represents the set of neighbor nodes of the i-th node, and W (L) is a learnable weight matrix.

[0227] j is a loop variable used in the accumulation process to control the range of accumulation. Specifically, j∈N(i) can represent the jth neighbor node in the neighbor node set of the i-th node, and α ij It refers to the attention coefficient between the i-th node and the j-th neighbor node of the i-th node.

[0228] After encoding through multiple layers of GNNs, pooling can be used to aggregate node features into a graph-level representation. Common pooling operations include average pooling, max pooling, and attention pooling. Attention pooling adaptively assigns weights based on the importance of node features, effectively aggregating node information and yielding a more expressive graph-level representation.

[0229] For example, the attention pooling method can be used to calculate the attention weight of each node:

[0230]

[0231] Among them, a iis the attention weight of the i-th node in the graph data, w is the learnable attention weight vector, represents the feature vector output by the i-th node in the L-th layer GNN, where L is an integer greater than or equal to 1, and w T Refers to the vector obtained by transposing the learnable attention weight vector w, and V represents the set of all nodes in the graph data. j is the loop variable used in the accumulation process to control the range of accumulation. Specifically, j∈V can represent the jth node among all the nodes in the graph data. Represents the feature vector output by the j-th node in the L-th layer GNN.

[0232] Next, by traversing each node in the graph data and using the attention weights to perform weighted summation on the node features, we can obtain a graph-level representation:

[0233]

[0234] Among them, h G Represents the graph structure feature information of the target text, V represents the set of all nodes in the graph data, α i is the attention weight of the i-th node in the graph data, Represents the feature vector output by the i-th node in the L-th layer GNN, where L is an integer greater than or equal to 1.

[0235] Graph neural networks can model the contextual dependencies of conversational text. By stacking multiple layers of graph neural networks, the model can capture long-range contextual dependencies in conversations and learn more comprehensive and accurate sentence representations. When processing conversational text that includes multiple rounds of dialogue, compared to directly using the intermediate semantic feature information encoded by a pre-trained language model to assist in the classification of abnormal speech data, first encoding the intermediate feature information, then converting it into graph data, and then performing feature extraction on the graph data to obtain graph structural feature information. Using this graph structural feature information to assist in the classification of speech data can better capture long-range contextual dependencies, make semantic representation more accurate, and thus help improve classification accuracy.

[0236] Step S208 , classifying the target speech data according to the emotion feature information and the graph structure feature information to obtain a classification result of the target speech data.

[0237] Speech detection refers to the process of analyzing and processing speech signals using various techniques and algorithms to identify, classify, or extract specific information from the speech. This process can involve multiple subtasks, including speech recognition, speaker identification, emotion recognition, and voice activity detection. The goal of speech detection is to extract useful information from speech signals and make decisions or perform specific actions based on this information.

[0238] Specifically, in this embodiment, the goal of classifying the target speech data may be to extract emotional feature information and graph structure feature information from the speech signal of the target speech data as useful information, and determine the classification result of the target speech data from multiple predetermined candidate classification results based on this information. In this embodiment, by classifying the target speech data, speech detection of the target speech data can be achieved.

[0239] The classification result of the target speech data obtained in this step may be the execution result of a pre-configured speech detection task.

[0240] That is, in the process of executing the speech detection task for the target speech data, the speech classification method provided in this embodiment can be used to process the target speech data to obtain the classification result of the target speech data, and the classification result can be used as the execution result of the speech detection task.

[0241] For each target speech data, the classification result of the target speech data may be one of a plurality of candidate classification results, and the plurality of candidate classification results may be determined by a pre-configured speech detection task.

[0242] For example, the speech detection task is used to detect whether the target speech data is related to abnormal behavior. The classification result of the target speech data may indicate that the target speech data is related to abnormal behavior, or the classification result of the target speech data may indicate that the target speech data is not related to abnormal behavior.

[0243] Alternatively, the speech detection task is used to detect whether the target speech data includes risk data content, and the classification result of the target speech data may indicate that the target speech data includes risk data content, or the classification result of the target speech data may indicate that the target speech data does not include risk data content.

[0244] Alternatively, the speech detection task is used to detect the emotional tendency of the target speech data, and the classification result of the target speech data can indicate that the emotional tendency of the target speech data is one of multiple preset emotional tendencies, and so on.

[0245] The aforementioned pre-configured speech detection task may be any detection task for detecting target speech data, and may use the semantic features and emotional features of the target speech data as a reference in the detection process.

[0246] In a specific implementation, the classification result of the target voice data may be a classification result based on the data content of the target voice data, or a classification result based on the emotional tendency of the target voice data, and so on.

[0247] When the classification result of the target voice data is the classification result for the data content of the target voice data, the reason for combining the emotional feature information and the graph structure feature information to classify the target voice data is that in real conversations between non-abnormal users, the speaker's emotional tendency and the speech content are usually matched, that is, there is an implicit correspondence between the emotional tendency and the speech content. However, if there is an abnormal user who wants to deceive the other party in the conversation, his speech content may seem normal. If his acting skills are not up to standard, his true emotions may be exposed in the conversation, making the emotional tendency inconsistent with the speech content. Therefore, combining the emotional feature information and the graph structure feature information as the classification basis can improve the classification accuracy of the data content of the target voice data.

[0248] When the classification result of the target speech data is the classification result of the emotional tendency of the target speech data, the reason for combining the emotional feature information and the graph structure feature information to classify the target speech data is that in real conversations, when the speaker is satirizing the other party, he may say very impolite words in a very friendly tone. If the emotional tendency of the target speech data is determined only from the emotional feature information, sarcasm and criticism may be mistaken for encouragement, resulting in a serious misunderstanding of the emotional tendency of the target speech data. Therefore, combining the emotional feature information and the graph structure feature information as the classification basis can improve the classification accuracy of the emotional tendency of the target speech data.

[0249] During specific implementation, the classification result of the target voice data may also be a classification result of whether the target voice data is related to abnormal behavior.

[0250] For example, in a call scenario, the voice classification method provided in this embodiment is used to classify the conversational voice data generated in a call to obtain a classification result of the conversational voice data, which can be used to indicate whether the call is related to abnormal behavior.

[0251] When the classification result of the target speech data is a classification result for whether the target speech data is related to abnormal behavior, the reason for combining the emotional feature information and the graph structure feature information to classify the target speech data is that, on the one hand, speech related to abnormal behavior usually has some fixed words, and on the other hand, the speaker related to abnormal behavior is likely to be deceptive. If the speaker's interlocutor is not fooled, the speaker is likely to reveal his true emotions during the speech. Therefore, combining the emotional feature information and the graph structure feature information as the classification basis can improve the accuracy of judging whether the target speech data is related to abnormal behavior.

[0252] For example, the classification result of the target voice data can be used to indicate whether the data content of the target voice data is abnormal. Alternatively, the classification result of the target voice data can also be used to indicate the emotional tendency of the target voice data. Alternatively, the classification result of the target voice data can also be used to indicate whether the target voice data is associated with abnormal behavior, etc.

[0253] The target speech data is classified and processed according to the emotional feature information and the graph structure feature information to obtain the classification result of the target speech data. The emotional feature information and the graph structure feature information can be fused to obtain comprehensive feature information, and the target speech data is classified and processed according to the comprehensive feature information to obtain the classification result of the target speech data.

[0254] The emotional feature information can reflect the emotional features of the target speech data and can be used as one of the classification bases of the target speech data.

[0255] The graph structure feature information can reflect the semantic features of the target speech data and can be used as one of the classification bases of the target speech data.

[0256] In some implementations, only one of the emotional feature information and the graph structure feature information may be used as the classification basis for the target speech data.

[0257] For example, the following classification methods may be used in the classification process: SVM (Support Vector Machine), decision trees, logistic regression, and the like.

[0258] The target speech data is classified and processed according to the comprehensive feature information to obtain the classification result of the target speech data. Alternatively, the comprehensive feature information is input into the fully connected layer for classification processing to obtain the classification result.

[0259] During the implementation of the speech classification method provided in this embodiment, any classification method may be used when classifying the target speech data.

[0260] The classification result may be one of a plurality of preset classification results.

[0261] Exemplarily, the plurality of preset classification results may include: abnormal, non-abnormal, undetermined, and the like.

[0262] The classification result of the target voice data may also be a target risk level, which is one of a plurality of preset risk levels.

[0263] When the target risk level is greater than the preset level threshold, the target voice data can be determined as abnormal voice data; when the target risk level is less than the preset level threshold, the target voice data can be determined as non-abnormal voice data.

[0264] In addition, the warning method for the target voice data can be determined according to the target risk level.

[0265] The emotional feature information is derived from the target speech data, and the graph structure feature information is derived from the target text. Therefore, the comprehensive feature information obtained by fusing the emotional feature information and the graph structure feature information can be a multimodal comprehensive feature representation.

[0266] By using techniques such as the attention mechanism, the importance of features from different modalities is dynamically adjusted, leveraging their complementary relationships. The fused, comprehensive feature information can be input into a classifier to predict whether each target speech data point is anomalous and output a corresponding confidence score. The confidence score indicates the reliability of the classification. For example, a higher confidence score indicates a more accurate prediction of whether the target speech data point is anomalous.

[0267] The fusion method used when fusing emotional feature information and graph structure feature information can be to splice two feature vectors representing different modes to obtain a comprehensive feature vector with a larger dimension.

[0268] Furthermore, a microservices architecture can be used to package models such as speech recognition and conversation understanding into independent service components. By defining inter-service call relationships, the system connects speech input to the service components that execute the speech classification method into an automated service process, encapsulating it as a pre-defined interface to support flexible business integration. By integrating with call center systems, the speech classification method provided in this embodiment can achieve real-time detection and early warning of abnormal communication conversations, significantly improving risk prevention and control efficiency.

[0269] In actual algorithm model deployment, any speech recognition framework can be used to encapsulate the speech recognition model into a streaming end-to-end service. Pre-configured components can be used to encapsulate the speech recognition and dialogue understanding models into independent microservices. The speech recognition microservice takes the target speech data as input and outputs the target text; the dialogue understanding microservice takes the target text as input and outputs the dialogue understanding results, which include both emotional and graph feature information.

[0270] Next, you can use the preset components to define the serial execution flow of these two microservices:

[0271] Pass the target speech data as input to the speech recognition microservice to obtain the target text.

[0272] Pass the target text as input to the dialogue understanding microservice.

[0273] The dialogue understanding microservice extracts the emotional feature information and graph structure feature information of the target text.

[0274] The extracted features are concatenated into a complete feature vector H.

[0275] The feature vector H is passed into the fully connected layer classification network for classification processing, that is, speech classification processing:

[0276] y=FC(H), (22)

[0277] Where y is the classification result, indicating whether the target speech data is abnormal speech data. FC (Fully Connected) can be used to represent a fully connected layer.

[0278] The above formula (22) indicates that the classification result y is obtained by classifying the feature vector H through the fully connected layer.

[0279] In summary, we use pre-defined decorators to define the speech recognition model and the conversation understanding model as independent service components. The conversation understanding model can be used to extract emotional and graph structure features. Pre-defined decorators are used to specify the dependency environment. Pre-defined components are used to define data transfer and execution order between services. Finally, a complete service workflow is built and packaged as a container image.

[0280] Through the above steps, the speech recognition and conversation understanding models can be encapsulated as independent microservices. The data transfer and execution process between them is defined, thus generating a complete service workflow that receives audio input, performs speech recognition and conversation understanding, and outputs speech classification results. This microservice architecture and workflow design makes the system more modular, scalable, and maintainable, while also facilitating model updates and deployment.

[0281] In a specific implementation, the graph structure feature information includes sub-feature information of the first sub-text and sub-feature information of the second sub-text; the target speech data is classified according to the emotional feature information and the graph structure feature information to obtain a classification result of the target speech data, including: updating the preset initial state information of the first sub-text according to the sub-feature information of the first sub-text to obtain first state information representing the dialogue state of the first sub-text; updating the first state information according to the sub-feature information of the second sub-text to obtain second state information representing the dialogue state of the target text; fusing the emotional feature information and the second state information to obtain a first comprehensive feature; and classifying the target speech data according to the first comprehensive feature to obtain a classification result.

[0282] In this implementation, taking the target text including the first subtext and the second subtext as an example, the specific implementation process of classifying the target speech data according to the emotional feature information and the graph structure feature information to obtain the classification result of the target speech data is exemplified.

[0283] During specific implementation, the target text may include two subtexts, or may include at least three subtexts. In the case where the target text includes at least three subtexts, reference may be made to the corresponding description of this implementation.

[0284] During processing of the sub-feature information of the first sub-text, preset initial state information of the first sub-text may be obtained; the initial state information may be updated according to the sub-feature information of the first sub-text to obtain first state information indicating the dialog state of the first sub-text.

[0285] In the case that the first subtext is the first subtext in the target text, the preset initial state information of the first subtext may be an initial value of state information of a custom-configured dialog state of the target text.

[0286] It should be noted that, in the process of processing target texts of different target speech data using the speech classification method provided by this embodiment, the initial value of the state information can be reused by the first subtext in each different target text.

[0287] For example, an initial value S0 of state information representing the dialog state of the target text is preconfigured.

[0288] The target text 1 includes a first subtext 1 and a second subtext 1 , and the first subtext 1 is located before the second subtext 1 .

[0289] In the process of processing the target text 1 , obtaining the preset initial state information of the first subtext 1 may be obtaining S0 .

[0290] The target text 2 includes a first subtext 2 , a second subtext 2 and a third subtext 2 .

[0291] In the process of processing the target text 1 , obtaining the preset initial state information of the first subtext 2 may be obtaining S0 .

[0292] The target text 3 includes a first subtext 3 , a second subtext 3 , a third subtext 3 , a fourth subtext 3 and a fifth subtext 3 .

[0293] In the process of processing the target text 3 , the preset initial state information of the first subtext 3 is obtained, which may be obtaining S0 .

[0294] In the process of processing the sub-feature information of the second sub-text, first state information representing the dialogue state of the first sub-text can be obtained; the first state information is updated according to the sub-feature information of the second sub-text to obtain second state information representing the dialogue state of the target text.

[0295] It should be emphasized that the second state information obtained after updating the first state information according to the sub-feature information of the second subtext is not only used to represent the dialogue state of the second subtext, but is used to represent the dialogue state of the target text composed of the first subtext and the second subtext.

[0296] The dialog state may be the understanding and recording of the current dialog context at a specific moment. Specifically in this embodiment, the second state information representing the dialog state of the target text may represent the semantics of the target text.

[0297] The second state information may be used as one of the classification bases for the target voice data.

[0298] The above-mentioned steps of "updating the preset initial state information of the first subtext according to the sub-feature information of the first subtext to obtain first state information representing the dialogue state of the first subtext; and updating the first state information according to the sub-feature information of the second subtext to obtain second state information representing the dialogue state of the target text" can be an exemplary specific implementation process of determining the second state information representing the dialogue state of the target text according to the graph structure feature information of the target text. In the case where the target text includes at least three subtexts, the corresponding explanatory part of the example in which the target text includes the first subtext and the second subtext can be referred to.

[0299] After determining the second state information, the emotional feature information and the second state information may be fused to obtain a first comprehensive feature, and then the target speech data may be classified based on the first comprehensive feature to obtain a classification result.

[0300] The emotional characteristic information can be represented in the form of a vector, for example, the emotional characteristic information can be represented by a first characteristic vector. The second state information can be represented in the form of a vector, for example, the second state information can be represented by a second characteristic vector.

[0301] The emotional feature information and the second state information are fused to obtain the first comprehensive feature, which can be obtained by concatenating the first feature vector representing the emotional feature information and the second feature vector representing the second state information.

[0302] The emotional feature information and the second state information are fused to obtain the first comprehensive feature, or the first comprehensive feature is obtained by performing weighted sum processing on the first feature vector and the second feature vector according to a pre-configured weight parameter.

[0303] The emotional feature information and the second state information are fused to obtain the first comprehensive feature, or the first comprehensive feature is obtained by performing a dot product calculation on the first feature vector and the second feature vector.

[0304] The various fusion processing methods listed above are only exemplary. In actual applications, the fusion method can be flexibly configured.

[0305] According to the first comprehensive feature, the target speech data is classified to obtain a classification result. The first comprehensive feature can be input into a fully connected layer to classify the target speech data according to the first comprehensive feature to obtain a classification result, and the classification result is used as the classification result of the target speech data.

[0306] The target speech data is classified based on the first comprehensive feature to obtain a classification result. Alternatively, the first comprehensive feature may be input into a classification unit of a logistic regression model to classify the target speech data based on the first comprehensive feature to obtain a classification result, and the classification result is used as the classification result of the target speech data. The classification unit of the logistic regression model uses a logistic regression algorithm to classify the target speech data based on the first comprehensive feature to obtain the classification result.

[0307] The two specific implementation processes listed above for classifying the target speech data according to the first comprehensive feature are merely exemplary. In the process of classifying the target speech data according to the first comprehensive feature, any classification method may be used.

[0308] By configuring a state tracking mechanism and designing a state update function, we can fuse the graph neural network output of the current sentence with the previous conversation state, dynamically tracking the progress of the conversation and capturing changes in the speaker's intent. The output of the state update function can be used to represent the semantics of the target text and assist in speech classification.

[0309] In a specific implementation method, the target speech data is classified according to the emotional feature information and the graph structure feature information to obtain the classification result of the target speech data, including: segmenting the target text to obtain multiple segmentations of the target text; counting the frequency of occurrence of multiple segmentations of the target text in the corpus set; and classifying the target speech data according to the emotional feature information, the graph structure feature information and the frequency to obtain the classification result.

[0310] By performing word segmentation processing on the target text, multiple word segments of the target text can be obtained, and each word segment can be a phrase.

[0311] The corpus set may include one or more corpora. The corpora may be phrases or sentences. When the speech classification method provided in this embodiment is applied to a speech detection task, the corpus set may be determined by the speech detection task.

[0312] For example, speech detection task 1 is used to determine whether speech is related to abnormal behavior. When the speech classification method is applied to speech detection task 1, the corpus set may be a set consisting of corpora related to abnormal behavior.

[0313] Alternatively, the speech detection task 2 is used to determine the emotional tendency of the speech. In the case where the speech classification method is applied to the speech detection task 2, the corpus set may be a set consisting of corpora of various preset emotional categories.

[0314] In the process of counting the frequency of occurrence of multiple word segments of the target text in the corpus collection, the following steps can be performed:

[0315] (1) For each of the multiple segmentations of the target text, a matching process is performed on each corpus in the corpus set and each segmentation result of the target text: if the corpus and the segmentation are the same, it is determined that the corpus matches the segmentation; otherwise, if the corpus and the segmentation are different, it is determined that the corpus does not match the segmentation.

[0316] (2) Based on the matching results, count the frequency of each word in the corpus.

[0317] For example, the word segmentation result of the target text includes: word segmentation a1. The corpus set includes: "a1", "a2", and "a3".

[0318] The corpus "a1" is matched with the word a1, and a matching result is obtained. This matching result indicates that the corpus "a1" and the word a1 match. The corpus "a2" is matched with the word a1, and a matching result is obtained. This matching result indicates that the corpus "a2" and the word a1 do not match. The corpus "a3" is matched with the word a1, and a matching result is obtained. This matching result indicates that the corpus "a3" and the word a1 do not match.

[0319] Based on the statistical processing of the three matching results, it can be determined that the frequency of the word "a1" in the corpus is 1 / 3.

[0320] Alternatively, in the process of counting the frequencies of occurrence of multiple word segments of the target text in the corpus, the following steps can also be performed:

[0321] (1) For each of the multiple segmentations of the target text, a matching process is performed on each corpus in the corpus set and each segmentation result of the target text: if the corpus includes the segmentation, it is determined that the corpus and the segmentation match; otherwise, if the corpus does not include the segmentation, it is determined that the corpus and the segmentation do not match.

[0322] (2) Based on the matching results, count the frequency of each word in the corpus.

[0323] For example, the target text segmentation results include: segmentation AB. The corpus collection includes: "ABC", "CDF", and "ABEF".

[0324] The corpus "ABC" is matched with the word AB, and a matching result is obtained. This matching result indicates that the corpus "ABC" and the word AB match. The corpus "CDF" is matched with the word AB, and a matching result is obtained. This matching result indicates that the corpus "CDF" and the word AB do not match. The corpus "ABEF" is matched with the word AB, and a matching result is obtained. This matching result indicates that the corpus "ABEF" and the word AB do not match.

[0325] Based on the statistical processing of the three matching results, it can be determined that the frequency of word segmentation AB in the corpus is 2 / 3.

[0326] It should be noted that in order to facilitate the use of frequency as a feature for classification, the statistical results can be converted to obtain frequency feature vectors representing the frequencies of occurrence of multiple word segments of the target text in the corpus set.

[0327] For example, after statistics, we obtain: the frequency x1 of word 1 in the corpus, the frequency x2 of word 2 in the corpus, the frequency x3 of word 3 in the corpus, etc. Multiple frequencies such as x1, x2, and x3 are assembled to obtain a frequency set, and the frequency set is encoded to obtain a frequency feature vector.

[0328] In a specific implementation method, the target speech data is classified according to the emotional feature information, the graph structure feature information and the frequency to obtain a classification result, including: determining the second state information representing the dialogue state of the target text according to the graph structure feature information; fusing the emotional feature information, the second state information and the frequency to obtain a second comprehensive feature; and classifying the target speech data according to the second comprehensive feature to obtain a classification result.

[0329] In a case where the target text includes a first subtext and a second subtext, and the graph structure feature information includes subfeature information of the first subtext and subfeature information of the second subtext, second state information representing the dialogue state of the target text is determined based on the graph structure feature information. This may be performed by updating preset initial state information of the first subtext based on the subfeature information of the first subtext to obtain first state information representing the dialogue state of the first subtext; and by updating the first state information based on the subfeature information of the second subtext to obtain second state information representing the dialogue state of the target text.

[0330] When the target text includes at least three subtexts, the specific implementation process of determining the second state information representing the dialog state of the target text according to the graph structure feature information can refer to the corresponding description that the target subtext includes the first subtext and the second subtext.

[0331] After determining the second state information representing the dialogue state of the target text based on the graph structure feature information, the emotional feature information, the second state information and the frequency can be fused to obtain a second comprehensive feature. The target speech data can then be classified based on the second comprehensive feature to obtain a classification result.

[0332] Sentimental feature information can be represented in the form of a vector, for example, by a first feature vector. Second state information can be represented in the form of a vector, for example, by a second feature vector. Frequency can be represented in the form of a vector, for example, by a third feature vector. This third feature vector is the aforementioned frequency feature vector representing the frequency of occurrence of multiple word segments of the target text in the corpus.

[0333] The emotional feature information, the second state information and the frequency are fused to obtain the second comprehensive feature. The second comprehensive feature can be obtained by splicing the first feature vector representing the emotional feature information, the second feature vector representing the second state information and the third feature vector representing the frequency.

[0334] The emotional feature information, the second state information and the frequency are fused to obtain the second comprehensive feature, or the first feature vector, the second feature vector and the third feature vector are weighted and summed according to pre-configured weight parameters to obtain the second comprehensive feature.

[0335] The emotional feature information, the second state information and the frequency are fused to obtain the second comprehensive feature, or the first feature vector, the second feature vector and the third feature vector are subjected to dot product calculation to obtain the second comprehensive feature.

[0336] The various fusion processing methods listed above are only exemplary. In actual applications, the fusion method can be flexibly configured.

[0337] According to the second comprehensive feature, the target speech data is classified and processed to obtain a classification result. The second comprehensive feature can be input into a fully connected layer to classify the target speech data according to the second comprehensive feature to obtain a classification result, and the classification result is used as the classification result of the target speech data.

[0338] The target speech data is classified based on the second comprehensive feature to obtain a classification result. Alternatively, the second comprehensive feature may be input into a classification unit of a logistic regression model to classify the target speech data based on the second comprehensive feature to obtain a classification result, and the classification result is used as the classification result of the target speech data. The classification unit of the logistic regression model uses a logistic regression algorithm to classify the target speech data based on the second comprehensive feature to obtain a classification result.

[0339] The two specific implementation processes of classifying the target speech data according to the second comprehensive features listed above are merely exemplary. In the process of classifying the target speech data according to the second comprehensive features, any classification method may be used.

[0340] In a specific implementation, a corpus set includes an abnormal corpus set and a non-abnormal corpus set; counting the frequencies of occurrence of multiple segmentations of a target text in the corpus set includes: counting the frequencies of occurrence of multiple segmentations in the abnormal corpus set to obtain a first evaluation score for each segmentation; counting the frequencies of occurrence of multiple segmentations in the non-abnormal corpus set to obtain a second evaluation score for each segmentation; screening the multiple segmentations according to the first evaluation score, the second evaluation score and a preset score threshold to obtain a target segmentation set; determining a target segmentation belonging to the abnormal corpus set in the target segmentation set; and generating a frequency according to the target segmentation and the first evaluation score of the target segmentation.

[0341] The abnormal corpus set may include multiple abnormal corpora, each of which may be collected in advance after determining that the speech data is related to abnormal behavior. The non-abnormal corpus set may include multiple non-abnormal corpora, each of which may be collected in advance after determining that the speech data is not related to abnormal behavior.

[0342] For example, the first evaluation score of each word segment may be the TF-IDF score of the word segment in the abnormal corpus set; the second evaluation score of each word segment may be the TF-IDF score of the word segment in the non-abnormal corpus set.

[0343] The first evaluation score may also be any evaluation parameter other than the TF-IDF score that can reflect the frequency of occurrence or importance of the corresponding word in the abnormal corpus set.

[0344] The second evaluation score may also be any evaluation parameter other than the TF-IDF score that can reflect the frequency of occurrence or importance of the corresponding word in the non-abnormal corpus set.

[0345] TF-IDF (Term Frequency-Inverse Document Frequency) is a statistical method commonly used in information retrieval and text mining to evaluate the importance of a word to a document collection or a document in a corpus.

[0346] TF-IDF consists of two parts: TF (Term Frequency) and IDF (Inverse Document Frequency).

[0347] TF refers to the frequency of a word in a document. It's typically calculated by dividing the number of times a word appears in a document by the total number of words in the document. A higher frequency indicates a more important word in the document.

[0348] IDF measures the prevalence of a term across all documents. It's typically calculated by dividing the total number of documents in the corpus by the logarithm of the number of documents containing the term. Higher IDF values ​​indicate a less common term, and therefore a more representative characterization of the document.

[0349] The TF-IDF score is the product of TF and IDF. It reflects the importance of a word to a document while taking into account its prevalence in the entire corpus. Specifically, the more frequently a word appears in a document and the less frequently it appears across all documents, the higher its TF-IDF score.

[0350] Counting the frequencies of multiple segmentations appearing in the abnormal corpus set to obtain a first evaluation score for each segmentation can be performed by matching each segmentation with the abnormal corpus set for each segmentation, and calculating the TF-IDF score of the segmentation in the abnormal corpus set based on the matching result.

[0351] The matching process of matching the segmented words with the abnormal corpus set can refer to the corresponding instructions for matching each segmented word in the multiple segmented words of the target text, and each segmented word in the segmented results of the target text in the corpus set is matched, which will not be repeated here.

[0352] For example, the target text is ABCDEFG, and the word segmentation result includes 3 words: AB, CD, EFG.

[0353] In the process of counting the frequency of the three segmentations in the abnormal corpus set and obtaining the first evaluation score of each segmentation:

[0354] The frequency of the word "AB" appearing in the abnormal corpus set is counted to obtain a first evaluation score of the word "AB". This can be done by first matching the word "AB" with the abnormal corpus set to obtain a matching result, and then calculating the TF-IDF score of the word "AB" in the abnormal corpus set based on the matching result.

[0355] The frequency of the word "CD" appearing in the abnormal corpus set is counted to obtain a first evaluation score of the word "CD". This can be done by first matching the word "CD" with the abnormal corpus set to obtain a matching result, and then calculating the TF-IDF score of the word "CD" in the abnormal corpus set based on the matching result.

[0356] The frequency of the word "EFG" appearing in the abnormal corpus set is counted to obtain a first evaluation score of the word "EFG". This can be done by first matching the word "EFG" with the abnormal corpus set to obtain a matching result, and then calculating the TF-IDF score of the word "EFG" in the abnormal corpus set based on the matching result.

[0357] Counting the frequencies of multiple segmentations appearing in a non-abnormal corpus set to obtain a second evaluation score for each segmentation can refer to the corresponding description portion of counting the frequencies of multiple segmentations appearing in an abnormal corpus set to obtain a first evaluation score for each segmentation.

[0358] According to the first evaluation score, the second evaluation score and the preset score threshold, multiple segmentations are screened to obtain a target segmentation set. The comprehensive evaluation score of each segmentation can be determined based on the first evaluation score and the second evaluation score, and the multiple segmentations are screened according to the comprehensive evaluation score of each segmentation and the preset score threshold to obtain a target segmentation set.

[0359] The target word segmentation set may include one or more word segments, each of which may or may not belong to the abnormal corpus set. In the target word segmentation set, a word belonging to the abnormal corpus set may be determined and used as the target word segmentation.

[0360] The frequency in this implementation may be a third feature vector representing the frequency at which multiple word segments of the target text appear in the corpus set.

[0361] Specifically, generating a frequency based on the target segmentation and the first evaluation score of the target segmentation can be performed by assembling the first evaluation scores of the target segmentations to obtain an evaluation score set, encoding the evaluation score set, and obtaining a third feature vector representing the frequency. The specific use of the third feature vector can refer to the corresponding description of the aforementioned fusion processing of the sentiment feature information, the second state information, and the frequency to obtain the second comprehensive feature.

[0362] Taking into account that when the target text includes multiple segmentations, some segmentations may have a relatively large reference role in the subsequent classification steps, while other segmentations are almost useless in the subsequent classification steps. Therefore, in this implementation method, the first evaluation score, the second evaluation score and the preset score threshold are used to screen multiple segmentations in the target text, and the target segmentations with a relatively large reference role in the subsequent classification steps are selected. Then, the first evaluation score of the target segmentation is used to generate a third eigenvector representing the frequency. The third eigenvector can be used as one of the classification bases used in the subsequent classification steps. This is conducive to reducing redundant data in the frequency used as the classification basis before classification.

[0363] In a specific implementation method, multiple word segments are screened and processed according to the first evaluation score, the second evaluation score and the preset score threshold to obtain a target word segmentation set, including: summing the first evaluation score and the second evaluation score to obtain a third evaluation score; taking the ratio of the first evaluation score to the third evaluation score as the comprehensive evaluation score; comparing the comprehensive evaluation score with the preset score threshold to obtain a comparison result; if the comparison result indicates that the comprehensive evaluation score is greater than or equal to the preset score threshold, the word segmentation corresponding to the comprehensive evaluation score is added to the target word segmentation set.

[0364] According to the first evaluation score and the second evaluation score, a corresponding weight parameter is generated for each word in the word segmentation result, which can be referred to the following formula (23):

[0365]

[0366] Among them, t can be the word sequence number, which is used to represent a corresponding word, D f rau drepresents the abnormal corpus collection, D normal represents a non-abnormal corpus set, TF-IDF(t, D fraud ) represents the TF-IDF value of word segment t in the abnormal corpus set, that is, the first evaluation score, TF-IDF(t, D normal ) represents the TF-IDF value of word t in the non-anomalous corpus, i.e., the second evaluation score. Weight(t) represents the weight parameter corresponding to word t.

[0367] Referring to the above formula (23), the first evaluation score and the second evaluation score are summed to obtain a third evaluation score, which can be expressed as TF-IDF(t, D fraud )+TF-IDF(t,D normal )express.

[0368] The ratio of the first evaluation score to the third evaluation score is used as the comprehensive evaluation score. That is, for any segmentation, the weight parameter corresponding to the segmentation can be used as the comprehensive evaluation score of the segmentation.

[0369] The comprehensive evaluation score is compared with the preset score threshold to obtain a comparison result.

[0370] If the comprehensive evaluation score is greater than or equal to the preset score threshold, the segmentation corresponding to the comprehensive evaluation score can be added to the target segmentation set; otherwise, it can be determined that the segmentation corresponding to the comprehensive evaluation score does not belong to the target segmentation set.

[0371] For example, a target segmentation set may be obtained by screening multiple segmentations of the target text, and the target segmentation set includes segmentations whose comprehensive evaluation scores are greater than a preset score threshold of 0.6.

[0372] In such Figure 2In the embodiment shown, first, target speech data and target text of the target speech data are acquired; then, feature extraction is performed on the target speech data to obtain emotional feature information; next, feature extraction is performed on the semantics of the target text to obtain intermediate feature information, and the intermediate feature information is converted into graph structure feature information; finally, the target speech data is classified according to the emotional feature information and the graph structure feature information to obtain a classification result of the target speech data. Therefore, on the one hand, although the data content of the target text and the target voice data is the same, the data in voice form and the data in text form have different characteristics. Compared with text, voice can better reflect the user's emotional fluctuations and is conducive to perceiving the feelings of the user who makes the voice. Compared with voice, text is more conducive to extracting semantic features. Therefore, the comprehensive use of graph structure feature information derived from the target text and emotional feature information derived from the target voice data for classification can fully utilize the data advantages of different modalities, improve classification accuracy, and make the classification results of the target voice data more accurate; on the other hand, the longer the time of the target voice data, the higher the difficulty of understanding the semantics of the target text of the target voice data. Therefore, the classification difficulty of the target voice data may increase with the increase of the time length of the target voice data. Considering that the graph structure can better capture long-distance contextual dependencies, by extracting intermediate feature information from the target text and converting the intermediate feature information into graph structure feature information, the advantages of the graph structure can be fully utilized to more accurately express the semantics of the target text, which is conducive to improving the classification accuracy of the target voice data.

[0373] The following is an example of an application of a speech classification method provided by this embodiment in a communication scenario. Figure 5 The speech classification method provided in this embodiment is further described. Figure 5 A processing flow chart of another speech classification method provided in an embodiment of the present application.

[0374] Step S502: Based on the telephone voice data provided by the cooperative organization, the target domain telephone voice data including normal conversations and abnormal conversations is labeled.

[0375] In the process of labeling telephone voice data in the target field containing normal conversations and abnormal conversations based on the telephone voice data provided by the cooperative organization, the following steps can be performed: obtaining real telephone recordings in the target field from the cooperative organization; anonymizing the obtained telephone recordings to protect user privacy; labeling the anonymized telephone recordings, marking normal samples and abnormal samples; and desensitizing the labeled telephone recordings, including sound deformation, to prevent the original sound of the telephone voice data from being recognized.

[0376] The specific process of sound deformation is as follows: use libraries such as librosa to extract the fundamental frequency (pitch) curve of the speech; apply a low-pass filter to smooth the curve; slightly change its shape by superimposing a small amount of random noise on the filtered fundamental frequency curve; and use a vocoder synthesizer (vocoder) to resynthesize the speech based on the modified fundamental frequency curve.

[0377] In step S504, an end-to-end speech recognition model is trained to transcribe the collected telephone voice data in real time to obtain multiple rounds of dialogue and abnormal keyword frequencies; at the same time, the emotional features are further obtained through the speech features of the speech recognition model.

[0378] The multi-round dialogue can refer to the target text in the aforementioned method embodiment. The frequency of abnormal keywords can refer to the corresponding description part of "the frequency of occurrence of multiple word segments of the target text in the corpus set" in the aforementioned method embodiment.

[0379] For real-time transcription of the collected telephone voice data to obtain multiple rounds of conversation, please refer to the corresponding description of step S202 in the aforementioned method embodiment. For real-time transcription of the collected telephone voice data to obtain abnormal keyword frequencies, please refer to the corresponding description of "segmenting the target text to obtain multiple target text segmentations; and counting the frequencies of the multiple target text segmentations in the corpus" in the aforementioned method embodiment.

[0380] At the same time, the emotion features can be further obtained through the speech features of the speech recognition model, and reference can be made to the corresponding description of step S204 in the aforementioned method embodiment.

[0381] Step S506: Build a conversation understanding model based on a large-scale pre-trained language model, integrate multi-round conversation features, emotional features, and abnormal keyword frequencies to complete abnormal intent modeling.

[0382] The multi-round dialogue features can refer to the corresponding description parts of the intermediate feature information and graph structure feature information in the aforementioned method embodiment.

[0383] Step S508: deploy real-time speech recognition and conversation understanding services to perform real-time detection and anomaly identification on the telephone speech transcription text.

[0384] For steps S506 to S508 , reference may be made to the corresponding description of step 208 in the aforementioned method embodiment.

[0385] Since the technical concept is the same, the description in this embodiment is relatively simple, and the relevant parts can refer to the corresponding description of the method embodiment provided above.

[0386] Figure 6 This is an algorithm architecture diagram of another speech classification method provided in an embodiment of the present application.

[0387] like Figure 6 As shown, the target speech data is encoded by a speech encoder 602 to obtain first encoded information. The first encoded information is decoded by a speech-to-text decoder 604 to obtain a target text. Intermediate feature information representing semantics can be extracted from the target text. Based on the intermediate feature vectors and the relationship 608 between sentences in the target text, graph data can be constructed and graph structure feature information 6126 can be extracted from it through a graph neural network. The first encoded information is decoded by a sentiment classification decoder 604 to obtain emotional feature information 6122. Based on the target text, frequency 610 can be statistically processed to obtain frequency feature information 6124.

[0388] By fusing the emotion feature information 6122 , the frequency feature information 6124 and the graph structure feature information 6126 , comprehensive feature information 612 can be obtained.

[0389] The comprehensive feature information 612 is subjected to nonlinear conversion processing through the fully connected layer 614, and the converted comprehensive feature information 612 is classified through abnormal probability prediction to obtain the classification result of the target speech data.

[0390] Since the technical concept is the same, the description in this embodiment is relatively simple, and the relevant parts can refer to the corresponding description of the method embodiment provided above.

[0391] An embodiment of a speech classification device provided in this specification is as follows:

[0392] In the above embodiment, a speech classification method is provided. Based on the same technical concept, the embodiment of the present application also provides a speech classification device, which is described below with reference to the accompanying drawings.

[0393] Figure 7 Schematic diagram of a speech classification device provided in an embodiment of the present application. This embodiment provides a speech classification device 700, comprising:

[0394] An acquiring unit 702 is configured to acquire target speech data and a target text of the target speech data;

[0395] Extraction unit 704, used to extract features from the target speech data to obtain emotional feature information;

[0396] The extraction unit 704 is further configured to extract features from the semantics of the target text to obtain intermediate feature information, and convert the intermediate feature information into graph structure feature information;

[0397] The classification unit 706 is configured to classify the target speech data according to the emotion feature information and the graph structure feature information to obtain a classification result of the target speech data.

[0398] Optionally, the target text includes a first subtext and a second subtext; the intermediate feature information includes semantic features of the first subtext and semantic features of the second subtext; and the extraction unit 704 performs the following steps when converting the intermediate feature information into graph structure feature information:

[0399] converting the semantic features of the first subtext into a first node representing the first subtext;

[0400] converting the semantic features of the second subtext into a second node representing the second subtext;

[0401] determining, based on a relationship between the first subtext and the second subtext, a first edge connecting the first node and the second node;

[0402] Constructing graph data according to the first node, the second node, and the first edge;

[0403] The graph data is encoded through a graph neural network to obtain the graph structure feature information.

[0404] Optionally, the graph structure feature information includes sub-feature information of the first subtext and sub-feature information of the second subtext; when the classification unit 706 classifies the target speech data according to the emotional feature information and the graph structure feature information to obtain a classification result of the target speech data, the classification unit 706 performs the following steps:

[0405] updating the preset initial state information of the first subtext according to the sub-feature information of the first subtext to obtain first state information indicating the conversation state of the first subtext;

[0406] updating the first state information according to the sub-feature information of the second sub-text to obtain second state information representing the dialog state of the target text;

[0407] fusing the emotion feature information and the second state information to obtain a first comprehensive feature;

[0408] The target speech data is classified according to the first comprehensive feature to obtain the classification result.

[0409] Optionally, when the classification unit 706 classifies the target speech data according to the emotion feature information and the graph structure feature information to obtain the classification result of the target speech data, it performs the following steps:

[0410] Segmenting the target text to obtain multiple segmentations of the target text;

[0411] Counting the frequencies of occurrence of multiple word segments of the target text in the corpus set;

[0412] The target speech data is classified according to the emotion feature information, the graph structure feature information, and the frequency to obtain the classification result.

[0413] Optionally, when the classification unit 706 classifies the target speech data according to the emotion feature information, the graph structure feature information, and the frequency to obtain the classification result, it performs the following steps:

[0414] determining, based on the graph structure feature information, second state information representing a dialog state of the target text;

[0415] fusing the emotion feature information, the second state information, and the frequency to obtain a second comprehensive feature;

[0416] The target speech data is classified according to the second comprehensive feature to obtain the classification result.

[0417] Optionally, the preset keyword set includes an abnormal corpus set and a non-abnormal corpus set; when counting the frequencies of occurrence of multiple word segments of the target text in the corpus set, the classification unit 706 performs the following steps:

[0418] Counting the frequencies of the multiple segmentations appearing in the abnormal corpus set to obtain a first evaluation score for each segmentation;

[0419] Counting the frequencies of the multiple segmentations appearing in the non-abnormal corpus set to obtain a second evaluation score for each segmentation;

[0420] Filtering the plurality of segmented words according to the first evaluation score, the second evaluation score, and a preset score threshold to obtain a target segmented word set;

[0421] Determining target segmentations belonging to the abnormal corpus set in the target segmentation set;

[0422] The frequency is generated according to the target word segment and the first evaluation score of the target word segment.

[0423] Optionally, when the classification unit 706 filters the word segmentation results according to the first evaluation score, the second evaluation score, and a preset score threshold to obtain a target word segmentation set, the classification unit 706 performs the following steps:

[0424] Summing the first evaluation score and the second evaluation score to obtain a third evaluation score;

[0425] taking the ratio of the first evaluation score to the third evaluation score as a comprehensive evaluation score;

[0426] Comparing the comprehensive evaluation score with the preset score threshold to obtain a comparison result;

[0427] If the comparison result indicates that the comprehensive evaluation score is greater than or equal to the preset score threshold, the segmentation corresponding to the comprehensive evaluation score is added to the target segmentation set.

[0428] Optionally, when extracting features from the target speech data to obtain emotional feature information, the extraction unit 704 performs the following steps:

[0429] Encoding the target voice data to obtain first encoding information of the target voice data;

[0430] Performing attention processing on the first encoded information to obtain an emotional feature vector;

[0431] Performing a nonlinear transformation on the emotion feature vector to obtain a transformed emotion feature vector;

[0432] The transformed emotion feature vector is classified to obtain probability distribution information of the target speech data in a plurality of preset emotion categories, and the probability distribution information is used as the emotion feature information.

[0433] In an embodiment of the present application, the speech classification device includes: an acquisition unit for acquiring target speech data and a target text of the target speech data; an extraction unit for performing feature extraction on the target speech data to obtain emotional feature information; the extraction unit is also used to perform feature extraction on the semantics of the target text to obtain intermediate feature information, and convert the intermediate feature information into graph structure feature information; a classification unit is used to classify the target speech data according to the emotional feature information and the graph structure feature information to obtain a classification result of the target speech data. Therefore, on the one hand, although the data content of the target text and the target voice data is the same, the data in voice form and the data in text form have different characteristics. Compared with text, voice can better reflect the user's emotional fluctuations and is conducive to perceiving the feelings of the user who makes the voice. Compared with voice, text is more conducive to extracting semantic features. Therefore, the comprehensive use of graph structure feature information derived from the target text and emotional feature information derived from the target voice data for classification can fully utilize the data advantages of different modalities, improve classification accuracy, and make the classification results of the target voice data more accurate; on the other hand, the longer the time of the target voice data, the higher the difficulty of understanding the semantics of the target text of the target voice data. Therefore, the classification difficulty of the target voice data may increase with the increase of the time length of the target voice data. Considering that the graph structure can better capture long-distance contextual dependencies, by extracting intermediate feature information from the target text and converting the intermediate feature information into graph structure feature information, the advantages of the graph structure can be fully utilized to more accurately express the semantics of the target text, which is conducive to improving the classification accuracy of the target voice data.

[0434] Corresponding to the above-described voice classification method, based on the same technical concept, an embodiment of the present application further provides an electronic device, which is used to execute the above-described voice classification method. Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0435] like Figure 8As shown, electronic devices may have relatively large differences due to different configurations or performances, and may include one or more processors 801 and memory 802, and the memory 802 may store one or more storage applications or data. Among them, the memory 802 can be a temporary storage or a persistent storage. The application stored in the memory 802 may include one or more modules (not shown in the figure), each module may include a series of computer-executable instructions in the electronic device. Furthermore, the processor 801 can be configured to communicate with the memory 802 to execute a series of computer-executable instructions in the memory 802 on the electronic device. The electronic device may also include one or more power supplies 803, one or more wired or wireless network interfaces 804, one or more input / output interfaces 805, one or more keyboards 806, etc.

[0436] In a specific embodiment, the electronic device includes a memory and one or more programs, wherein the one or more programs are stored in the memory, and the one or more programs may include one or more modules, and each module may include a series of computer-executable instructions for the electronic device, and the one or more programs are configured to be executed by one or more processors, including computer-executable instructions for performing the following:

[0437] Acquire target voice data and target text of the target voice data;

[0438] Extracting features from the target speech data to obtain emotional feature information;

[0439] Extracting features from the semantics of the target text to obtain intermediate feature information, and converting the intermediate feature information into graph structure feature information;

[0440] The target speech data is classified according to the emotion feature information and the graph structure feature information to obtain a classification result of the target speech data.

[0441] The computer-readable storage medium provided in this embodiment is used to store computer-executable instructions. When the computer-executable instructions are executed by a processor, the following process is implemented:

[0442] Acquire target voice data and target text of the target voice data;

[0443] Extracting features from the target speech data to obtain emotional feature information;

[0444] Extracting features from the semantics of the target text to obtain intermediate feature information, and converting the intermediate feature information into graph structure feature information;

[0445] The target speech data is classified according to the emotion feature information and the graph structure feature information to obtain a classification result of the target speech data.

[0446] It should be noted that the embodiment of the computer-readable storage medium in this specification and the embodiment of the speech classification method in this specification are based on the same inventive concept. Therefore, the specific implementation of this embodiment can refer to the implementation of the corresponding method mentioned above, and the repeated parts will not be repeated.

[0447] Another embodiment of the present disclosure further provides a computer program product, the computer program product including a computer program, which implements the following process when executed by a processor:

[0448] Acquire target voice data and target text of the target voice data;

[0449] Extracting features from the target speech data to obtain emotional feature information;

[0450] Extracting features from the semantics of the target text to obtain intermediate feature information, and converting the intermediate feature information into graph structure feature information;

[0451] The target speech data is classified according to the emotion feature information and the graph structure feature information to obtain a classification result of the target speech data.

[0452] The computer program product in the embodiment of the present disclosure can implement each process of the above-mentioned speech classification method embodiment and achieve the same effects and functions, which will not be repeated here.

[0453] In an embodiment of the present application, first, target speech data and target text of the target speech data are obtained; then, feature extraction is performed on the target speech data to obtain emotional feature information; then, feature extraction is performed on the semantics of the target text to obtain intermediate feature information, and the intermediate feature information is converted into graph structure feature information; finally, the target speech data is classified according to the emotional feature information and the graph structure feature information to obtain a classification result of the target speech data. Therefore, on the one hand, although the data content of the target text and the target voice data is the same, the data in voice form and the data in text form have different characteristics. Compared with text, voice can better reflect the user's emotional fluctuations and is conducive to perceiving the feelings of the user who makes the voice. Compared with voice, text is more conducive to extracting semantic features. Therefore, the comprehensive use of graph structure feature information derived from the target text and emotional feature information derived from the target voice data for classification can fully utilize the data advantages of different modalities, improve classification accuracy, and make the classification results of the target voice data more accurate; on the other hand, the longer the time of the target voice data, the higher the difficulty of understanding the semantics of the target text of the target voice data. Therefore, the classification difficulty of the target voice data may increase with the increase of the time length of the target voice data. Considering that the graph structure can better capture long-distance contextual dependencies, by extracting intermediate feature information from the target text and converting the intermediate feature information into graph structure feature information, the advantages of the graph structure can be fully utilized to more accurately express the semantics of the target text, which is conducive to improving the classification accuracy of the target voice data.

[0454] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0455] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the embodiments of the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, this specification may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0456] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable electronic device to produce a machine, so that the instructions executed by the processor of the computer or other programmable electronic device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0457] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable electronic device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0458] These computer program instructions can also be loaded onto a computer or other programmable electronic device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable device to implement the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0459] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0460] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0461] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0462] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0463] The embodiments of the present application may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. One or more embodiments of the present specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0464] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.

[0465] The foregoing description is merely an example of the present invention and is not intended to limit the present invention. Persons skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be included within the scope of the claims herein.

Claims

1. A speech classification method, characterized in that: include: Acquire target voice data and target text of the target voice data; Extracting features from the target speech data to obtain emotional feature information; Extracting features from the semantics of the target text to obtain intermediate feature information, and converting the intermediate feature information into graph structure feature information; The target speech data is classified according to the emotion feature information and the graph structure feature information to obtain a classification result of the target speech data.

2. The method according to claim 1, characterized in that The target text includes a first subtext and a second subtext; the intermediate feature information includes semantic features of the first subtext and semantic features of the second subtext; and converting the intermediate feature information into graph structure feature information includes: converting the semantic features of the first subtext into a first node representing the first subtext; converting the semantic features of the second subtext into a second node representing the second subtext; determining, based on a relationship between the first subtext and the second subtext, a first edge connecting the first node and the second node; Constructing graph data according to the first node, the second node, and the first edge; The graph data is encoded through a graph neural network to obtain the graph structure feature information.

3. The method according to claim 2, characterized in that The graph structure feature information includes sub-feature information of the first subtext and sub-feature information of the second subtext; and classifying the target speech data according to the emotion feature information and the graph structure feature information to obtain a classification result of the target speech data includes: updating the preset initial state information of the first subtext according to the sub-feature information of the first subtext to obtain first state information indicating the conversation state of the first subtext; updating the first state information according to the sub-feature information of the second sub-text to obtain second state information representing the dialog state of the target text; fusing the emotion feature information and the second state information to obtain a first comprehensive feature; The target speech data is classified according to the first comprehensive feature to obtain the classification result.

4. The method according to claim 1, wherein The classifying the target speech data according to the emotion feature information and the graph structure feature information to obtain a classification result of the target speech data includes: Segmenting the target text to obtain multiple segmentations of the target text; Counting the frequencies of occurrence of multiple word segments of the target text in the corpus set; The target speech data is classified according to the emotion feature information, the graph structure feature information, and the frequency to obtain the classification result.

5. The method according to claim 4, characterized in that The classifying the target speech data according to the emotion feature information, the graph structure feature information, and the frequency to obtain the classification result includes: determining, based on the graph structure feature information, second state information representing a dialog state of the target text; fusing the emotion feature information, the second state information, and the frequency to obtain a second comprehensive feature; The target speech data is classified according to the second comprehensive feature to obtain the classification result.

6. The method according to claim 4, characterized in that The corpus set includes an abnormal corpus set and a non-abnormal corpus set; the counting of the frequencies of occurrence of multiple word segments of the target text in the corpus set includes: Counting the frequencies of the multiple segmentations appearing in the abnormal corpus set to obtain a first evaluation score for each segmentation; Counting the frequencies of the multiple segmented words appearing in the non-abnormal entertainment set to obtain a second evaluation score for each segmented word; Filtering the plurality of segmented words according to the first evaluation score, the second evaluation score, and a preset score threshold to obtain a target segmented word set; Determining target segmentations belonging to the abnormal corpus set in the target segmentation set; The frequency is generated according to the target word segment and the first evaluation score of the target word segment.

7. The method according to claim 6, characterized in that The step of screening the plurality of segmented words according to the first evaluation score, the second evaluation score, and a preset score threshold to obtain a target segmented word set includes: Summing the first evaluation score and the second evaluation score to obtain a third evaluation score; taking the ratio of the first evaluation score to the third evaluation score as a comprehensive evaluation score; Comparing the comprehensive evaluation score with the preset score threshold to obtain a comparison result; If the comparison result indicates that the comprehensive evaluation score is greater than or equal to the preset score threshold, the segmentation corresponding to the comprehensive evaluation score is added to the target segmentation set.

8. The method according to claim 1, characterized in that The step of extracting features from the target speech data to obtain emotional feature information includes: Encoding the target voice data to obtain first encoding information of the target voice data; Performing attention processing on the first encoded information to obtain an emotional feature vector; Performing a nonlinear transformation on the emotion feature vector to obtain a transformed emotion feature vector; The transformed emotion feature vector is classified to obtain probability distribution information of the target speech data in a plurality of preset emotion categories, and the probability distribution information is used as the emotion feature information.

9. A speech classification device, characterized in that: include: An acquisition unit, configured to acquire target speech data and a target text of the target speech data; An extraction unit, configured to extract features from the target speech data to obtain emotional feature information; The extraction unit is further configured to extract features from the semantics of the target text to obtain intermediate feature information, and convert the intermediate feature information into graph structure feature information; The classification unit is used to classify the target speech data according to the emotional feature information and the graph structure feature information to obtain a classification result of the target speech data.

10. An electronic device, characterized in that: include: processor; and a memory configured to store computer-executable instructions, which, when executed, cause the processor to perform the speech classification method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that The computer-readable storage medium is used to store computer-executable instructions, and when the computer-executable instructions are executed by a processor, the speech classification method according to any one of claims 1 to 8 is implemented.

12. A computer program product, characterized in that The invention comprises a computer program, which implements the speech classification method according to any one of claims 1 to 8 when executed by a processor.

Images