A voice quality inspection method, device and equipment

Abstract

The application provides a voice quality inspection method and device, and relates to the technical field of voice recognition. The method comprises the following steps: acquiring voice data to be detected; the voice data to be detected comprises a voice sample set with dialect characteristics input by a user and a voice signal set to be detected; performing data preprocessing on the voice sample set, and determining the multi-modal characteristics of the processed voice sample set; determining a voice emotion recognition model according to the multi-modal characteristics; inputting the voice signal set to be detected into the voice emotion recognition model, and determining the emotional state of the user. Compared with the existing voice emotion recognition scheme, the technical scheme of the application has the advantages that multiple dialect emotions can be recognized, and the local Mandarin with dialect characteristics also has good effects.

Description

Technical Field

[0001] This application relates to the field of speech recognition technology, and in particular to a speech quality inspection method, apparatus and device. Background Technology

[0002] Currently, telephone customer service has become an indispensable part of handling various industry services, with many traditional industries shifting their services from counter service to telephone customer service. In telephone customer service, human agents hold a large share due to their advantages such as direct and convenient communication, fast processing speed, and low learning cost for customers. However, many problems have also arisen in human telephone customer service, such as poor service quality and attitude, and instances of overly aggressive language from customers. Therefore, quality control of telephone conversations and voice recordings has become a crucial aspect of telephone customer service.

[0003] Currently, voice quality inspection methods mainly include: 1) The more traditional manual method: This involves manually screening and randomly selecting samples from all recorded customer service voice messages to identify those with service issues; 2) The text extraction method: This involves transcribing the voice conversation and then extracting the text using machine learning and other methods to assess the service quality of the telephone customer service representative; 3) The keyword matching method: This involves transcribing the voice conversation and then matching it against a pre-built dictionary of sensitive words. If more than a certain number of sensitive words are found, the service attitude is considered poor. Currently, except for industries with older systems or lower customer service traffic that use manual review for voice customer service quality inspection, most systems use automated voice quality inspection methods.

[0004] The existing technology has the following problems: 1) It requires speech recognition results and the network structure is relatively complex; 2) The effect cannot be guaranteed when the speech recognition results are not very accurate; 3) It cannot process dialogue audio with dialects or heavy accents. Summary of the Invention

[0005] This application provides a voice quality inspection method, apparatus, and device to solve the problem that the prior art cannot process dialogue audio with dialects or heavy accents.

[0006] To solve the above-mentioned technical problems, this application adopts the following technical solution:

[0007] This application provides a voice quality inspection method, including:

[0008] Acquire the speech data to be detected; the speech data to be detected includes a set of speech samples with dialect features input by the user and a set of speech signals to be detected;

[0009] The speech sample set is preprocessed to determine the multimodal features of the processed speech sample set; the multimodal features include: input feature vector, embedded embedding output features, and preset feature set; the embedded embedding output features are determined by dialect embedding model encoding and are used to indicate each dialect in the speech sample set;

[0010] Based on the aforementioned multimodal features, a speech emotion recognition model is determined;

[0011] The set of voice signals to be detected is input into the voice emotion recognition model to determine the user's emotional state.

[0012] Optionally, the set of speech signals to be detected is input into the speech emotion recognition model to determine the user's emotional state, including:

[0013] Based on the preset embedding algorithm and the input feature vector, the embedding output feature of the first target dialect in the speech sample set is determined; the first target dialect is one of the dialects in the speech sample set.

[0014] Based on the set of speech signals to be detected and the embedding output features of the first target dialect, the dialect type of the set of speech signals to be detected is determined;

[0015] Based on the dialect type and the speech emotion recognition model, the emotion probability vector of the second target dialect in the set of speech signals to be detected is determined; wherein, the speech emotion recognition model includes a mixed dialect emotion model and a Mandarin emotion model; the second target dialect is one of the dialects in the set of speech signals to be detected;

[0016] The emotion result of the set of speech signals to be detected is determined based on the emotion probability vector of the second target dialect.

[0017] Optionally, based on the dialect type and the speech emotion recognition model, the emotion probability vector of the second target dialect in the set of speech signals to be detected is determined, including:

[0018] According to the formula: Determine the emotion probability vector of the second target dialect in the set of speech signals to be detected;

[0019] Where P is the emotion probability vector of the second target dialect; P i P0 is the probability vector of each emotion obtained from the mixed dialect emotion model; P0 is the probability vector of each emotion obtained from the Mandarin emotion model; L0 is the Mandarin centroid distance of the speech sample set; L0i The dialect centroid distance of the speech sample set is denoted as .

[0020] Optionally, based on the set of speech signals to be detected and the embedding output features of the first target dialect, the dialect type of the set of speech signals to be detected is determined, including:

[0021] Based on the embedding output features of the first target dialect, determine the first centroid vector of the first target dialect and the discrimination threshold of the first target dialect;

[0022] Based on the set of speech signals to be detected and the preset embedding algorithm, determine the second centroid vector of the second target dialect in the set of speech signals to be detected;

[0023] If the distance between the second centroid vector and the corresponding first centroid vector is less than the discrimination threshold, then the set of speech signals to be detected is determined to be a mixed dialect; otherwise, the set of speech signals to be detected is determined to be Mandarin.

[0024] Optionally, determining the embedding output features of the first target dialect in the speech sample set based on the preset embedding algorithm and the input feature vector includes:

[0025] Based on the dialect embedding model and the preset embedding algorithm, the compiled encoded feature vector is determined;

[0026] The compiled encoded feature vector and the input feature vector are concatenated to determine the embedding output feature of the first target dialect in the speech sample set.

[0027] Optionally, the dialect embedding model is constructed and trained, including:

[0028] Construct the dialect embedding model; the input value of the dialect embedding model is the input feature vector, and the output value of the dialect embedding model is the one-hot code corresponding to the dialect to which the speech sample set belongs;

[0029] The triplet loss function is used as the loss function, and the dialect embedding model is trained based on the speech sample set.

[0030] Optionally, determining the speech emotion recognition model based on the multimodal features includes:

[0031] The input feature vector, the embedding output feature, and the preset feature set are concatenated to determine the concatenated feature vector.

[0032] Based on the spliced ​​feature vector and the preset recurrent neural network, the speech emotion fusion vector is determined;

[0033] The speech emotion fusion vector is input into two fully connected layers, and a preset softmax function is used to calculate the confidence of the emotion category to determine the speech emotion recognition model.

[0034] Optionally, the acquisition of a set of speech samples with dialect features in the acquisition of the speech data to be detected includes:

[0035] Collect dialect voice dialogue data and remove voice data without emotion from the dialect voice dialogue data;

[0036] The audio format of the dialect speech dialogue data after dialect removal is standardized and determined to be unified audio data.

[0037] The unified audio data is segmented and determined into multiple audio files limited to a preset duration.

[0038] The emotion category is marked based on all audio files within the preset duration.

[0039] Optionally, the input feature vector in the multimodal features of the processed speech sample set is determined, including:

[0040] All audio files within the preset duration after marking are processed for noise reduction using a preset filtering method to obtain the noise-reduced audio files;

[0041] The noise-reduced audio file is processed by a convolutional recurrent neural network to determine the input feature vector after preprocessing the speech sample set.

[0042] Optionally, the denoised audio file is processed by a convolutional recurrent neural network to determine the preprocessed input feature vector of the speech sample set, including:

[0043] Perform a short-time Fourier transform on the denoised audio file to determine the processed spectral signal;

[0044] The Mel spectrum is determined by multiplying the spectral signal with the Mel filter;

[0045] The Mel spectrum is input into the convolutional recurrent neural network to determine the input feature vector after preprocessing the speech sample set.

[0046] This application also provides a voice quality inspection device, including:

[0047] The acquisition module is used to acquire the speech data to be detected; the speech data to be detected includes a set of speech samples with dialect features input by the user and a set of speech signals to be detected;

[0048] The first determining module is used to preprocess the speech sample set to determine the multimodal features of the processed speech sample set; the multimodal features include: input feature vector, embedded embedding output features, and preset feature set; the embedded embedding output features are determined by dialect embedding model encoding and are used to indicate each dialect in the speech sample set;

[0049] The second determining module is used to determine the speech emotion recognition model based on the multimodal features;

[0050] The third determining module is used to input the set of voice signals to be detected into the voice emotion recognition model to determine the user's emotional state.

[0051] This application also provides a voice quality inspection device, including: a processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the voice quality inspection method as described in any of the preceding claims.

[0052] This application also provides a readable storage medium storing a program that, when executed by a processor, implements the voice quality inspection method as described in any of the preceding claims.

[0053] The beneficial effects of this application are:

[0054] In the above technical solution, the user's emotional state is determined by recognizing the language data input by the user in dialect, thus achieving the purpose of detecting dialect-based speech. Compared with existing speech emotion recognition solutions, the technical solution of this application has advantages such as being able to recognize emotions in multiple dialects, and also having good results with local Mandarin with dialectal characteristics. Attached Figure Description

[0055] Figure 1 This is a flowchart illustrating the voice quality inspection method provided in an embodiment of this application;

[0056] Figure 2 This is a schematic diagram of one of the modules of the voice quality inspection device provided in the embodiments of this application;

[0057] Figure 3 This is a second schematic diagram of the module of the voice quality inspection device provided in the embodiments of this application;

[0058] Figure 4This is the third schematic diagram of the module of the voice quality inspection device provided in the embodiments of this application. Detailed Implementation

[0059] To make the technical problems, technical solutions, and advantages of this application clearer, a detailed description will be provided below in conjunction with the accompanying drawings and specific embodiments. In the following description, specific details such as particular configurations and components are provided merely to aid in a comprehensive understanding of the embodiments of this application. Therefore, those skilled in the art should understand that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Furthermore, for clarity and brevity, descriptions of known functions and structures have been omitted.

[0060] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments.

[0061] In the various embodiments of this application, it should be understood that the sequence number of each process described below does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0062] This application addresses the problem that existing technologies cannot handle dialogue audio with dialects or heavy accents by providing a voice quality inspection method, apparatus, and device.

[0063] like Figure 1 As shown, an optional embodiment of the present invention provides a voice quality inspection method, including:

[0064] Step 100: Obtain the speech data to be detected; the speech data to be detected includes a set of speech samples with dialect features input by the user and a set of speech signals to be detected;

[0065] Here, a set of speech samples with dialect features is obtained, including a dialect dataset and a dialect dialogue dataset with emotion, and the speech sample set is subjected to audio track separation and sentence segmentation processing. Step 200: The speech sample set is preprocessed to determine the multimodal features of the processed speech sample set; the multimodal features include: input feature vector, embedded embedding output features, and a preset feature set; the embedded embedding output features are determined by dialect embedding model encoding and are used to indicate each dialect in the speech sample set;

[0066] In this embodiment, the data preprocessing of the speech sample set can prioritize determining the input feature vector; based on the preset embedding algorithm and the input feature vector, the embedding output feature of the first target dialect in the speech sample set is determined; the first target dialect is one of the dialects in the speech sample set.

[0067] It should be noted that the GeMAPS feature set comprises 62 features, all of which are HFS (high-level statistical functions) derived from LLDs (low-level descriptors) through statistical analysis. The 18 LLD features include 6 frequency-related features, 3 energy / amplitude-related features, and 9 spectral features.

[0068] Step 300: Determine the speech emotion recognition model based on the multimodal features;

[0069] Here, a speech emotion recognition model is trained based on the input feature vector, the embedding output features, and a preset feature set. In this embodiment, three feature sources are used as inputs to the speech emotion recognition model: the spectrogram of the input feature vector, the preset feature set (GeMAPS feature set), and the embedding output features (language encoding of the speech through a previously trained embedding model).

[0070] Step 400: Input the set of voice signals to be detected into the voice emotion recognition model to determine the user's emotional state.

[0071] In this embodiment, the set of speech signals to be detected is input into a pre-trained speech emotion recognition model to determine the speech recognition result. The technical solution of this invention, by employing embedding encoding followed by training a classification network model, determines the speech recognition result, achieving recognition of dialectal speech and increasing the diversity of speech recognition.

[0072] It should be noted that before starting speech recognition, the speech signal must first be preprocessed. Preprocessing consists of three stages: analog signal digitization, endpoint detection, and framing. Only after the speech signal is framed can we perform detailed analysis. Framing refers to dividing a complete speech signal into many equal-length segments. Each segment is called a frame. Framing is generally implemented using a moving window function. The first and second frames have only one superposition. Typically, a frame length is 25 milliseconds, and the superposition length between any two frames is 10 milliseconds. After framing the speech signal, feature parameters are extracted. The dialects mentioned include, but are not limited to: Northeastern Mandarin, Wu dialect, Xiang dialect, Min dialect, Hakka dialect, Yue dialect, and Mandarin.

[0073] Here, the emotional states mentioned include, but are not limited to, the following: happy, angry, sad, afraid, surprised, and neutral emotional states.

[0074] In summary, by recognizing the language data input by the user in dialect, the emotional state of the user can be determined, thus achieving the goal of detecting dialect-based speech. Compared with existing speech emotion recognition schemes, this method has advantages such as being able to recognize emotions in multiple dialects and also having good results with local Mandarin with dialectal characteristics.

[0075] Specifically, step 400 includes: steps 410 to 440;

[0076] In step 410, the embedding output features of a first target dialect in the speech sample set are determined according to a preset embedding algorithm and the input feature vector; the first target dialect is a dialect in the speech sample set.

[0077] In this embodiment, the preset embedding algorithm is a preset embedding encoding. For dialect discrimination, a mode of training a classification network model is adopted after pre-embedding encoding. The input of this encoding is the mel spectrum, and the output is the one-hot encoding corresponding to the dialect type of the audio. The weights of the layer before the output layer of the embedding network are used as the encoding of that language. At the same time, a certain number of encodings are extracted to obtain the center points, which are then concatenated with the input feature vector and added to the classification network model. Here, a weighted summation method is used to obtain the embedding center points of a dialect.

[0078] Optionally, step 410 includes:

[0079] Step 411: Determine the compiled encoding feature vector based on the dialect embedding model and the preset embedding algorithm; here, the preset embedding algorithm is the preset embedding encoding.

[0080] Step 412: Concatenate the compiled encoded feature vector and the input feature vector to determine the embedded output features of the first target dialect in the speech sample set.

[0081] In this embodiment, an embedding encoding scheme for different dialects is adopted, which achieves the technical effect that the dialect embedding model can effectively perform multi-dialect recognition.

[0082] Specifically, constructing and training a dialect embedding model includes:

[0083] Construct a dialect embedding model; the input value of the dialect embedding model is the input feature vector, and the output value of the dialect embedding model is the one-hot code corresponding to the dialect to which the speech sample set belongs;

[0084] In this embodiment, the input value of the dialect embedding model is the mel spectrum of the input feature vector, and the output is the one-hot encoding corresponding to the dialect type to which the audio belongs. The weights of the layer before the output layer of the embedding network are used as the encoding of the language.

[0085] The triplet loss function is used as the loss function, and the dialect embedding model is trained based on the speech sample set.

[0086] In this embodiment, triplet loss is used as the loss function for training the dialect embedding model. This method can effectively increase the distance between different labeled data in a specified dimension, ensuring that different dialects have sufficient distinguishability and preparing for subsequent memory emotion weighting.

[0087] The Triplet loss formula is: L = max(d(a, p) - d(a, n) + margin, 0), where d(a, p) represents the distance between a sample and a positive sample, and d(a, n) represents the distance between a sample and a negative sample. The intention here is to make the distance to a positive sample less than the distance to a negative sample by a margin, aiming to shorten the distance between a and p and widen the distance between a and n. Here, 80% of the data in the speech sample set is used as the training set, and 20% as the test set. The test set for each language is encoded, and the centroid of each dialect is calculated. After encoding the dialect data in the test set, the mean of the same dimension for each dialect is used as the centroid of that dimension. Simultaneously, a discrimination threshold for that language is set based on the distance from the encoded data to the centroid, specifically 95% of the total number of test set data points to the Euclidean distance of the centroid.

[0088] In step 420, the dialect type of the speech signal set to be detected is determined based on the embedding output features of the speech signal set to be detected and the first target dialect.

[0089] In step 430, based on the dialect type and the speech emotion recognition model, the emotion probability vector of the second target dialect in the set of speech signals to be detected is determined; wherein, the speech emotion recognition model includes a mixed dialect emotion model and a Mandarin emotion model; the second target dialect is one of the dialects in the set of speech signals to be detected.

[0090] It should be noted that the first target dialect is a dialect in the speech sample set, i.e., the dialect in the training set; the second target dialect is a dialect in the speech signal set to be detected.

[0091] In step 440, the emotion result of the set of voice signals to be detected is determined based on the emotion probability vector of the second target dialect.

[0092] In this embodiment, by determining the dialect type of the set of speech signals to be detected, when the language is determined to be a mixed dialect emotion type, i.e., when the language is determined to be a dialect, the output result is the output result of the trained mixed dialect emotion type, and the final result is the probability of each emotion and the emotion with the highest probability. If the determination result is a Mandarin emotion model, and the Mandarin emotion model includes Mandarin with regional accents or is entirely Mandarin speech, then the output result of the Mandarin emotion model is determined to be used.

[0093] Specifically, the 430 includes:

[0094] According to the formula: Determine the emotion probability vector of the second target dialect in the set of speech signals to be detected;

[0095] Where P is the emotion probability vector of the second target dialect; P i P0 is the probability vector of each emotion obtained from the mixed dialect emotion model; P0 is the probability vector of each emotion obtained from the Mandarin emotion model; L0 is the Mandarin centroid distance of the speech sample set; L0 i The dialect centroid distance of the speech sample set is denoted as .

[0096] In this embodiment, the emotion represented by the audio segment is determined by the emotion label to which the maximum value in the probability vector P of each emotion belongs. At the same time, the proportion of each other emotion is also given to facilitate the user's judgment.

[0097] This invention takes into account that in many scenarios, the speaker does not use a dialect or Mandarin, which causes great trouble for many systems that can only support a single language. Therefore, the speech emotion recognition model of this invention incorporates speech emotion result weighting, that is, effectively weighting the speech emotions obtained in the model to make them more accurate in emotion classification.

[0098] Specifically, step 420 includes:

[0099] Based on the embedding output features of the first target dialect, determine the first centroid vector of the first target dialect and the discrimination threshold of the first target dialect;

[0100] Based on the set of speech signals to be detected and the preset embedding algorithm, the second centroid vector of the second target dialect in the set of speech signals to be detected is determined; here, the preset embedding algorithm is the preset embedding encoding.

[0101] If the distance between the second centroid vector and the corresponding first centroid vector is less than the discrimination threshold, then the set of speech signals to be detected is determined to be a mixed dialect; otherwise, the set of speech signals to be detected is determined to be Mandarin.

[0102] It should be noted that the dialect embedding model can obtain centroid vectors for eight languages ​​(Northeastern dialect, Wu dialect, Xiang dialect, Min dialect, Hakka dialect, Yue dialect, and Mandarin). These vectors are combined into a matrix X, where the first row X0 represents the centroid vector after Mandarin embedding. A language discrimination threshold L can also be obtained. If the minimum distance between a preset speech segment after embedding and each centroid is less than L, the speech is determined to be the dialect corresponding to that preset speech; otherwise, it is Mandarin with a local accent. The distance is calculated using Euclidean distance.

[0103] In this embodiment, by determining the first centroid vector of the first target dialect, the discrimination threshold of the first target dialect, and the second centroid vector of the second target dialect in the set of speech signals to be detected, the dialect type of the set of speech signals to be detected is determined by judging the relationship between the three.

[0104] Optionally, step 300 includes:

[0105] The input feature vector, the embedded output feature, and the preset feature set are concatenated to determine the concatenated feature vector.

[0106] Based on the spliced ​​feature vector and the preset recurrent neural network, the speech emotion fusion vector is determined;

[0107] The speech emotion fusion vector is input into two fully connected layers, and a preset softmax function is used to calculate the confidence of the emotion category to determine the speech emotion recognition model.

[0108] Here, features are extracted from the spectrogram using a CNN (Convolutional Neural Network). The extracted features are then expanded in the time dimension and concatenated with GeMaps features and embedded output features. An RNN (Recurrent Neural Network) is then used to fully utilize the temporal connections. Finally, the network structure of a DNN (Deep Neural Network) and a preset classifier are added to train the speech emotion recognition model. The preset classifier is preferably a softmax classifier.

[0109] Specifically, to better adapt to scenarios involving both Mandarin and dialects, two models are trained: one using a mixture of emotion data from all dialects, and the other using only Mandarin. Since the amount of Mandarin data is large, training solely with Mandarin yields good results. The specific steps are as follows: 1. Extract features from the preprocessed audio dataset using three methods: MEL spectrogram transformation, GeMaps feature set extraction, and generating corresponding vectors using a previously trained embedding network. 2. Expand the spectrogram features using a CNN convolutional neural network and concatenate them with the GeMaps feature set and embedding features. 3. Input the concatenated features into an RNN recurrent neural network. 4. After passing through two fully connected layers, the network is connected to a softmax classifier, and the output is the corresponding emotion label.

[0110] Optionally, obtaining a set of speech samples with dialect features in step 100 includes:

[0111] Collect dialect voice dialogue data and remove voice data without emotion from the dialect voice dialogue data;

[0112] The audio format of the dialect speech dialogue data after dialect removal is standardized and determined to be unified audio data.

[0113] The unified audio data is segmented and determined into multiple audio files limited to a preset duration.

[0114] The emotion category is marked based on all audio files within the preset duration.

[0115] In this embodiment, dialect dialogue data is preferably collected via telephone conversations. For ease of processing, a single speaker uses only one dialect throughout a dialogue. Further, after determining the unified audio data, each dialogue segment is divided into sentences to avoid excessively long single audio segments. This is achieved by defining multiple audio files within a preset duration. Sentence segmentation is primarily based on vellus noise detection (VAD), with the longest segment not exceeding 10 seconds. The segment closest to 10 seconds is used for segmentation. Finally, all audio files within the preset duration are labeled with their respective emotion categories, indicating the dialect of the sentence, which can be represented numerically, thus completing the dialect encoding dataset collection.

[0116] Optionally, determining the input feature vector in step 200 includes:

[0117] All audio files within the preset duration after marking are processed for noise reduction using a preset filtering method to obtain the noise-reduced audio files;

[0118] The noise-reduced audio file is processed by a convolutional recurrent neural network to determine the input feature vector after preprocessing the speech sample set.

[0119] In this embodiment, since noise is common in speech signals during acquisition and the speech signals have uneven distribution characteristics in the frequency domain, some preprocessing is required. All audio files within a preset duration after marking are denoised using a preset filtering method to obtain denoised audio files. Here, in terms of speech denoising and enhancement, the selected methods include, but are not limited to: using LMS (Vibration Noise Test System) self-filtering for denoising, basic spectrum subtraction, Wiener filtering for denoising, etc. Specifically, in this invention, Kalman filtering is preferred for speech enhancement.

[0120] It should be noted that the Kalman filter is a linear least mean square estimator for the state of a discrete linear system. Its basic idea is to predict first and then correct, achieving system estimation through time update equations and observation update equations. In speech system modeling, the ultimate goal is to obtain an estimate of the actual speech signal. Its advantage lies in that it does not require using all past observations for estimation, making it suitable for real-time processing. Furthermore, it uses a recursive method for computation, making it suitable for computer solutions. Other preprocessing methods include pre-emphasis for uneven frequency distributions, and, since the entire speech signal cannot be processed at once, framing, windowing, endpoint detection, and short-time analysis.

[0121] In this embodiment, the denoised audio file is processed using a convolutional recurrent neural network (CRNN) to determine the input feature vector after preprocessing the speech sample set. First, the audio signal is converted into a spectrogram, which serves as the input source for the deep learning network, for feature extraction. The main advantage of this method is that the network input largely retains all the information inherent in the speech itself, while also eliminating the complex procedure of secondary screening of manual feature sets, and achieving good recognition results.

[0122] Specifically, the noise-reduced audio file is processed using a convolutional recurrent neural network to determine the preprocessed input feature vector of the speech sample set, including:

[0123] Perform a short-time Fourier transform on the denoised audio file to determine the processed spectral signal;

[0124] The Mel spectrum is determined by multiplying the spectral signal with the Mel filter;

[0125] The Mel spectrum is input into the convolutional recurrent neural network to determine the input feature vector after preprocessing the speech sample set.

[0126] In this embodiment, the steps for obtaining the Mel spectrum are as follows: performing a Short-Time Fourier Transform (STFT) on the signal; multiplying the generated spectral signal with a Mel filter to obtain the Mel spectrum. Further, after obtaining the Mel spectrum, a CRNN network model is built, and the Mel spectrum is input into the convolutional recurrent neural network, where the emotion category is used as the network output. This completes network training and determines the input feature vector after preprocessing the speech sample set. This invention extracts features from the Mel spectrum, extracts features from the Mel spectrogram using CRNN, and employs a scheme of training the model using multiple dialects. This effectively improves the robustness of the speech quality inspection system and achieves the technical effect of supporting automated speech quality inspection for multiple dialects.

[0127] like Figure 2As shown, an optional embodiment of the present invention also provides a voice quality inspection device, comprising:

[0128] Voice collection and transmission module 01 is used to collect and transmit voice data from voice customer service calls;

[0129] Voice storage module 02 is used to receive voice data transmitted from the voice transmission module and store it in the server;

[0130] The voice preprocessing module 03 performs voice preprocessing on the voice data that will be subject to quality inspection.

[0131] AI (Artificial Intelligence) Algorithm Module 04 uses AI algorithm models to derive multi-dialect emotion recognition results from pre-processed data.

[0132] The results feedback module 05 receives the recognition results from the AI ​​algorithm module, records the emotions in each data point, and reports dialogues containing non-compliant emotions.

[0133] like Figure 3 As shown, specifically, the voice quality inspection device may include: an audio collection module, a communication bus, a preprocessing module, a feature extraction module, a trained AI model, and a result output module.

[0134] In this embodiment, the audio collection module mainly uses front-end hardware devices, such as mobile phones and landline phones, to collect the voice to be recognized.

[0135] In this embodiment, the result output module is a server that stores the customer service voice recognition results and a database that records non-compliant emotional dialogues, used to query and display the emotion recognition results.

[0136] In this embodiment, the communication bus is mainly used for communication between modules, including the Serial Peripheral Interface communication bus and the I2C (Inter-integrated Circuit, two-wire serial bus) communication bus. Its main functions are to upload the collected audio data to the cloud server and send the recognition results to the result output module.

[0137] In this embodiment, the preprocessing module mainly performs speech noise reduction and enhancement and preprocessing operations on the received speech signal so that the subsequent functional modules can proceed smoothly.

[0138] In this embodiment, the feature extraction module mainly extracts features from the preprocessed speech data. In this document, the main tasks are generating Mel spectrum and extracting GeMAPS feature sets.

[0139] In this embodiment, the main function of the trained AI model is to identify and encode different dialects based on the features obtained by the feature extraction module, to weight the emotions of Mandarin containing dialects, and to perform emotion recognition on speech using the model obtained from the training data.

[0140] like Figure 4 As shown, an optional embodiment of the present invention also provides a voice quality inspection device, comprising:

[0141] The acquisition module 10 is used to acquire the speech data to be detected; the speech data to be detected includes a set of speech samples with dialect features input by the user and a set of speech signals to be detected;

[0142] The first determining module 20 is used to preprocess the speech sample set to determine the multimodal features of the processed speech sample set; the multimodal features include: input feature vector, embedded embedding output features, and preset feature set; the embedded embedding output features are determined by dialect embedding model encoding and are used to indicate each dialect in the speech sample set;

[0143] The second determining module 30 is used to determine a speech emotion recognition model based on the multimodal features;

[0144] The third determining module 40 is used to input the set of voice signals to be detected into the voice emotion recognition model to determine the user's emotional state.

[0145] Optionally, the third determining module 40 includes:

[0146] The first determining unit is configured to determine the embedding output features of a first target dialect in the speech sample set based on a preset embedding algorithm and the input feature vector; the first target dialect is a dialect in the speech sample set.

[0147] The second determining unit is used to determine the dialect type of the speech signal set to be detected based on the embedding output features of the speech signal set to be detected and the first target dialect.

[0148] The third determining unit is used to determine the emotion probability vector of the second target dialect in the set of speech signals to be detected based on the dialect type and the speech emotion recognition model; wherein, the speech emotion recognition model includes a mixed dialect emotion model and a Mandarin emotion model; and the second target dialect is one of the dialects in the set of speech signals to be detected.

[0149] The fourth determining unit is used to determine the emotion result of the set of voice signals to be detected based on the emotion probability vector of the second target dialect.

[0150] Specifically, the third determining unit is used for:

[0151] According to the formula: Determine the emotion probability vector of the second target dialect in the set of speech signals to be detected;

[0152] Where P is the emotion probability vector of the second target dialect; P i P0 is the probability vector of each emotion obtained from the mixed dialect emotion model; P0 is the probability vector of each emotion obtained from the Mandarin emotion model; L0 is the Mandarin centroid distance of the speech sample set; L0 i The dialect centroid distance of the speech sample set is denoted as .

[0153] Optionally, the second determining unit includes:

[0154] The first determining subunit is used to determine the first centroid vector of the first target dialect and the discrimination threshold of the first target dialect based on the embedding output features of the first target dialect.

[0155] The second determining subunit is used to determine the second centroid vector of the second target dialect in the set of speech signals to be detected based on the set of speech signals to be detected and the preset embedding algorithm.

[0156] The third determining subunit is used to determine the set of speech signals to be detected as a mixed dialect if the distance between the second centroid vector and the corresponding first centroid vector is less than the discrimination threshold; otherwise, it determines the set of speech signals to be detected as Mandarin.

[0157] Optionally, the first determining unit includes:

[0158] The fourth determining subunit is used to determine the compiled encoding feature vector based on the dialect embedding model and the preset embedding algorithm;

[0159] The fifth determining subunit is used to concatenate the compiled encoded feature vector and the input feature vector to determine the embedding output feature of the first target dialect in the speech sample set.

[0160] Optionally, the device further includes:

[0161] A model is constructed to build the dialect embedding model; the input value of the dialect embedding model is the input feature vector, and the output value of the dialect embedding model is the one-hot code corresponding to the dialect to which the speech sample set belongs;

[0162] A training model is used to train the dialect embedding model based on the set of speech samples, using the triplet loss function as the loss function.

[0163] Optionally, the second determining module 30 includes:

[0164] The fifth determining unit is used to perform feature concatenation on the input feature vector, the embedding output feature, and the preset feature set to determine the concatenated feature vector;

[0165] The sixth determining unit is used to determine the speech emotion fusion vector based on the spliced ​​feature vector and the preset recurrent neural network;

[0166] The seventh determining unit is used to input the speech emotion fusion vector into two fully connected layers and use a preset softmax function to calculate the confidence of the emotion category and determine the speech emotion recognition model.

[0167] Optionally, the acquisition module 10 includes:

[0168] A collection unit is used to collect dialect speech dialogue data and remove speech data without emotion from the dialect speech dialogue data.

[0169] The eighth determining unit is used to unify the audio format of the dialect speech dialogue data after removal, and determine it as unified audio data;

[0170] The ninth determining unit is used to segment the unified audio data and determine it into multiple audio files limited to a preset duration;

[0171] The tagging unit is used to tag the emotion category of all audio files within the preset duration.

[0172] Optionally, the first determining module 20 includes:

[0173] The acquisition unit is used to perform noise reduction processing on all audio files within a preset duration after marking, using a preset filtering method, and to acquire the noise-reduced audio files.

[0174] The tenth determining unit is used to determine the input feature vector of the preprocessed speech sample set by using a convolutional recurrent neural network on the denoised audio file.

[0175] Optionally, the tenth determining unit includes:

[0176] The sixth determining subunit is used to perform a short-time Fourier transform on the noise-reduced audio file to determine the processed spectral signal;

[0177] The seventh determining subunit is used to determine the Mel spectrum by multiplying the spectral signal and the Mel filter;

[0178] The eighth determining subunit is used to input the Mel spectrum into the convolutional recurrent neural network to determine the input feature vector after preprocessing the speech sample set.

[0179] An optional embodiment of the present invention also provides a voice quality inspection device, comprising: a processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the voice quality inspection method as described in any of the preceding claims.

[0180] An optional embodiment of the present invention also provides a readable storage medium storing a program. When executed by a processor, the program implements the various processes of the speech quality inspection method embodiments described above and achieves the same technical effect. To avoid repetition, it will not be described again here. The readable storage medium may be, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0181] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0182] The above describes the preferred embodiments of this application. It should be noted that those skilled in the art can make several improvements and modifications without departing from the principles described in this application, and these improvements and modifications are also within the protection scope of this application.

Claims

1. A voice quality inspection method, characterized by, include: Acquire the speech data to be detected; the speech data to be detected includes a set of speech samples with dialect features input by the user and a set of speech signals to be detected; The speech sample set is preprocessed to determine the multimodal features of the processed speech sample set; the multimodal features include: input feature vector, embedded embedding output features, and preset feature set; the embedded embedding output features are determined by dialect embedding model encoding and are used to indicate each dialect in the speech sample set; Based on the aforementioned multimodal features, a speech emotion recognition model is determined; The process of inputting the set of speech signals to be detected into the speech emotion recognition model to determine the user's emotional state includes: determining the embedding output features of a first target dialect in the speech sample set based on a preset embedding algorithm and the input feature vector; the first target dialect being one of the dialects in the speech sample set; determining the dialect type of the set of speech signals to be detected based on the embedding output features of the first target dialect; determining the emotion probability vector of a second target dialect in the set of speech signals to be detected based on the dialect type and the speech emotion recognition model; wherein the speech emotion recognition model includes a mixed dialect emotion model and a Mandarin emotion model; the second target dialect being one of the dialects in the set of speech signals to be detected; and determining the emotion result of the set of speech signals to be detected based on the emotion probability vector of the second target dialect.

2. The method of claim 1, wherein, Based on the dialect type and the speech emotion recognition model, the emotion probability vector of the second target dialect in the set of speech signals to be detected is determined, including: According to the formula: , the emotion probability vector of the second target dialect in the to-be-detected speech signal set is determined; P is an emotion probability vector of the second target dialect; a probability vector of each emotion obtained for the mixed-dialect emotion model; a probability vector of each emotion obtained for the Mandarin emotion model; a Mandarin centroid distance of the set of speech samples; a dialect centroid distance of the set of speech samples.

3. The method of claim 1, wherein, Based on the set of speech signals to be detected and the embedding output features of the first target dialect, the dialect type of the set of speech signals to be detected is determined, including: Based on the embedding output features of the first target dialect, determine the first centroid vector of the first target dialect and the discrimination threshold of the first target dialect; Based on the set of speech signals to be detected and the preset embedding algorithm, determine the second centroid vector of the second target dialect in the set of speech signals to be detected; If the distance between the second centroid vector and the corresponding first centroid vector is less than the discrimination threshold, then the set of speech signals to be detected is determined to be a mixed dialect; otherwise, the set of speech signals to be detected is determined to be Mandarin.

4. The method according to claim 1, characterized in that, The step of determining the embedding output features of the first target dialect in the speech sample set according to the preset embedding algorithm and the input feature vector includes: Based on the dialect embedding model and the preset embedding algorithm, the compiled encoded feature vector is determined; The compiled encoded feature vector and the input feature vector are concatenated to determine the embedding output feature of the first target dialect in the speech sample set.

5. The method according to claim 1, characterized in that, Constructing and training the dialect embedding model includes: Construct the dialect embedding model; the input value of the dialect embedding model is the input feature vector, and the output value of the dialect embedding model is the one-hot code corresponding to the dialect to which the speech sample set belongs; The triplet loss function is used as the loss function, and the dialect embedding model is trained based on the speech sample set.

6. The method according to claim 1, characterized in that, The step of determining the speech emotion recognition model based on the multimodal features includes: The input feature vector, the embedding output feature, and the preset feature set are concatenated to determine the concatenated feature vector. Based on the spliced ​​feature vector and the preset recurrent neural network, the speech emotion fusion vector is determined; The speech emotion fusion vector is input into two fully connected layers, and a preset softmax function is used to calculate the confidence of the emotion category to determine the speech emotion recognition model.

7. The method according to claim 1, characterized in that, The acquisition of the speech data to be detected includes acquiring a set of speech samples with dialect features, including: Collect dialect voice dialogue data and remove voice data without emotion from the dialect voice dialogue data; The audio format of the dialect speech dialogue data after dialect removal is standardized and determined to be unified audio data. The unified audio data is segmented and determined into multiple audio files limited to a preset duration. The emotion category is marked based on all audio files within the preset duration.

8. The method according to claim 1, characterized in that, Determine the input feature vector in the multimodal features of the processed speech sample set, including: All audio files within the preset duration after marking are processed for noise reduction using a preset filtering method to obtain the noise-reduced audio files; The noise-reduced audio file is processed by a convolutional recurrent neural network to determine the input feature vector after preprocessing the speech sample set.

9. The method according to claim 8, characterized in that, The denoised audio file is processed by a convolutional recurrent neural network to determine the preprocessed input feature vector of the speech sample set, including: Perform a short-time Fourier transform on the denoised audio file to determine the processed spectral signal; The Mel spectrum is determined by multiplying the spectral signal with the Mel filter; The Mel spectrum is input into the convolutional recurrent neural network to determine the input feature vector after preprocessing the speech sample set.

10. A voice quality inspection device, characterized in that, include: The acquisition module is used to acquire the speech data to be detected; the speech data to be detected includes a set of speech samples with dialect features input by the user and a set of speech signals to be detected; The first determining module is used to preprocess the speech sample set to determine the multimodal features of the processed speech sample set; the multimodal features include: input feature vector, embedded embedding output features, and preset feature set; the embedded embedding output features are determined by dialect embedding model encoding and are used to indicate each dialect in the speech sample set; The second determining module is used to determine the speech emotion recognition model based on the multimodal features; The third determining module is used to input the set of voice signals to be detected into the voice emotion recognition model to determine the user's emotional state. The third determining module includes: a first determining unit, configured to determine the embedding output features of a first target dialect in the speech sample set based on a preset embedding algorithm and the input feature vector; the first target dialect is one of the dialects in the speech sample set; a second determining unit, configured to determine the dialect type of the speech signal set to be detected based on the speech signal set to be detected and the embedding output features of the first target dialect; a third determining unit, configured to determine the emotion probability vector of a second target dialect in the speech signal set to be detected based on the dialect type and the speech emotion recognition model; wherein the speech emotion recognition model includes a mixed dialect emotion model and a Mandarin emotion model; the second target dialect is one of the dialects in the speech signal set to be detected; and a fourth determining unit, configured to determine the emotion result of the speech signal set to be detected based on the emotion probability vector of the second target dialect.

11. A voice quality inspection device, characterized in that, include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the voice quality inspection method as described in any one of claims 1 to 9.

12. A readable storage medium, characterized in that, The readable storage medium stores a program that, when executed by a processor, implements the voice quality inspection method as described in any one of claims 1 to 9.

Images