Speech evaluation method and device, computer device and storage medium

Abstract

The application relates to a voice evaluation method and device based on cloud technology, computer equipment, a storage medium and a computer program product. The method comprises the following steps: performing voice activity detection on original voice to obtain corresponding to-be-detected voice; performing time-frequency conversion processing on the to-be-detected voice, and performing feature extraction on a voice signal obtained through the time-frequency conversion processing to obtain acoustic features corresponding to the to-be-detected voice; performing convolution processing on the acoustic features to obtain deep features at different time points; and performing quality evaluation processing based on the deep features at the different time points to obtain a quality evaluation result corresponding to the original voice. The method can effectively perform quality monitoring and quality evaluation on real-time voice on a line.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a speech evaluation method, apparatus, computer equipment, storage medium, and computer program product. Background Technology

[0002] With the development of computer technology, speech quality assessment technology has emerged. This technology is used to detect the quality of speech and can be applied in various scenarios such as speech synthesis, speech conversion, and speech enhancement.

[0003] Traditional speech quality assessment techniques often employ reference-based methods. These methods first adjust the levels and time of both the reference and damaged speech samples to ensure that the differences between them are solely due to quality variations. Then, they perform auditory transformation on the speech pairs and finally compare the differences between the transformed samples to estimate the quality score of the test speech. It is evident that reference-based speech quality assessment methods require lossless reference speech for comparison, calculating the difference between the damaged speech signal and the reference signal in order to estimate the corresponding score.

[0004] However, current reference-based speech quality assessment methods require lossless reference signals as a reference, which are difficult to obtain in real-world scenarios. Therefore, these methods can only be used for offline assessment and cannot achieve real-time online speech quality assessment. Summary of the Invention

[0005] Therefore, it is necessary to provide a voice evaluation method, apparatus, computer device, computer-readable storage medium, and computer program product capable of real-time evaluation to address the aforementioned technical problems.

[0006] This application provides a speech evaluation method, the method comprising:

[0007] Speech activity detection is performed on the original speech to obtain the corresponding speech to be tested;

[0008] The speech to be tested is subjected to time-frequency conversion processing, and the speech signal obtained by the time-frequency conversion processing is subjected to feature extraction to obtain the acoustic features corresponding to the speech to be tested.

[0009] The acoustic features are convolved to obtain depth features at different time points;

[0010] Based on the depth features at different times, a quality assessment process is performed to obtain the quality assessment result corresponding to the original speech.

[0011] This application also provides a speech evaluation device, the device comprising:

[0012] The detection module is used to detect speech activity in the original speech to obtain the corresponding speech to be tested;

[0013] The conversion module is used to perform time-frequency conversion processing on the speech to be tested, and to extract features from the speech signal obtained by the time-frequency conversion processing to obtain the acoustic features corresponding to the speech to be tested.

[0014] The convolution module is used to perform convolution processing on the acoustic features to obtain depth features at different times;

[0015] The processing module is used to perform quality assessment processing based on the depth features at different times to obtain the quality assessment result corresponding to the original speech.

[0016] In one embodiment, the detection module is further configured to perform frame-segmentation processing on the original speech to obtain corresponding frame speech signals; determine the short-time energy and short-time zero-crossing rate corresponding to each frame speech signal; and based on the short-time energy and the short-time zero-crossing rate, filter out valid signal frames from each frame speech signal to obtain the speech to be tested.

[0017] In one embodiment, the detection module is further configured to perform windowing processing on each frame of the speech signal, and perform time-frequency conversion processing on the windowed speech signal to obtain the corresponding spectrum; determine the short-time energy of each frame of the speech signal based on the spectrum corresponding to each frame of the speech signal; obtain the time-domain waveform corresponding to each frame of the speech signal, and determine the short-time zero-crossing rate of the corresponding speech signal based on the time-domain waveform.

[0018] In one embodiment, the convolution module is further configured to perform frame segmentation processing on the acoustic features to obtain feature frames at different times; and to perform convolution and normalization processing on the feature frames at different times to obtain depth features at different times.

[0019] In one embodiment, the processing module is further configured to: determine the output features of the current moment based on the input features of the current moment and the hidden layer features output by the preceding moments, and determine the update parameters of the feature object at the current moment; the input features of the current moment are the depth features of the corresponding moment, and the feature object represents the path of feature transmission; determine the feature object of the current moment based on the feature object of the preceding moments, the output features of the current moment, and the update parameters; determine the hidden layer features output by the current moment based on the hidden layer features output by the preceding moments, the output features of the current moment, and the feature object of the current moment; take the input features of the next moment as the input features of the current moment, and return to the step of determining the output features of the current moment based on the input features of the current moment and the hidden layer features output by the preceding moments, and continue to execute until the hidden layer features output by the last moment are obtained; determine the quality assessment result corresponding to the original speech based on the hidden layer features output by the last moment.

[0020] In one embodiment, the device is applied to a target speech evaluation model; the target speech evaluation model includes a forget gate, an input gate, and an output gate, and the feature object is a cell; the processing module is further configured to: determine the output feature of the forget gate at the current time based on the input features at the current time and the hidden layer features output at the previous time based on the input features at the current time; determine the output feature of the input gate at the current time based on the input features at the current time and the hidden layer features output at the previous time based on the input features at the current time and the hidden layer features output at the previous time based on the input features at the current time and the hidden layer features output at the previous time based on the input features at the current time and the hidden layer features output at the previous time based on the input features at the current time; determine the feature object at the current time based on the output feature of the forget gate at the current time, the output feature of the input gate at the current time, the update parameters, and the feature object at the previous time; determine the output feature of the output gate at the current time based on the input features at the current time and the hidden layer features output at the previous time based on the input features at the current time and the cell at the current time; and determine the hidden layer features output at the current time based on the output feature of the output gate at the current time and the cell at the current time.

[0021] In one embodiment, the device is applied to a video conference; the detection module is further configured to detect voice activity in the video conference and obtain the corresponding real-time voice.

[0022] The conversion module is also used to perform time-frequency conversion processing on the real-time speech, and to extract features from the speech signal obtained by the time-frequency conversion processing to obtain the real-time acoustic features corresponding to the real-time speech.

[0023] The convolution module is also used to perform convolution processing on the real-time acoustic features to obtain target depth features at different times.

[0024] The processing module is also used to perform quality assessment processing based on the target depth features at different times to obtain the quality assessment score of the real-time voice in the video conference.

[0025] The device further includes a trigger module, which is used to trigger a reminder operation if the quality assessment score of the real-time voice does not meet the preset quality conditions.

[0026] In one embodiment, the apparatus is applied to a target speech evaluation model, the training steps of which include:

[0027] The acquisition module is used to acquire a first speech sample and a second speech sample that is different from the first speech sample;

[0028] The evaluation module is used to perform quality evaluation processing on the first speech sample and the second speech sample respectively through the speech evaluation model to be trained, so as to obtain the first prediction result and the second prediction result.

[0029] A quality difference determination module is used to determine the quality difference between the first prediction result and the second prediction result and their respective corresponding evaluation labels.

[0030] The ranking loss determination module is used to determine the ranking loss between the first speech sample and the second speech sample based on the first prediction result and the second prediction result.

[0031] A construction module is used to construct a target loss function based on the quality differences and the ranking loss.

[0032] The training module is used to train the speech evaluation model to be trained using the target loss function until a stopping condition is met, thereby obtaining a trained target speech evaluation model; the target speech evaluation model is used to evaluate the quality of the original speech.

[0033] This application also provides a computer device, the computer device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0034] Speech activity detection is performed on the original speech to obtain the corresponding speech to be tested; time-frequency conversion processing is performed on the speech to be tested, and feature extraction is performed on the speech signal obtained by time-frequency conversion processing to obtain the acoustic features corresponding to the speech to be tested; convolution processing is performed on the acoustic features to obtain depth features at different times; quality assessment processing is performed based on the depth features at different times to obtain the quality assessment result corresponding to the original speech.

[0035] This application also provides a computer-readable storage medium having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0036] Speech activity detection is performed on the original speech to obtain the corresponding speech to be tested; time-frequency conversion processing is performed on the speech to be tested, and feature extraction is performed on the speech signal obtained by time-frequency conversion processing to obtain the acoustic features corresponding to the speech to be tested; convolution processing is performed on the acoustic features to obtain depth features at different times; quality assessment processing is performed based on the depth features at different times to obtain the quality assessment result corresponding to the original speech.

[0037] This application also provides a computer program product, which includes a computer program that, when executed by a processor, performs the following steps:

[0038] Speech activity detection is performed on the original speech to obtain the corresponding speech to be tested; time-frequency conversion processing is performed on the speech to be tested, and feature extraction is performed on the speech signal obtained by time-frequency conversion processing to obtain the acoustic features corresponding to the speech to be tested; convolution processing is performed on the acoustic features to obtain depth features at different times; quality assessment processing is performed based on the depth features at different times to obtain the quality assessment result corresponding to the original speech.

[0039] The aforementioned speech evaluation method, apparatus, computer equipment, storage medium, and computer program product, by detecting speech activity in the original speech, can eliminate silences in the original speech and obtain valid test speech. Time-frequency conversion processing is performed on the test speech to reflect the relevant information of the signal frequency change over time. Feature extraction is performed on the speech signal obtained from the time-frequency conversion processing to obtain acoustic features reflecting the time-frequency relationship. Convolution processing is performed on the acoustic features to obtain higher-dimensional and deeper-level depth features at different times. Quality evaluation processing based on these higher-dimensional and deeper-level depth features at different times can accurately evaluate the quality of the original speech, thereby obtaining the corresponding quality evaluation result. Furthermore, this embodiment does not require a reference signal to participate in the speech quality evaluation process; only the original speech signal is needed to effectively evaluate the speech quality, thus enabling real-time speech quality evaluation.

[0040] This application provides a method for training a speech evaluation model, the method comprising:

[0041] Acquire a first speech sample and a second speech sample that is different from the first speech sample;

[0042] The first speech sample and the second speech sample are subjected to quality assessment processing by the speech evaluation model to be trained, respectively, to obtain the first prediction result and the second prediction result.

[0043] Determine the quality differences between the first prediction result and the second prediction result and their respective corresponding evaluation labels;

[0044] Based on the first prediction result and the second prediction result, determine the ranking loss between the first speech sample and the second speech sample;

[0045] Based on the aforementioned quality differences and ranking losses, a target loss function is constructed;

[0046] The speech evaluation model to be trained is trained using the target loss function until a stopping condition is met, resulting in a trained target speech evaluation model; the target speech evaluation model is used to evaluate the quality of the original speech.

[0047] This application also provides a speech evaluation model training device, the device comprising:

[0048] The acquisition module is used to acquire a first speech sample and a second speech sample that is different from the first speech sample;

[0049] The evaluation module is used to perform quality evaluation processing on the first speech sample and the second speech sample respectively through the speech evaluation model to be trained, so as to obtain the first prediction result and the second prediction result.

[0050] A quality difference determination module is used to determine the quality difference between the first prediction result and the second prediction result and their respective corresponding evaluation labels.

[0051] The ranking loss determination module is used to determine the ranking loss between the first speech sample and the second speech sample based on the first prediction result and the second prediction result.

[0052] A construction module is used to construct a target loss function based on the quality differences and the ranking loss.

[0053] The training module is used to train the speech evaluation model to be trained using the target loss function until a stopping condition is met, thus obtaining the trained target speech evaluation model; the target speech evaluation model is used to evaluate the quality of the speech to be tested.

[0054] In one embodiment, the ranking loss determination module is further configured to determine, based on the first prediction result and the second prediction result, a prediction probability that the quality of the first speech sample is higher than that of the second speech sample; determine the correctness of the relative relationship between the first speech sample and the second speech sample based on the prediction probability and the corresponding probability label; and determine the ranking loss between the first speech sample and the second speech sample based on the correctness of the relative relationship.

[0055] In one embodiment, the ranking loss determination module is further configured to determine the uncertainty coefficient of the relative relationship based on the evaluation labels of the first prediction result and the evaluation labels of the second prediction result;

[0056] Based on the correctness of the relative relationship and the uncertainty coefficient of the relative relationship, the ranking loss between the first speech sample and the second speech sample is determined.

[0057] This application also provides a computer device, the computer device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0058] A first speech sample and a second speech sample different from the first speech sample are acquired. The first speech sample and the second speech sample are then subjected to quality assessment processing using a speech evaluation model to be trained, resulting in a first prediction result and a second prediction result. The quality differences between the first prediction result and the second prediction result and their respective corresponding evaluation labels are determined. Based on the first prediction result and the second prediction result, a ranking loss is determined between the first speech sample and the second speech sample. A target loss function is constructed based on the quality differences and the ranking loss. The speech evaluation model to be trained is trained using the target loss function until a stopping condition is met, resulting in a trained target speech evaluation model. The target speech evaluation model is used to evaluate the quality of the speech to be tested.

[0059] This application also provides a computer-readable storage medium having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0060] A first speech sample and a second speech sample different from the first speech sample are acquired. The first speech sample and the second speech sample are then subjected to quality assessment processing using a speech evaluation model to be trained, resulting in a first prediction result and a second prediction result. The quality differences between the first prediction result and the second prediction result and their respective corresponding evaluation labels are determined. Based on the first prediction result and the second prediction result, a ranking loss is determined between the first speech sample and the second speech sample. A target loss function is constructed based on the quality differences and the ranking loss. The speech evaluation model to be trained is trained using the target loss function until a stopping condition is met, resulting in a trained target speech evaluation model. The target speech evaluation model is used to evaluate the quality of the speech to be tested.

[0061] This application also provides a computer program product, which includes a computer program that, when executed by a processor, performs the following steps:

[0062] A first speech sample and a second speech sample different from the first speech sample are acquired. The first speech sample and the second speech sample are then subjected to quality assessment processing using a speech evaluation model to be trained, resulting in a first prediction result and a second prediction result. The quality differences between the first prediction result and the second prediction result and their respective corresponding evaluation labels are determined. Based on the first prediction result and the second prediction result, a ranking loss is determined between the first speech sample and the second speech sample. A target loss function is constructed based on the quality differences and the ranking loss. The speech evaluation model to be trained is trained using the target loss function until a stopping condition is met, resulting in a trained target speech evaluation model. The target speech evaluation model is used to evaluate the quality of the speech to be tested.

[0063] The aforementioned speech evaluation model training method, apparatus, computer equipment, storage medium, and computer program product perform quality evaluation on first and second speech samples with different levels of quality loss using the speech evaluation model to be trained, obtaining corresponding first and second prediction results. Determining the quality difference between the first and second prediction results and their respective corresponding evaluation labels allows for the identification of the gap between the model's predictions and the actual results. Based on the first and second prediction results, the ranking loss between the first and second speech samples is determined, which measures the relative quality distance between speech sample pairs. Based on the quality differences and ranking losses, a target loss function is constructed, incorporating both ranking loss and quality differences. The speech evaluation model is repeatedly trained to adjust model parameters, thereby improving the accuracy of model evaluation. When a stopping condition is met, a well-trained target speech evaluation model is obtained. This target speech evaluation model enables accurate and rapid quality evaluation of raw speech and real-time speech, improving the timeliness and efficiency of speech evaluation. Attached Figure Description

[0064] Figure 1 This is a diagram illustrating the application environment of the speech evaluation method in one embodiment;

[0065] Figure 2 This is a flowchart illustrating a speech evaluation method in one embodiment;

[0066] Figure 3 This is a schematic diagram of frame segmentation of the original speech;

[0067] Figure 4 This is a schematic diagram of the structure of a convolutional neural network in one embodiment;

[0068] Figure 5 This is a flowchart illustrating the process of performing quality assessment based on depth features at different times to obtain the quality assessment result corresponding to the original speech in one embodiment.

[0069] Figure 6 This is a schematic diagram of the structure of a long short-term memory network in one embodiment;

[0070] Figure 7 This is a flowchart illustrating the process of performing quality assessment based on depth features at different times to obtain the quality assessment result corresponding to the original speech in another embodiment.

[0071] Figure 8a This is a schematic diagram of the forget gate in one embodiment;

[0072] Figure 8b This is a schematic diagram of the input gate structure in one embodiment;

[0073] Figure 8c This is a schematic diagram of the cell cell renewal structure in one embodiment;

[0074] Figure 8d This is a schematic diagram of the output gate in one embodiment;

[0075] Figure 9 A framework diagram of the speech evaluation method in another embodiment;

[0076] Figure 10 This is a flowchart illustrating a speech evaluation model training method in one embodiment;

[0077] Figure 11 This is a schematic diagram illustrating the determination of the ranking loss for speech sample pairs in one embodiment;

[0078] Figure 12 This is a structural block diagram of a speech evaluation device in one embodiment;

[0079] Figure 13 This is a structural block diagram of a speech evaluation device in one embodiment;

[0080] Figure 14 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0081] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0082] This application relates to the field of Artificial Intelligence (AI) technology. AI is the theory, methods, techniques, and application systems that utilize digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine capable of reacting in a manner similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making capabilities. The solutions provided in this application relate to AI-based speech evaluation methods, which are specifically illustrated through the following embodiments.

[0083] The speech evaluation method provided in this application embodiment can be applied to, for example, Figure 1In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated on server 104 or placed on a cloud or other network server. Both terminal 102 and server 104 can independently execute the speech evaluation method provided in this embodiment. Terminal 102 and server 104 can also work together to execute the speech evaluation method provided in this embodiment. When terminal 102 and server 104 work together to execute the speech evaluation method provided in this embodiment, terminal 102 acquires the original speech to be evaluated and sends it to server 104. Server 104 performs speech activity detection on the original speech to obtain the corresponding speech to be tested. Server 104 performs time-frequency conversion processing on the speech to be tested and extracts features from the speech signal obtained by the time-frequency conversion processing to obtain the acoustic features corresponding to the speech to be tested. Server 104 performs convolution processing on the acoustic features to obtain depth features at different times. Server 104 performs quality evaluation processing based on the depth features at different times to obtain the quality evaluation result corresponding to the original speech. Server 104 returns the quality assessment results of the original speech to terminal 102.

[0084] The server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal 102 can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, and this application does not impose any restrictions.

[0085] In one embodiment, such as Figure 2 As shown, a speech evaluation method is provided, which is applied to a computer device (specifically, the computer device may be...). Figure 1 Taking a terminal or server as an example, the following steps are included:

[0086] Step S202: Perform speech activity detection on the original speech to obtain the corresponding speech to be tested.

[0087] Speech, in this context, refers to the material form of language, its external expression. Raw speech is directly collected, unprocessed audio, including spoken words, singing, sounds from musical instruments, and audio from videos, which can be at least one sourced from movies, TV series, programs, and animations. Test speech is the effective speech extracted from the raw speech and used for quality assessment.

[0088] Voice Activity Detection (VAD), also known as speech endpoint detection or speech boundary detection, is used to identify and eliminate long periods of silence in a sound signal stream.

[0089] Specifically, computer devices can acquire raw speech from local devices or other devices or networks, or they can obtain audio or video through recording, using the recorded audio as the raw speech, or extracting audio from video as the raw speech. The computer devices perform speech activity detection on the raw speech to identify and remove silences, thus obtaining the corresponding speech to be tested.

[0090] Step S204: Perform time-frequency conversion processing on the speech to be tested, and extract features from the speech signal obtained by the time-frequency conversion processing to obtain the acoustic features corresponding to the speech to be tested.

[0091] Acoustic features characterize key information in the tested speech and may include at least one of pitch features, intensity features, duration features, and timbre features. Further, acoustic features refer to the expression of key information in the tested speech in the time and frequency domains. The time domain describes the relationship between a mathematical function or physical signal and time, such as the relationship between a speech signal and time. The frequency domain is the coordinate system used to describe the frequency characteristics of a signal. For example, an acoustic feature could be Mel-Frequency Cipstal Coefficients (MFCC) features.

[0092] Pitch characteristics represent the highness or lowness of a sound, intensity characteristics represent the strength or weakness of a sound, duration characteristics represent the length of a sound, and timbre characteristics represent the unique features and essence of a sound, also known as "sound quality".

[0093] Time-frequency conversion (TFCC) is used to determine the relationship between time and frequency in a speech signal, revealing the frequency components and their changes over time. Specifically, TFCC involves converting the speech signal from the time domain to the frequency domain using time-frequency analysis, while preserving both time and frequency information to reflect the time-frequency relationship. Time-frequency analysis methods can include Short-Time Fourier Transform (STFT) or Wavelet Transform (WT). The STFT is a mathematical transformation related to the Fourier Transform, performing a Fourier transform on a short-time signal to determine the frequency and phase of a local sinusoidal wave in a time-varying signal. The Wavelet Transform is a newer transform analysis method that inherits and develops the localization concept of the STFT while overcoming the drawback of the window size not changing with frequency, providing a "time-frequency" window that varies with frequency.

[0094] Specifically, the computer equipment can perform time-frequency conversion processing on the speech to be tested to obtain a spectrogram corresponding to the speech. Feature extraction is then performed on the spectrogram to obtain the corresponding acoustic features.

[0095] Time-frequency conversion processing can be Fourier transform or short-time Fourier transform. Short-time Fourier transform involves dividing a long speech signal into frames, windowing it, and then performing a Fourier transform on each frame. The results of each frame are then stacked along another dimension to obtain a graph, which is the spectrogram.

[0096] The computer device can perform frame-based processing on the speech signal to be tested, and then window the speech signal obtained from each frame-based processing. The computer device then performs a Fourier transform on each windowed speech signal to obtain a spectrogram corresponding to the speech signal to be tested. Feature extraction is performed on the spectrogram to obtain the corresponding acoustic features. Further, the computer device performs a Fourier transform on each windowed speech signal to obtain the spectrum corresponding to each frame of the speech signal. The spectrograms of each frame are stacked along the time dimension to obtain the spectrogram corresponding to the speech signal to be tested.

[0097] In one embodiment, a computer device can perform a Mel-spectral transform on the spectrogram to obtain the corresponding Mel spectrum, which is then used as the acoustic feature of the speech to be tested. Further, a Mel-spectral transform can be performed on the spectrogram using Mel-scale filter banks to obtain the Mel spectrum mel(f), highlighting the low-frequency components of the speech signal. The transform formula is as follows:

[0098]

[0099] Where f is the frequency of the speech signal.

[0100] Step S206: Perform convolution processing on the acoustic features to obtain depth features at different times.

[0101] Among them, deep features refer to the deeper, higher-dimensional, and more abstract key information obtained by further feature extraction of acoustic features.

[0102] Specifically, the computer device performs multi-layer convolution processing on the acoustic features to obtain depth features at different times. Further, the computer device performs frame segmentation processing on the acoustic features, and then performs multi-layer convolution processing on the feature frames obtained from the frame segmentation at different times to obtain depth features at different times.

[0103] In one embodiment, the computer device inputs feature frames obtained from frame segmentation at different times into a convolutional neural network, and performs convolution processing through multiple convolutional layers of the convolutional neural network to obtain depth features at different times.

[0104] In one embodiment, the Mel spectrum can be framed to obtain the Mel spectrum corresponding to the speech signal at each time moment, such as the Mel spectrum for 50ms. Using the Mel spectrum at each time moment as a feature frame, feature frames for different time moments can be obtained.

[0105] In one embodiment, a 48-Melb filter bank can be used to extract features from the spectrogram of the speech under test, obtaining the Mel spectrum. The Mel spectrum is then segmented into frames, and a 50ms segment of the Mel spectrum is used as the input to the neural network. Therefore, the Mel spectrum feature dimension input to the convolutional neural network is T. d ×48×15, where T d The duration of the speech signal.

[0106] Step S208: Perform quality assessment processing based on the deep features at different times to obtain the quality assessment result corresponding to the original speech.

[0107] The quality assessment result refers to the evaluation result obtained by assessing the quality of the original speech. Specifically, the quality assessment result can be a quality assessment score, such as the mean opinion score (MOS), meaning the MOS value can be used to evaluate the quality of the speech.

[0108] Specifically, the computer equipment performs further feature extraction and feature fusion processing on the deep features at different times to obtain hidden layer features that incorporate more feature information. Fully connected processing is then applied to the hidden layer features to obtain the quality assessment result corresponding to the original speech.

[0109] In one embodiment, the quality assessment result may include a MOS value. The computer device performs further feature extraction and feature fusion processing on the deep features at different times to obtain high-dimensional hidden layer features. Fully connected processing is then applied to the hidden layer features to obtain the MOS value corresponding to the original speech, and this MOS value is used as the quality assessment result for the original speech.

[0110] In this embodiment, speech activity detection is performed on the original speech to eliminate silences and obtain valid test speech. Time-frequency conversion is then applied to the test speech to reflect the frequency variation of the signal over time. Feature extraction is performed on the speech signal obtained from the time-frequency conversion to obtain acoustic features reflecting the time-frequency relationship. Convolution processing is applied to the acoustic features to obtain higher-dimensional, deeper-level depth features at different times. Quality assessment is then performed based on these higher-dimensional, deeper-level depth features at different times, accurately evaluating the quality of the original speech and obtaining the corresponding quality assessment result. Furthermore, this embodiment does not require a reference signal for speech quality assessment; only the original speech signal is needed to effectively evaluate speech quality, enabling real-time speech quality assessment.

[0111] Traditional speech evaluation methods require both the signal of the speech to be evaluated and a reference signal to determine its quality. However, the reference signal is unavailable in real-time audio and video scenarios. The speech evaluation method described in this embodiment, however, only requires the signal of the speech to be evaluated to effectively perform online real-time speech quality evaluation and improves the efficiency of speech quality evaluation.

[0112] In one embodiment, speech activity detection is performed on the original speech to obtain the corresponding speech to be tested, including:

[0113] The original speech is segmented into frames to obtain corresponding speech signals for each frame; the short-time energy and short-time zero-crossing rate of each speech signal are determined; based on the short-time energy and short-time zero-crossing rate, valid signal frames are selected from each speech signal to obtain the speech to be tested.

[0114] Short-time energy refers to the speech energy calculated over a short period of time, typically within one frame. The short-time energy for silent segments is zero, meaning the short-time energy during silence is zero.

[0115] Short-time zero-crossing rate is a characteristic parameter in the time-domain analysis of speech signals, referring to the number of times the signal crosses zero within each frame. For continuous speech signals with a time horizontal axis, the time-domain waveform of the speech can be observed crossing the horizontal axis. In the case of discrete-time speech signals, if adjacent samples have different algebraic signs, a zero-crossing is said to have occurred, and therefore the number of zero-crossings can be calculated. The number of zero-crossings per unit time is called the zero-crossing rate. The zero-crossing rate over a long period is called the average zero-crossing rate.

[0116] Specifically, the computer equipment can segment the original speech into frames according to a preset duration to obtain corresponding frame speech signals. For example, segmenting the speech into frames every 20 milliseconds (ms) yields a 20-millisecond speech signal per frame. The computer equipment can then calculate the short-time energy and short-time zero-crossing rate of each frame speech signal. Based on these parameters, silent segments in the original speech are filtered out to obtain valid signal frames, thus acquiring the speech to be tested. Each frame of the speech to be tested is a valid signal, i.e., a non-silent signal.

[0117] like Figure 3 The diagram shown illustrates the framing of the original speech in one embodiment. The entire speech signal is divided into several segments, each of which is a frame. During framing, adjacent frames are designed to overlap, creating overlapping frames. For example, there is an overlapping signal between the k-th frame and the (k-1)-th frame, and an overlapping signal between the (k+1)-th frame and the k-th frame.

[0118] In this embodiment, the original speech is processed into frames to obtain corresponding speech signals for each frame. The short-time energy and short-time zero-crossing rate corresponding to each frame of speech signal are determined. The short-time energy and short-time zero-crossing rate are used as conditions for filtering valid signal frames. This can effectively remove silent signals from each frame of speech signal and filter out valid signal frames, thereby accurately extracting the speech to be tested from the original speech.

[0119] In one embodiment, determining the short-time energy and short-time zero-crossing rate corresponding to each frame of the speech signal includes:

[0120] Windowing is applied to each frame of the speech signal, and time-frequency conversion is performed on the windowed speech signal to obtain the corresponding spectrum. Based on the spectrum of each frame of the speech signal, the short-time energy of each frame of the speech signal is determined. The time-domain waveform of each frame of the speech signal is obtained, and the short-time zero-crossing rate of the corresponding speech signal is determined based on the time-domain waveform.

[0121] Windowing processing refers to multiplying the speech signal with a specific window function to achieve time-frequency localization of the speech signal. Window functions include rectangular windows, triangular windows, and Hanning windows, and generally have low-pass characteristics. The spectrum, also known as the vibration spectrum, is short for frequency spectral density, which is the distribution curve of frequencies. The time-domain waveform refers to the waveform of the speech signal in the time domain. The time-domain waveform of a speech signal can express the changes of the speech signal over time.

[0122] Specifically, the computer device can obtain a window function, and then perform windowing processing on each frame of the speech signal to obtain the windowed speech signal for each frame.

[0123] For example, the window function is the Hanning window. The Hanning window has a wider main lobe and significantly smaller side lobes, which can better prevent spectral leakage. The function expression w(n) of the Hanning window is shown below:

[0124]

[0125] Where S is the window length, and n represents the nth frame of the speech signal. The window length and step size for windowing each frame of the speech signal are 20ms*sr or 10ms*sr, respectively. Here, sr represents the sampling rate of the speech signal.

[0126] The computer device can perform time-frequency conversion on each frame of the windowed speech signal to obtain the spectrum corresponding to each frame. Furthermore, the computer device can perform a Fourier transform on each frame of the speech signal to obtain the spectrum corresponding to each frame. Based on the spectrum of the speech signal, the short-time energy of the speech signal can be calculated.

[0127] In one embodiment, the computer device can perform wavelet transform on each frame of the speech signal to obtain the spectrum corresponding to each frame of the speech signal.

[0128] For example, if each frame of the speech signal is x(n), then the windowed speech signal is... Then, a Fourier transform is performed on each frame of the windowed speech signal as shown in the following equation. The corresponding spectrum can be obtained after the Fourier transform.

[0129]

[0130] Where f is the frequency of the speech signal, j represents a complex number, and N2 represents the length of each frame of the speech signal.

[0131] The short-time energy of each frame of the speech signal can then be calculated using the following formula:

[0132]

[0133] in, N represents the short-time energy of each frame, and N1 is the number of frequency bands of the speech signal.

[0134] Computer equipment can acquire the time-domain waveform corresponding to each frame of the speech signal, determine the number of times the time-domain waveform of a single frame of the speech signal crosses zero within that frame time, and the number of zero crossings is the short-time zero-crossing rate of that single frame of the speech signal. Following the same processing method, the short-time zero-crossing rate corresponding to each frame of the speech signal can be obtained.

[0135] In one embodiment, a computer device can determine the time-domain waveform of a single frame of audio signal in the time domain and determine the number of times the time-domain waveform crosses the horizontal axis. The number of times it crosses the horizontal axis is the number of zero crossings, and the corresponding short-time zero crossing rate can be obtained.

[0136] In one embodiment, after obtaining the short-time energy and short-time zero-crossing rate of each frame of speech signal, the short-time energy and short-time zero-crossing rate can be combined as an effective means of VAD, and the evaluation rules are as follows:

[0137] Silent frames have lower short-time energy and short-time zero-crossing rate;

[0138] The transition from silent to non-silent mode is accompanied by a short-term surge in energy.

[0139] Background noise has relatively small short-term energy fluctuations;

[0140] The transition from silent to non-silent mode is accompanied by a short-term surge in the zero-crossing rate.

[0141] Based on the various evaluation rules, silence signal frames can be effectively filtered out from each frame of speech signal to obtain valid signal frames.

[0142] In this embodiment, window functions generally have low-pass characteristics. Windowing reduces spectral leakage of the speech signal, allowing frequency components that are previously invisible due to leaked energy to be clearly displayed. The windowed speech signal undergoes time-frequency conversion to obtain its frequency distribution curve, i.e., the spectrum, thus reflecting the changes in the speech signal over time. Based on the spectrum corresponding to each frame of the speech signal, the short-time energy of each frame is accurately determined, and based on the time-domain waveform corresponding to each frame, the short-time zero-crossing rate of the corresponding speech signal is accurately determined. This allows the short-time energy and short-time zero-crossing rate to be used as conditions for removing silence segments from the original speech, enabling accurate extraction of effective speech. Because silence segments are removed from the test speech, the features extracted from the test speech more significantly reflect the speech characteristics, helping to improve the accuracy of subsequent objective evaluation of speech quality.

[0143] In one embodiment, the acoustic features are convolved to obtain depth features at different time points, including:

[0144] The acoustic features are segmented into frames to obtain feature frames at different times; convolution and normalization are then performed on the feature frames at different times to obtain depth features at different times.

[0145] Specifically, the computer device performs frame segmentation on the acoustic features to obtain feature frames at different times. The computer device then performs convolution processing on the feature frames at different times and normalizes the features obtained from the convolution processing to obtain depth features at different times.

[0146] In one embodiment, the computer device performs multiple convolutional processes on feature frames at different times, and performs normalization processing after each convolutional process, that is, normalizes the feature frames after convolution. After the normalization process is completed, the normalized feature frames are subjected to the next convolutional and normalization process until the feature frames at different times after the last normalization process are obtained.

[0147] In one embodiment, convolution and normalization are performed on feature frames at different times to obtain depth features at different times, including: performing convolution on feature frames at different times and normalizing the convolutionally processed feature frames to obtain depth features at different times.

[0148] In one embodiment, the convolutional feature frames are normalized to obtain depth features at different time points, including:

[0149] The mean of the feature frames at different time points after convolution is determined, and the variance of the feature frames at different time points is determined based on the mean and the feature frames at different time points after convolution. The feature frames at different time points are normalized according to the mean and variance to obtain the depth features at different time points.

[0150] In one embodiment, a computer device inputs feature frames from different time points into a convolutional neural network (CNN), and performs convolution processing through multiple convolutional layers to obtain depth features at different time points. The structure of the convolutional neural network is as follows: Figure 4 As shown, it includes convolutional layers, pooling layers, and fully connected layers, specifically six convolutional layers. Each convolutional layer is followed by a corresponding non-linear activation function ReLU and a normalization function BatchNorm. The specific parameters of each convolutional layer, pooling layer, and fully connected layer, such as input size, kernel size, stride, number of channels, and output feature size, are shown below. Figure 4 As shown.

[0151] The normalization function is as follows:

[0152]

[0153] Among them, BN γ,βFor BatchNorm normalization function, x i The input features are m, and the mini-batch size is m. as well as x i The mean and variance of γ and β. γ and β are BN. γ,β The learnable parameters of the function represent the scaling and translation parameters, respectively. (BN) i For BN γ,β The depth features obtained after function computation. BN γ,β The function shifts the input distribution from the saturated region to the unsaturated region, reducing gradient vanishing, improving training speed, and significantly accelerating the convergence process. Simultaneously, a max-pooling layer is added after the convolutional layer, reducing feature dimensionality while preserving convolutional features, thereby reducing network computation. The output of the last convolutional layer is then flattened, followed by a fully connected layer to further compress high-dimensional features. The convolutional neural network in this embodiment is a deep separable network, which can further compress the model size, greatly reducing the number of network parameters and computational cost, avoiding the problem of excessive computation in traditional convolutional neural networks.

[0154] In this embodiment, the acoustic features are segmented into frames to obtain feature frames at different times. Convolution processing is then performed on the feature frames at different times to further extract high-dimensional, deep-level key features. Normalization processing can reduce the feature dimension while retaining the key features extracted by convolution, thereby reducing the computational load of the network.

[0155] In one embodiment, such as Figure 5 As shown, quality assessment is performed based on deep features at different times to obtain the quality assessment results corresponding to the original speech, including:

[0156] Step S502: Based on the input features at the current time and the hidden layer features output at the previous time, determine the output features at the current time and determine the update parameters of the feature object at the current time; the input features at the current time are the depth features at the corresponding time, and the feature object represents the path of feature transmission.

[0157] Among them, the preceding time refers to the time point before the current time. The preceding time can be the N times before the current time, such as the previous time, the two times before, etc., but is not limited to this.

[0158] Specifically, the computer device inputs deep features at different times into a Long Short-Term Memory (LSTM) network, enabling the LSTM network to perform quality assessment based on these features and obtain the corresponding quality assessment results. Feature objects represent the paths used for feature transfer within the LSTM network, including the features that need to be transferred to the next processing node.

[0159] Understandably, deep features from different time steps are used as input features at different time steps. If the first time step has no preceding time steps, then there are no hidden features output from preceding time steps. The Long Short-Term Memory (LSTM) network processes the input features of the first time step to determine the output features, and the update parameters of the feature objects at the first time step are also determined. The feature objects at the first time step include the output features of that first time step.

[0160] Starting from the second time step, the Long Short-Term Memory network uses the deep features corresponding to the current time step as the input features for the current time step, and calculates the output features for the current time step based on the input features for the current time step and the hidden layer features of the previous time step.

[0161] Furthermore, the Long Short-Term Memory network determines the update parameters of the feature object at the current time step based on the input features at the current time step and the hidden layer features at the previous time step.

[0162] Step S504: Determine the feature object at the current time based on the feature object of the previous time step, the output feature of the current time step, and the update parameters.

[0163] Specifically, the Long Short-Term Memory network obtains the feature objects corresponding to previous time steps, and determines the feature objects at the current time step based on the feature objects at previous time steps, the output features at the current time step, and the update parameters of the feature objects at the current time step.

[0164] For example, a Long Short-Term Memory (LSTM) network obtains the feature objects corresponding to the previous time step, and determines the feature objects at the current time step based on the feature objects at the previous time step, the output features at the current time step, and the update parameters of the feature objects at the current time step.

[0165] Step S506: Based on the hidden layer features output at the previous time step, the output features at the current time step, and the feature objects at the current time step, determine the hidden layer features output at the current time step.

[0166] Specifically, the Long Short-Term Memory network obtains the hidden features of the output of the previous time step, and determines the hidden features of the output of the current time step based on the hidden features of the output of the previous time step, the output features of the current time step, and the feature objects of the current time step.

[0167] Step S508: Determine whether the current time is the last time. If not, proceed to step S510. If yes, proceed to step S512.

[0168] Step S510: Take the input features of the next time step as the input features of the current time step, and return the steps of determining the output features of the current time step based on the input features of the current time step and the hidden features output by the previous time step, and continue to execute until the hidden features output by the last time step are obtained.

[0169] Specifically, it is determined whether the current time is the last time step. If so, step S512 is executed; otherwise, after obtaining the hidden layer features output at the current time step, the depth features of the next time step are used as the input features of that next time step. The input features of the next time step are used as the input features of the current time step, and the steps for determining the output features of the current time step based on the input features of the current time step and the hidden layer features output at the previous time step are returned and executed again, thereby obtaining the new hidden layer features output at the current time step, and processing is performed for the next time step, until the hidden layer features output at the last time step are obtained.

[0170] Step S512: Based on the hidden layer features output at the last moment, determine the quality assessment result corresponding to the original speech.

[0171] Specifically, the computer device uses a Long Short-Term Memory (LSTM) network to determine the quality assessment result corresponding to the original speech based on the hidden layer features output at the last time step. Further, a fully connected processing is performed on the hidden layer features output at the last time step to obtain the quality assessment result corresponding to the original speech.

[0172] In this embodiment, deeper feature extraction and fusion are performed on the deep features at different time points. This ensures that the hidden features obtained at each processing time point incorporate the hidden features from previous time points, allowing the hidden features obtained in later processing to acquire more useful information. Consequently, the hidden features output at the final time point integrate all the key information from the deep features at different time points. The quality of the original speech can be accurately evaluated using the hidden features output at the final time point. Furthermore, the speech signal framing strategy and feature framing strategy reduce the real-time computation load, enabling more efficient online deployment.

[0173] In one embodiment, such as Figure 6 The diagram shows the structure of a Long Short-Term Memory (LSTM) network in one embodiment. The internal state of an LSTM network can exhibit dynamic temporal behavior, utilizing internal memory to process variable-length input sequences and effectively addressing long-term dependencies. An LSTM network includes three gates: a forget gate, an input gate, and an output gate, as well as cells and hidden layers. A cell is like a conveyor belt, i.e., the path for feature transmission, containing the features currently being transmitted. The forget gate determines the information to be discarded from the cell, i.e., which currently transmitted features to discard. The input gate determines the new features to be added to the cell, comprising two parts: a sigmoid function determines which values ​​will be updated, and a tanh function determines a new candidate value vector. Cell updates are achieved by multiplying the features contained in the previous cell with the output features of the forget gate to discard unwanted information, and adding this to the output features of the input gate to add useful new information. The output gate determines the features to be output based on the cell. Where f... tLet i be the output feature of the forget gate at time t. t Let o be the output feature of the input gate at time t. t Let C be the output characteristic of the output gate at time t. t h represents the features contained in a cell element at time t. t x represents the hidden layer features output by the hidden layer at time t. t The input features at the current time, C represents the update parameters of a cell at time t. t-1 h represents the cell element at time (t-1). t-1 This represents the hidden layer features from the previous time step. σ is the activation function used to enhance the nonlinear learning capability of the LSTM network; in this embodiment, Sigmoid is used as the activation function.

[0174] In one embodiment, such as Figure 7 As shown, this method is applied to a target speech evaluation model; the target speech evaluation model includes a forget gate, an input gate, and an output gate, and the feature object is a cell; based on the input features at the current time and the hidden layer features output from the previous time step, the output features at the current time are determined, and the update parameters of the feature object at the current time are determined, including steps S702 to S706:

[0175] Step S702: Determine the output features of the forget gate at the current time by using the input features at the current time and the hidden layer features output at the previous time step.

[0176] Specifically, the speech evaluation method is applied to a target speech evaluation model, which includes a forget gate, an input gate, an output gate, and cells, where cells include the features of the current transmission.

[0177] The computer device inputs the raw speech into the target speech evaluation model. After speech activity detection, time-frequency conversion processing, feature extraction, and convolution processing performed by the target speech evaluation model, deep features at different time points are obtained. The deep features at different time points are used as the input features at the corresponding time points. The forget gate of the target speech evaluation model determines the output features of the forget gate at the current time point based on the input features at the current time point and the hidden layer features output from the previous time point.

[0178] In one embodiment, the forget gate is used to determine the feature to be discarded at the current moment, and the output feature of the forget gate at the current moment is the feature to be discarded.

[0179] It is understandable that if there is no preceding time step in the first time step, then there are no hidden features output by the preceding time step in the first time step, and therefore the forget gate has no features to discard in the first time step.

[0180] Starting from the second time step, the forget gate calculates the features to be discarded at the current time step, i.e., the output features, based on the input features at the current time step and the hidden features at the previous time step.

[0181] like Figure 8a The diagram shows the structure of the forget gate in one embodiment, i.e. Figure 8a The structure shown by the thick black line. f t Let C be the output feature of the forget gate at the current time step. t-1 Which features are used to calculate C? t f t It is a vector, where each element is in the range [0, 1]. The forget gate's f... t It can be obtained from the following formula:

[0182] f t =σ(W f ·[h t-1 ,x t ]+b f )

[0183] Among them, h t-1 W represents the hidden layer features at time t-1. f b f ... t As can be seen from the calculation formula, the output characteristics of the forget gate are not only related to the input x at the current time. t It is related to the hidden layer characteristics h of the hidden layer at the previous time step. t-1 related.

[0184] Step S704: Determine the output features of the input gate at the current time based on the input features at the current time and the hidden layer features output at the previous time.

[0185] Step S706: Based on the input features at the current time and the hidden layer features output from the previous time step, determine the update parameters corresponding to the cell element at the current time step through the input gate.

[0186] Specifically, the deep features at different times are used as the input features at the corresponding times. The input gate of the target speech evaluation model determines the output features of the input gate at the current time based on the input features at the current time and the hidden layer features output at the previous time.

[0187] Furthermore, the input gate determines the update parameters of the cell element at the current time step based on the input features at the current time step and the hidden layer features output at the previous time step.

[0188] It is understandable that the processes of determining the output features of the input gate at the current time and determining the update parameters of the cell element at the current time can be executed concurrently to improve processing efficiency, or they can be executed sequentially.

[0189] It is understandable that the processing of the forget gate and the processing of the input gate can be executed concurrently or sequentially, without much restriction here.

[0190] In one embodiment, if the input gate is used to determine the feature to be added at the current moment, then the output feature of the input gate at the current moment is the feature to be added.

[0191] like Figure 8b The diagram shows the structure of the input gate in one embodiment, namely... Figure 8b The structure is shown by the bold black line. The target speech evaluation model determines which new useful features to input through input gates and cell updates. The specific mathematical expression is shown below:

[0192]

[0193] in, i represents the update parameters of a cell at time t. t W represents the output characteristic of the input gate at time t. i b i These are the learnable parameters of the input gate, W. C b C These are the learnable parameters of the cell's update state. From the above equation, we can see that the cell's update parameters... From input features x t and hidden layer features h t-1 It is obtained through a neural network layer. The output feature i of the input gate... t Output feature f of the forget gate t It is also a vector within the interval [0, 1], given by x. t and h t-1 Calculated via the Sigmoid activation function, used to control Which features are used to update C? t .

[0194] Based on the feature objects of the previous time step, the output features of the current time step, and the update parameters, the feature objects of the current time step are determined, including:

[0195] Step S708: Based on the output features of the forget gate at the current time, the output features of the input gate at the current time, the update parameters, and the feature objects of the previous time steps, determine the feature objects at the current time.

[0196] Specifically, the target speech evaluation model can obtain the feature objects of the previous time step, update the feature objects based on the output features of the forget gate at the current time step, the output features of the input gate at the current time step, the update parameters, and the feature objects of the previous time step step, and obtain the feature objects of the current time step.

[0197] like Figure 8c The diagram shows the structure of cell renewal in one embodiment, namely... Figure 8c The structure is shown by the bold black line. The output characteristic i of the input gate at time t is utilized. t The output characteristic f of the forget gate at time t t The previous moment, cell element C t-1 and the cell update parameters at the current moment. The new cell element is determined by the following expression:

[0198]

[0199] Among them, C t Let be the cell element state at time t, i.e., the cell element at time t.

[0200] Based on the hidden features output from previous time steps, the output features from the current time step, and the feature objects from the current time step, the hidden features output from the current time step are determined, including:

[0201] Step S710: Determine the output features of the output gate at the current time based on the input features at the current time and the hidden layer features of the output at the previous time.

[0202] Step S712: Based on the output features of the output gate at the current time and the cell element at the current time, determine the hidden layer features of the output at the current time.

[0203] Specifically, the output gate of the target speech evaluation model determines its output features at the current time step based on the input features at the current time step and the hidden layer features of the output at the previous time step. Furthermore, the hidden layer features of the output at the current time step are determined by the output gate based on its output features at the current time step and the cell elements at the current time step.

[0204] like Figure 8d The diagram shows the structure of the output gate in one embodiment. The output gate is used to calculate the hidden layer feature h output by the hidden layer at time t. t Its expression is as follows:

[0205]

[0206] Among them, o t W represents the output characteristics of the output gate at time t. o b o σ represents the learnable parameters of the output gate, and σ is the activation function.

[0207] In this embodiment, the cell is equivalent to a conveyor belt, i.e., the path for feature transmission, enabling features to be transferred from the current node to the next node for processing. The forget gate, based on the input features at the current moment and the hidden features output from the previous moment, determines irrelevant or unimportant features with little information to be discarded from the transmitted features, reducing computational load. The input gate, based on the input features at the current moment and the hidden features output from the previous moment, determines new features to be added to the transmitted features, increasing new useful information. Furthermore, the output gate also needs to determine the update parameters of the feature transmission path for updating. By multiplying the features contained in the previous cell with the output features of the forget gate to discard unnecessary information, and adding this to the output features of the input gate to add useful new information, the cell is effectively updated. By using the output gate to determine the hidden features output at the current moment based on the cell and the output features of the output gate at the current moment, the deep features from each moment can be further extracted and fused, accurately assessing the quality of the original speech. This embodiment uses a target speech evaluation model to evaluate speech quality, which is an objective evaluation method that is highly correlated with human subjective perception. It is also simpler, more convenient, and the evaluation results are more robust.

[0208] In one embodiment, the voice evaluation method is applied to a video conference; the method further includes:

[0209] Speech activity detection is performed on the original speech to obtain the corresponding speech to be tested, including: speech activity detection is performed on video conferences to obtain the corresponding real-time speech;

[0210] The test speech is subjected to time-frequency conversion processing, and the speech signal obtained by the time-frequency conversion processing is subjected to feature extraction to obtain the acoustic features corresponding to the test speech. This includes: performing time-frequency conversion processing on real-time speech, and extracting features from the speech signal obtained by the time-frequency conversion processing to obtain the real-time acoustic features corresponding to the real-time speech.

[0211] The acoustic features are convolved to obtain depth features at different times, including: convolving real-time acoustic features to obtain target depth features at different times.

[0212] Quality assessment is performed based on depth features at different times to obtain the quality assessment results corresponding to the original speech, including: quality assessment is performed based on target depth features at different times to obtain the quality assessment score of real-time speech in video conferencing.

[0213] The method also includes:

[0214] If the real-time voice quality assessment score does not meet the preset quality conditions, an alert will be triggered.

[0215] Specifically, this speech evaluation method can be applied to video conferencing. It involves real-time acquisition of speech generated during the video conference, followed by speech activity detection to eliminate silence and obtain the corresponding real-time speech. Time-frequency conversion is then performed on the real-time speech to obtain the corresponding spectrogram, and feature extraction is performed on the spectrogram to obtain the real-time acoustic features corresponding to the real-time speech. The real-time acoustic features are then segmented into frames to obtain real-time feature frames at different times. Convolution processing is then performed on these real-time feature frames at different times to obtain target depth features at different times. Further feature extraction and feature fusion processing based on the target depth features at different times are then performed, followed by fully connected processing to obtain the quality evaluation score of the real-time speech in the video conference.

[0216] It is understood that the process of evaluating the quality of real-time audio in this video conference is the same as the process of evaluating the quality of raw audio in the above embodiments. For specific processing, please refer to the various embodiments of raw audio described above, which will not be repeated here.

[0217] If the real-time voice quality assessment score does not meet the preset quality conditions, an alert is triggered to promptly remind the user. Furthermore, if the real-time voice quality assessment score is less than or equal to the assessment threshold, an alert is triggered again.

[0218] In this embodiment, the voice evaluation method can also be applied to video conferencing. It can monitor voice quality in real time in the audio and video transmission link, further locate the events that cause low-quality voice, and remind the user, thereby effectively improving the quality of real-time voice calls.

[0219] In one embodiment, the voice evaluation method can also be applied to audio conferencing and any scenario with real-time voice, such as audio calls, video calls, live video streaming, and other real-time audio communication scenarios. It can monitor the voice quality of calls in real time, ensure the voice quality of calls, and improve the user experience.

[0220] like Figure 9 The diagram shown illustrates the framework of a speech evaluation method in one embodiment. The speech evaluation method in this embodiment takes the input speech signal to be tested (i.e., the original speech signal) as input, performs Voice Activity Detection (VAD) to obtain valid speech segments from the speech signal, and then uses Short Time Fourier Transform (STFT) and Mel Transform to obtain the Mel spectrum features of the valid speech segments. Subsequently, a designed Convolutional Neural Network (CNN) is used to extract high-dimensional features from the speech signal to be tested, and finally, a Long Short-Term Memory (LSTM) network is used to fuse the high-dimensional features from different time periods to obtain the quality score of the speech signal to be tested.

[0221] In this embodiment, the target speech evaluation model employs a network topology combining CNN and LSTM, with the feedforward network using a two-layer multilayer perceptron structure. It is understood that this application does not specifically limit the model type and topology of the deep neural network; it can be replaced with various other effective novel model structures, such as attention models, or other network structures, such as time-delay networks, gated convolutional neural networks, etc. The neural network and feedforward network topologies illustrated above can be extended or simplified according to the limitations of model memory usage and the requirements for detection accuracy in actual applications.

[0222] In one embodiment, the method is applied to a target speech evaluation model, the training steps of which include:

[0223] A first speech sample and a second speech sample different from the first speech sample are obtained. The first and second speech samples are then subjected to quality assessment processing using a speech evaluation model to be trained, resulting in a first prediction result and a second prediction result. The quality differences between the first and second prediction results and their respective corresponding evaluation labels are determined. Based on the first and second prediction results, the ranking loss between the first and second speech samples is determined. A target loss function is constructed based on the quality differences and ranking losses. The speech evaluation model to be trained is trained using the target loss function until a stopping condition is met, resulting in a trained target speech evaluation model. The target speech evaluation model is used to evaluate the quality of the speech to be tested.

[0224] The first prediction result is the quality assessment result corresponding to the first speech sample, and the second prediction result is the quality assessment result corresponding to the second speech sample.

[0225] Specifically, both the first and second speech samples are used to train the speech evaluation model, and the first speech sample differs from the second speech sample. Furthermore, both the first and second speech samples are speech samples with quality loss; the first speech sample differs from the second speech sample in that the degree of quality loss in the first speech sample is different from the degree of quality loss in the second speech sample.

[0226] In one embodiment, the first and second speech samples may have different levels of quality loss, but their speech content may be related or completely unrelated.

[0227] The computer device can input a first speech sample and a second speech sample into a speech evaluation model to be trained. The speech evaluation model to be trained performs quality evaluation processing on the first speech sample to obtain a first prediction result. The speech evaluation model to be trained then performs quality evaluation processing on the second speech sample to obtain a second prediction result.

[0228] The computer device can acquire evaluation labels corresponding to a first speech sample and evaluation labels corresponding to a second speech sample. The evaluation label corresponding to the first speech sample represents the true speech quality of the first speech sample, and the evaluation label corresponding to the second speech sample represents the true speech quality of the second speech sample. The computer device determines the quality difference between a first prediction result and its corresponding evaluation label for the first speech sample, and determines the quality difference between a second prediction result and its corresponding evaluation label for the second speech sample.

[0229] The computer device determines the ranking loss between the first speech sample and the second speech sample based on the first prediction result and the second prediction result.

[0230] In one embodiment, the computer device determines a predicted probability that the quality of the first speech sample is higher than the quality of the second speech sample based on a first prediction result and a second prediction result. Based on the predicted probability and the corresponding probability label, it determines the ranking loss between the first and second speech samples. The probability label indicates whether the quality of the first speech sample is higher or lower than the quality of the second speech sample, and the probability label can be represented as 0 or 1.

[0231] The computer device can construct a target loss function based on the quality difference between the first prediction result and its corresponding evaluation label, the quality difference between the second prediction result and its corresponding evaluation label, and the ranking loss between the first and second speech samples. Further, the computer device constructs the target loss function based on each quality difference, the ranking loss, and their respective weights. In one embodiment, the weights corresponding to each quality difference can be the same.

[0232] In one embodiment, the computer device sums the quality difference between the first prediction result and its corresponding evaluation label with the quality difference between the second prediction result and its corresponding evaluation label to obtain a sum of quality differences. A target loss function is then constructed based on the sum of quality differences and the ranking loss. Further, the computer device can construct the target loss function based on the sum of quality differences, the ranking loss, and their respective weights.

[0233] In this embodiment, the speech evaluation model to be trained performs quality evaluation on first and second speech samples with different levels of quality loss, respectively, to obtain corresponding first and second prediction results. Determining the quality difference between the first and second prediction results and their respective evaluation labels allows us to identify the gap between the model's predictions and the actual results. Based on the first and second prediction results, the ranking loss between the first and second speech samples is determined, which measures the relative quality distance between the speech sample pairs. Based on the quality differences and ranking losses, a target loss function is constructed, incorporating both ranking loss and quality differences. The speech evaluation model is repeatedly trained to adjust model parameters, thereby improving the accuracy of the model evaluation. When the stopping condition is met, the trained target speech evaluation model is obtained. This target speech evaluation model can accurately and quickly evaluate the quality of raw speech and can also evaluate real-time speech, improving the timeliness and efficiency of speech evaluation.

[0234] In one embodiment, such as Figure 10 As shown, a method for training a speech evaluation model is provided, which can be applied to a computer device (specifically, the computer device can be...). Figure 1 Taking a terminal or server as an example, the following steps are included:

[0235] Step S1002: Obtain a first speech sample and a second speech sample that is different from the first speech sample.

[0236] Specifically, both the first speech sample and the second speech sample are used to train the speech evaluation model, and the first speech sample is different from the second speech sample.

[0237] Furthermore, both the first and second speech samples are speech samples with quality loss. The first speech sample is different from the second speech sample in that the degree of quality loss of the first speech sample is different from the degree of quality loss of the second speech sample.

[0238] In one embodiment, the first and second speech samples may have different levels of quality loss, but their speech content may be related or completely unrelated.

[0239] In one embodiment, obtaining a first speech sample and a second speech sample different from the first speech sample includes: performing speech activity detection on a first training speech to obtain a corresponding first speech sample; performing speech activity detection on a second training speech to obtain a corresponding second speech sample; wherein the second training speech is different from the first training speech.

[0240] In one embodiment, performing speech activity detection on the first training speech to obtain the corresponding first speech sample includes: performing frame segmentation on the first training speech to obtain corresponding first speech signals for each frame; determining the short-time energy and short-time zero-crossing rate corresponding to each frame of the first speech signal; and filtering out valid signal frames from each frame of the first speech signal based on the short-time energy and short-time zero-crossing rate of each first speech signal to obtain the first speech sample.

[0241] Speech activity detection is performed on the second training speech to obtain the corresponding second speech samples, including: performing frame segmentation on the second training speech to obtain the corresponding second speech signals for each frame; determining the short-time energy and short-time zero-crossing rate corresponding to each frame of the second speech signal; and selecting valid signal frames from each frame of the second speech signal based on the short-time energy and short-time zero-crossing rate of each second speech signal to obtain the second speech samples.

[0242] In one embodiment, determining the short-time energy and short-time zero-crossing rate corresponding to each frame of the first speech signal includes: performing windowing processing on each frame of the first speech signal, and performing time-frequency conversion processing on the windowed speech signal to obtain a first spectrum corresponding to each frame of the first speech signal; determining the short-time energy of each frame of the first speech signal based on each first spectrum; and determining the short-time zero-crossing rate of each frame of the first speech signal according to the first time-domain waveform corresponding to each frame of the first speech signal.

[0243] Determining the short-time energy and short-time zero-crossing rate corresponding to each frame of the second speech signal includes: performing windowing processing on each frame of the second speech signal, and performing time-frequency conversion processing on the windowed speech signal to obtain the second spectrum corresponding to each frame of the second speech signal; determining the short-time energy of each frame of the second speech signal based on each second spectrum; and determining the short-time zero-crossing rate of each frame of the second speech signal according to the second time-domain waveform corresponding to each frame of the second speech signal.

[0244] It is understood that the detailed processing of performing speech activity detection on the first and second training speech to obtain the corresponding first and second speech samples is the same as the detailed processing of performing activity detection on the original speech to obtain the speech to be tested in the above embodiments, and will not be repeated here.

[0245] Step S1004: The first speech sample and the second speech sample are subjected to quality assessment processing by the speech evaluation model to be trained, respectively, to obtain the first prediction result and the second prediction result.

[0246] Specifically, the computer device can input a first speech sample and a second speech sample into a speech evaluation model to be trained. The speech evaluation model to be trained performs quality evaluation processing on the first speech sample to obtain a first prediction result. The speech evaluation model to be trained then performs quality evaluation processing on the second speech sample to obtain a second prediction result.

[0247] In one embodiment, the first speech sample and the second speech sample are subjected to quality assessment processing by a speech evaluation model to be trained, respectively, to obtain a first prediction result and a second prediction result, including:

[0248] The first speech sample is subjected to time-frequency conversion processing using the speech evaluation model to be trained, and the speech signal obtained by time-frequency conversion processing is subjected to feature extraction to obtain the corresponding first acoustic features; the second speech sample is subjected to time-frequency conversion processing, and the speech signal obtained by time-frequency conversion processing is subjected to feature extraction to obtain the corresponding second acoustic features; quality evaluation processing is performed based on the first acoustic features to obtain the first prediction result corresponding to the first speech sample; quality evaluation processing is performed based on the second acoustic features to obtain the second prediction result corresponding to the second speech sample.

[0249] In one embodiment, a quality assessment process based on a first acoustic feature is performed to obtain a first prediction result corresponding to a first speech sample, including: performing frame segmentation on the first acoustic feature to obtain each first feature frame corresponding to the first acoustic feature; performing normalization on each first feature frame to obtain the input feature at each time step corresponding to the first acoustic feature; and performing a quality assessment process based on the input feature at each time step corresponding to the first acoustic feature to obtain the first prediction result corresponding to the first speech sample.

[0250] The quality assessment process based on the second acoustic features yields the second prediction result corresponding to the second speech sample. This process includes: performing frame segmentation on the second acoustic features to obtain each second feature frame corresponding to the second acoustic features; performing normalization on each second feature frame to obtain the input features at each time step corresponding to the second acoustic features; and performing quality assessment on the input features at each time step corresponding to the second acoustic features to obtain the second prediction result corresponding to the second speech sample.

[0251] In one embodiment, normalization processing is performed on each first feature frame to obtain the input features corresponding to each time step of the first acoustic feature, including: normalizing each first feature frame based on the mean and variance corresponding to each first feature frame to obtain the input features corresponding to each time step of the first acoustic feature.

[0252] Normalization is performed on each second feature frame to obtain the input features corresponding to each time step of the second acoustic feature. This includes: normalizing each second feature frame based on the mean and variance of each second feature frame to obtain the input features corresponding to each time step of the second acoustic feature.

[0253] In one embodiment, quality assessment processing based on first acoustic features to obtain a first prediction result corresponding to a first speech sample includes: determining the output features at the current moment based on the input features at the current moment and the hidden features output at the previous moment, and determining the update parameters of the feature object at the current moment; the input features are obtained by temporal partitioning based on the first acoustic features, and the feature object represents the path of feature transmission; determining the feature object at the current moment based on the feature object at the previous moment, the output features at the current moment, and the update parameters; determining the hidden features output at the current moment based on the hidden features output at the previous moment, the output features at the current moment, and the feature object at the current moment; taking the input features of the next moment as the input features of the current moment, and returning the step of determining the output features at the current moment based on the input features of the current moment and the hidden features output at the previous moment, and continuing to execute until the hidden features output at the last moment are obtained; and determining the first prediction result corresponding to the first speech sample based on the hidden features output at the last moment.

[0254] In one embodiment, the speech evaluation model includes a forget gate, an input gate, and an output gate, with the feature object being a cell. Based on the input features at the current time step and the hidden layer features output from the previous time step, the output features at the current time step are determined, and the update parameters of the feature object at the current time step are determined, including:

[0255] The forget gate is used to determine the output features of the forget gate at the current time based on the input features at the current time and the hidden layer features output at the previous time. The input gate is used to determine the output features of the input gate at the current time based on the input features at the current time and the hidden layer features output at the previous time. The update parameters of the cell element at the current time are determined by using the input gate based on the input features at the current time and the hidden layer features output at the previous time.

[0256] Based on the feature objects of the previous time step, the output features of the current time step, and the update parameters, the feature objects of the current time step are determined, including: based on the output features of the forget gate at the current time step, the output features of the input gate at the current time step, the update parameters, and the feature objects of the previous time step, the feature objects of the current time step are determined.

[0257] Based on the hidden features output at the previous time step, the output features at the current time step, and the feature objects at the current time step, the hidden features output at the current time step are determined, including: determining the output features of the output gate at the current time step based on the input features at the current time step and the hidden features output at the previous time step; and determining the hidden features output at the current time step based on the output features of the output gate at the current time step and the cell elements at the current time step.

[0258] It is understood that the detailed processing procedure for performing quality assessment on the first speech sample and the second speech sample to obtain the first prediction result and the second prediction result is the same as the detailed processing procedure for performing quality assessment on the depth features of the original speech at different times to obtain the corresponding quality assessment result of the original speech in the above embodiments, and will not be repeated here.

[0259] Step S1006: Determine the quality differences between the first prediction result and the second prediction result and their respective corresponding evaluation labels.

[0260] Specifically, the computer device can obtain evaluation labels corresponding to the first speech sample and evaluation labels corresponding to the second speech sample. The evaluation label corresponding to the first speech sample is used to represent the actual speech quality of the first speech sample, and the evaluation label corresponding to the second speech sample is used to represent the actual speech quality of the second speech sample.

[0261] The computer device determines the quality difference between the first prediction result and the corresponding evaluation label for the first speech sample, and determines the quality difference between the second prediction result and the corresponding evaluation label for the second speech sample.

[0262] In one embodiment, the computer device uses a Least Square Error (LSE) loss function to determine the quality difference, as shown in the following expression:

[0263]

[0264] Where L2 represents the LSE loss function, f(x) i ) represents the prediction result of the speech evaluation model, which can be the MOS value, y i This represents the evaluation label, which corresponds to the actual MOS value. i Let n represent the number of speech samples.

[0265] Step S1008: Determine the ranking loss between the first speech sample and the second speech sample based on the first prediction result and the second prediction result.

[0266] Specifically, the computer device determines the ranking loss between the first speech sample and the second speech sample based on the first prediction result and the second prediction result.

[0267] In one embodiment, the computer device determines a predicted probability that the quality of the first speech sample is higher than the quality of the second speech sample based on a first prediction result and a second prediction result. Based on the predicted probability and the corresponding probability label, it determines the ranking loss between the first and second speech samples. The probability label indicates whether the quality of the first speech sample is higher or lower than the quality of the second speech sample, and the probability label can be represented as 0 or 1.

[0268] Step S1010: Construct the target loss function based on the quality differences and ranking losses.

[0269] Specifically, the computer device can construct a target loss function based on the quality difference between the first prediction result and its corresponding evaluation label, the quality difference between the second prediction result and its corresponding evaluation label, and the ranking loss between the first and second speech samples. Further, the computer device constructs the target loss function based on each quality difference, the ranking loss, and their respective weights. In one embodiment, the weights corresponding to each quality difference can be the same.

[0270] In one embodiment, the computer device sums the quality difference between the first prediction result and its corresponding evaluation label with the quality difference between the second prediction result and its corresponding evaluation label to obtain a sum of quality differences. A target loss function is then constructed based on the sum of quality differences and the ranking loss. Further, the computer device can construct the target loss function based on the sum of quality differences, the ranking loss, and their respective weights.

[0271] In one embodiment, the target loss function is as follows:

[0272] L totall =α×L2+(1-α)×L rank

[0273] Among them, L totall The objective loss function is derived from the L2 loss function and the ranking loss function L. rank Composition. The L2 loss function measures the quality gap between the predicted result and the true evaluation label of a speech sample, that is, the quality difference between the speech samples. rank The method measures whether the relative magnitude of the estimated speech sample pairs is correct, such as the first and second speech samples. α∈[0,1] is the weighting coefficient between the two, and in this embodiment, α=0.5. A larger α indicates a greater emphasis on the L2 loss function, and a larger (1-α) indicates a greater emphasis on the ranking loss function. rank .

[0274] Step S1012: Train the speech evaluation model to be trained using the target loss function until the stopping condition is met, and obtain the trained target speech evaluation model; this target speech evaluation model is used to evaluate the quality of the speech to be tested.

[0275] Specifically, the computer equipment updates and trains the speech evaluation model to be trained using the target loss function. During training, the parameters are adjusted based on the loss value generated by the speech evaluation model, and training continues until the stopping condition is met, thus obtaining the trained target speech evaluation model.

[0276] The stopping condition can be reaching the required number of iterations, or the target loss value being less than or equal to the loss threshold.

[0277] For example, during training, the target loss value of the speech evaluation model is determined based on the target loss function. When the target loss value is greater than the loss threshold, the parameters of the speech evaluation model are adjusted and training continues until the target loss value of the speech evaluation model is less than or equal to the loss threshold, at which point the training stops and the target speech evaluation model is obtained.

[0278] This target speech evaluation model is used to evaluate the quality of the original speech and obtain the corresponding quality evaluation results.

[0279] In this embodiment, the speech evaluation model to be trained performs quality evaluation on first and second speech samples with different levels of quality loss, respectively, to obtain corresponding first and second prediction results. Determining the quality difference between the first and second prediction results and their respective evaluation labels allows us to identify the gap between the model's predictions and the actual results. Based on the first and second prediction results, the ranking loss between the first and second speech samples is determined, which measures the relative quality distance between the speech sample pairs. Based on the quality differences and ranking losses, a target loss function is constructed, incorporating both ranking loss and quality differences. The speech evaluation model is repeatedly trained to adjust model parameters, thereby improving the accuracy of the model evaluation. When the stopping condition is met, the trained target speech evaluation model is obtained. This target speech evaluation model can accurately and quickly evaluate the quality of raw speech and can also evaluate real-time speech, improving the timeliness and efficiency of speech evaluation.

[0280] Traditional speech evaluation models require both the signal of the speech to be evaluated and a reference signal to perform quality assessment in order to predict the speech quality. However, the reference signal is unavailable in real-time audio and video scenarios. The speech evaluation model in this embodiment only requires the input of the speech to be evaluated to effectively perform real-time speech quality assessment and improve the efficiency of speech evaluation.

[0281] In one embodiment, although the L2 loss function can statistically measure the difference between the predicted and actual MOS scores of a single speech segment, it cannot accurately measure the difference for speech pairs with significant discriminative power. For example, consider a speech pair A and B. Where A... p A g B p B g Table 1 shows the predicted MOS values ​​and the actual MOS values ​​for speech A and B, respectively. Traditional speech prediction models result in two scenarios, as illustrated in Table 1:

[0282] Table 1: Different scenarios for MOS estimation of speech pairs A and B

[0283] situation <![CDATA[A g (NOT)]]> <![CDATA[A p (NOT)]]> <![CDATA[B p (NOT)]]> <![CDATA[B p (NOT)]]> Scenario 1 3 2.2 2 2.5 Scenario 2 3 3.8 2 2.5

[0284] As shown in the table, the L2 loss function estimated by the model is the same in both cases, but the estimated A is different in case 1. p Less than B p However, in reality, the quality of speech A is significantly higher than that of speech B. Therefore, another loss function is needed to constrain the relative magnitude of the estimated speech pair qualities, namely the ranking loss function.

[0285] In one embodiment, determining the ranking loss between the first speech sample and the second speech sample based on the first prediction result and the second prediction result includes:

[0286] Based on the first prediction result and the second prediction result, determine the prediction probability that the quality of the first speech sample is higher than that of the second speech sample; based on the prediction probability and the corresponding probability label, determine the correctness of the relative relationship between the first speech sample and the second speech sample; based on the correctness of the relative relationship, determine the ranking loss between the first speech sample and the second speech sample.

[0287] Specifically, the computer device determines the difference between the first prediction result and the second prediction result, and based on the difference between the first prediction result and the second prediction result, determines the prediction probability that the quality of the first speech sample is higher than the quality of the second speech sample.

[0288] The computer device acquires probability labels, which indicate whether the quality of the first speech sample is higher or lower than that of the second speech sample. Based on the predicted probabilities, the corresponding probability labels, the first prediction result, and the second prediction result, the computer device determines the correctness of the relative relationship between the first and second speech samples.

[0289] The computer device determines the ranking loss between the first speech sample and the second speech sample based on the correctness of the relative relationship and the corresponding weight coefficients.

[0290] For example, using a pair of speech samples xi x j The speech sample x is predicted by the speech evaluation model. i x j The corresponding prediction results are used to measure x using the ranking loss function. i With x j The relative size relationship between them is shown in the following formula, which is used to calculate the speech sample x. i The quality is higher than that of the speech sample x j Predicted probability P ij (f):

[0291]

[0292] Where, f(x) i ) represents the speech sample x predicted by the speech evaluation model. i The prediction result, f(x) j The speech sample x predicted by the speech evaluation model j The prediction result can be specifically the MOS value.

[0293] Using the cross-entropy loss function Metrics the accuracy of estimating the relative quality of speech sample pairs:

[0294]

[0295] in, Representing speech sample x i The actual quality is higher than that of the speech sample x j The probability label of the true quality of the speech sample x can be represented as 0 or 1, indicating the true quality of the speech sample x. i The actual quality is lower or higher than the speech sample x j The true quality is represented by Pij, which is the corresponding predicted probability.

[0296] In this embodiment, based on the first and second prediction results, the predicted probability that the quality of the first speech sample is higher than that of the second speech sample is determined, thereby determining the relative relationship between the first and second speech samples in terms of speech quality. Based on the predicted probability and the corresponding probability label, the correctness of the relative relationship between the first and second speech samples is determined, thus effectively measuring the accuracy of the predicted probability that the quality of the first speech sample is higher than that of the second speech sample. Determining the ranking loss between the first and second speech samples based on the correctness of the relative relationship effectively constrains the relative relationship between the speech sample pairs in terms of speech quality.

[0297] In one embodiment, determining the ranking loss between the first and second speech samples based on the correctness of the relative relationship includes:

[0298] Based on the evaluation labels of the first prediction result and the second prediction result, the uncertainty coefficient of the relative relationship is determined; based on the correctness of the relative relationship and the uncertainty coefficient of the relative relationship, the ranking loss between the first speech sample and the second speech sample is determined.

[0299] Specifically, the relative relationship between the first speech sample and the second speech sample determined by the first prediction result and the second prediction result is uncertain. For example, it is not possible for different users to completely determine that the quality of the first speech sample is higher than that of the second speech sample. Therefore, it is necessary to set an uncertainty coefficient for the relative relationship to measure the importance of the relative relationship.

[0300] The computer device can calculate the label difference between the evaluation labels of the first prediction result and the evaluation labels of the second prediction result, and obtain a label difference threshold. Based on the label difference and the label difference threshold, an uncertainty coefficient for the relative relationship is determined.

[0301] In one embodiment, when the label difference is less than or equal to the label difference threshold, the computer device determines the uncertainty coefficient of the relative relationship based on the ratio of the label difference to the label difference threshold. When the label difference is greater than the label difference threshold, the uncertainty coefficient for determining the relative relationship is zero.

[0302] For example, the uncertainty coefficient U(T) of the relative relationship can be determined by the following expression:

[0303]

[0304] Where U(T) is the uncertainty coefficient, T = abs(y) i -y j ), T c =0.5 sets the label difference threshold. T≤T c This indicates that the speech sample x i Is the actual quality greater than the speech sample x? j In terms of actual quality, it can be distinguished by most people. i For speech sample x i The corresponding evaluation label, i.e., speech sample x i The true quality, y j For speech sample x j The corresponding evaluation label, i.e., speech sample x j The true quality. Then the uncertainty coefficient is multiplied by the cross-entropy loss function. The sorting loss function L can be obtained. b (f):

[0305]

[0306] in, This represents the batch size for training. From the above formula, it can be seen that speech pairs with lower uncertainty (i.e., smaller uncertainty coefficients) contribute more to the ranking loss, while speech pairs with higher uncertainty (i.e., larger uncertainty coefficients) contribute less. Finally, the derivative of the learnable parameter w of the speech evaluation model can be obtained from the following formula:

[0307]

[0308] Where w represents all learnable parameters of the speech evaluation model.

[0309] In this embodiment, the uncertainty coefficient of the relative relationship is determined based on the evaluation labels of the first and second prediction results. This uncertainty coefficient allows for an accurate determination of the contribution of the relative relationship between the first and second speech samples to the quality assessment, thereby accurately determining the ranking loss between the first and second speech samples. Training the speech assessment model by combining the ranking loss and the quality difference can improve the accuracy of the model's evaluation.

[0310] like Figure 11 The diagram illustrates the process of determining the ranking loss of speech sample pairs in one embodiment. The computer device acquires speech sample pairs with different degrees of speech quality loss, i.e., speech sample x. i and speech sample x j For speech sample x i and speech sample x j Mel spectrum features were collected separately, and feature extraction was performed using a convolutional neural network (CNN) to obtain speech samples x. i The corresponding depth features at each time point, and the speech sample x j The corresponding depth features at each time step. The speech sample x... i The deep features at each time step are input into the Long Short-Term Memory (LSTM) network to obtain the speech sample x. i The MOS value of the speech sample x. j The deep features at each time step are input into the Long Short-Term Memory (LSTM) network to obtain the speech sample x. j MOS value. Based on speech sample x i MOS value and speech sample x j The MOS value is used to construct the ranking loss function to calculate the speech sample x. i and speech sample x j The sorting loss.

[0311] In one embodiment, a speech evaluation method is provided, executed through a target speech evaluation model applied to a computer device, comprising:

[0312] The target speech evaluation model, upon obtaining the original speech input, will undergo the following processing:

[0313] The original speech is segmented into frames to obtain the corresponding speech signals for each frame; each frame of speech signal is then windowed, and the windowed speech signals are then time-frequency converted to obtain the corresponding spectrum.

[0314] Based on the spectrum corresponding to each frame of the speech signal, the short-time energy of each frame of the speech signal is determined.

[0315] Obtain the time-domain waveform corresponding to each frame of the speech signal, and determine the short-time zero-crossing rate of the corresponding speech signal based on the time-domain waveform.

[0316] Based on short-time energy and short-time zero-crossing rate, valid signal frames are selected from each frame of speech signal to obtain the speech to be tested.

[0317] The speech to be tested is subjected to time-frequency conversion processing, and the speech signal obtained by time-frequency conversion processing is used to extract features to obtain the acoustic features corresponding to the speech to be tested.

[0318] The acoustic features are segmented into frames to obtain feature frames at different times; convolution and normalization are then performed on the feature frames at different times to obtain depth features at different times.

[0319] The output features of the forget gate at the current time are determined by using the input features at the current time and the hidden features of the output at the previous time step.

[0320] The output features of the input gate at the current time are determined by using the input features at the current time step and the hidden layer features of the output at the previous time step.

[0321] The update parameters of the cell element at the current time are determined by using the input gate based on the input features at the current time step and the hidden layer features output from the previous time step.

[0322] Based on the output characteristics of the forget gate at the current time, the output characteristics of the input gate at the current time, the update parameters, and the feature objects of the previous time step, the feature objects at the current time step are determined.

[0323] The output features of the output gate at the current time are determined by using the input features at the current time step and the hidden layer features of the output at the previous time step.

[0324] Based on the output characteristics of the output gate at the current time and the cell element at the current time, the hidden layer characteristics of the output at the current time are determined.

[0325] When the current time is not the last time, the input features of the next time are used as the input features of the current time, and the hidden features output by the previous time based on the input features of the current time and the hidden features of the previous time are returned. The steps to determine the output features of the current time are continued until the hidden features output by the last time are obtained.

[0326] When the current time is the last time step, the quality assessment result corresponding to the original speech is determined based on the hidden layer features output at the last time step.

[0327] In this embodiment, window functions generally have low-pass characteristics. Windowing reduces spectral leakage of the speech signal, allowing frequency components that are previously invisible due to leaked energy to be clearly displayed. The windowed speech signal undergoes time-frequency conversion to obtain its frequency distribution curve, i.e., the spectrum, thus reflecting the changes in the speech signal over time. Based on the spectrum corresponding to each frame of the speech signal, the short-time energy of each frame is accurately determined, and based on the time-domain waveform corresponding to each frame, the short-time zero-crossing rate of the corresponding speech signal is accurately determined. This allows the short-time energy and short-time zero-crossing rate to be used as conditions for removing silence segments from the original speech, enabling accurate extraction of effective speech. Because silence segments are removed from the test speech, the features extracted from the test speech more significantly reflect the speech characteristics, helping to improve the accuracy of subsequent objective evaluation of speech quality.

[0328] The acoustic features are segmented into frames to obtain feature frames at different times. Convolution is then performed on these feature frames to further extract high-dimensional, deep-level key features. Normalization is then applied to reduce the feature dimension while preserving the key features extracted by convolution, thereby reducing the computational cost of the model.

[0329] Cells act as conveyor belts, representing the paths for feature transmission from the current node to the next for processing. The forget gate, based on the current input features and the hidden features from previous time steps, determines irrelevant or unimportant features to discard from the transmitted features, reducing computational load. The input gate, based on the current input features and the hidden features from previous time steps, determines new features to add to the transmitted features, increasing useful information. Furthermore, the output gate determines the update parameters for the feature transmission path. By multiplying the features from the previous cell with the output features of the forget gate to discard unnecessary information and adding them to the output features of the input gate to add useful new information, cell updates are effectively achieved. The output gate, based on the current cell and output features, determines the hidden features at the current time step, enabling further feature extraction and fusion of deep features from different time steps, accurately assessing the quality of the original speech. This embodiment uses a target speech evaluation model to evaluate speech quality, which is an objective evaluation method that is highly correlated with human subjective perception. It is also simpler, more convenient, and the evaluation results are more robust.

[0330] This embodiment does not require a reference signal to participate in the speech quality assessment process. It only needs to use the original speech signal to effectively assess the speech quality, thereby enabling effective online real-time speech quality assessment with low computational load and improved efficiency of speech quality assessment.

[0331] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0332] Based on the same inventive concept, this application also provides a speech evaluation apparatus for implementing the speech evaluation method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more speech evaluation apparatus embodiments provided below can be found in the limitations of the speech evaluation method described above, and will not be repeated here.

[0333] In one embodiment, such as Figure 12 As shown, a speech evaluation device 1200 is provided. This device can be a software module, a hardware module, or a combination of both as part of a computer device. Specifically, the device includes: a detection module 1202, a conversion module 1204, a convolution module 1206, and a processing module 1208, wherein:

[0334] The detection module 1202 is used to perform speech activity detection on the original speech to obtain the corresponding speech to be tested.

[0335] The conversion module 1204 is used to perform time-frequency conversion processing on the speech to be tested, and to extract features from the speech signal obtained by the time-frequency conversion processing to obtain the acoustic features corresponding to the speech to be tested.

[0336] The convolution module 1206 is used to perform convolution processing on acoustic features to obtain depth features at different times.

[0337] The processing module 1208 is used to perform quality assessment processing based on the depth features at different times to obtain the quality assessment results corresponding to the original speech.

[0338] In this embodiment, speech activity detection is performed on the original speech to eliminate silences and obtain valid test speech. Time-frequency conversion is then applied to the test speech to reflect the frequency variation of the signal over time. Feature extraction is performed on the speech signal obtained from the time-frequency conversion to obtain acoustic features reflecting the time-frequency relationship. Convolution processing is applied to the acoustic features to obtain higher-dimensional, deeper-level depth features at different times. Quality assessment is then performed based on these higher-dimensional, deeper-level depth features at different times, accurately evaluating the quality of the original speech and obtaining the corresponding quality assessment result. Furthermore, this embodiment does not require a reference signal for speech quality assessment; only the original speech signal is needed to effectively evaluate speech quality, enabling real-time speech quality assessment.

[0339] Traditional speech evaluation devices require both the signal of the speech to be evaluated and a reference signal to determine the speech quality. However, the reference signal is unavailable in real-time audio and video scenarios. The speech evaluation device of this embodiment, on the other hand, only requires the signal of the speech to be evaluated to effectively perform real-time speech quality evaluation and improve the efficiency of speech evaluation.

[0340] In one embodiment, the detection module 1202 is further configured to perform frame-segmentation processing on the original speech to obtain corresponding speech signals for each frame; determine the short-time energy and short-time zero-crossing rate corresponding to each frame of speech signal; and based on the short-time energy and short-time zero-crossing rate, filter out valid signal frames from each frame of speech signal to obtain the speech to be tested.

[0341] In this embodiment, the original speech is processed into frames to obtain corresponding speech signals for each frame. The short-time energy and short-time zero-crossing rate corresponding to each frame of speech signal are determined. The short-time energy and short-time zero-crossing rate are used as conditions for filtering valid signal frames. This can effectively remove silent signals from each frame of speech signal and filter out valid signal frames, thereby accurately extracting the speech to be tested from the original speech.

[0342] In one embodiment, the detection module 1202 is further configured to perform windowing processing on each frame of speech signal, and perform time-frequency conversion processing on the windowed speech signal to obtain the corresponding spectrum; determine the short-time energy of each frame of speech signal based on the spectrum corresponding to each frame of speech signal; acquire the time-domain waveform corresponding to each frame of speech signal, and determine the short-time zero-crossing rate of the corresponding speech signal based on the time-domain waveform.

[0343] In this embodiment, window functions generally have low-pass characteristics. Windowing reduces spectral leakage of the speech signal, allowing frequency components that are previously invisible due to leaked energy to be clearly displayed. The windowed speech signal undergoes time-frequency conversion to obtain its frequency distribution curve, i.e., the spectrum, thus reflecting the changes in the speech signal over time. Based on the spectrum corresponding to each frame of the speech signal, the short-time energy of each frame is accurately determined, and based on the time-domain waveform corresponding to each frame, the short-time zero-crossing rate of the corresponding speech signal is accurately determined. This allows the short-time energy and short-time zero-crossing rate to be used as conditions for removing silence segments from the original speech, enabling accurate extraction of effective speech. Because silence segments are removed from the test speech, the features extracted from the test speech more significantly reflect the speech characteristics, helping to improve the accuracy of subsequent objective evaluation of speech quality.

[0344] In one embodiment, the convolution module 1206 is further configured to perform frame segmentation processing on the acoustic features to obtain feature frames at different times; and to perform convolution and normalization processing on the feature frames at different times to obtain depth features at different times.

[0345] In this embodiment, the acoustic features are segmented into frames to obtain feature frames at different times. Convolution processing is then performed on the feature frames at different times to further extract high-dimensional, deep-level key features. Normalization processing can reduce the feature dimension while retaining the key features extracted by convolution, thereby reducing the computational load of the network.

[0346] In one embodiment, the processing module 1208 is further configured to determine the output features of the current time based on the input features of the current time and the hidden layer features output by the previous time step, and to determine the update parameters of the feature object at the current time; the input features of the current time are the depth features of the corresponding time step, and the feature object represents the path of feature transmission; the feature object of the current time is determined based on the feature object of the previous time step, the output features of the current time step, and the update parameters; the hidden layer features output by the previous time step, the output features of the current time step, and the feature object of the current time step are determined based on the hidden layer features output by the previous time step, the output features of the current time step, and the feature object of the current time step; the input features of the next time step are used as the input features of the current time step, and the steps of determining the output features of the current time step based on the input features of the current time step and the hidden layer features output by the previous time step are returned and executed until the hidden layer features output by the last time step are obtained; the quality assessment result corresponding to the original speech is determined based on the hidden layer features output by the last time step.

[0347] In this embodiment, deeper feature extraction and fusion are performed on the deep features at different time points. This ensures that the hidden features obtained at each time point incorporate the hidden features from previous time points, allowing the hidden features obtained in later processing to acquire more useful information. Consequently, the hidden features output at the final time point incorporate all the key information from the deep features at different time points. The quality of the original speech can be accurately evaluated using the hidden features output at the final time point.

[0348] In one embodiment, the device is applied to a target speech evaluation model; the target speech evaluation model includes a forget gate, an input gate, and an output gate, and the feature object is a cell; the processing module 1208 is further configured to: determine the output feature of the forget gate at the current time based on the input feature at the current time and the hidden layer feature output at the previous time using the forget gate; determine the output feature of the input gate at the current time based on the input feature at the current time and the hidden layer feature output at the previous time using the input gate; determine the update parameter corresponding to the cell at the current time based on the input feature at the current time and the hidden layer feature output at the previous time using the input feature at the current time; determine the feature object at the current time based on the output feature of the forget gate at the current time, the output feature of the input gate at the current time, the update parameter, and the feature object at the previous time; determine the output feature of the output gate at the current time based on the input feature at the current time and the hidden layer feature output at the previous time using the output gate; and determine the hidden layer feature output at the current time based on the output feature of the output gate at the current time and the cell at the current time.

[0349] In this embodiment, the cell is equivalent to a conveyor belt, i.e., the path for feature transmission, enabling features to be transferred from the current node to the next node for processing. The forget gate, based on the input features at the current moment and the hidden features output from the previous moment, determines irrelevant or unimportant features with little information to be discarded from the transmitted features, reducing computational load. The input gate, based on the input features at the current moment and the hidden features output from the previous moment, determines new features to be added to the transmitted features, increasing new useful information. Furthermore, the output gate also needs to determine the update parameters of the feature transmission path for updating. By multiplying the features contained in the previous cell with the output features of the forget gate to discard unnecessary information, and adding this to the output features of the input gate to add useful new information, the cell is effectively updated. By using the output gate to determine the hidden features output at the current moment based on the cell and the output features of the output gate at the current moment, the deep features from each moment can be further extracted and fused, accurately assessing the quality of the original speech.

[0350] In one embodiment, the device is applied to a video conference; the detection module 1202 is also used to detect voice activity in the video conference and obtain the corresponding real-time voice.

[0351] The conversion module 1204 is also used to perform time-frequency conversion processing on real-time speech, and to extract features from the speech signal obtained by the time-frequency conversion processing to obtain the real-time acoustic features corresponding to the real-time speech.

[0352] The convolution module 1206 is also used to perform convolution processing on real-time acoustic features to obtain target depth features at different times.

[0353] The processing module 1208 is also used to perform quality assessment processing based on the target depth features at different times to obtain the quality assessment score of real-time speech in video conferencing.

[0354] The device also includes a trigger module, which is used to trigger a reminder operation if the real-time voice quality assessment score does not meet the preset quality conditions.

[0355] In this embodiment, the voice evaluation method can also be applied to video conferencing. It can monitor voice quality in real time in the audio and video transmission link, further locate the events that cause low-quality voice, and remind the user, thereby effectively improving the quality of real-time voice calls.

[0356] In one embodiment, the device is applied to a target speech evaluation model, the training steps of which include:

[0357] The acquisition module is used to acquire a first speech sample and a second speech sample that is different from the first speech sample;

[0358] The evaluation module is used to perform quality evaluation processing on the first speech sample and the second speech sample respectively through the speech evaluation model to be trained, so as to obtain the first prediction result and the second prediction result.

[0359] The quality difference determination module is used to determine the quality difference between the first prediction result and the second prediction result and their respective corresponding evaluation labels.

[0360] The ranking loss determination module is used to determine the ranking loss between the first speech sample and the second speech sample based on the first prediction result and the second prediction result.

[0361] The building block is used to construct the target loss function based on each quality difference and ranking loss;

[0362] The training module is used to train the speech evaluation model to be trained using the target loss function until the stopping condition is met, thus obtaining the trained target speech evaluation model; the target speech evaluation model is used to evaluate the quality of the original speech.

[0363] In this embodiment, the speech evaluation model to be trained performs quality evaluation on first and second speech samples with different levels of quality loss, respectively, to obtain corresponding first and second prediction results. Determining the quality difference between the first and second prediction results and their respective evaluation labels allows us to identify the gap between the model's predictions and the actual results. Based on the first and second prediction results, the ranking loss between the first and second speech samples is determined, which measures the relative quality distance between the speech sample pairs. Based on the quality differences and ranking losses, a target loss function is constructed, incorporating both ranking loss and quality differences. The speech evaluation model is repeatedly trained to adjust model parameters, thereby improving the accuracy of the model evaluation. When the stopping condition is met, the trained target speech evaluation model is obtained. This target speech evaluation model can accurately and quickly evaluate the quality of raw speech and can also evaluate real-time speech, improving the timeliness and efficiency of speech evaluation.

[0364] Based on the same inventive concept, this application also provides a speech evaluation model training apparatus for implementing the speech evaluation model training method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more embodiments of the speech evaluation model training apparatus provided below can be found in the limitations of the speech evaluation model training method described above, and will not be repeated here.

[0365] In one embodiment, such as Figure 13As shown, a speech evaluation model training device 1300 is provided, including: an acquisition module 1302, an evaluation module 1304, a quality difference determination module 1306, a ranking loss determination module 1308, a construction module 1310, and a training module 1312, wherein,

[0366] The acquisition module 1302 is used to acquire a first speech sample and a second speech sample that is different from the first speech sample.

[0367] The evaluation module 1304 is used to perform quality evaluation processing on the first speech sample and the second speech sample respectively through the speech evaluation model to be trained, so as to obtain the first prediction result and the second prediction result.

[0368] The quality difference determination module 1306 is used to determine the quality difference between the first prediction result and the second prediction result and their respective corresponding evaluation labels.

[0369] The ranking loss determination module 1308 is used to determine the ranking loss between the first speech sample and the second speech sample based on the first prediction result and the second prediction result.

[0370] Module 1310 is used to construct the target loss function based on each quality difference and ranking loss.

[0371] Training module 1312 is used to train the speech evaluation model to be trained using the target loss function until the stopping condition is met, thus obtaining the trained target speech evaluation model; the target speech evaluation model is used to evaluate the quality of the original speech.

[0372] In this embodiment, the speech evaluation model to be trained performs quality evaluation on first and second speech samples with different levels of quality loss, respectively, to obtain corresponding first and second prediction results. Determining the quality difference between the first and second prediction results and their respective evaluation labels allows us to identify the gap between the model's predictions and the actual results. Based on the first and second prediction results, the ranking loss between the first and second speech samples is determined, which measures the relative quality distance between the speech sample pairs. Based on the quality differences and ranking losses, a target loss function is constructed, incorporating both ranking loss and quality differences. The speech evaluation model is repeatedly trained to adjust model parameters, thereby improving the accuracy of the model evaluation. When the stopping condition is met, the trained target speech evaluation model is obtained. This target speech evaluation model can accurately and quickly evaluate the quality of raw speech and can also evaluate real-time speech, improving the timeliness and efficiency of speech evaluation.

[0373] In one embodiment, the acquisition module 1302 is further configured to perform speech activity detection on the first training speech to obtain a corresponding first speech sample; perform speech activity detection on the second training speech to obtain a corresponding second speech sample; the second training speech is different from the first training speech.

[0374] In one embodiment, the acquisition module 1302 is further configured to perform frame-segmentation processing on the first training speech to obtain corresponding first speech signals for each frame; determine the short-time energy and short-time zero-crossing rate corresponding to each frame of the first speech signal; and based on the short-time energy and short-time zero-crossing rate of each first speech signal, select valid signal frames from each frame of the first speech signal to obtain first speech samples.

[0375] The acquisition module 1302 is also used to perform frame segmentation processing on the second training speech to obtain the corresponding second speech signals of each frame; determine the short-time energy and short-time zero-crossing rate corresponding to each frame of the second speech signal; and select effective signal frames from each frame of the second speech signal based on the short-time energy and short-time zero-crossing rate of each second speech signal to obtain the second speech samples.

[0376] In one embodiment, the acquisition module 1302 is further configured to perform windowing processing on each frame of the first speech signal, and perform time-frequency conversion processing on the windowed speech signal to obtain a first spectrum corresponding to each frame of the first speech signal; determine the short-time energy of each frame of the first speech signal based on each first spectrum; and determine the short-time zero-crossing rate of each frame of the first speech signal according to the first time-domain waveform corresponding to each frame of the first speech signal.

[0377] The acquisition module 1302 is further configured to perform windowing processing on each frame of the second speech signal, and perform time-frequency conversion processing on the windowed speech signal to obtain the second spectrum corresponding to each frame of the second speech signal; determine the short-time energy of each frame of the second speech signal based on each second spectrum; and determine the short-time zero-crossing rate of each frame of the second speech signal according to the second time-domain waveform corresponding to each frame of the second speech signal.

[0378] In one embodiment, the evaluation module 1304 is further configured to perform time-frequency conversion processing on the first speech sample using the speech evaluation model to be trained, and extract features from the speech signal obtained by the time-frequency conversion processing to obtain corresponding first acoustic features; perform time-frequency conversion processing on the second speech sample, and extract features from the speech signal obtained by the time-frequency conversion processing to obtain corresponding second acoustic features; perform quality evaluation processing based on the first acoustic features to obtain a first prediction result corresponding to the first speech sample; and perform quality evaluation processing based on the second acoustic features to obtain a second prediction result corresponding to the second speech sample.

[0379] In one embodiment, the evaluation module 1304 is further configured to perform frame segmentation processing on the first acoustic feature to obtain each first feature frame corresponding to the first acoustic feature; perform normalization processing on each first feature frame to obtain the input feature at each time corresponding to the first acoustic feature; and perform quality evaluation processing based on the input feature at each time corresponding to the first acoustic feature to obtain the first prediction result corresponding to the first speech sample.

[0380] The evaluation module 1304 is also used to perform frame segmentation processing on the second acoustic feature to obtain each second feature frame corresponding to the second acoustic feature; to perform normalization processing on each second feature frame to obtain the input feature at each time corresponding to the second acoustic feature; and to perform quality evaluation processing based on the input feature at each time corresponding to the second acoustic feature to obtain the second prediction result corresponding to the second speech sample.

[0381] In one embodiment, the evaluation module 1304 is further configured to normalize each first feature frame based on the mean and variance corresponding to each first feature frame to obtain the input features corresponding to each moment of the first acoustic feature.

[0382] The evaluation module 1304 is also used to normalize each second feature frame based on the mean and variance of each second feature frame to obtain the input features corresponding to each time step of the second acoustic feature.

[0383] In one embodiment, the evaluation module 1304 is further configured to determine the output features at the current moment based on the input features at the current moment and the hidden features output at the previous moment, and to determine the update parameters of the feature object at the current moment; the input features are obtained by temporal partitioning based on the first acoustic features, and the feature object represents the path of feature transmission; the feature object at the current moment is determined based on the feature object at the previous moment, the output features at the current moment, and the update parameters; the hidden features output at the current moment are determined based on the hidden features output at the previous moment, the output features at the current moment, and the feature object at the current moment; the input features at the next moment are used as the input features at the current moment, and the steps of determining the output features at the current moment based on the input features at the current moment and the hidden features output at the previous moment are returned and executed until the hidden features output at the last moment are obtained; the first prediction result corresponding to the first speech sample is determined based on the hidden features output at the last moment.

[0384] In one embodiment, the speech evaluation model includes a forget gate, an input gate, and an output gate, with the feature object being a cell. The evaluation module 1304 is further configured to: determine the output feature of the forget gate at the current time based on the input features at the current time and the hidden layer features output at the previous time using the forget gate; determine the output feature of the input gate at the current time based on the input features at the current time and the hidden layer features output at the previous time using the input gate; determine the update parameters corresponding to the cell at the current time based on the input features at the current time and the hidden layer features output at the previous time using the input gate; determine the feature object at the current time based on the output feature of the forget gate at the current time, the output feature of the input gate at the current time, the update parameters, and the feature object at the previous time; determine the output feature of the output gate at the current time based on the input features at the current time and the hidden layer features output at the previous time using the output gate; and determine the hidden layer features of the output at the current time based on the output feature of the output gate at the current time and the cell at the current time.

[0385] In one embodiment, the ranking loss determination module 1308 is further configured to determine, based on the first prediction result and the second prediction result, the prediction probability that the quality of the first speech sample is higher than that of the second speech sample; determine the correctness of the relative relationship between the first speech sample and the second speech sample based on the prediction probability and the corresponding probability label; and determine the ranking loss between the first speech sample and the second speech sample based on the correctness of the relative relationship.

[0386] In this embodiment, based on the first and second prediction results, the predicted probability that the quality of the first speech sample is higher than that of the second speech sample is determined, thereby determining the relative relationship between the first and second speech samples in terms of speech quality. Based on the predicted probability and the corresponding probability label, the correctness of the relative relationship between the first and second speech samples is determined, thus effectively measuring the accuracy of the predicted probability that the quality of the first speech sample is higher than that of the second speech sample. Determining the ranking loss between the first and second speech samples based on the correctness of the relative relationship effectively constrains the relative relationship between the speech sample pairs in terms of speech quality.

[0387] In one embodiment, the ranking loss determination module 1308 is further configured to determine the uncertainty coefficient of the relative relationship based on the evaluation label of the first prediction result and the evaluation label of the second prediction result.

[0388] Based on the correctness of the relative relationship and the uncertainty coefficient of the relative relationship, the ranking loss between the first speech sample and the second speech sample is determined.

[0389] In this embodiment, the uncertainty coefficient of the relative relationship is determined based on the evaluation labels of the first and second prediction results. This uncertainty coefficient allows for an accurate determination of the contribution of the relative relationship between the first and second speech samples to the quality assessment, thereby accurately determining the ranking loss between the first and second speech samples. Training the speech assessment model by combining the ranking loss and the quality difference can improve the accuracy of the model's evaluation.

[0390] Each module in the aforementioned speech evaluation device and speech evaluation model training device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0391] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 14 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores speech evaluation data and training data for speech evaluation models. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer program implements a speech evaluation method and a speech evaluation model training method.

[0392] Those skilled in the art will understand that Figure 14 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0393] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0394] In one embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0395] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and executes the computer instructions, causing the computer device to perform the steps in the above method embodiments.

[0396] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0397] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0398] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0399] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A speech evaluation method, characterized in that, The method includes: Speech activity detection is performed on the original speech to obtain the corresponding speech to be tested; The speech to be tested is subjected to time-frequency conversion processing, and the speech signal obtained by the time-frequency conversion processing is subjected to feature extraction to obtain the acoustic features corresponding to the speech to be tested. The acoustic features are convolved to obtain depth features at different time points; Based on the input features at the current moment and the hidden layer features output at the previous moment, the output features at the current moment are determined, and the update parameters of the feature object at the current moment are determined; the input features at the current moment are the depth features at the corresponding moment, and the feature object represents the path of feature transmission. The feature object at the current time is determined based on the feature object at the previous time step, the output feature at the current time step, and the update parameter. Based on the hidden layer features output at the previous time step, the output features at the current time step, and the feature object at the current time step, the hidden layer features output at the current time step are determined. The steps of taking the input features of the next time step as the input features of the current time step, returning the hidden layer features output by the previous time step based on the input features of the current time step, determining the output features of the current time step, and continuing to execute until the hidden layer features output by the last time step are obtained. Based on the hidden layer features output at the last moment, the quality assessment result corresponding to the original speech is determined.

2. The method according to claim 1, characterized in that, The process of detecting speech activity in the original speech to obtain the corresponding speech to be tested includes: The original speech is segmented into frames to obtain the corresponding speech signals for each frame. Determine the short-time energy and short-time zero-crossing rate corresponding to each frame of the speech signal; Based on the short-time energy and the short-time zero-crossing rate, valid signal frames are selected from each frame of the speech signal to obtain the speech to be tested.

3. The method according to claim 2, characterized in that, Determining the short-time energy and short-time zero-crossing rate corresponding to each frame of the speech signal includes: Each frame of the audio signal is windowed, and the windowed audio signal is then subjected to time-frequency conversion to obtain the corresponding spectrum. Based on the spectrum corresponding to each frame of the speech signal, the short-time energy of each frame of the speech signal is determined; Obtain the time-domain waveform corresponding to each frame of the speech signal, and determine the short-time zero-crossing rate of the corresponding speech signal based on the time-domain waveform.

4. The method according to claim 1, characterized in that, The convolutional processing of the acoustic features to obtain depth features at different time points includes: The acoustic features are segmented into frames to obtain feature frames at different times; Convolution and normalization processes are performed on the feature frames at different times to obtain depth features at different times.

5. The method according to claim 1, characterized in that, The method is applied to a target speech evaluation model; the target speech evaluation model includes a forget gate, an input gate, and an output gate, and the feature object is a cell; the method involves determining the output feature at the current time based on the input feature at the current time and the hidden layer feature output from the previous time step, and determining the update parameters of the feature object at the current time, including: The output features of the forget gate at the current time are determined by the input features of the forget gate based on the input features at the current time and the hidden layer features of the output at the previous time. The output features of the input gate at the current time are determined by the input gate based on the input features at the current time and the hidden layer features output at the previous time. The update parameters of the cell element at the current time are determined by the input gate based on the input features at the current time and the hidden layer features output at the previous time. The step of determining the feature object at the current time based on the feature object at the previous time step, the output feature at the current time step, and the update parameter includes: Based on the output characteristics of the forget gate at the current time, the output characteristics of the input gate at the current time, the update parameters, and the feature objects of the previous time, the feature objects at the current time are determined; The step of determining the hidden layer features output at the current time based on the hidden layer features output at the previous time step, the output features at the current time step, and the feature object at the current time step includes: The output characteristics of the output gate at the current time are determined by the output gate based on the input characteristics at the current time and the hidden layer characteristics of the output at the previous time. Based on the output characteristics of the output gate at the current time and the cell element at the current time, the hidden layer characteristics of the output at the current time are determined.

6. The method according to claim 1, characterized in that, The speech evaluation method is applied to video conferencing, and the quality evaluation result is the quality evaluation score of the real-time speech in the video conferencing; the step of performing speech activity detection on the original speech to obtain the corresponding speech to be tested includes: Perform voice activity detection on video conferences to obtain the corresponding real-time voice; The step of performing time-frequency conversion processing on the speech to be tested, and extracting features from the speech signal obtained by the time-frequency conversion processing to obtain the acoustic features corresponding to the speech to be tested, includes: The real-time speech is subjected to time-frequency conversion processing, and the speech signal obtained by the time-frequency conversion processing is subjected to feature extraction to obtain the real-time acoustic features corresponding to the real-time speech. The convolutional processing of the acoustic features to obtain depth features at different time points includes: The real-time acoustic features are convolved to obtain target depth features at different times; The method further includes: If the real-time voice quality assessment score does not meet the preset quality conditions, a reminder operation will be triggered.

7. The method according to any one of claims 1 to 6, characterized in that, The method is applied to a target speech evaluation model, and the training steps of the target speech evaluation model include: Acquire a first speech sample and a second speech sample that is different from the first speech sample; The first speech sample and the second speech sample are subjected to quality assessment processing by the speech evaluation model to be trained, respectively, to obtain the first prediction result and the second prediction result. Determine the quality differences between the first prediction result and the second prediction result and their respective corresponding evaluation labels; Based on the first prediction result and the second prediction result, determine the ranking loss between the first speech sample and the second speech sample; Based on the aforementioned quality differences and ranking losses, a target loss function is constructed; The speech evaluation model to be trained is trained using the target loss function until a stopping condition is met, resulting in a trained target speech evaluation model; the target speech evaluation model is used to evaluate the quality of the original speech.

8. A method for training a speech evaluation model, characterized in that, The method includes: Acquire a first speech sample and a second speech sample that is different from the first speech sample; The first and second speech samples are subjected to quality assessment processing using a speech evaluation model to be trained, respectively, to obtain a first prediction result and a second prediction result. The speech evaluation model to be trained is used to detect speech activity on the first speech sample, and after time-frequency conversion of the obtained speech sample, extract first acoustic features. The first acoustic features are convolved to obtain depth features at different times. Furthermore, based on the input features at the current time and the hidden layer features output from the preceding time steps, the output features at the current time are determined, and the update parameters of the feature object at the current time are determined. The input features at the current time are the depth features at the corresponding time step, and the feature object represents the... The path of feature transmission is determined based on the feature object of the previous time step, the output feature of the current time step, and the update parameters. The feature object of the current time step is determined based on the hidden layer features output by the previous time step, the output feature of the current time step, and the feature object of the current time step. The input feature of the next time step is used as the input feature of the current time step. The step of determining the output feature of the current time step based on the input feature of the current time step and the hidden layer features output by the previous time step is returned and the process continues until the hidden layer features output by the last time step are obtained. The first prediction result is determined based on the hidden layer features output by the last time step. Determine the quality differences between the first prediction result and the second prediction result and their respective corresponding evaluation labels; Based on the first prediction result and the second prediction result, determine the ranking loss between the first speech sample and the second speech sample; Based on the aforementioned quality differences and ranking losses, a target loss function is constructed; The speech evaluation model to be trained is trained using the target loss function until a stopping condition is met, resulting in a trained target speech evaluation model; the target speech evaluation model is used to evaluate the quality of the original speech.

9. The method according to claim 8, characterized in that, The step of determining the ranking loss between the first speech sample and the second speech sample based on the first prediction result and the second prediction result includes: Based on the first prediction result and the second prediction result, determine the predicted probability that the quality of the first speech sample is higher than that of the second speech sample; Based on the predicted probabilities and the corresponding probability labels, the correctness of the relative relationship between the first speech sample and the second speech sample is determined; Based on the correctness of the relative relationship, the ranking loss between the first speech sample and the second speech sample is determined.

10. The method according to claim 9, characterized in that, The determination of the ranking loss between the first speech sample and the second speech sample based on the correctness of the relative relationship includes: The uncertainty coefficient of the relative relationship is determined based on the evaluation labels of the first prediction result and the evaluation labels of the second prediction result; Based on the correctness of the relative relationship and the uncertainty coefficient of the relative relationship, the ranking loss between the first speech sample and the second speech sample is determined.

11. A voice evaluation device, characterized in that, The device includes: The detection module is used to detect speech activity in the original speech to obtain the corresponding speech to be tested; The conversion module is used to perform time-frequency conversion processing on the speech to be tested, and to extract features from the speech signal obtained by the time-frequency conversion processing to obtain the acoustic features corresponding to the speech to be tested. The convolution module is used to perform convolution processing on the acoustic features to obtain depth features at different times; The processing module is configured to determine the output features of the current moment based on the input features of the current moment and the hidden layer features output from the previous moment, and to determine the update parameters of the feature object at the current moment; the input features of the current moment are the depth features of the corresponding moment, and the feature object represents the path of feature transmission; the module determines the feature object of the current moment based on the feature object of the previous moment, the output features of the current moment, and the update parameters; the module determines the hidden layer features output at the current moment based on the hidden layer features output from the previous moment, the output features of the current moment, and the feature object of the current moment; the module uses the input features of the next moment as the input features of the current moment, and returns the step of determining the output features of the current moment based on the input features of the current moment and the hidden layer features output from the previous moment, and continues to execute until the hidden layer features output at the last moment are obtained; and the module determines the quality assessment result corresponding to the original speech based on the hidden layer features output at the last moment.

12. The apparatus according to claim 11, characterized in that, The detection module is further configured to perform frame-segmentation processing on the original speech to obtain corresponding speech signals for each frame; determine the short-time energy and short-time zero-crossing rate corresponding to each frame of the speech signal; and based on the short-time energy and the short-time zero-crossing rate, filter out valid signal frames from each frame of the speech signal to obtain the speech to be tested.

13. The apparatus according to claim 12, characterized in that, The detection module is further configured to perform windowing processing on each frame of the speech signal, and perform time-frequency conversion processing on the windowed speech signal to obtain the corresponding spectrum; determine the short-time energy of each frame of the speech signal based on the spectrum corresponding to each frame of the speech signal; obtain the time-domain waveform corresponding to each frame of the speech signal, and determine the short-time zero-crossing rate of the corresponding speech signal based on the time-domain waveform.

14. The apparatus according to claim 11, characterized in that, The convolution module is also used to perform frame segmentation processing on the acoustic features to obtain feature frames at different times; and to perform convolution and normalization processing on the feature frames at different times to obtain depth features at different times.

15. The apparatus according to claim 11, characterized in that, The device is applied to a target speech evaluation model; the target speech evaluation model includes a forget gate, an input gate, and an output gate, and the feature object is a cell element; The processing module is further configured to: determine the output feature of the forget gate at the current time based on the input features at the current time and the hidden layer features output at the previous time using the forget gate; determine the output feature of the input gate at the current time based on the input features at the current time and the hidden layer features output at the previous time using the input gate; determine the update parameters corresponding to the cell at the current time based on the input features at the current time and the hidden layer features output at the previous time using the input gate; determine the feature object at the current time based on the output feature of the forget gate at the current time, the output feature of the input gate at the current time, the update parameters, and the feature object at the previous time; and determine the output feature of the output gate at the current time based on the input features at the current time and the hidden layer features output at the previous time using the output gate. Based on the output characteristics of the output gate at the current time and the cell element at the current time, the hidden layer characteristics of the output at the current time are determined.

16. The apparatus according to claim 11, characterized in that, The device is used in video conferencing, and the quality assessment result is the quality assessment score of real-time audio in the video conferencing. The detection module is also used to detect voice activity in the video conference and obtain the corresponding real-time voice. The conversion module is also used to perform time-frequency conversion processing on the real-time speech, and to extract features from the speech signal obtained by the time-frequency conversion processing to obtain the real-time acoustic features corresponding to the real-time speech. The convolution module is also used to perform convolution processing on the real-time acoustic features to obtain target depth features at different times. The device further includes a trigger module; the trigger module is used to trigger a reminder operation if the quality assessment score of the real-time voice does not meet the preset quality conditions.

17. The apparatus according to any one of claims 11 to 16, characterized in that, The device is applied to a target speech evaluation model, and the training steps of the target speech evaluation model include: The acquisition module is used to acquire a first speech sample and a second speech sample that is different from the first speech sample; The evaluation module is used to perform quality evaluation processing on the first speech sample and the second speech sample respectively through the speech evaluation model to be trained, so as to obtain the first prediction result and the second prediction result. A quality difference determination module is used to determine the quality difference between the first prediction result and the second prediction result and their respective corresponding evaluation labels. The ranking loss determination module is used to determine the ranking loss between the first speech sample and the second speech sample based on the first prediction result and the second prediction result. A construction module is used to construct a target loss function based on the quality differences and the ranking loss. The training module is used to train the speech evaluation model to be trained using the target loss function until a stopping condition is met, thereby obtaining a trained target speech evaluation model; the target speech evaluation model is used to evaluate the quality of the original speech.

18. A speech evaluation model training device, characterized in that, The device includes: The acquisition module is used to acquire a first speech sample and a second speech sample that is different from the first speech sample; An evaluation module is used to perform quality evaluation processing on the first speech sample and the second speech sample respectively using a speech evaluation model to be trained, to obtain a first prediction result and a second prediction result; the speech evaluation model to be trained is used to perform speech activity detection on the first speech sample, and extract the first acoustic feature after time-frequency conversion of the obtained speech sample, convolve the first acoustic feature to obtain depth features at different times, and determine the output feature at the current time based on the input feature at the current time and the hidden layer feature output at the previous time, and determine the update parameters of the feature object at the current time, wherein the input feature at the current time is the depth feature at the corresponding time, and the feature object... The image represents the path of feature transmission. Based on the feature object of the previous time step, the output feature of the current time step, and the update parameters, the feature object of the current time step is determined. Based on the hidden layer features output by the previous time step, the output feature of the current time step, and the feature object of the current time step, the hidden layer features output by the current time step are determined. The input feature of the next time step is used as the input feature of the current time step. The steps of determining the output feature of the current time step based on the input feature of the current time step and the hidden layer features output by the previous time step are returned and the process continues until the hidden layer features output by the last time step are obtained. The first prediction result is determined based on the hidden layer features output by the last time step. A quality difference determination module is used to determine the quality difference between the first prediction result and the second prediction result and their respective corresponding evaluation labels. The ranking loss determination module is used to determine the ranking loss between the first speech sample and the second speech sample based on the first prediction result and the second prediction result. A construction module is used to construct a target loss function based on the quality differences and the ranking loss. The training module is used to train the speech evaluation model to be trained using the target loss function until a stopping condition is met, thus obtaining the trained target speech evaluation model; the target speech evaluation model is used to evaluate the quality of the original speech.

19. The apparatus according to claim 18, characterized in that, The ranking loss determination module is further configured to determine, based on the first prediction result and the second prediction result, the prediction probability that the quality of the first speech sample is higher than that of the second speech sample; based on the prediction probability and the corresponding probability label, determine the correctness of the relative relationship between the first speech sample and the second speech sample; and based on the correctness of the relative relationship, determine the ranking loss between the first speech sample and the second speech sample.

20. The apparatus according to claim 19, characterized in that, The ranking loss determination module is further configured to determine the uncertainty coefficient of the relative relationship based on the evaluation labels of the first prediction result and the evaluation labels of the second prediction result; and to determine the ranking loss between the first speech sample and the second speech sample based on the correctness of the relative relationship and the uncertainty coefficient of the relative relationship.

21. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 10.

22. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 10.

23. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 10.

Images