A method and related apparatus for evaluating speech pronunciation and prosody

Abstract

This application discloses a method and related apparatus for evaluating speech pronunciation and prosody, relating to the field of speech evaluation technology. The evaluation method includes: acquiring the speech to be evaluated from a target user; extracting hidden state features representing the speech content from the speech to be evaluated; using a pre-trained generative acoustic model, generating a personalized standard acoustic feature sequence based on the hidden state features and the timbre representation vector of the target user. The personalized standard acoustic feature sequence carries speech content consistent with the speech to be evaluated and contains standard pronunciation and prosody information, while the timbre features are consistent with the timbre features of the target user; synthesizing personalized standard speech based on the personalized standard acoustic feature sequence; and evaluating the pronunciation and prosody of the speech to be evaluated using the personalized standard speech as the evaluation comparison benchmark. The evaluation method disclosed in this application ensures accurate and reliable evaluation results by generating standard speech consistent with the timbre of the target user as the evaluation comparison benchmark.

Description

Technical Field

[0001] This application relates to the field of speech evaluation technology, and in particular to a method and related apparatus for evaluating speech pronunciation and prosody. Background Technology

[0002] With the acceleration of globalization and the widespread application of online education, the market demand for efficient and accurate language pronunciation learning and training tools is becoming increasingly urgent. In various specific application scenarios such as language learning, speech training, broadcasting, and speech therapy, assessing pronunciation accuracy and rhythmic appropriateness is a core requirement.

[0003] Traditional pronunciation and prosody assessment schemes are based on human assessment, meaning that qualified assessors subjectively evaluate pronunciation and prosody through auditory perception and experience. Traditional pronunciation assessment schemes rely heavily on the human auditory judgment of professionals, which generally suffers from inherent drawbacks such as high subjectivity, low accuracy, low efficiency, high labor costs, and difficulty in large-scale application. Summary of the Invention

[0004] In view of this, this application provides a method and related apparatus for speech pronunciation and prosody evaluation, which solves the problems of traditional pronunciation and prosody evaluation schemes, such as strong subjectivity, low evaluation accuracy, low evaluation efficiency, high labor costs, and difficulty in large-scale application. The technical solution is as follows:

[0005] The first aspect of this application provides a method for evaluating speech pronunciation and prosody, including:

[0006] Acquire the target user's voice recordings to be evaluated;

[0007] Extract hidden state features that represent the speech content from the speech to be evaluated;

[0008] Using a generative acoustic model, a personalized standard acoustic feature sequence is generated based on the hidden state features and the timbre representation vector of the target user. The generative acoustic model is trained using training speech, the timbre representation vector of the user to whom the training speech belongs, and a reference acoustic feature sequence corresponding to the training speech. The reference acoustic feature sequence is the acoustic feature sequence of a standard speaker reading the text corresponding to the training speech. The personalized standard acoustic feature sequence carries speech content consistent with the speech to be evaluated and contains standard pronunciation and prosodic information. At the same time, the timbre features are consistent with the timbre features of the target user.

[0009] Personalized standard speech is obtained by synthesizing speech based on the personalized standard acoustic feature sequence;

[0010] Using the personalized standard speech as the evaluation benchmark, the pronunciation and rhythm of the speech to be evaluated are assessed.

[0011] In one possible implementation, the training objective of the generative acoustic model includes:

[0012] Make the timbre features of the standard acoustic feature sequence generated based on the hidden state features of the training speech and the timbre representation vector of the user to which the training speech belongs consistent with the timbre features of the user to which the training speech belongs;

[0013] Make the prosodic features of the standard acoustic feature sequence generated based on the hidden state features of the training speech and the timbre representation vector of the user to which the training speech belongs consistent with the prosodic features of the reference acoustic feature sequence;

[0014] The standard acoustic feature sequence generated based on the hidden state features of the training speech and the timbre representation vector of the user to whom the training speech belongs is accurate, conforms to the natural distribution of acoustic features, and is internally coherent.

[0015] In one possible implementation, the speech pronunciation and prosody evaluation method further includes:

[0016] Obtain the text corresponding to the speech to be evaluated;

[0017] Based on the prosodic annotation model, the text corresponding to the speech to be evaluated is prosodicly annotated, and the prosodic annotation results are used as prosodic prior information. The prosodic annotation model is trained using training text annotated with prosodic information.

[0018] The process of generating a personalized standard acoustic feature sequence using a generative acoustic model, based on the hidden state features and the timbre representation vector of the target user, includes:

[0019] Using a generative acoustic model, a personalized standard acoustic feature sequence is generated based on the hidden state features and the timbre representation vector of the target user, supplemented by the prosodic prior information.

[0020] In one possible implementation, the step of utilizing a generative acoustic model to generate a personalized standard acoustic feature sequence based on the hidden state features and the timbre representation vector of the target user, supplemented by the prosodic prior information, includes:

[0021] The hidden state features, the timbre representation vector of the target user, and the prosodic prior information are fused to obtain multimodal fusion features;

[0022] Using a generative acoustic model, a personalized standard acoustic feature sequence is generated based on the multimodal fusion features.

[0023] In one possible implementation, the step of using a generative acoustic model to generate a personalized standard acoustic feature sequence based on the multimodal fusion features includes:

[0024] Using a generative acoustic model, personalized standard acoustic features are predicted frame by frame in an autoregressive manner based on the multimodal fusion features, resulting in a personalized standard acoustic feature sequence.

[0025] Specifically, during each prediction step, the personalized standard acoustic features for the current step are predicted based on the multimodal fusion features and the personalized standard acoustic features predicted in the previous step.

[0026] In one possible implementation, the evaluation of the pronunciation and prosody of the speech to be evaluated, using the personalized standard speech as the evaluation comparison benchmark, includes:

[0027] The phoneme-level and word-level timestamp information corresponding to the speech to be evaluated and the personalized standard speech are obtained respectively. The phoneme-level and word-level timestamp information is obtained by aligning the corresponding speech with the text corresponding to the speech to be evaluated.

[0028] By combining the word-level timestamp information corresponding to the speech to be evaluated and the personalized standard speech, prosodic event information of the speech to be evaluated and the personalized standard speech are generated respectively. By comparing the prosodic event information of the speech to be evaluated with the prosodic event information of the personalized standard speech, the prosodic evaluation result of the speech to be evaluated is obtained.

[0029] By combining the phoneme-level timestamp information corresponding to the speech to be evaluated and the personalized standard speech, a phoneme-level pronunciation evaluation is performed on the speech to be evaluated, and the pronunciation evaluation result of the speech to be evaluated is obtained.

[0030] In one possible implementation, the step of combining the word-level timestamp information corresponding to the speech to be evaluated and the personalized standard speech to generate prosodic event information for the speech to be evaluated and the personalized standard speech respectively includes:

[0031] For each speech item in the speech to be evaluated and the personalized standard speech:

[0032] The average energy or average fundamental frequency of each word's speech segment is determined by combining the corresponding word-level timestamp information. Based on the average energy or average fundamental frequency of each word's speech segment and the average energy or average fundamental frequency of adjacent words' speech segments, it is determined whether each word is a stressed word. Structured stressed word event information is generated for the determined stressed words.

[0033] And / or, determine the silence duration of the word boundary based on the corresponding word-level timestamp information, determine whether the pause at the word boundary is a semantic pause based on the silence duration of the word boundary, and generate structured semantic pause event information for the determined semantic pause.

[0034] And / or, combine the corresponding word-level timestamp information to determine the silence duration of the word boundary, and obtain the fundamental frequency curve of the speech. Based on the silence duration of the word boundary and the continuity of the fundamental frequency curve at the word boundary, determine whether a connected speech has occurred, and generate structured connected speech event information for words that have been connected.

[0035] And / or, combine the corresponding word-level timestamp information to obtain the fundamental frequency sequence of the speech segment of the sentence-ending word, perform linear fitting on the fundamental frequency sequence, determine the intonation direction based on the slope of the fitted line, and generate structured pitch rise and fall event information for the determined intonation direction.

[0036] In one possible implementation, the step of obtaining the prosodic evaluation result of the speech to be evaluated by comparing the prosodic event information of the speech to be evaluated with the prosodic event information of the personalized standard speech includes:

[0037] By comparing the prosodic event information of the speech to be evaluated with the prosodic event information of the personalized standard speech, the prosodic difference information between the speech to be evaluated and the personalized standard speech is obtained.

[0038] Based on the prosodic difference information, it is determined whether there are prosodic errors in the speech to be evaluated and the type of error if there are errors. The prosodic difference information is then converted into a score to obtain the prosodic score of the speech to be evaluated.

[0039] In one possible implementation, the step of combining the phoneme-level timestamp information corresponding to the speech to be evaluated and the personalized standard speech to perform phoneme-level pronunciation evaluation of the speech to be evaluated, and obtaining the pronunciation evaluation result of the speech to be evaluated, includes:

[0040] The frame-level posterior probability corresponding to the speech to be evaluated is obtained. Based on the phoneme-level timestamp information and the frame-level posterior probability corresponding to the speech to be evaluated, the phoneme-level pronunciation accuracy is determined, and the pronunciation accuracy evaluation result is obtained. The frame-level posterior probability is the probability estimate of each frame of speech belonging to each phoneme by the acoustic model, given that the timestamp information of each phoneme is known.

[0041] And / or, obtain the vowel speech segment to be evaluated based on the phoneme-level timestamp information corresponding to the speech to be evaluated, and obtain the standard vowel speech segment corresponding to the vowel speech segment to be evaluated based on the phoneme-level timestamp information corresponding to the personalized standard speech, determine the formant distance between the vowel speech segment to be evaluated and the corresponding standard vowel speech segment, and obtain the vowel sound quality evaluation result.

[0042] And / or, obtain the consonant speech segment to be evaluated based on the phoneme-level timestamp information corresponding to the speech to be evaluated. If the consonant speech segment to be evaluated is a plosive speech segment, determine the plosive energy of the plosive speech segment and determine the consonant intelligibility evaluation result based on the plosive energy. If the consonant speech segment to be evaluated is a fricative speech segment, determine the spectral centroid of the fricative speech segment and determine the consonant intelligibility evaluation result based on the spectral centroid.

[0043] In one possible implementation, the speech pronunciation and prosody evaluation method further includes:

[0044] Based on the pronunciation and prosody evaluation results of the speech to be evaluated, an evaluation feedback report of the speech to be evaluated is generated;

[0045] The evaluation feedback report of the speech to be evaluated includes some or all of the following: the pronunciation and prosody evaluation score of the speech to be evaluated, the acoustic curve comparison between the speech to be evaluated and the personalized standard speech, the pronunciation and prosody error information of the speech to be evaluated, and the corresponding correction suggestions for the speech to be evaluated.

[0046] A second aspect of this application provides an electronic device, including at least one processor and a memory connected to the processor, wherein:

[0047] The memory is used to store computer programs;

[0048] The processor is used to execute the computer program so that the electronic device can implement any of the above-described speech pronunciation and prosody evaluation methods.

[0049] A third aspect of this application provides a computer storage medium carrying one or more computer programs, which, when executed by an electronic device, enable the electronic device to implement any of the above-described speech pronunciation and prosody evaluation methods.

[0050] A fourth aspect of this application provides a computer program product including computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement any of the above-described speech pronunciation and prosody evaluation methods.

[0051] Using the above-mentioned technical solution, the speech pronunciation and prosody evaluation method provided in this application first obtains the speech to be evaluated from the target user, then extracts hidden state features representing the speech content from the speech to be evaluated, and then uses a pre-trained generative acoustic model to generate a personalized standard acoustic feature sequence based on the hidden state features and the timbre representation vector of the target user. Next, personalized standard speech is synthesized based on the personalized standard acoustic feature sequence, and finally, the personalized standard speech is used as the evaluation comparison benchmark to complete the automated evaluation of the speech pronunciation and prosody. The speech pronunciation and prosody evaluation method provided in this application achieves full-process automation of speech evaluation, effectively overcoming the limitations of traditional evaluation schemes such as low evaluation efficiency, high labor costs, and difficulty in scaling. At the same time, by generating personalized standard speech with standard pronunciation and prosody, timbre consistent with the target user's timbre, and speech content consistent with the speech content of the speech to be evaluated as an objective evaluation comparison benchmark, the subjectivity of manual judgment is eliminated, ensuring the accuracy and reliability of the evaluation results. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0053] Figure 1 A flowchart illustrating the speech pronunciation and prosody evaluation method provided in the embodiments of this application;

[0054] Figure 2 A schematic diagram illustrating the process of generating personalized standard speech as provided in the embodiments of this application;

[0055] Figure 3 A schematic diagram illustrating the training process of the generative acoustic model provided in this application embodiment;

[0056] Figure 4 This is a flowchart illustrating the process of evaluating the pronunciation and prosody of the speech to be evaluated using personalized standard speech as the evaluation and comparison benchmark, as provided in the embodiments of this application.

[0057] Figure 5 This is a schematic diagram of the structure of the speech pronunciation and prosody evaluation device provided in the embodiments of this application. Detailed Implementation

[0058] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0059] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0060] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0061] Given that traditional pronunciation and prosody assessment schemes rely on the human auditory judgment of professionals, which generally suffer from inherent defects such as strong subjectivity, low assessment accuracy, low assessment efficiency, and high labor costs, this case attempts to propose an automatic assessment scheme. To this end, research was conducted. The initial idea was to acquire the speech to be assessed from the target user, and use the pre-recorded speech of a standard speaker reading the same text as the assessment comparison benchmark to assess the pronunciation and prosody of the speech to be assessed.

[0062] Research on the above-mentioned automatic evaluation scheme revealed that there is a significant difference in timbre between the standard speaker and the target user. This significant difference in timbre severely interferes with the judgment of pronunciation and rhythm. Therefore, the evaluation accuracy of the above-mentioned automatic evaluation scheme is not high. In addition, since the above-mentioned automatic evaluation scheme requires the prior acquisition of the standard speaker's reading of a fixed text, it can only evaluate the speech of fixed texts and cannot cope with free speech scenarios.

[0063] To address the problems of the aforementioned automatic evaluation schemes, further research was conducted. Through continuous research, a more effective speech pronunciation and prosody evaluation method was finally proposed. This evaluation method overcomes the shortcomings of traditional pronunciation and prosody evaluation schemes, as well as the shortcomings of the aforementioned automatic evaluation schemes.

[0064] The speech pronunciation and prosody evaluation method provided in this application will be described below through the following embodiments.

[0065] Please see Figure 1 The diagram illustrates a flowchart of a speech pronunciation and prosody evaluation method provided in an embodiment of this application. This speech pronunciation and prosody evaluation method may include:

[0066] Step S101: Obtain the voice recordings of the target user to be evaluated.

[0067] The audio to be evaluated can be any type of audio, such as audio of the target user reading or reciting a specified text, or audio of the target user speaking freely.

[0068] In this embodiment, the speech to be evaluated can be a complete recorded speech segment or a real-time speech data stream (a speech data stream that is being continuously generated and has not yet ended). That is, the speech pronunciation and prosody evaluation method provided in this application embodiment can evaluate a complete speech segment or a real-time speech data stream.

[0069] Step S102: Extract hidden state features representing the speech content of the speech to be evaluated, and obtain the hidden state features of the speech to be evaluated.

[0070] In one possible implementation, such as Figure 2 As shown, speech recognition can be performed on the speech to be evaluated based on a speech recognition model to extract hidden state features that represent the speech content. During the speech recognition process, the speech recognition model generates high-quality hidden state features that perfectly represent the speech content of the speech to be evaluated. These hidden state features are rich in linguistic information such as phonemes and syllables, and are aligned with the recognized text of the speech to be evaluated.

[0071] It should be noted that if the speech to be evaluated is a real-time speech data stream, an end-to-end streaming speech recognition model (such as TransformerED, RNNT, etc.) can be used to perform speech recognition on the speech to extract hidden state features that represent the speech content.

[0072] In addition to obtaining hidden state features representing speech content through the methods mentioned above, other methods can also be used to obtain hidden state features representing speech content. For example, self-supervised learning speech base models (such as wav2vec2.0, HuberT, WavLM, etc.) can be used to obtain hidden state features representing speech content for the speech to be evaluated.

[0073] Step S103: Using a generative acoustic model, generate a personalized standard acoustic feature sequence based on the hidden state features of the speech to be evaluated and the timbre representation vector of the target user.

[0074] Here, the timbre representation vector of the target user is a feature vector representing the timbre of the target user. The timbre representation vector of the target user can be obtained in advance, specifically, as shown in... Figure 2As shown, the target user's registration voice can be obtained before the evaluation (for example, the user reads a text aloud to register before the evaluation). Then, the target user's timbre representation vector can be obtained based on the target user's registration voice. More specifically, a high-dimensional voiceprint embedding vector can be extracted from the target user's registration voice based on the voiceprint extraction model. This voiceprint embedding vector is a low-dimensional, dense vector that uniquely represents the user's timbre. The high-dimensional voiceprint embedding vector is used as the target user's timbre representation vector.

[0075] This embodiment uses the hidden state features of the speech to be evaluated as the content condition input of the generative acoustic model. It encodes the temporal structure and phoneme-level acoustic information of the speech to be evaluated. It can more accurately guide the generative acoustic model when to generate which acoustic features than plain text. The timbre representation vector of the target user is used as the timbre condition input of the generative acoustic model, which provides personalized timbre for the generative acoustic model.

[0076] In this embodiment, the generative acoustic model is pre-trained using training speech, the timbre representation vector of the user to whom the training speech belongs, and the reference acoustic feature sequence corresponding to the training speech. The reference acoustic feature sequence corresponding to the training speech is the acoustic feature sequence of the standard speaker's reading of the text corresponding to the training speech, which contains standard pronunciation and prosodic information.

[0077] During training, the generative acoustic model learns to generate acoustic feature sequences that possess standard pronunciation and prosody, whose timbre features are consistent with the specified timbre, and whose carried speech content is consistent with the specified speech content, based on specified speech content (represented by the hidden state representation of the specified speech) and specified timbre (represented by the timbre representation vector). Furthermore, the personalized standard acoustic feature sequence generated by the generative acoustic model based on the hidden state features of the speech to be evaluated and the timbre representation vector of the target user carries speech content consistent with the speech to be evaluated, contains standard pronunciation and prosodic information, and maintains consistency with the timbre features of the target user.

[0078] Step S104: Synthesize speech based on the personalized standard acoustic feature sequence to obtain personalized standard speech.

[0079] Personalized standard speech synthesized based on personalized standard acoustic feature sequences has standard pronunciation and rhythm, its timbre is consistent with that of the target user, and its speech content is the same as that of the speech to be evaluated.

[0080] Optional, such as Figure 2As shown, a high-performance neural vocoder (such as HiFi-GAN) can be used to synthesize speech based on a personalized standard acoustic feature sequence to obtain personalized standard speech. That is, the personalized standard acoustic feature sequence is input into the neural vocoder, and the neural vocoder synthesizes high-fidelity, audible personalized standard speech based on the personalized standard acoustic features.

[0081] It should be noted that a neural vocoder is a model based on deep neural networks that can reverse the acoustic feature sequence to reconstruct high-quality, natural waveform audio. It uses deep neural networks (such as WaveNet, GAN, Diffusion, etc.) to directly learn the complex mapping relationship from features to original waveform sample points. The generated speech is extremely natural, close to human voice, has high fidelity, and can simulate complex pronunciation details and rhythms.

[0082] Step S105: Using personalized standard speech as the evaluation comparison benchmark, evaluate the pronunciation and rhythm of the speech to be evaluated.

[0083] This application constructs a personalized evaluation benchmark (i.e., a standard voice consistent with the target user's timbre) for the target user's speech to be evaluated. This eliminates the influence of timbre on the evaluation, thereby achieving a more accurate, fair, and instructive evaluation. This is not only a technological advancement but also an upgrade in user experience and evaluation philosophy, and has extremely high application value in fields such as advanced language learning, professional broadcaster training, and speech pathology.

[0084] The speech pronunciation and prosody evaluation method provided in this application first acquires the speech of the target user to be evaluated. Then, it extracts hidden state features representing the speech content from the speech. Next, using a pre-trained generative acoustic model, it generates a personalized standard acoustic feature sequence based on the hidden state features and the target user's timbre representation vector. Then, it synthesizes personalized standard speech based on the personalized standard acoustic feature sequence. Finally, it uses the personalized standard speech as the evaluation comparison benchmark to complete the automated evaluation of the speech pronunciation and prosody. The speech pronunciation and prosody evaluation method provided in this application realizes full-process automation of speech evaluation, effectively overcoming the limitations of traditional manual evaluation, such as low efficiency, high cost, and difficulty in scaling. At the same time, by generating standard speech consistent with the user's timbre as an objective evaluation comparison benchmark, it eliminates the subjectivity of manual evaluation and ensures the accuracy and reliability of the evaluation results. Furthermore, considering that when using the standard voice of a standard speaker as the evaluation benchmark, the target user's timbre differs significantly from that of the standard speaker, which can severely interfere with the evaluation of pronunciation and rhythm, this application generates personalized standard voice (with standard pronunciation and rhythm while maintaining the target user's timbre) for the voice to be evaluated. This allows the evaluation benchmark and the voice to be evaluated to be highly aligned in timbre, enabling the evaluation process to focus on the essential differences in pronunciation and rhythm, thereby greatly improving the accuracy and fairness of the evaluation. In addition, since this application dynamically generates personalized standard voice for the voice to be evaluated, it does not rely on standard voice pre-recorded for a fixed text. Therefore, it can be flexibly applied to free speaking scenarios, expanding the scope of application and practicality.

[0085] As mentioned in the above embodiments, a generative acoustic model can be used to generate a personalized standard acoustic feature sequence based on the hidden state features of the speech to be evaluated and the timbre representation vector of the target user. This process is described in some embodiments of this application.

[0086] In one possible implementation, the process of generating a personalized standard acoustic feature sequence based on the hidden state features of the speech to be evaluated and the timbre representation vector of the target user, using a generative acoustic model, may include:

[0087] Step a1: Fuse the hidden state features of the speech to be evaluated with the timbre representation vector of the target user to obtain multimodal fusion features.

[0088] In one possible implementation, the hidden state features of the speech to be evaluated can be fused with the timbre representation vector of the target user based on a cross-attention mechanism. Specifically, the timbre representation vector of the target user is first processed through a linear projection layer to make its dimension consistent with the dimension of the hidden state features of the speech to be evaluated (if the hidden state features of the speech to be evaluated are obtained based on a speech recognition model, then the dimension of the timbre representation vector of the target user is processed to the latent space dimension of the speech recognition model). Then, the processed timbre representation vector and the hidden state features of the speech to be evaluated are cross-attention calculated to obtain the multimodal fusion features.

[0089] This embodiment is not limited to using a cross-attention mechanism-based fusion method to fuse the hidden state features of the speech to be evaluated with the timbre representation vector of the target user. In another possible implementation, a direct concatenation or summation fusion method can be used to fuse the hidden state features of the speech to be evaluated with the timbre representation vector of the target user. That is, the timbre representation vector of the target user is first processed through a linear projection layer to make its dimension consistent with the dimension of the hidden state features of the speech to be evaluated. Then, the processed timbre representation vector is concatenated or summed with the hidden state features of the speech to be evaluated to obtain multimodal fusion features.

[0090] Step a2: Using a generative acoustic model, generate a personalized standard acoustic feature sequence based on multimodal fusion features.

[0091] The multimodal fusion features are input into the generative acoustic model, which then generates a personalized standard acoustic feature sequence based on the multimodal fusion features.

[0092] In one possible implementation, the generative acoustic model can employ an autoregressive model with a Transformer architecture (such as VALL-E or SpeechT5 structures). That is, the generative acoustic model predicts personalized standard acoustic features frame-by-frame using an autoregressive approach based on multimodal fusion features, resulting in a sequence of personalized standard acoustic features. During each prediction step, the generative acoustic model predicts the personalized standard acoustic features for the current step based on the multimodal fusion features and the personalized standard acoustic features predicted in the previous step (such as Mel spectrograms), ensuring the temporal coherence of the generated standard acoustic features and their consistency with the conditions (modal fusion features).

[0093] This embodiment does not limit the generative acoustic model to an autoregressive model based on the Transformer architecture. For example, the generative acoustic model can also be a diffusion model.

[0094] To improve the accuracy of generating personalized standard acoustic feature sequences, this application provides another method for generating personalized standard acoustic feature sequences:

[0095] Step b1: Obtain the text corresponding to the speech to be evaluated.

[0096] If the text corresponding to the speech to be evaluated already exists, then the text can be obtained directly. For example, if the speech to be evaluated is the target user reading a specified text, then the specified text is the text corresponding to the speech to be evaluated. If the text corresponding to the speech to be evaluated does not exist, for example, in a free-speaking scenario where there is no text corresponding to the speech to be evaluated, then the text corresponding to the speech to be evaluated can be obtained by performing speech recognition on the speech to be evaluated. Based on the speech recognition model, performing speech recognition on the speech to be evaluated can simultaneously obtain the recognized text and hidden state features.

[0097] Step b2: Based on the prosodic annotation model, perform prosodic annotation on the text corresponding to the speech to be evaluated, and use the obtained prosodic annotation results as prosodic prior information.

[0098] The prosody annotation model is trained using training text labeled with prosody information.

[0099] This embodiment uses a pre-trained prosodic annotation model to perform prosodic annotation on the text corresponding to the speech to be evaluated, so as to obtain the prosodic information (such as stress, pause level, etc.) of the text corresponding to the speech to be evaluated, and uses the prosodic information of the text corresponding to the speech to be evaluated as the prosodic prior information.

[0100] Step b3: Using a generative acoustic model, based on the hidden state features of the speech to be evaluated and the timbre representation vector of the target user, supplemented by prosodic prior information, a personalized standard acoustic feature sequence is generated.

[0101] This embodiment additionally introduces prosodic prior information as a weak condition to guide the generative acoustic model to generate personalized standard acoustic feature sequences with more accurate prosodices.

[0102] This embodiment is based on text and timbre-driven principles, supplemented by prosodic priors to generate personalized standard acoustic feature sequences. Specifically, for example... Figure 2 As shown, the hidden state features of the speech to be evaluated, the timbre representation vector of the target user, and the prosodic prior information are fused (for example, the hidden state features of the speech to be evaluated are fused with the timbre representation vector of the target user based on the cross-attention mechanism, and the fused features are concatenated with the prosodic prior information; or, for example, the hidden state features of the speech to be evaluated, the timbre representation vector of the target user, and the prosodic prior information are directly concatenated) to obtain multimodal fusion features. The multimodal fusion features are then input into the generative acoustic model, which generates a personalized standard acoustic feature sequence based on the input multimodal fusion features.

[0103] As mentioned in the above embodiments, the generative acoustic model is trained using training speech, the timbre representation vector of the user to whom the training speech belongs, and the reference acoustic feature sequence corresponding to the training speech. In some embodiments of this application, the training process of the generative acoustic model is described.

[0104] like Figure 3 As shown, the training process for a generative acoustic model may include:

[0105] Step S301: Obtain training samples from the training dataset. The training samples include training speech, the timbre representation vector of the user to which the training speech belongs, and the reference acoustic feature sequence corresponding to the training speech.

[0106] A training dataset is pre-built, which includes multiple training samples. Each training sample includes training speech, the timbre representation vector of the user to which the training speech belongs, and the reference acoustic feature sequence corresponding to the training speech. It may also include the text corresponding to the training speech and the timbre representation vector of the standard speaker. When training the generative acoustic model, training samples are obtained from the training dataset.

[0107] The reference acoustic feature sequence corresponding to the training speech can be obtained in the following way: obtain the reading speech of the text corresponding to the training speech by a standard speaker to obtain the reference speech corresponding to the training speech (the reference speech corresponding to the training speech has standard pronunciation and rhythm), obtain the acoustic feature sequence of the reference speech corresponding to the training speech, and obtain the reference acoustic feature sequence corresponding to the training speech.

[0108] Step S302: Extract hidden state features representing the speech content from the training speech to obtain the hidden state features of the training speech.

[0109] Optionally, speech recognition can be performed on the training speech based on the speech recognition model to extract hidden state features that represent the speech content, thereby obtaining the hidden state features of the training speech.

[0110] Step S303: Using a generative acoustic model, generate a standard acoustic feature sequence based on the hidden state features of the training speech and the timbre representation vector of the user to whom the training speech belongs.

[0111] Specifically, the hidden state features of the training speech can be fused with the timbre representation vector of the user to whom the training speech belongs (for example, fusion based on cross-attention mechanism, or direct concatenation or summation) to obtain multimodal fusion features. The multimodal fusion features are then input into the generative acoustic model, which generates a standard acoustic feature sequence based on the input multimodal fusion features.

[0112] Optionally, prosodic annotation can be performed on the text corresponding to the training speech based on a pre-trained prosodic annotation model to obtain the prosodic information of the text corresponding to the training speech. This prosodic information is then used as prosodic prior information. A generative acoustic model is then used to generate a standard acoustic feature sequence based on the hidden state features of the training speech, the timbre representation vector of the user to which the training speech belongs, and the prosodic prior information. Specifically, the hidden state features of the training speech, the timbre representation vector of the user to which the training speech belongs, and the prosodic information of the text corresponding to the training speech are fused to obtain multimodal fusion features. These multimodal fusion features are then input into the generative acoustic model, which generates a standard acoustic feature sequence based on the input multimodal fusion features.

[0113] Step S304a: Determine the timbre consistency loss based on the generated standard acoustic feature sequence, the reference acoustic feature sequence corresponding to the training speech, the timbre representation vector of the user to which the training speech belongs, and the timbre representation vector of the standard speaker.

[0114] Optionally, the tone uniformity loss L can be calculated using the following formula. voice :

[0115] (1).

[0116] Among them, g i The acoustic features of the i-th frame in the standard acoustic feature sequence generated for the generative acoustic model, t i To train the acoustic features of the i-th frame in the reference acoustic feature sequence corresponding to the speech, v user To train the timbre representation vector of the user to which the speech belongs, v ref is the timbre representation vector of the standard speaker, and N is the total number of frames.

[0117] The purpose of introducing timbre consistency loss in this embodiment is to ensure that the timbre features of the standard acoustic feature sequence generated by the generative acoustic model based on the hidden state features of the training speech and the timbre representation vector of the user to which the training speech belongs are consistent with the timbre features of the user to which the training speech belongs.

[0118] Step S304b: Obtain the prosodic features of the generated standard acoustic feature sequence and the prosodic features of the reference acoustic feature sequence corresponding to the training speech, and determine the prosodic alignment loss based on the prosodic features of the generated standard acoustic feature sequence and the prosodic features of the reference acoustic feature sequence corresponding to the training speech.

[0119] Optionally, the prosodic alignment loss L can be calculated using the following formula. prosody :

[0120] (2).

[0121] in, This represents the prosodic features of a standard acoustic feature sequence generated by a generative acoustic model for training speech. This represents the prosodic features of the reference acoustic feature sequence corresponding to the training speech.

[0122] The purpose of introducing prosodic alignment loss in this embodiment is to ensure that the prosodic features of the standard acoustic feature sequence generated by the generative acoustic model based on the hidden state features of the training speech and the timbre representation vector of the user to which the training speech belongs are consistent with the prosodic features of the reference acoustic feature sequence. That is, to make the standard acoustic feature sequence generated by the generative acoustic model have standard prosody.

[0123] Step S304c: Determine the acoustic feature generation loss based on the acoustic feature prediction probability of the generative acoustic model.

[0124] For example, if the generative acoustic model is an autoregressive model based on the Transformer architecture, then the acoustic feature generation loss is an autoregressive generation loss, which can be expressed as a negative log-likelihood loss, as shown in the following equation:

[0125] (3).

[0126] in, Let θ be the predicted probability of the acoustic features in the i-th frame, θ be the model parameters, and N be the total number of frames.

[0127] The purpose of introducing acoustic feature generation loss in this embodiment is to ensure that the standard acoustic feature sequence generated by the generative acoustic model based on the hidden state features of the training speech and the timbre representation vector of the user to which the training speech belongs is accurate, conforms to the natural distribution of acoustic features, and is internally coherent.

[0128] As mentioned in the above embodiments, the generative acoustic model can also be a diffusion model. If the generative acoustic model is a diffusion model, then the acoustic feature generation loss can be a variational lower bound loss.

[0129] Step S305: Fuse the timbre consistency loss, prosodic alignment loss, and acoustic feature generation loss to obtain the fused loss.

[0130] In one possible implementation, the timbre consistency loss, prosodic alignment loss, and acoustic feature generation loss can be fused by direct summation, resulting in a fused loss L. total for:

[0131] L total =L voice + L prosody +L NLL (4).

[0132] This embodiment does not limit the fusion of timbre consistency loss, prosodic alignment loss, and acoustic feature generation loss by direct summation. For example, weights can be set for timbre consistency loss, prosodic alignment loss, and acoustic feature generation loss respectively, and then a weighted summation method can be used to fuse timbre consistency loss, prosodic alignment loss, and acoustic feature generation loss. The weights for timbre consistency loss, prosodic alignment loss, and acoustic feature generation loss can be set according to the actual application scenario.

[0133] Step S306: Update the parameters of the generative acoustic model based on the fusion loss.

[0134] The generative acoustic model is trained iteratively multiple times in the manner described above until the training termination condition is met (e.g., the model converges, or the preset number of training iterations is reached).

[0135] Through training, the generative acoustic model has learned the following ability: given a text content (represented by hidden state features) and a specified timbre (represented by timbre representation vector), it generates speech (represented by personalized acoustic feature sequence) that can read the text in a standard and fluent manner like a standard speaker, which realizes the fusion of timbre transfer and pronunciation / prosody standardization.

[0136] After training a generative acoustic model capable of generating personalized acoustic feature sequences, the trained generative acoustic model can be used to generate personalized standard acoustic feature sequences based on the hidden state features of the speech to be evaluated and the timbre representation vector of the target user. After obtaining the personalized standard acoustic feature sequences, personalized standard speech can be synthesized based on the personalized standard acoustic feature sequences. Then, the personalized standard speech can be used as the evaluation comparison benchmark to evaluate the pronunciation and prosody of the speech to be evaluated.

[0137] In some embodiments of this application, the process of evaluating the pronunciation and prosody of the speech to be evaluated using personalized standard speech as the evaluation comparison benchmark is described.

[0138] like Figure 4 As shown, using personalized standard speech as the evaluation benchmark, the process of evaluating the pronunciation and prosody of the speech to be evaluated may include:

[0139] Step S401: Obtain the phoneme-level and word-level timestamp information corresponding to the speech to be evaluated and the personalized standard speech, respectively.

[0140] Phoneme-level and word-level timestamp information is obtained by aligning the corresponding speech with the text corresponding to the speech to be evaluated.

[0141] It should be noted that some speech recognition models can output phoneme-level and word-level timestamp information when recognizing input speech. In this way, when obtaining the recognized text and hidden state features of the speech to be evaluated based on the speech recognition model, the phoneme-level and word-level timestamp information corresponding to the speech to be evaluated can be obtained at the same time.

[0142] If the speech recognition model cannot output phoneme-level and word-level timestamp information, the phoneme-level and word-level timestamp information corresponding to the speech to be evaluated and the personalized standard speech can be obtained as follows: For each speech in the speech to be evaluated and the personalized standard speech, a decoding network can be constructed based on the text corresponding to the speech, and an acoustic feature sequence can be extracted from the speech. The extracted acoustic feature sequence is decoded through the standard acoustic model and the decoding network to align the speech with the text corresponding to the speech to be evaluated, so as to obtain the phoneme-level and word-level timestamp information corresponding to the speech, that is, the start and end time of each phoneme and the start and end time of each word.

[0143] Step S402a: Combine the word-level timestamp information corresponding to the speech to be evaluated and the personalized standard speech to generate prosodic event information for the speech to be evaluated and the personalized standard speech respectively. By comparing the prosodic event information of the speech to be evaluated with the prosodic event information of the personalized standard speech, the prosodic evaluation result of the speech to be evaluated is obtained.

[0144] The prosodic event information of the personalized standard speech describes how the text should be read aloud under standard, beautiful pronunciation habits, while the prosodic event information of the speech to be evaluated describes how the target user actually reads it aloud.

[0145] The prosodic event information of the speech to be evaluated and the personalized standard speech may include, but is not limited to, one or more of the following types of prosodic events: stress events, sense group pauses, linking events, rising and falling tone events, etc.

[0146] For each speech segment in the evaluation speech and personalized standard speech, the repetition event information of that speech segment can be obtained as follows: Combining the word-level timestamp information corresponding to the speech segment, determine the average energy or average fundamental frequency of each word's speech segment. Based on the average energy or average fundamental frequency of each word's speech segment and the average energy or average fundamental frequency of adjacent word speech segments, determine whether each word is a repetition word. For the determined repetition words, generate structured repetition event information. The structured repetition event information may, but is not limited to, include event type, repetition word, timestamp information of the repetition word, repetition intensity, etc.

[0147] Specifically, the speech segments of each word can be extracted from the speech based on the word-level timestamp information corresponding to the speech, and the average energy or average fundamental frequency of the speech segments of each word can be determined. For each word, whether the word is a stressed word can be determined based on the average energy or average fundamental frequency of the speech segments of the word and the average energy or average fundamental frequency of the speech segments of adjacent words. For example, if the average energy of the speech segments of the word exceeds the average energy of the speech segments of adjacent words by a certain proportion (such as 120%), then the word is determined to be a stressed word.

[0148] For each speech item in the evaluation speech and personalized standard speech, the meaning group pause event information can be obtained as follows: Based on the word-level timestamp information corresponding to the speech item, determine the silence duration at the word boundary (i.e., the silence duration between two adjacent words). Based on the silence duration at the word boundary, determine whether the pause occurring at the word boundary is a meaning group pause. Generate structured meaning group pause event information for the determined meaning group pauses. It should be noted that a meaning group pause refers to the pause between several relatively complete and closely connected word groups (i.e., "meaning groups") when speaking or reading aloud, based on semantic and grammatical structure, rather than pausing after each word. Meaning group pause event information may include, but is not limited to, event type, pause position, pause level, pause start time, and pause duration.

[0149] Specifically, the process of determining whether a pause at a word boundary is a semantic pause based on the silence duration at the word boundary can include: if the silence duration at the word boundary is less than a preset first duration threshold (e.g., 50ms), then the pause at the word boundary is determined to be a short pause; if the silence duration at the word boundary is greater than a preset second duration threshold (the preset second duration threshold is greater than the preset first duration threshold, e.g., 200ms), then the pause at the word boundary is determined to be a semantic pause; and then, structured semantic pause event information is generated for the detected semantic pauses.

[0150] For each speech item in the evaluation speech and the personalized standard speech, the connected speech event information can be obtained as follows: The silence duration at the word boundary is determined by combining the corresponding word-level timestamp information, and the fundamental frequency curve of the speech is obtained. Whether connected speech has occurred is determined based on the silence duration at the word boundary and the continuity of the fundamental frequency curve at the word boundary. Structured connected speech event information is generated for words where connected speech has occurred. This connected speech event information may include, but is not limited to, the event type, the word in which connected speech occurred, the location of the connected speech, and the acoustic characteristics of the connected speech.

[0151] Specifically, for each word boundary, it can be determined whether the silence duration of the word boundary is less than the preset third duration threshold (e.g., 20ms), and whether the fundamental frequency curve at the word boundary transitions smoothly. If the silence duration of the word boundary is less than the preset third duration threshold, and the fundamental frequency curve at the word boundary transitions smoothly, then it is determined that the words before and after the word boundary have been read together.

[0152] For each speech item in the evaluation speech and personalized standard speech, the pitch rise and fall event information of that speech item can be obtained as follows: Combine the word-level timestamp information corresponding to that speech item, obtain the fundamental frequency sequence of the speech segment of the sentence-ending word, perform linear fitting on the fundamental frequency sequence of the speech segment of the sentence-ending word, determine the intonation direction (i.e., rising or falling tone) based on the slope of the fitted line, and generate structured pitch rise and fall event information for the determined intonation direction. Pitch rise and fall event information may include, but is not limited to, the event type, the word in which the intonation change occurs, the intonation outline (e.g., rising), and the slope.

[0153] The process of determining the tone direction based on the slope of the fitted line includes: if the slope of the fitted line is greater than the first slope threshold (e.g., 10 Hz / s), the tone direction is determined to be rising; if the slope of the fitted line is less than the second slope threshold (e.g., -10 Hz / s), the tone direction is determined to be falling.

[0154] The following table shows an example of prosodic event information for the speech to be evaluated and prosodic event information for a personalized standard speech:

[0155] Table 1. Examples of prosodic event information between the speech to be evaluated and the personalized standard speech.

[0156]

[0157] The process of obtaining prosodic evaluation results for the speech to be evaluated by comparing the prosodic event information of the speech to be evaluated with the prosodic event information of a personalized standard speech may include:

[0158] Step S402a-1: By comparing the prosodic event information of the speech to be evaluated with the prosodic event information of the personalized standard speech, the prosodic difference information between the speech to be evaluated and the personalized standard speech is obtained.

[0159] For example, prosodic event information includes stress event information, sense group pause event information, linking event information, and pitch rise and fall event information. Then, prosodic difference information can include the difference information between the speech to be evaluated and the personalized standard speech in stress events, the difference information in sense group pause events, the difference information in linking events, and the difference information in pitch rise and fall events. Among them, the difference information between the speech to be evaluated and the personalized standard speech in stress events may include, but is not limited to, time shift, intensity difference, word matching degree, etc. The difference information between the speech to be evaluated and the personalized standard speech in sense group pause events may include, but is not limited to, pause position deviation, pause duration difference, pause level matching degree, etc. The difference information between the speech to be evaluated and the personalized standard speech in linking events may include, but is not limited to, the difference in whether linking occurs, the difference in linking sound change features, etc. The difference information between the speech to be evaluated and the personalized standard speech in pitch rise and fall events may include, but is not limited to, intonation contour difference (the intonation contour is a curve shape formed by connecting a series of fundamental frequency points), slope difference, etc.

[0160] Step S402a-2: Based on the prosodic difference information, determine whether there are prosodic errors in the speech to be evaluated and the type of error if there are.

[0161] In one possible implementation, an error detection rule based on expert knowledge can be used to determine whether there are prosodic errors in the speech to be evaluated, and if so, the type of error. In another possible implementation, a pre-trained classifier can be used to determine whether there are prosodic errors in the speech to be evaluated, and if so, the type of error (e.g., "correct", "minor error", "serious error"), based on prosodic difference information.

[0162] Step S402a-3: Convert prosodic difference information into scores to obtain the prosodic score of the speech to be evaluated.

[0163] In one possible implementation, for each type of prosodic event, the difference between the speech to be evaluated and the personalized standard speech on that prosodic event can be quantified into a difference degree, and then the difference degree can be mapped to a score. For example, an exponential decay function can be used to map the difference degree to a score. After obtaining the scores of the speech to be evaluated on each type of prosodic event, the scores of the speech to be evaluated on each type of prosodic event can be fused to obtain the prosodic score of the speech to be evaluated. In another possible implementation, the scores of the speech to be evaluated on each type of prosodic event can be directly summed to obtain the prosodic score of the speech to be evaluated. In yet another possible implementation, the weights corresponding to each type of prosodic event can be preset, and then a weighted summation fusion method can be used to fuse the scores of the speech to be evaluated on each type of prosodic event to obtain the prosodic score of the speech to be evaluated.

[0164] Step S402b: Combining the phoneme-level timestamp information corresponding to the speech to be evaluated and the personalized standard speech respectively, perform phoneme-level pronunciation evaluation on the speech to be evaluated to obtain the pronunciation evaluation result of the speech to be evaluated.

[0165] The phoneme-level pronunciation assessment of the speech to be evaluated may include, but is not limited to, one or more of the following assessments: pronunciation accuracy assessment, vowel quality assessment, and consonant clarity assessment.

[0166] The process of evaluating the pronunciation accuracy of the speech to be evaluated at the phoneme level may include: obtaining the frame-level posterior probability corresponding to the speech to be evaluated; determining the phoneme-level pronunciation accuracy based on the phoneme-level timestamp information and the frame-level posterior probability; and obtaining the pronunciation accuracy evaluation result. The frame-level posterior probability corresponding to the speech to be evaluated is the probability estimate by the acoustic model of each frame of the speech to be evaluated belonging to each phoneme, given the known timestamp information of each phoneme.

[0167] The process of evaluating the vowel quality of the speech to be evaluated may include: obtaining the vowel speech segment to be evaluated from the speech to be evaluated based on the phoneme-level timestamp information corresponding to the speech to be evaluated; obtaining the standard vowel speech segment corresponding to the vowel speech segment to be evaluated from the personalized standard speech based on the phoneme-level timestamp information corresponding to the personalized standard speech; determining the formant distance between the vowel speech segment to be evaluated and the corresponding standard vowel speech segment; and obtaining the vowel quality evaluation result.

[0168] The formant distance between the vowel segment to be evaluated and the corresponding standard vowel segment can be calculated using the following formula:

[0169] VowelDistance = sqrt( (F1_user - F1_std)² + (F2_user - F2_std)² )(5).

[0170] Wherein, F1_user represents the first formant frequency of the vowel speech segment to be evaluated, F1_std represents the first formant frequency of the standard vowel speech segment corresponding to the vowel speech segment to be evaluated, F2_user represents the second formant frequency of the vowel speech segment to be evaluated, F2_std represents the second formant frequency of the standard vowel speech segment corresponding to the vowel speech segment to be evaluated, and VowelDistance is the formant distance between the vowel speech segment to be evaluated and the corresponding standard vowel speech segment, that is, the distance between the pronunciation point of the vowel speech segment to be evaluated and the pronunciation point of the corresponding standard vowel speech segment. The larger the distance, the farther the vowel pronunciation of the vowel speech segment to be evaluated deviates from the standard vowel pronunciation.

[0171] The process of evaluating consonant intelligibility in the speech to be evaluated may include: obtaining the consonant speech segment to be evaluated from the speech to be evaluated based on the phoneme-level timestamp information corresponding to the speech to be evaluated; if the consonant speech segment to be evaluated is a plosive speech segment, determining the plosive energy of the plosive speech segment, and determining the consonant intelligibility evaluation result based on the plosive energy; if the consonant speech segment to be evaluated is a fricative speech segment, determining the spectral centroid of the fricative based on the fricative speech segment, and determining the consonant intelligibility evaluation result based on the spectral centroid.

[0172] After obtaining the pronunciation evaluation results of the above dimensions, the pronunciation score of the speech to be evaluated can be determined based on the pronunciation evaluation results of the above dimensions.

[0173] After obtaining the pronunciation score and prosody score of the speech to be evaluated, the comprehensive score of the speech to be evaluated can be determined based on the pronunciation score and prosody score. For example, the pronunciation score and prosody score can be summed (or the weights corresponding to the pronunciation dimension and the prosody dimension can be preset respectively, and the comprehensive score can be obtained by weighted summation).

[0174] In some embodiments of this application, the speech pronunciation and prosody evaluation method may further include: generating an evaluation feedback report of the speech to be evaluated based on the pronunciation and prosody evaluation results of the speech to be evaluated.

[0175] The evaluation feedback report for the speech to be evaluated may, but is not limited to, include some or all of the following: the pronunciation and prosody evaluation scores of the speech to be evaluated; the acoustic curve comparison between the speech to be evaluated and the personalized standard speech; pronunciation and prosody error information of the speech to be evaluated (which may be presented in chronological order); and correction suggestions for the speech to be evaluated (e.g., "Your stress intensity on the word 'very' is only 70% of the standard; please try to increase the intensity of your pronunciation by about 30%...").

[0176] Optionally, the pronunciation and prosody evaluation score of the speech to be evaluated may include some or all of the following scores: pronunciation score of the speech to be evaluated, prosody score of the speech to be evaluated, and overall score of the speech to be evaluated.

[0177] Optionally, the acoustic curve comparison between the speech to be evaluated and the personalized standard speech may include, but is not limited to, the following types of acoustic curves: sound wave diagram, fundamental frequency curve, and energy curve.

[0178] Optionally, the pronunciation and prosody error information of the speech to be evaluated may include, but is not limited to, some or all of the following information: error type, error location (corresponding text and time point), severity, etc.

[0179] The speech pronunciation and prosody evaluation method provided in this application can dynamically generate personalized standard speech for the speech to be evaluated of the target user. The pronunciation and prosody standards, timbre and timbre of the personalized standard speech are consistent with those of the target user, and the speech content is consistent with the speech content of the speech to be evaluated. Thus, the personalized standard speech can be used as the evaluation comparison benchmark to accurately evaluate the pronunciation and prosody of the speech to be evaluated of the target user, realizing a fundamental change from "general" to "specific" evaluation benchmark.

[0180] The speech pronunciation and prosody evaluation method provided in this application has the following advantages: First, the evaluation comparison benchmark is more accurate. The timbre of the dynamically generated personalized standard speech is highly consistent with that of the target user, completely eliminating acoustic feature interference caused by timbre differences and significantly improving the accuracy of prosodic feature comparison. Second, the prosodic standard is more natural. The generative acoustic model can learn the optimal prosodic pattern from massive amounts of high-quality speech data, making the generated personalized standard speech prosody more natural, rich, and beautiful, thereby providing a more reasonable, human-like, and high-precision evaluation benchmark. The evaluation results are more in line with human auditory perception; thirdly, the feedback and guidance are personalized, adopting a "one-on-one" personalized guidance model. The evaluation benchmark (i.e., personalized standard speech) is fully adapted to the user's own timbre and vocal range, making the evaluation feedback more targeted and the guidance effect more significant; fourthly, the technology paradigm is intelligent, upgrading the technology paradigm from "static reference" to "dynamic mirroring". It can generate the optimal evaluation comparison benchmark in real time according to the specific user and speech content, greatly improving the flexibility and intelligence level, and laying a solid foundation for future expansion (such as adapting to different speaking styles).

[0181] This application also provides a speech pronunciation and prosody evaluation device, such as... Figure 5 As shown, the speech pronunciation and prosody evaluation device may include: a timbre representation vector acquisition unit 500, a speech acquisition unit 501 to be evaluated, a hidden state feature acquisition unit 502, a personalized standard acoustic feature sequence generation unit 503, a personalized standard speech synthesis unit 504, and a pronunciation and prosody evaluation unit 505.

[0182] The timbre representation vector acquisition unit 500 is used to acquire the timbre representation vector of the target user based on the registered voice of the target user.

[0183] The voice acquisition unit 501 is used to acquire the voice to be evaluated from the target user.

[0184] The hidden state feature acquisition unit 502 is used to extract hidden state features that represent the speech content of the speech to be evaluated.

[0185] The personalized standard acoustic feature sequence generation unit 503 is used to generate a personalized standard acoustic feature sequence based on the hidden state features and the timbre representation vector of the target user using a generative acoustic model. The generative acoustic model is trained using training speech, the timbre representation vector of the user to whom the training speech belongs, and a reference acoustic feature sequence corresponding to the training speech. The reference acoustic feature sequence is the acoustic feature sequence of a standard speaker reading the text corresponding to the training speech. The personalized standard acoustic feature sequence carries speech content consistent with the speech to be evaluated, and contains standard pronunciation and prosodic information, while maintaining consistency with the timbre features of the target user.

[0186] The personalized standard speech synthesis unit 504 is used to synthesize speech based on the personalized standard acoustic feature sequence to obtain personalized standard speech.

[0187] The pronunciation and prosody evaluation unit 505 is used to evaluate the pronunciation and prosody of the speech to be evaluated, using a personalized standard speech as the evaluation comparison benchmark.

[0188] In one possible implementation, the training objectives of the generative acoustic model include: ensuring that the timbre features of the standard acoustic feature sequence generated from the hidden state features of the training speech and the timbre representation vector of the user to which the training speech belongs are consistent with the timbre features of the user to which the training speech belongs; ensuring that the prosodic features of the standard acoustic feature sequence generated from the hidden state features of the training speech and the timbre representation vector of the user to which the training speech belongs are consistent with the prosodic features of the reference acoustic feature sequence; and ensuring that the standard acoustic feature sequence generated from the hidden state features of the training speech and the timbre representation vector of the user to which the training speech belongs is accurate, conforms to the natural distribution of acoustic features, and is internally coherent.

[0189] In one possible implementation, the speech pronunciation and prosody evaluation device may further include: a prosody prior information acquisition unit.

[0190] The prosodic prior information acquisition unit is used to acquire the text corresponding to the speech to be evaluated. Based on the prosodic annotation model, the text corresponding to the speech to be evaluated is prosodic annotated, and the prosodic annotation result is used as the prosodic prior information. The prosodic annotation model is trained using training text annotated with prosodic information.

[0191] The personalized standard acoustic feature sequence generation unit 503, when generating personalized standard acoustic feature sequences using a generative acoustic model based on hidden state features and the target user's timbre representation vector, is specifically used for:

[0192] By using a generative acoustic model, a personalized standard acoustic feature sequence is generated based on the hidden state features and the timbre representation vector of the target user, supplemented by prosodic prior information.

[0193] In one possible implementation, the personalized standard acoustic feature sequence generation unit 503, when generating a personalized standard acoustic feature sequence using a generative acoustic model based on hidden state features and the timbre representation vector of the target user, supplemented by prosodic prior information, is specifically used for:

[0194] By fusing hidden state features, the target user's timbre representation vector, and prosodic prior information, a multimodal fusion feature is obtained.

[0195] By using a generative acoustic model, personalized standard acoustic feature sequences are generated based on multimodal fusion features.

[0196] In one possible implementation, the personalized standard acoustic feature sequence generation unit 503, when generating a personalized standard acoustic feature sequence using a generative acoustic model based on multimodal fusion features, is specifically used for:

[0197] Using a generative acoustic model, personalized standard acoustic features are predicted frame by frame in an autoregressive manner based on multimodal fusion features, resulting in a personalized standard acoustic feature sequence.

[0198] In each prediction step, the personalized standard acoustic features for the current step are predicted based on the multimodal fusion features and the personalized standard acoustic features predicted in the previous step.

[0199] In one possible implementation, the pronunciation and prosody evaluation unit 505, when evaluating the pronunciation and prosody of the speech to be evaluated using a personalized standard speech as the evaluation comparison benchmark, is specifically used for:

[0200] Obtain the phoneme-level and word-level timestamp information corresponding to the speech to be evaluated and the personalized standard speech, respectively. The phoneme-level and word-level timestamp information is obtained by aligning the corresponding speech with the text corresponding to the speech to be evaluated.

[0201] By combining the word-level timestamp information corresponding to the speech to be evaluated and the personalized standard speech, prosodic event information of the speech to be evaluated and the personalized standard speech are generated respectively. By comparing the prosodic event information of the speech to be evaluated with the prosodic event information of the personalized standard speech, the prosodic evaluation result of the speech to be evaluated is obtained.

[0202] By combining the phoneme-level timestamp information corresponding to the speech to be evaluated and the personalized standard speech, the speech to be evaluated is subjected to phoneme-level pronunciation evaluation, and the pronunciation evaluation result of the speech to be evaluated is obtained.

[0203] In one possible implementation, when the pronunciation and prosody evaluation unit 505 generates prosodic event information for the speech to be evaluated and the personalized standard speech by combining the word-level timestamp information corresponding to the speech to be evaluated and the personalized standard speech respectively, it is specifically used for:

[0204] For each voice recording in the test voice and the personalized standard voice:

[0205] The average energy or average fundamental frequency of each word's speech segment is determined by combining the corresponding word-level timestamp information. Based on the average energy or average fundamental frequency of each word's speech segment and the average energy or average fundamental frequency of adjacent words' speech segments, it is determined whether each word is a stressed word. Structured stressed word event information is generated for the determined stressed words.

[0206] And / or, determine the silence duration of the word boundary based on the corresponding word-level timestamp information, determine whether the pause at the word boundary is a semantic pause based on the silence duration of the word boundary, and generate structured semantic pause event information for the determined semantic pause.

[0207] And / or, combine the corresponding word-level timestamp information to determine the silence duration of the word boundary, and obtain the fundamental frequency curve of the speech. Based on the silence duration of the word boundary and the continuity of the fundamental frequency curve at the word boundary, determine whether a connected speech has occurred, and generate structured connected speech event information for words that have been connected.

[0208] And / or, combine the corresponding word-level timestamp information to obtain the fundamental frequency sequence of the speech segment of the last word of the sentence, perform linear fitting on the fundamental frequency sequence, determine the intonation direction based on the slope of the fitted line, and generate structured rise and fall tone event information for the determined intonation direction.

[0209] In one possible implementation, when the pronunciation and prosody evaluation unit 505 compares the prosodic event information of the speech to be evaluated with the prosodic event information of the personalized standard speech to obtain the prosodic evaluation result of the speech to be evaluated, it is specifically used for:

[0210] By comparing the prosodic event information of the speech to be evaluated with the prosodic event information of the personalized standard speech, the prosodic difference information between the speech to be evaluated and the personalized standard speech is obtained.

[0211] Based on prosodic difference information, it is determined whether there are prosodic errors in the speech to be evaluated and the type of error if so. The prosodic difference information is then converted into a score to obtain the prosodic score of the speech to be evaluated.

[0212] In one possible implementation, when the pronunciation and prosody evaluation unit 505 performs phoneme-level pronunciation evaluation on the speech to be evaluated by combining the phoneme-level timestamp information corresponding to the speech to be evaluated and the personalized standard speech, and obtains the pronunciation evaluation result of the speech to be evaluated, it is specifically used for:

[0213] The frame-level posterior probability corresponding to the speech to be evaluated is obtained. Based on the phoneme-level timestamp information and the frame-level posterior probability corresponding to the speech to be evaluated, the phoneme-level pronunciation accuracy is determined, and the pronunciation accuracy evaluation result is obtained. Among them, the frame-level posterior probability is the probability estimate of each frame of speech belonging to each phoneme by the acoustic model under the premise that the timestamp information of each phoneme is known.

[0214] And / or, obtain the vowel speech segment to be evaluated based on the phoneme-level timestamp information corresponding to the speech to be evaluated, and obtain the standard vowel speech segment corresponding to the vowel speech segment to be evaluated based on the phoneme-level timestamp information corresponding to the personalized standard speech, determine the formant similarity between the vowel speech segment to be evaluated and the corresponding standard vowel speech segment, and obtain the vowel sound quality evaluation result.

[0215] And / or, obtain the consonant speech segment to be evaluated based on the phoneme-level timestamp information corresponding to the speech to be evaluated. If the consonant speech segment to be evaluated is a plosive speech segment, determine the plosive energy of the plosive speech segment and determine the consonant intelligibility evaluation result based on the plosive energy. If the consonant speech segment to be evaluated is a fricative speech segment, determine the spectral centroid of the fricative speech segment and determine the consonant intelligibility evaluation result based on the spectral centroid.

[0216] In one possible implementation, the speech pronunciation and prosody evaluation device may further include an evaluation feedback report generation unit.

[0217] The evaluation feedback report generation unit is used to generate an evaluation feedback report for the speech to be evaluated based on the evaluation results of the pronunciation and prosody of the speech to be evaluated.

[0218] The evaluation feedback report for the speech to be evaluated includes some or all of the following: the pronunciation and prosody evaluation score of the speech to be evaluated, the acoustic curve comparison between the speech to be evaluated and the personalized standard speech, the pronunciation and prosody error information of the speech to be evaluated, and the corresponding correction suggestions for the speech to be evaluated.

[0219] The speech pronunciation and prosody evaluation device provided in this application embodiment can generate personalized standard speech for the speech to be evaluated, which has the standard pronunciation and prosody, the timbre consistent with the timbre of the target user, and the speech content consistent with the speech content of the speech to be evaluated. Using this personalized standard speech as the evaluation comparison benchmark can greatly improve the accuracy and fairness of the evaluation. In addition, since the speech pronunciation and prosody evaluation device provided in this application embodiment dynamically generates personalized standard speech for the speech to be evaluated, it does not need to rely on standard speech pre-recorded for fixed text. Therefore, it can be flexibly applied to free speaking scenarios, expanding the application scope and practicality.

[0220] This application also provides an electronic device, which may include at least one processor and a memory connected to the processor.

[0221] The processor may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application; the memory may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage.

[0222] The memory is used to store computer programs, and the processor is used to execute the computer programs so that the electronic device can implement the speech pronunciation and prosody evaluation method provided in the above embodiments.

[0223] This application also provides a computer storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device is able to implement the speech pronunciation and prosody evaluation method provided in the above embodiments.

[0224] This application also provides a computer program product, including computer-readable instructions, which, when executed on an electronic device, enable the electronic device to implement the speech pronunciation and prosody evaluation method provided in the above embodiments.

[0225] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0226] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0227] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0228] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A method of speech articulation and prosody evaluation, characterized by, The method comprises the steps of: obtaining a speech to be evaluated of a target user; extracting a hidden state feature representing the content of the speech to be evaluated; generating a personalized standard acoustic feature sequence from the hidden state feature and a timbre feature vector of the target user by using a generative acoustic model, wherein the generative acoustic model is trained by using training speech, a timbre feature vector of a user to which the training speech belongs, and a reference acoustic feature sequence corresponding to the training speech, the reference acoustic feature sequence being an acoustic feature sequence of a reading speech of a standard speaker for a text corresponding to the training speech, the personalized standard acoustic feature sequence carrying the same speech content as the speech to be evaluated, containing standard pronunciation and prosody information, and having the same timbre feature as the target user; synthesizing speech from the personalized standard acoustic feature sequence to obtain a personalized standard speech; using the personalized standard speech as a reference for evaluation, evaluating the pronunciation and prosody of the speech to be evaluated.

2. The method of claim 1, wherein, The training target of the generative acoustic model comprises: making the timbre feature of the standard acoustic feature sequence generated from the hidden state feature of the training speech and the timbre feature vector of the user to which the training speech belongs consistent with the timbre feature of the user to which the training speech belongs; making the prosody feature of the standard acoustic feature sequence generated from the hidden state feature of the training speech and the timbre feature vector of the user to which the training speech belongs consistent with the prosody feature of the reference acoustic feature sequence; making the standard acoustic feature sequence generated from the hidden state feature of the training speech and the timbre feature vector of the user to which the training speech belongs accurate, consistent with the natural distribution of acoustic features, and coherent within the sequence.

3. The method of claim 1, wherein, Further comprising: obtaining a text corresponding to the speech to be evaluated; annotating the text corresponding to the speech to be evaluated based on a prosody annotation model to obtain a prosody annotation result as prosody prior information, wherein the prosody annotation model is trained by using training text annotated with prosody information; The method comprises the steps of: generating a personalized standard acoustic feature sequence from the hidden state feature and the timbre feature vector of the target user by using a generative acoustic model, supplemented by the prosody prior information.

4. The method of claim 3, wherein, The method comprises the steps of: fusing the hidden state feature, the timbre feature vector of the target user, and the prosody prior information to obtain a multi-modal fusion feature; generating a personalized standard acoustic feature sequence from the multi-modal fusion feature by using a generative acoustic model.

5. The method of claim 4, wherein, The method comprises the steps of: The personalized standard acoustic features are predicted in a self-recurrent manner according to the multi-modal fusion features by using a generative acoustic model, and a sequence of the personalized standard acoustic features is obtained. In each step of prediction, the personalized standard acoustic features of the current step are predicted according to the multi-modal fusion features and the personalized standard acoustic features of the previous step.

6. The method of claim 1, wherein, The pronunciation and prosody of the speech to be evaluated are evaluated by taking the personalized standard speech as an evaluation comparison reference, including: Obtaining phoneme-level and word-level timestamp information corresponding to the speech to be evaluated and the personalized standard speech respectively, the phoneme-level and word-level timestamp information being obtained by aligning the corresponding speech with the text corresponding to the speech to be evaluated; Combining the word-level timestamp information corresponding to the speech to be evaluated and the personalized standard speech respectively, generating prosody event information of the speech to be evaluated and the personalized standard speech respectively, and obtaining prosody evaluation results of the speech to be evaluated by comparing the prosody event information of the speech to be evaluated with the prosody event information of the personalized standard speech; Combining the phoneme-level timestamp information corresponding to the speech to be evaluated and the personalized standard speech respectively, performing phoneme-level pronunciation evaluation on the speech to be evaluated, and obtaining pronunciation evaluation results of the speech to be evaluated.

7. The method of claim 6, wherein the prosody evaluation is performed by using a prosody evaluation model trained by using a prosody evaluation dataset including a plurality of prosody evaluation examples. The combination of the word-level timestamp information corresponding to the speech to be evaluated and the personalized standard speech respectively includes: For each speech in the speech to be evaluated and the personalized standard speech: Combining the corresponding word-level timestamp information to determine the average energy or average fundamental frequency of the speech segment of each word, determining whether each word is a stress word according to the average energy or average fundamental frequency of the speech segment of each word and the average energy or average fundamental frequency of the speech segment of adjacent words, and generating structured stress event information for the determined stress word; And / or, determining the silence duration of the word boundary according to the corresponding word-level timestamp information, determining whether the pause at the word boundary is a sense group pause according to the silence duration of the word boundary, and generating structured sense group pause event information for the determined sense group pause; And / or, combining the corresponding word-level timestamp information to determine the silence duration of the word boundary, and obtaining the fundamental frequency curve of the speech, determining whether there is an elision according to the silence duration of the word boundary and the continuity of the fundamental frequency curve at the word boundary, and generating structured elision event information for the word with elision; And / or, combining the corresponding word-level timestamp information to obtain the fundamental frequency sequence of the speech segment of the last word of the sentence, performing linear fitting on the fundamental frequency sequence, determining the tone trend according to the slope of the straight line obtained by fitting, and generating structured rising and falling tone event information for the determined tone trend.

8. The method of claim 6, wherein the prosody evaluation is performed by using a prosody evaluation model. The prosody evaluation results of the speech to be evaluated are obtained by comparing the prosody event information of the speech to be evaluated with the prosody event information of the personalized standard speech, including: The prosody difference information of the speech to be evaluated and the personalized standard speech is obtained by comparing the prosody event information of the speech to be evaluated with the prosody event information of the personalized standard speech. According to the prosody difference information, it is determined whether the prosody of the speech to be evaluated has errors and the error type when errors exist, and the prosody difference information is converted into a score to obtain a prosody score of the speech to be evaluated.

9. The method of claim 6, wherein the prosody evaluation is performed by using a prosody evaluation model trained by using a prosody evaluation dataset including a plurality of prosody evaluation examples. The prosodic evaluation of the speech to be evaluated is performed at the phoneme level by combining the phoneme-level timestamp information corresponding to the speech to be evaluated and the individualized standard speech, to obtain a pronunciation evaluation result of the speech to be evaluated, including: The frame-level posterior probability corresponding to the speech to be evaluated is obtained, and the pronunciation accuracy at the phoneme level is determined according to the phoneme-level timestamp information and the frame-level posterior probability corresponding to the speech to be evaluated, to obtain a pronunciation accuracy evaluation result; wherein the frame-level posterior probability is a probability estimation of an acoustic model for each frame of speech belonging to each phoneme under the premise that the timestamp information of each phoneme is known; And / or, the to-be-evaluated vowel speech segment is obtained according to the phoneme-level timestamp information corresponding to the speech to be evaluated, and the standard vowel speech segment corresponding to the to-be-evaluated vowel speech segment is obtained according to the phoneme-level timestamp information corresponding to the individualized standard speech, to determine the formant distance between the to-be-evaluated vowel speech segment and the corresponding standard vowel speech segment, and obtain a vowel quality evaluation result; And / or, the to-be-evaluated consonant speech segment is obtained according to the phoneme-level timestamp information corresponding to the speech to be evaluated, if the to-be-evaluated consonant speech segment is a stop consonant speech segment, the burst energy of the stop consonant is determined according to the stop consonant speech segment, the consonant articulation evaluation result is determined according to the burst energy, if the to-be-evaluated consonant speech segment is a fricative consonant speech segment, the spectral centroid of the fricative consonant is determined according to the fricative consonant speech segment, and the consonant articulation evaluation result is determined according to the spectral centroid.

10. The method of claim 1-9, wherein, Further comprising: According to the pronunciation and prosody evaluation results of the speech to be evaluated, an evaluation feedback report of the speech to be evaluated is generated. The evaluation feedback report of the speech to be evaluated includes part or all of the following contents: pronunciation and prosody evaluation scores of the speech to be evaluated, acoustic curve comparison between the speech to be evaluated and the individualized standard speech, pronunciation and prosody error information of the speech to be evaluated, and correction suggestions corresponding to the speech to be evaluated.

11. An electronic device, comprising: The electronic device includes at least one processor and a memory connected to the processor, wherein: The memory is used to store a computer program; The processor is used to execute the computer program, so that the electronic device can implement the speech pronunciation and prosody evaluation method according to any one of claims 1-10.

12. A computer storage medium, characterized in that The storage medium carries one or more computer programs, which can enable the electronic device to implement the speech pronunciation and prosody evaluation method according to any one of claims 1-10 when the one or more computer programs are executed by the electronic device.

13. A computer program product, characterised in that, The computer readable instructions enable the electronic device to implement the speech pronunciation and prosody evaluation method according to any one of claims 1-10 when the computer readable instructions are run on the electronic device.

Images