Voice evaluation method and device, medium and program product

Abstract

The invention provides a voice evaluation method, a voice evaluation device, a non-transitory storage medium and a computer program product. The voice evaluation method comprises the following steps: extracting a first person voice with a first tone based on a to-be-evaluated voice; converting the voice of the first person into voice with a second tone; and obtaining an evaluation result of the voice of the first person based on the converted voice with the second tone and the standard voice of the one or more accent.

Description

Technical Field

[0001] The present disclosure relates to speech processing technology, and more particularly, to a speech evaluation method, a speech evaluation device, a non-transitory storage medium, and a computer program product. Background Art

[0002] In the process of English learning, the learning of spoken language requires continuous feedback and guidance from teachers to make progress. However, the improvement of spoken language level requires a large amount of practice, and the cost of manual teaching and error correction is high, and it is also restricted by time and space. With the progress of artificial intelligence technology, the technology of automatic speech evaluation for spoken audio has also developed. Therefore, it has become an irresistible trend to improve the automatic speech evaluation technology through artificial intelligence technology. Summary of the Invention

[0003] According to one aspect of the present disclosure, there is provided a speech evaluation method, including: extracting a first person's speech with a first timbre based on the speech to be evaluated; converting the first person's speech into a speech with a second timbre; obtaining an evaluation result of the first person's speech based on the converted speech with the second timbre and the standard speech of one or more accents.

[0004] According to one aspect of the present disclosure, there is provided a speech evaluation device, including: an extraction device configured to extract a first person's speech with a first timbre based on the speech to be evaluated; a conversion device configured to convert the first person's speech into a speech with a second timbre; an evaluation device configured to obtain an evaluation result of the first person's speech based on the converted speech with the second timbre and the standard speech of one or more accents.

[0005] According to another aspect of the present disclosure, there is provided a speech evaluation device, including: a memory for storing instructions; a processor for reading the instructions in the memory and executing the method according to at least one embodiment of the present disclosure.

[0006] According to another aspect of the present disclosure, there is provided a non-transitory storage medium having instructions stored thereon, wherein when the instructions are executed by a processor, the processor is caused to execute the method according to at least one embodiment of the present disclosure.

[0007] According to one aspect of the present disclosure, there is provided a computer program product having instructions stored thereon, wherein when the instructions are executed by a processor, the processor is caused to execute the method according to at least one embodiment of the present disclosure. Brief Description of the Drawings

[0008] To more clearly illustrate the technical solutions in the embodiments of the present disclosure or the prior art, the following will briefly introduce the accompanying drawings required for the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only some embodiments of the present disclosure. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0009] Figure 1 The flowchart of the speech evaluation method according to at least one embodiment of the present disclosure is shown.

[0010] Figure 2 The schematic diagram of the process of performing speech evaluation by the speech evaluation system according to at least one embodiment of the present disclosure is shown.

[0011] Figure 3 The schematic diagram of the identity vector extraction model according to at least one embodiment of the present disclosure is shown.

[0012] Figure 4 The schematic diagram of the four types of speech that make up each training sample according to at least one embodiment of the present disclosure is shown.

[0013] Figure 5 It is a schematic diagram of the training and inference stages of the feature enhancement model according to at least one embodiment of the present disclosure.

[0014] Figure 6 The speech transformation module from the children's voice of the first timbre to the adult voice of the second timbre according to at least one embodiment of the present disclosure is shown.

[0015] Figure 7 The block diagram of the phoneme classification model according to at least one embodiment of the present disclosure is shown.

[0016] Figure 8 The block diagram of the speech evaluation device according to at least one embodiment of the present disclosure is shown.

[0017] Figure 9 The block diagram of the speech evaluation device according to at least one embodiment of the present disclosure is shown. Detailed Embodiments

[0018] Now, specific embodiments of the present disclosure will be described in detail. Examples of the present disclosure are illustrated in the accompanying drawings. Although the present disclosure will be described in conjunction with specific embodiments, it will be understood that it is not intended to limit the present disclosure to the described embodiments. On the contrary, it is intended to cover modifications, variations, and equivalents included within the spirit and scope of the present disclosure as defined by the appended claims. It should be noted that the method steps described herein can be implemented by any functional block or functional arrangement, and any functional block or functional arrangement can be implemented as a physical entity or a logical entity, or a combination of both.

[0019] Traditional speech evaluation algorithms model speech signals based on Hidden Markov Models (HMMs) plus deep learning models. They train acoustic models with a large amount of data and then score the pronunciation quality according to the likelihood values output by the acoustic models, enabling students to improve their spoken pronunciation based on the pronunciation quality scores.

[0020] However, traditional speech evaluation systems have some significant drawbacks:

[0021] 1. Relying solely on a speech recognition system for speech evaluation makes it difficult to solve the problem of noisy environments in usage scenarios. Moreover, many child users need parental guidance during oral practice, so the voices of parents and children often appear simultaneously during recording, resulting in a deteriorated recognition effect for children's speech.

[0022] 2. General American English evaluations cannot adapt to Chinese accent evaluations, as there are significant differences between Chinese and American pronunciations. If only the general American standard English is used to evaluate children's Chinese-accented English, the evaluation scores may be too low to accurately assess the correctness of children's pronunciations.

[0023] 3. A Chinese English evaluation system cannot simultaneously adapt to American English evaluations.

[0024] 4. There has always been an inaccuracy problem in the evaluation of Chinese children's English. Since it is very difficult to obtain the standard Chinese English of children aged 3 - 6, it is difficult to train the pronunciation evaluation model for young children users.

[0025] The present disclosure provides a method and device for diagnosing and correcting Chinese children's oral English. The method includes: extracting the speech of a first person (such as a child) with a first timbre from the speech to be evaluated; converting the speech of the first person into speech with a second timbre (such as an adult); obtaining the evaluation results (including pronunciation quality, pronunciation fluency, pronunciation speed measurement, etc.) of the speech of the first person based on the converted speech with the second timbre and the standard speech of one or more accents (including Chinese English, American English, British English, Indian English, Japanese English, Korean English, etc.). Then, a detailed evaluation feedback report can be generated based on the evaluation results for children to improve their spoken pronunciation. It realizes the diagnosis of the oral English of children and adult learners with different accents including Chinese accents and provides feedback close to that of a real teacher.

[0026] Note that in the present disclosure, the speech of a certain child is used as the speech of the first person for illustration, but it can also be applied to the speech of other people (such as another child or an adult) by modifying the speech model of this child to that of others. Here, the first person is only for reference and is not restricted to the first person or only one person or other restrictive meanings.

[0027] Figure 1 The flowchart of a speech evaluation method according to at least one embodiment of the present disclosure is shown.

[0028] As Figure 1 shown, the speech evaluation method includes step 110, step 120, and step 130.

[0029] In step 110, a first-person speech with a first timbre is extracted based on the speech to be evaluated.

[0030] The first-person speech can be the speech of a first child, and the first timbre can be a child timbre.

[0031] In step 120, the first-person speech is converted into a speech with a second timbre.

[0032] The second timbre can be an adult timbre. Since there is usually only the standard English speech of adults, but no standard English speech with a child timbre, traditionally using the standard English speech of adults to evaluate the pronunciation of children may lead to inaccuracy. Therefore, by converting the speech with a child timbre into a speech with an adult timbre, the subsequent speech evaluation can be made more accurate. Of course, conversely, the first timbre can be an adult timbre and the second timbre can be a child timbre, which can also well convert the adult timbre into a child timbre for evaluation in the case of having the standard English speech of children. Or the first timbre can be a male timbre and the second timbre can be a female timbre, etc., and no further examples are given here.

[0033] In step 130, an evaluation result of the first-person speech is obtained based on the converted speech with the second timbre and the standard speech of one or more accents.

[0034] One or more accents include at least one of the following: Chinglish, American English, British English, Indian English, Japanese English, Korean English.

[0035] In this way, creatively considering the influence of different timbres and different accents can reduce the influence of environmental noise and parental speech, and more standardly evaluate the pronunciation quality (including pronunciation quality, pronunciation fluency, pronunciation speed measurement, etc.) of different spoken languages (such as Chinglish, American English) of the first-person speech of children, for example, so that children can improve their spoken pronunciation.

[0036] Figure 2 The schematic diagram of the process of performing speech evaluation by a speech evaluation system according to at least one embodiment of the present disclosure is shown.

[0037] Figure 2The children's voice enhancement module 210, children's voice extraction module 220, text regularization module 230, voice alignment module 240, voice evaluation module 250, pronunciation diagnosis module 260, and voice voice conversion module 270 of the terminal device are shown.

[0038] The children's voice enhancement module 210 and children's voice extraction module 220 of the terminal device can extract the first children's voice based on the voice to be evaluated in step 110.

[0039] These modules are introduced below.

[0040] First, the children's voice enhancement module 210 is described.

[0041] The goal of the children's voice enhancement module 210 is to enhance the features of the first children's voice so that the children's voice extraction module 220 can extract, for example, the first children's voice from the voice to be evaluated including other sounds (including adult voices, other noises) through the enhanced features of the first children's voice, achieving the effects of separation and noise reduction.

[0042] To extract the first person's voice with the first timbre based on the voice to be evaluated, the children's voice enhancement module 210 can extract splicing features from the voice to be evaluated; extract the identity vector of the first person from the recorded voice of the first person through the identity vector extraction model; and obtain the enhanced features of the first person of the voice to be evaluated based on the splicing features and the identity vector of the first person through the feature enhancement model. The children's voice extraction module 220 can extract the first person's voice from the voice to be evaluated based on the enhanced features of the first person of the voice to be evaluated.

[0043] The specific description is as follows:

[0044] First, the extraction of the splicing features and identity vector of the first child is described.

[0045] Define the splicing features.

[0046] Adopt: 13 filter bank (FBANK) features (select the coefficients of the first 12 dimensions, and then splice the energy of the current voice frame, a total of 13 dimensions); 3-dimensional pitch features; timbre features for splicing to obtain the splicing features.

[0047] FBANK features are features extracted from the voice signal by applying a set of triangular filters in different frequency bandwidths. These filters are usually distributed according to the Mel Scale, which is a frequency scale based on human auditory perception and simulates the sensitivity of the human ear to different frequencies. Therefore, FBANK features can better reflect the actual perception of sound by the human auditory system.

[0048] The FBANK features can be calculated through the following steps: 1. Pre-emphasis: Since high-frequency signals are more likely to attenuate during audio recording, the loss of high-frequency components may lead to unclear formants of phonemes, making the acoustic model less capable of modeling these phonemes. Pre-emphasis is a first-order high-pass filter that can increase the energy of the high-frequency part of the signal, thereby enhancing the representation of high-frequency components. 2. Framing: The speech signal is a time-varying and non-stationary signal, but it can be considered time-invariant and stationary within a short time range (such as 10 - 30 ms). Therefore, the speech signal can be segmented into multiple short-time frames for processing. 3. Windowing: After framing, a finite-length window needs to be used for weighted processing, that is, windowing. Commonly used windowing functions include the Hamming window and the Hanning window. 4. Fast Fourier Transform (FFT): Perform FFT transformation on each windowed frame signal to obtain spectral information. 5. Mel filter bank processing: Apply a set of Mel filters to the result of the FFT, and each filter corresponds to a specific frequency range. These filters filter the spectrum and extract information in different frequency bands. 6. Energy calculation: Calculate the energy or power of the output of each filter to obtain the FBANK features.

[0049] The 3D pitch features can be extracted by the short-time autocorrelation function method, and this algorithm is more suitable for feature extraction in a noisy environment. Of course, other algorithms can also be used for extraction.

[0050] For the timbre feature, considering that the ratio of the heights of formants determines the timbre, and noting that the timbre difference is an important basis for distinguishing children (especially before the age of 14) from adults, therefore, the difference in the logarithms of formants F1, F2, and F3 is used as the timbre feature. Among them, F1 (the first formant) is mainly related to the height (degree of opening) of the vowel. The greater the degree of opening of the vowel, the lower the frequency of F1; conversely, the smaller the degree of opening of the vowel, the higher the frequency of F1. For example, in the vowel / a / (such as the vowel in "father"), the frequency of F1 is usually low; while in the vowel / i / (such as the vowel in "sit"), the frequency of F1 is high. F2 (the second formant) is mainly related to the front-back position of the vowel. The more forward the vowel pronunciation position, the higher the frequency of F2; conversely, the more backward the vowel pronunciation position, the lower the frequency of F2. For example, in the vowels / i / and / e / (such as the vowels in "sit" and "bed"), the frequency of F2 is high; while in the vowels / u / and / o / (such as the vowels in "boot" and "goat"), the frequency of F2 is low. F3 (the third formant) is mainly related to whether the vowel is rounded or not, but this relationship is not as clear as that of F1 and F2. The F3 frequency of rounded vowels (such as / u / and / o / ) is usually low; while the F3 frequency of unrounded vowels (such as / i / and / a / ) may be high. However, F3 is also affected by other factors, such as the height and front-back position of the tongue, etc., so its change is relatively complex.

[0051] Therefore, by performing the above three types of feature extraction on the speech, the extracted features can be spliced.

[0052] This spliced feature is specifically defined based on the speech feature differences between children and adults, and can more accurately extract children's speech rather than adult speech or noise in at least one embodiment of the present disclosure.

[0053] Secondly, the identity vector extraction of children is performed.

[0054] Figure 3 The schematic diagram of the identity vector extraction model according to at least one embodiment of the present disclosure is shown.

[0055] The identity vector of the first child uses Figure 3 the identity vector extraction model to complete the extraction.

[0056] As Figure 3 shown, when each child user registers, it is necessary to guide the child user to record a certain duration of speech (the recorded speech of the first person) by reading the specified content, and extract the identity vector (d-vector) of the first person from the recorded speech of the first person through the identity vector extraction model. The identity vector extraction model is as Figure 3As shown in the figure, it includes: a Mel Frequency Cepstral Coefficient (MFCC) feature extraction module 301 for extracting Mel Frequency Cepstral Coefficient (MFCC) features from the recorded speech of the first person; an N1-layer fully connected neural network layer 302 for receiving the Mel Frequency Cepstral Coefficient (MFCC) features as input and outputting a first output, where N1 is a positive integer; an N2-layer convolutional neural network layer (convolutional layer) 303 for receiving the first output as input and outputting a second output, where N2 is a positive integer; a global average pooling layer 304 for receiving the second output as input and outputting a third output; a concatenation layer 306 for concatenating the third output with the fourth output output by inputting the third output and the first output into a dimensionality reduction layer 305 to obtain a fifth output; a multi-head attention mechanism layer 307 for receiving the fifth output as input to obtain an identity vector of the first person. The identity vector of the first person is for identifying the speech features of different speakers.

[0057] The Mel Frequency Cepstral Coefficient (MFCC) features can be obtained through multiple steps such as pre-emphasis, framing, windowing, fast Fourier transform (FFT), Mel filter bank, taking the logarithm, discrete cosine transform (DCT), etc.

[0058] Since the above-mentioned concatenated features can obtain more discriminative speech features of the first person, and the identity vector of the first person strengthens the features of the first person, therefore, in at least one embodiment of the present disclosure, by inputting both the concatenated features and the identity vector of the first person into a feature enhancement model to obtain the enhanced features of the first person, such features that can better distinguish the speech of the first person and others can be obtained, so as to more accurately extract the speech of the first person.

[0059] Next, the training process of the feature enhancement model is described.

[0060] First, the training data preparation is described.

[0061] Prepare children's speech corpus and adult speech corpus in English. The data format can be a 16-bit waveform audio format (Waveform Audio Format, WAV) with a size of 16k and its corresponding content text (the content text can be in the form of words, phrases or phonemes, etc., but ultimately must be converted into a phoneme sequence format). Preferably, each corpus contains only one speaker, and at least 4 training samples are collected for each child speaker.

[0062] Figure 4 The figure shows a schematic diagram of 4 kinds of speech that make up each training sample according to at least one embodiment of the present disclosure.

[0063] For the first child speaker, select a clean audio of the first child speaker.

[0064] Select another piece of children's speech belonging to the first child speaker as the reference audio.

[0065] The mixed speech includes a first mixed audio (such as the mixed audio 1 in Figure 4 ) generated by mixing a recorded speech (clean audio) of a first person and the speech from one or more second persons (such as another child speaker), and a second mixed audio (such as the mixed audio 2 in Figure 4 ) generated by mixing a recorded speech (clean audio) of a first person and other speech from one or more other speakers (such as an adult speaker). That is, select one piece of children's audio and one piece of adult audio from another child speaker different from the first child speaker, and superimpose them on the clean audio of the first child speaker to generate two mixed audios (such as the mixed audios 1 and 2 in Figure 4 ) as the mixed speech.

[0066] The reference audio, mixed audio 1, and 2 {reference audio , mixed audio } can be called the training audio triple.

[0067] If the duration of the randomly selected mixed speech is less than the clean audio of the first child speaker, more adult speech can be randomly selected and spliced with it until the duration of the spliced speech is greater than or equal to the duration of the clean audio. This can ensure that the duration of the training samples is sufficient.

[0068] For the training samples, the speech can be framed with a frame shift of 10 ms and a frame length of 25 ms (160 sampling points, window size 400 sampling points).

[0069] Next, the training and inference processes of the feature enhancement model are described.

[0070] Figure 5 is a schematic diagram of the training and inference phases of the feature enhancement model according to at least one embodiment of the present disclosure.

[0071] The training phase scheme steps are as follows:

[0072] Obtain the mixed speech 501 including the recorded speech of a first person (where the mixed speech includes mixed audio 1 and 2).

[0073] Extract the FBANK features 502 of the mixed speech.

[0074] Input the FBANK features of the mixed speech and the identity vector 500 of the first person into the feature enhancement model 503 to obtain the enhanced speech features 504.

[0075] The feature enhancement model 503 is trained by minimizing the loss function 507 between the enhanced speech features 504 and the FBANK features 506 of a recorded speech (e.g., clean audio or reference audio) 505 of the first person.

[0076] Among them, the goal of minimizing the loss function is to maximize the similarity between the features output by the feature enhancement model and the FBANK features of the reference audio. The feature enhancement model trained in this way can extract features that are the same as or similar to the speaking audio (e.g., reference audio) of the first person from the speech to be evaluated, which includes various human voices or noises, thus helping to accurately extract the speaking audio of the first person, filtering out the interference of other various human voices or noises, and improving the accuracy of subsequent evaluation of the speaking audio of the first person.

[0077] Next, the inference process of the feature enhancement model is described.

[0078] As before, when each child user registers, the child user needs to be guided to record a certain duration of speech (the recorded speech of the first person) by reading the specified content, and then the identity vector d-vector of the first person is extracted for each user according to these speeches and stored in the system voiceprint library. During the inference process, the same identity vector d-vector 500 of the first person still needs to be used.

[0079] Extract the FBANK speech features from the speech to be evaluated (input audio) 508.

[0080] The FBANK speech features are concatenated with the three-dimensional pitch features and timbre features of the speech to be evaluated to obtain the concatenated features 510.

[0081] The identity vector d-vector 500 of the first person and the obtained concatenated features 510 are input into the trained feature enhancement model to extract the enhanced features of the first person of the speech to be evaluated. Here, creatively, the concatenated features of the speech to be evaluated when the user is currently speaking and the identity vector d-vector of the first person are sent into the feature enhancement model together, and the enhanced features of the first person of the speech to be evaluated obtained can be more characteristic of the first person than the individual features of the speech to be evaluated, thus better distinguishing the first person.

[0082] Based on the enhanced features of the first person of the speech to be evaluated, the speech of the first person is extracted from the speech to be evaluated and sent to the speech evaluation module 511 for speech quality evaluation.

[0083] Next, the speech evaluation module is introduced. The speech evaluation module can complete step 120 of converting the speech of the first person into speech with a second timbre and step 130 of obtaining the evaluation result of the speech of the first person based on the converted speech with a second timbre and the standard speech of one or more accents.

[0084] Return reference Figure 2 , Next, the voice conversion module 270 in the voice evaluation module will be introduced. The voice conversion module 270 can complete the conversion of the first person's voice into a voice with a second timbre in step 120. Here, the second timbre can be an adult timbre.

[0085] Generally, the distinction between a child's timbre and an adult's timbre can be made by age division. For example, the timbre of a person under 14 years old is called a child's timbre, and the timbre of a person over 18 years old is called an adult's timbre. Of course, this is only an example and not a limitation. Alternatively, the distinction between a child's timbre and an adult's timbre can be made by FBANK features. The frequency distribution of a child's voice signal may be more concentrated in the high-frequency band, resulting in larger coefficients of the FBANK features in the high-frequency band and smaller coefficients in the low-frequency band. The frequency distribution of an adult's voice signal may be more extensive, resulting in significant manifestations of the FBANK features in multiple frequency bands, and the coefficients in both the low-frequency and high-frequency bands may be larger.

[0086] Figure 6 Shows a voice conversion module from a child's voice of a first timbre to an adult's voice of a second timbre according to at least one embodiment of the present disclosure.

[0087] The voice conversion module includes a voice feature MFCC extractor 600, a speaker encoder 601, a phoneme recognition content encoder 602, a decoder 603, and a content encoder 604.

[0088] The voice feature MFCC extractor 600 extracts MFCC features from the first person's voice segment. The voice feature MFCC extractor 600 can first remove the silence from the first person's voice segment, normalize the volume, and convert the sampling rate to 24KHZ as described above. It is segmented with a frame length of 50ms and a frame shift of 12.5ms, and then the MFCC features are extracted. The MFCC features are input to the speaker encoder 601 and the phoneme recognition content encoder 602. The speaker encoder 601 is the same as Figure 3 shown, and can output a speaker identity vector d-vector from the input extracted MFCC features. The obtained speaker identity vector d-vector is input to a fully connected neural network layer and passes through the Mish activation function to obtain a first output. At the same time, the phoneme recognition content encoder 602 predicts a content vector for the input MFCC features, and the output content vector is concatenated with the first output to obtain a second output, and the second output is input to the decoder 603.

[0089] The content encoder 602 can encode the real phoneme sequence corresponding to the first person's voice segment to obtain a real content vector, concatenate it with the first output to obtain a third output, and input it to the decoder 603.

[0090] Among them, the decoder 603 includes a Dropout layer 6031 with a total of M1 layers, a fully connected neural network layer (FC layer) 6032, a Swish activation function layer 6033, and a bidirectional long short-term memory (BI-Lstm) layer 6034 with M2 layers, as well as a predicted speech feature MFCC extractor 6035 and a general voice generator 6036 for generating voices with a second timbre for the output of the predicted speech feature MFCC extractor 6035, where M1 and M2 are positive integers.

[0091] The Dropout layer 6031 works by randomly dropping (i.e., setting to zero) a certain proportion of neurons or units in the network during the training process.

[0092] The fully connected neural network layer 6032 includes the following steps: Input and weights: The input of the fully connected layer usually comes from the output of the previous layer. Each input is multiplied by a weight coefficient, which will be continuously optimized during the training process. Weighted sum: The sum of the products of all inputs and their corresponding weights, plus a bias term, constitutes the net input of the neuron.

[0093] The mathematical expression of the Swish activation function of the Swish activation function layer 6033 is usually:

[0094] Swish(x) = x * σ(x) = x / (1 + e^(-x))

[0095] Among them, σ(x) represents the Sigmoid function, that is, σ(x) = 1 / (1 + e^(-x)).

[0096] The Swish activation function layer 6033 performs a non-linear transformation to produce the output of the neuron.

[0097] The bidirectional long short-term memory (BI-Lstm) layer 6034 involves two long short-term memory (LSTM) networks running in parallel: the forward LSTM and the backward LSTM. These two networks process the forward and backward information of the input sequence respectively, and then combine their outputs to produce the final prediction result.

[0098] The predicted speech feature MFCC extractor 6035 reconstructs the speech features to obtain the predicted speech feature MFCC.

[0099] The general voice generator 6036 outputs the voice that converts the child's timbre to the adult's timbre and then converts it to standard English. Note that the "standard" here refers to the standard of English pronunciation.

[0100] Among them, in order to train the decoder, standard adult English data can be input for training. The standard adult English data is obtained by collecting the audio data of the following speakers reading English: the speaker reading Chinglish data; the speaker reading American English speakers. A total of speakers, and the speech of each speaker recording English is not less than sentences, so that the total standard adult English data is sentences, where , , are positive integers.

[0101] Input the standard adult English data into Figure 6 the voice conversion module shown in ), so as to train the decoder 603 by minimizing the difference between the value obtained by performing MFCC features on the standard adult English data and the value of the predicted speech feature MFCC output by the predicted speech feature MFCC extractor 6035 (loss function

[0102] Among them, Figure 6 the loss function shown in is the difference between the predicted content vector and the true content vector, and is used to measure the difference degree of the two output speech features in the same space during the training process. Use to represent, and the calculation method is as follows:

[0103]

[0104] In the calculation formula, , represents the element of the indicator matrix. respectively represent the indices of the phoneme recognition content encoding and the true content phoneme sequence length, represents the true phoneme sequence length, which is also the vector encoding length after the text is one-hot encoded and passed through the content encoder. In the formula the calculation method is as shown in the following formula. Among them , respectively represent the output vector of the phoneme recognition content encoder and the true phoneme sequence vector.

[0105]

[0106] To train the decoder, it can be achieved by minimizing the loss function and the loss function (such as their sum, or their product, or the weighted sum of both, etc.).

[0107] After the decoder is trained, the voice conversion module Figure 6 shown can be used to convert the children's English data with the first timbre into the voice-converted standard English data with the second timbre.

[0108] In this way, creatively converting the children's English data with the first timbre into the voice-converted standard English data with the second timbre solves the problem of being unable to obtain children's standard English data for comparison with the voices read by children, solves the mapping problem of the voice spaces with different timbres, and can compare the children's voices converted into adult voices with the existing adult standard English data or perform other processing.

[0109] Return Figure 2 , before the voice conversion module 270, a voice alignment module 240 is also required to align the extracted first-person voice with the text after regularization by the text regularization module 230 for the text read by the first person to obtain voice alignment information, and then input the voice alignment information into the voice evaluation module 250 to judge the phoneme classification, and then perform a pronunciation diagnosis on the phoneme classification through the pronunciation diagnosis module 260. The voice alignment information output by the voice alignment module 240 is also used by the voice conversion module 270 to align the speaker's voice and the text read by the speaker. The voice alignment module 240 solves the contradiction between the requirements of model generalization and pronunciation correction, and can provide accurate alignment information and accurately correct pronunciation.

[0110] The voice evaluation module 250 and the pronunciation diagnosis module 260 can complete the evaluation result of the first-person voice obtained from the converted voice with the second timbre and the standard voice with one or more accents in step 130.

[0111] The voice evaluation module 250 can input the converted voice with the second timbre into a phoneme classification model to obtain the posterior probability of each phoneme of the first-person voice. The pronunciation diagnosis module 260 can calculate the pronunciation quality based on the posterior probability of each phoneme of the first-person voice; calculate the pronunciation fluency based on the pauses of the first-person voice; calculate the pronunciation speed measure based on the pronunciation speed metric of the first-person voice, which measures the number of correct words read per unit time; and obtain the evaluation result of the first-person voice based on the pronunciation quality, pronunciation fluency, and pronunciation speed measure.

[0112] Note that a phoneme is the smallest unit or the smallest speech segment that constitutes a syllable, and it is the smallest sound unit in a language that can distinguish meanings. Phonemes can be divided into two major categories: vowel phonemes and consonant phonemes. Vowel phonemes can be further divided into monophthongs and diphthongs. Monophthongs are like [i], , [u], and diphthongs are like [ei], [ai], . Consonant phonemes can include plosives (such as [p], [t], [k]), fricatives (such as [s], , [f]), affricates, etc. In addition, consonant phonemes can be further subdivided according to the place of articulation and the manner of articulation, such as labial sounds (such as [p], [b], [m]), dental sounds (such as [θ], [ð], [s], [z]), palatal sounds, etc., as well as voiceless consonants, voiced consonants, aspirated sounds, and unaspirated sounds.

[0113] Figure 7 A block diagram of a phoneme classification model according to at least one embodiment of the present disclosure is shown.

[0114] The phoneme classification model is trained in the following manner: Input standard speech of one or more accents (such as standard Chinese English and standard American English) and standard speech converted from standard speech with a first timbre to standard speech with a second timbre (i.e., voice conversion standard English) into the phoneme classification model to obtain a phoneme classification result; train the phoneme classification model by minimizing the loss function between the phoneme classification result and the phoneme classification label.

[0115] The phoneme classification model includes: a K-layer time-delay neural network TDNN layer for extracting features of standard speech of one or more accents and standard speech with a second timbre, where K is a positive integer; different branch layers for receiving features of different standard speeches, and each branch layer includes 2 fully connected layers and 1 softmax layer.

[0116] That is, all data of the phoneme classification model share the underlying time-delay neural network TDNN layer. For the top layer, voice conversion standard English, standard Chinese English, and standard American English have their respective branches to learn their respective weights, and finally, through the same fully connected layers and softmax, the classification result is obtained for calculating the loss function. The idea is that the shared TDNN layer is used to extract the underlying features of each speech information, and finally, before phoneme recognition, through two layers of non-linear transformation, it is mapped into the same space. The phoneme classification model is composed of multiple TDNN layers, two fully connected layers, and one softmax layer. Except for the two fully connected layers, other layers are shared by all data, and the fully connected layers are calculated separately on different data sets, that is, each maintains its own parameters.

[0117] Input the voice with the second timbre converted from the voice with the first timbre extracted from the voice to be evaluated into the phoneme classification model to obtain the posterior probabilities of each phoneme of the first person's voice. For example, generally, it is determined that the phoneme corresponding to the maximum posterior probability among the posterior probabilities of each phoneme in the above example is the phoneme uttered in the currently evaluated voice, such as [θ].

[0118] To make the phoneme classification more accurate, the present application creatively samples speakers from the database at any time according to a uniform distribution speakers, and respectively convert the children's voices into the voices of speakers, where M is a positive integer. Extract features from the voices of speakers respectively, and obtain the posterior probabilities through the M-layer TDNN posterior probability calculation network. Calculate the average value of the posterior probabilities of the same phoneme for speakers, that is, obtain the posterior probability of this phoneme. Store the posterior probability of this phoneme as the basic measurement value used by the pronunciation diagnosis module 260 to calculate the pronunciation quality.

[0119] The following introduces the specific steps of the pronunciation diagnosis module 260.

[0120] The pronunciation diagnosis module 260 can, based on the posterior probabilities of each phoneme of the first person's voice, determine that the cut-off time boundary of the first phoneme among each phoneme is the time position where the posterior probability of the first phoneme exceeds the probability maximum value; based on the cut-off time boundary of the first phoneme among each phoneme, determine the phoneme duration of the first phoneme; calculate the logarithmic average of the posterior probabilities of all speech frames staying in the first speech state within the phoneme duration of the first phoneme as the pronunciation quality; based on the logarithmic average of the posterior probabilities of the first phoneme, determine the phoneme read by the first person; compare the determined phoneme read by the first person with the phoneme of the text actually read by the first person to determine the pronunciation quality.

[0121] Specifically, define the posterior probability of the phoneme in the phoneme state as the logarithmic average of the posterior probabilities of all speech frames staying in this phoneme state. In this way, the computational consumption in calculating the duration of all phoneme states is avoided. The calculation time is only limited to the state duration. The correlation between the same state types on different phonemes is often not high.

[0122]

[0123] For the phoneme The acoustic observation vector within the occurrence time period (which contains a series of eigenvalue extracted from the acoustic signal. These eigenvalues can be acoustic parameters such as amplitude, frequency, phase, etc., or frequency domain or time-frequency domain features obtained through transformation (such as Fourier transform, wavelet transform, etc.).). S is the set of states of phonemes (the states of different phonemes can be determined through data clustering). respectively represent the start frame and the end frame of phoneme alignment.

[0124]

[0125] Assume that N is the number of HMM states of the phoneme and the posterior probability of the pronunciation of the phoneme is defined as the average of the posterior probabilities of all states contained in the phoneme:

[0126]

[0127] The present disclosure creatively uses the calculation of Truncated Phone Log-Posterior Probability (TPLPP) to measure the posterior probability value in the stable stage of the phoneme. The truncated time boundary of the phoneme is calculated as the time position where the posterior probability value at this point exceeds the maximum probability value (for example,). Therefore, the defined truncated start time and end time are respectively changed from to and the TPLPP calculation formula is as follows:

[0128] In the formula, represents the coefficient, represents the maximum posterior probability value within the interval, and the whole represents a discounted value of the maximum probability value.

[0129]

[0130]

[0131]

[0132]

[0133] Among them, N is the total number of phonemes, is the probability of observing O for the phoneme , is the prior probability of the phoneme , is the set of all phonemes, represents the length of the truncated observation sequence, a represents a coefficient, such as 0.8, represents the maximum a posteriori probability value within the interval, a× as a whole represents a discounted value of the maximum probability value. Assuming that the prior probabilities of all phonemes are equal, that is , the above formula is simplified to the following formula:

[0134]

[0135] Finally, for take the average over M speakers to obtain the following :

[0136]

[0137] Next, based on the logarithmic average of the posterior probabilities of the first phoneme , determine the phoneme read by the first person (for example, if the logarithmic average of a certain phoneme is the highest, then it is determined that the first person reads that phoneme); compare the determined phoneme read by the first person with the phonemes in the text actually read by the first person to determine the pronunciation quality. For example, if the determined phoneme read by the first person is [s], but the phoneme in the text actually read by the first person is [θ], then the pronunciation quality is determined to be poor.

[0138] Next, describe the process of calculating the pronunciation fluency based on the pauses in the speech of the first person.

[0139] The measurement of pronunciation fluency is used for two aspects: the measurement of the duration of pauses and the measurement of the pause rhythm in the student's reading speech. If the pause time is too long, it indicates that the student's fluency is poor. Among them, the measurement of the pause rhythm can reflect the student's ability of continuous thinking. Too many pauses also indicate that the student's fluency is poor.

[0140] First, examine the pause correctness measurement. Examine whether there are appropriate pauses in the speech read by the user at the main punctuation marks in the text read by the user.

[0141] Classify the pauses in the speech of the first person into short pauses with a pause time less than or equal to the first threshold , medium pauses with a pause time greater than the first threshold and less than or equal to the second threshold , and long pauses with a pause time greater than the second threshold and less than or equal to the third threshold . As shown in the following table:

[0142]

[0143] Based on the time alignment information, the pause correctness metric is determined according to the degree to which short pauses, medium pauses, and long pauses meet the standard pause conditions of punctuation marks. Here, the time alignment information can be the alignment information between the first-person speech and the text read by the first person, that is, to know which part of the text the current first-person speech is reading. In this way, the present application can creatively examine the pause correctness metric by considering whether the pauses in the first-person speech correspond to the corresponding punctuation marks in the text.

[0144] Secondly, consider the overall pause distribution metric. The present application creatively examines the overall distribution of the user's pauses throughout the paragraph. The pause duration distribution can examine the fluency and rhythm measure of the user's reading of the article. Among them, the duration of the pauses in the first-person speech is extracted as a time series random signal; the time series random signal is subjected to a frequency domain conversion to obtain a frequency signal; based on the frequency distribution of the frequency signal after the frequency domain conversion, the pronunciation fluency is determined, where the more high-frequency components in the frequency distribution, the worse the pronunciation fluency.

[0145] Specifically, the pause durations after all the words in the user's reading recording are extracted and regarded as a time series random signal (the non-pause duration is 0, the sampling interval uses the average duration of a word, and the punctuation mark positions are not included). The Fourier transform is completed for this signal, and the time sampling interval is set to a fixed length T, where T is the minimum interval time value between two adjacent pause times in the current sequence. For the positions without pauses, the time length is filled with 0. After the Fourier transform, the frequency information of the reading pause time can be obtained, that is, the frequency and degree information of the occurrence of pauses. If there are more relatively high-frequency components, it means that the user's reading pauses frequently and the rhythm is poor. On the contrary, the fluency and rhythm are relatively normal. In this way, the fluency level and rhythm information of the user's entire reading can be measured to obtain the pronunciation fluency.

[0146] The calculation formula for the above process is as follows.

[0147] The pause duration sequence is expressed as: represents the total number of pauses.

[0148] The Fourier transform of the pause duration sequence is implemented according to the Fourier standard formula as follows:

[0149]

[0150] In the above formula, represents the value of the k-th component of the fluency. The k value represents the number of frequencies included during the pause time. The richer the included frequency information (for example, the more k), the more unstable the user's internal pause pattern tends to be, and the weaker the ability to control the reading. On the contrary, the less the connotative frequency information, the fewer the number of pauses, the more stable the user's reading rhythm, and the stronger the ability to control the reading.

[0151] Next, the pronunciation speed measure is described.

[0152] Specifically, based on the pronunciation speed metric of the first person's voice, which measures the number of correctly pronounced words read per unit time, the pronunciation speed measure is calculated, including: determining, according to the posterior probability of each phoneme, the number of words composed of phonemes whose posterior probability per unit time is greater than or equal to a predetermined threshold as the number of correctly read words per unit time; and determining the pronunciation speed measure according to the number of correctly read words, where the more the number of correctly read words per unit time, the better the pronunciation speed measure.

[0153] The pronunciation speed measure can describe the number of correctly pronounced words read by the user per unit time, and based on this number, it can be determined which proficiency level the user is at.

[0154] In summary, after obtaining the pronunciation quality, pronunciation fluency, and pronunciation speed measure, the evaluation result of the first person's voice can be obtained based on the pronunciation quality, pronunciation fluency, and pronunciation speed measure. The evaluation result can present the respective scores of the pronunciation quality, pronunciation fluency, and pronunciation speed measure, or combine the scores of the pronunciation quality, pronunciation fluency, and pronunciation speed measure to obtain a total score, which is not limited here. In this way, according to the various features of the creativity of the present application, the multi-dimensional evaluation results enable English learners to obtain more comprehensive feedback.

[0155] In summary, according to the various features of the creativity of the present application, the influence of environmental noise and the parent's voice can be reduced, and the pronunciation quality (including pronunciation quality, pronunciation fluency, pronunciation speed measure, etc.) of different spoken languages (such as Chinglish, American English) of the first person's voice of, for example, a child can be evaluated more standardly for the child to improve their spoken pronunciation.

[0156] Figure 8 FIG. shows a block diagram of a speech evaluation device 800 according to at least one embodiment of the present disclosure.

[0157] The speech evaluation device 800 includes: an extraction device 810, a conversion device 820, and an evaluation device 830.

[0158] The extraction device 810 can be configured to extract the first person's voice with a first timbre based on the voice to be evaluated.

[0159] The conversion device 820 can be configured to convert the first person's voice into a voice with a second timbre.

[0160] The evaluation device 830 can be configured to obtain the evaluation result of the first person's voice based on the converted voice with the second timbre and the standard voice of one or more accents.

[0161] In some embodiments, the first person's voice is the voice of a first child, the first timbre is a child's timbre, the second timbre is an adult's timbre, and one or more accents include at least one of the following: Chinglish, American English, British English, Indian English, Japanese English, Korean English.

[0162] In some embodiments, extracting the first person's voice with the first timbre from the voice to be evaluated includes: extracting splicing features from the voice to be evaluated; extracting the identity vector of the first person from the recorded voice of the first person through an identity vector extraction model; obtaining the enhanced feature of the first person of the voice to be evaluated based on the splicing features and the identity vector of the first person through a feature enhancement model; and extracting the first person's voice from the voice to be evaluated based on the enhanced feature of the first person of the voice to be evaluated.

[0163] In some embodiments, the splicing features include the splicing of the following features: 13-dimensional filter bank FBANK features, three-dimensional pitch features, and timbre features.

[0164] In some embodiments, the feature enhancement model is trained in the following manner: obtaining a mixed voice including the recorded voice of the first person; extracting the FBANK features of the mixed voice; inputting the FBANK features of the mixed voice and the identity vector of the first person into the feature enhancement model to obtain enhanced voice features; and training the feature enhancement model by minimizing the loss function between the enhanced voice features and the FBANK features of a recorded voice of the first person.

[0165] In some embodiments, the mixed voice includes a first mixed audio generated by mixing the recorded voice of the first person and the voices of one or more second persons, and a second mixed audio generated by mixing the recorded voice of the first person and other voices of one or more other speakers.

[0166] In some embodiments, the identity vector extraction model includes: an FBANK feature extraction module for extracting FBANK features from the recorded voice of the first person; an N1-layer fully connected neural network layer for receiving the FBANK features as input and outputting a first output, where N1 is a positive integer; an N2-layer convolutional layer for receiving the first output as input and outputting a second output, where N2 is a positive integer; a global average pooling layer for receiving the second output as input and outputting a third output; a splicing layer for splicing the third output with a fourth output output by inputting the first output into a dimensionality reduction layer to obtain a fifth output; and a multi-head attention mechanism layer for receiving the fifth output as input to obtain the identity vector of the first person.

[0167] In some embodiments, converting the first person's speech into speech with a second timbre includes: converting the content of the first person's speech into text by a speech recognizer; performing phonetic posterior probability PPG encoding on the text to obtain a content vector; concatenating the first output obtained by passing the identity vector of the first person through a fully connected neural network and a Mish activation function with the content vector to obtain a second output, and inputting the second output into a decoder to output speech with a second timbre, where the decoder includes a total of M1 layers of Dropout layers, fully connected neural network layers, Swish activation function layers, and M2 layers of bidirectional long short-term memory BI-Lstm layers, as well as a Fourier transform magnitude spectrum layer and a general vocoder for generating speech with a second timbre for the output of the decoder; wherein, the decoder is trained by minimizing the loss function between the value obtained by performing Fourier transform on the first person's speech and the value output by the Fourier transform magnitude spectrum layer.

[0168] In some embodiments, obtaining an evaluation result of the first person's speech based on the converted speech with a second timbre and the standard speech of one or more accents includes: inputting the converted speech with a second timbre into a phoneme classification model to obtain the posterior probability of each phoneme of the first person's speech; calculating the pronunciation quality based on the posterior probability of each phoneme of the first person's speech; calculating the pronunciation fluency based on the pauses of the first person's speech; calculating the pronunciation speech rate by measuring the number of correctly pronounced words read per unit time based on the pronunciation speech rate of the first person's speech; and obtaining the evaluation result of the first person's speech based on the pronunciation quality, pronunciation fluency, and pronunciation speech rate.

[0169] In some embodiments, the phoneme classification model is trained in the following manner: inputting the standard speech of one or more accents and the standard speech converted from the standard speech with a first timbre to the second timbre into the phoneme classification model to obtain a phoneme classification result; and training the phoneme classification model by minimizing the loss function between the phoneme classification result and the phoneme classification label.

[0170] In some embodiments, the phoneme classification model includes: K layers of time delay neural network TDNN layers for extracting the features of the standard speech of one or more accents and the standard speech with a second timbre, where K is a positive integer; and different branch layers for receiving the features of different standard speeches, where each branch layer includes 2 fully connected layers and 1 softmax layer.

[0171] In some embodiments, the pronunciation quality is calculated based on the posterior probabilities of the phonemes of the first person's speech, including: based on the posterior probabilities of the phonemes of the first person's speech, determining the truncation time boundary of the first phoneme among the phonemes as the time position where the posterior probability of the first phoneme exceeds the maximum probability; based on the truncation time boundary of the first phoneme among the phonemes, determining the phoneme duration of the first phoneme; calculating the logarithmic average of the posterior probabilities of all speech frames staying in the first speech state within the phoneme duration of the first phoneme; based on the logarithmic average of the posterior probabilities of the first phoneme, determining the phonemes read by the first person; and comparing the determined phonemes read by the first person with the phonemes of the text actually read by the first person to determine the pronunciation quality.

[0172] In some embodiments, the pronunciation fluency is calculated based on the pauses in the first person's speech, including: classifying the pauses in the first person's speech into short pauses with a pause time less than or equal to a first threshold, medium pauses with a pause time greater than the first threshold and less than or equal to a second threshold, and long pauses with a pause time greater than the second threshold and less than or equal to a third threshold; based on the time alignment information, determining the pause correctness metric according to the degree to which the short pauses, medium pauses, and long pauses meet the standard pause conditions of punctuation marks; extracting the duration of the pauses in the first person's speech as a time series random signal; performing a frequency domain transformation on the time series random signal to obtain a frequency signal; and determining the pronunciation fluency based on the frequency distribution of the frequency signal after the frequency domain transformation, wherein the more high-frequency components in the frequency distribution, the worse the pronunciation fluency.

[0173] In some embodiments, the pronunciation speed measure is calculated based on the pronunciation speed metric of the first person's speech, which measures the number of correctly read words per unit time, including: according to the posterior probabilities of the phonemes, determining the number of words composed of phonemes whose posterior probabilities per unit time are greater than or equal to a predetermined threshold as the number of correctly read words per unit time; and determining the pronunciation speed measure according to the number of correctly read words, wherein the more the number of correctly read words per unit time, the better the pronunciation speed measure.

[0174] In this way, the influence of environmental noise and the speech of parents can be reduced, and the pronunciation quality (including pronunciation quality, pronunciation fluency, pronunciation speed measure, etc.) of different spoken languages (such as Chinglish, American English) of the first person's speech, such as that of children, can be more standardly evaluated for children to improve their spoken pronunciation.

[0175] Figure 9 The block diagram of a speech evaluation device according to at least one embodiment of the present disclosure is shown.

[0176] The speech evaluation device may include a processor (H1); a storage medium (H2), coupled to the processor (H1), and storing computer-executable instructions therein for performing the steps of the various methods of at least one embodiment of the present disclosure when executed by the processor.

[0177] The processor (H1) may include, but is not limited to, for example, one or more processors or microprocessors, etc.

[0178] The storage medium (H2) may include, but is not limited to, for example, random access memory (RAM), read-only memory (ROM), flash memory, EPROM memory, EEPROM memory, registers, computer storage media (such as hard disks, floppy disks, solid state drives, removable disks, CD-ROMs, DVD-ROMs, Blu-ray discs, etc.).

[0179] In addition, the speech evaluation device may further include (but is not limited to) a data bus (H3), an input / output (I / O) bus (H4), a display (H5), and input / output devices (H6) (such as, for example, keyboards, mice, speakers, etc.).

[0180] The processor (H1) may communicate with external devices (H5, H6, etc.) via the I / O bus (H4) through a wired or wireless network (not shown).

[0181] The storage medium (H2) may also store at least one computer-executable instruction for performing the steps of the various functions and / or methods in the embodiments described in this technology when run by the processor (H1).

[0182] In one embodiment, the at least one computer-executable instruction may also be compiled into or constitute a software product, and when one or more computer-executable instructions are run by the processor, the steps of the various functions and / or methods in the embodiments described in this technology are performed.

[0183] In one embodiment, a computer-readable storage medium is provided, having instructions stored thereon, such as computer-readable instructions. When the computer-readable instructions are run by a processor, the various methods described above may be performed. The computer-readable storage medium includes, but is not limited to, for example, volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory, etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disks, flash memory, etc. For example, the computer-readable storage medium may be connected to a computing device such as a computer, and then, when the computing device runs the computer-readable instructions stored on the computer-readable storage medium, the various methods described above may be performed.

[0184] The block diagrams of the devices, apparatuses, equipment, and systems involved in the present disclosure are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open-ended terms, meaning "including but not limited to", and can be used interchangeably with each other. The words "or" and "and" used herein refer to the phrase "and / or", and can be used interchangeably with it, unless the context clearly indicates otherwise. The phrase "such as" used herein refers to the phrase "such as but not limited to", and can be used interchangeably with it.

[0185] In addition, the steps and devices in each embodiment of the present disclosure are not limited to being implemented in a certain embodiment. In fact, according to the concept of the present disclosure, relevant partial steps and partial devices in each embodiment of the present disclosure can be combined to conceive new embodiments, and these new embodiments are also included within the scope of the present disclosure.

Claims

1. A speech evaluation method, comprising: extracting a first person's speech with a first timbre from the speech to be evaluated; converting the first person's speech into a speech with a second timbre; obtaining an evaluation result of the first person's speech based on the converted speech with the second timbre and the standard speech of one or more accents.

2. The method according to claim 1, wherein the first person's speech is the speech of a first child, the first timbre is a child timbre, the second timbre is an adult timbre, and the one or more accents include at least one of the following: Chinglish, American English, British English, Indian English, Japanese English, Korean English.

3. The method according to claim 1, wherein the extracting a first person's speech with a first timbre from the speech to be evaluated includes: extracting splicing features from the speech to be evaluated; extracting the identity vector of the first person from the recorded speech of the first person through an identity vector extraction model; obtaining the enhanced feature of the first person of the speech to be evaluated based on the splicing features and the identity vector of the first person through a feature enhancement model; extracting the first person's speech from the speech to be evaluated based on the enhanced feature of the first person of the speech to be evaluated.

4. The method according to claim 3, wherein, The splicing features include the splicing of the following features: 13-dimensional filter bank FBANK features, three-dimensional pitch features, timbre features.

5. The method according to claim 3, wherein the feature enhancement model is trained in the following manner: obtaining a mixed speech including the recorded speech of the first person; extracting the FBANK features of the mixed speech; inputting the FBANK features of the mixed speech and the identity vector of the first person into the feature enhancement model to obtain enhanced speech features; training the feature enhancement model by minimizing the loss function between the enhanced speech features and the FBANK features of a recorded speech of the first person.

6. The method according to claim 5, wherein the mixed speech includes a first mixed audio generated by mixing the recorded speech of the first person and the speech of one or more second persons, and a second mixed audio generated by mixing the recorded speech of the first person and other speech of one or more other speakers.

7. The method according to claim 3, wherein The identity vector extraction model includes: a Mel cepstrum MFCC feature extraction module for extracting FBANK features from the recorded speech of the first person; an N1-layer fully connected neural network layer for receiving the Mel cepstrum MFCC features as inputs to output a first output, where N1 is a positive integer; an N2-layer convolutional layer for receiving the first output as an input to output a second output, where N2 is a positive integer; a global average pooling layer for receiving the second output as an input to output a third output; a splicing layer for splicing the third output and the fourth output output by the dimensionality reduction layer with the first output input thereto to obtain a fifth output; a multi-head attention mechanism layer for receiving the fifth output as an input to obtain the identity vector of the first person.

8. The method according to claim 1, wherein The converting the first person's speech into a speech with a second timbre includes: Convert the content of the first person's speech into text through a speech recognizer; Perform phonetic posterior probability PPG encoding on the text to obtain a content vector; Concatenate the first output obtained by passing the identity vector of the first person through a fully connected neural network and a Mish activation function with the content vector to obtain a second output, and input the second output into a decoder to output the speech with the second timbre, where the decoder includes a total of M1 layers of Dropout layers, fully connected neural network layers, Swish activation function layers, and M2 layers of bidirectional long short-term memory BI-Lstm layers, as well as a Fourier transform magnitude spectrum layer and a general vocoder for generating sound with the second timbre for the output of the decoder, where M1 and M2 are positive integers; Wherein, the decoder is trained by minimizing the loss function between the value obtained by performing Fourier transform on the first person's speech and the value output by the Fourier transform magnitude spectrum layer.

9. The method according to claim 1, wherein The evaluation result of the first person's speech obtained based on the converted speech with the second timbre and the standard speech of one or more accents includes: Input the converted speech with the second timbre into a phoneme classification model to obtain the posterior probability of each phoneme of the first person's speech; Calculate the pronunciation quality based on the posterior probability of each phoneme of the first person's speech; Calculate the pronunciation fluency based on the pauses in the first person's speech; Calculate the pronunciation speech rate measure based on the pronunciation speech rate of the first person's speech, which measures the number of correctly read words per unit time; Obtain the evaluation result of the first person's speech based on the pronunciation quality, pronunciation fluency, and pronunciation speech rate measure.

10. The method according to claim 9, wherein, Train the phoneme classification model in the following way: Input the standard speech of one or more accents and the standard speech converted from the standard speech with the first timbre to the second timbre into the phoneme classification model to obtain a phoneme classification result; Train the phoneme classification model by minimizing the loss function between the phoneme classification result and the phoneme classification label.

11. The method according to claim 10, wherein the phoneme classification model includes: K layers of time delay neural network TDNN layers for extracting the features of the standard speech of one or more accents and the standard speech with the second timbre, where K is a positive integer; A different branch layer for receiving the features of different standard speeches, where each branch layer includes 2 fully connected layers and 1 softmax layer.

12. The method according to claim 9, wherein, The calculation of the pronunciation quality based on the posterior probability of each phoneme of the first person's speech includes: Based on the posterior probability of each phoneme of the first person's speech, determine the truncation time boundary of the first phoneme among the phonemes as the time position where the posterior probability of the first phoneme exceeds the probability maximum value; Based on the truncation time boundary of the first phoneme among the phonemes, determine the phoneme duration of the first phoneme; Calculate the logarithmic average value of the posterior probabilities of all speech frames staying in the first speech state within the phoneme duration of the first phoneme; Determine the phoneme read by the first person based on the logarithmic average value of the posterior probability of the first phoneme; The pronunciation quality is determined by comparing the phonemes pronounced by the determined first person with the phonemes of the text actually pronounced by the first person.

13. The method according to claim 9, wherein calculating the pronunciation fluency based on the pauses in the first person's speech includes: classifying the pauses in the first person's speech into short pauses with a pause time less than or equal to a first threshold, medium pauses with a pause time greater than the first threshold and less than or equal to a second threshold, and long pauses with a pause time greater than the second threshold and less than or equal to a third threshold; based on the time alignment information, determining a pause correctness metric according to the degree to which the short pauses, the medium pauses, and the long pauses meet the standard pause conditions of punctuation marks; extracting the duration of the pauses in the first person's speech as a time series random signal; performing a frequency domain transformation on the time series random signal to obtain a frequency signal; determining the pronunciation fluency based on the frequency distribution of the frequency signal after the frequency domain transformation, wherein the more high-frequency components in the frequency distribution, the worse the pronunciation fluency.

14. The method according to claim 9, wherein calculating the pronunciation speed measure based on the pronunciation speed metric of the first person's speech, which measures the number of correctly pronounced words per unit time, includes: determining, according to the posterior probabilities of the respective phonemes, the number of words composed of phonemes with posterior probabilities greater than or equal to a predetermined threshold per unit time as the number of correctly pronounced words per unit time; determining a pronunciation speed measure according to the number of correctly pronounced words, wherein the more the number of correctly pronounced words per unit time, the better the pronunciation speed measure.

15. A speech evaluation device, comprising: an extraction device configured to extract the speech of a first person with a first timbre based on the speech to be evaluated; a conversion device configured to convert the speech of the first person into speech with a second timbre; an evaluation device configured to obtain an evaluation result of the speech of the first person based on the converted speech with the second timbre and the standard speech of one or more accents.

16. A speech evaluation device, comprising: a memory for storing instructions; a processor for reading the instructions in the memory and executing the method according to any one of claims 1-14.

17. A non-transitory storage medium having instructions stored thereon, Among them, wherein when the instructions are executed by a processor, the processor executes the method according to any one of claims 1-14.

18. A computer program product having instructions stored thereon, Among them, wherein when the instructions are executed by a processor, the processor executes the method according to any one of claims 1-14.

Images