Pronunciation evaluation method, training method, device and equipment for pronunciation evaluation system

Abstract

The present application discloses a pronunciation evaluation method, a training method, a device and equipment for a pronunciation evaluation system, and belongs to the field of artificial intelligence technology. The method comprises: obtaining a pronunciation audio to be evaluated; extracting the phoneme information of the pronunciation audio to be evaluated; obtaining the similarity between the pronunciation audio to be evaluated and the teacher's pronunciation audio based on the phoneme information of the pronunciation audio to be evaluated and the phoneme information of the teacher's pronunciation audio corresponding to the pronunciation audio to be evaluated; and determining the pronunciation score of the pronunciation audio to be evaluated based on the similarity. In the technical solution provided by the embodiment of the present application, the similarity between the student's pronunciation and the teacher's pronunciation is determined by comparing the universal phonemes applicable to multiple languages corresponding to the student's pronunciation in any language with the teacher's pronunciation, thereby obtaining the score of the student's pronunciation. This eliminates the need for multiple models to realize pronunciation evaluation in multiple languages, effectively reduces the complexity of pronunciation evaluation and saves storage resources.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a pronunciation evaluation method, a training method, a device and equipment for a pronunciation evaluation system. Background Art

[0002] In online learning scenarios such as foreign languages, evaluating users' pronunciation is the most commonly used function by users who are learning online.

[0003] Currently, the commonly used method for evaluating user pronunciation is to design different pronunciation evaluation algorithms or models for language learning scenarios in different languages and score the pronunciation audio of different languages.

[0004] The above method for evaluating user pronunciation is highly complex and consumes a lot of storage resources. Summary of the Invention

[0005] The present invention provides a pronunciation evaluation method, a training method, an apparatus, and a device for a pronunciation evaluation system, which can implement pronunciation evaluation in multiple languages, effectively reduce the complexity of pronunciation evaluation, and save storage resources. The technical solution is as follows:

[0006] According to one aspect of an embodiment of the present application, a pronunciation evaluation method is provided, the method comprising:

[0007] Get the pronunciation audio to be evaluated;

[0008] Extracting phoneme information of the pronunciation audio to be evaluated, wherein the phoneme information includes common phonemes of each pronunciation in the pronunciation audio, and the common phonemes are phonemes obtained by integrating proprietary phonemes of multiple languages;

[0009] Based on the phoneme information of the pronunciation audio to be evaluated and the phoneme information of the teacher pronunciation audio corresponding to the pronunciation audio to be evaluated, obtaining the similarity between the pronunciation audio to be evaluated and the teacher pronunciation audio;

[0010] Based on the similarity, a pronunciation score of the pronunciation audio to be evaluated is determined.

[0011] By comparing the student's pronunciation in any language with the teacher's pronunciation using universal phonemes applicable to multiple languages, the similarity between the student's pronunciation and the teacher's pronunciation is determined, and a score for the student's pronunciation is then derived. This eliminates the need for multiple models to implement pronunciation assessment in multiple languages, effectively reducing the complexity of pronunciation assessment while saving storage resources.

[0012] In one possible design, extracting phoneme information of the pronunciation audio to be evaluated includes:

[0013] Dividing the pronunciation audio to be evaluated into at least one audio frame;

[0014] Obtaining frequency domain features of each audio frame in the pronunciation audio to be evaluated;

[0015] Processing the frequency domain features of each audio frame in the audio to be evaluated using a multilingual phoneme detection model to obtain phoneme information of the audio to be evaluated, wherein the phoneme information of the audio to be evaluated includes a probability distribution of each audio frame in the audio to be evaluated, wherein the probability distribution reflects the probability of the pronunciation of the audio frame on each universal phoneme;

[0016] The multilingual phoneme detection model is a machine learning model used to determine the universal phonemes corresponding to each audio frame in the pronunciation audio.

[0017] Through the above method, the probability of detecting the audio frame on each phoneme is achieved, and the phoneme information is quantified.

[0018] In one possible design, the multilingual phoneme detection model is used to process the frequency domain features of each audio frame in the audio to be evaluated to obtain the phoneme information of the audio to be evaluated, including:

[0019] Obtaining the target language of the pronunciation audio to be evaluated;

[0020] Selecting target model parameters corresponding to the target language; wherein the multilingual phoneme detection model has multiple sets of model parameters, and different model parameters correspond to different languages;

[0021] The frequency domain features of each audio frame in the pronunciation audio to be evaluated are processed by the multilingual detection model according to the target model parameters to obtain the phoneme information of the pronunciation audio to be evaluated.

[0022] Through the above method, phoneme recognition is achieved according to the model parameters of different languages, making phoneme detection more targeted and improving the accuracy of multilingual phoneme detection.

[0023] In one possible design, obtaining the similarity between the pronunciation audio to be evaluated and the teacher pronunciation audio based on the phoneme information of the pronunciation audio to be evaluated and the phoneme information of the teacher pronunciation audio corresponding to the pronunciation audio to be evaluated includes:

[0024] Performing dynamic time normalization processing on the distribution probability vector of each audio frame in the audio of the pronunciation to be evaluated and the distribution probability vector of each audio frame in the audio of the teacher's pronunciation to obtain a timing matching result, wherein the timing matching result is a combination of matching distribution probability vectors obtained by aligning the audio of the pronunciation to be evaluated and the audio of the teacher's pronunciation according to the same universal phoneme in time sequence;

[0025] Based on the similarity between the matched distribution probability vectors determined based on the time series matching result, the similarity between the pronunciation audio to be evaluated and the teacher's pronunciation audio is obtained.

[0026] Through the above method, the effect of accurately matching the distribution probability vector is achieved.

[0027] In one possible design, determining the pronunciation score of the pronunciation audio to be evaluated based on the similarity includes:

[0028] The pronunciation score of the pronunciation audio to be evaluated is determined according to the similarity through a scoring model, and the scoring model is a regression model for quantitatively describing the statistical relationship between the similarity and the pronunciation score.

[0029] Through the above method, the effect of accurately outputting pronunciation scores is achieved.

[0030] According to one aspect of an embodiment of the present application, a training method for a pronunciation evaluation system is provided, the method comprising:

[0031] Obtain a first training sample set and a second training sample set, wherein the first training sample set includes at least one first training sample, sample data of the first training sample includes teacher pronunciation audio in any language, and label data of the first training sample includes common phonemes of each pronunciation in the teacher pronunciation audio; the second training sample set includes at least one second training sample, sample data of the second training sample includes teacher pronunciation audio and student pronunciation audio for the same paragraph, and label data of the second training sample includes pronunciation scores of the student pronunciation audio;

[0032] Training the multilingual phoneme detection model based on the first training sample set, wherein the multilingual phoneme detection model is used to extract phoneme information from the pronunciation audio, wherein the phoneme information includes common phonemes of each pronunciation in the pronunciation audio, wherein the common phonemes are phonemes obtained by integrating specific phonemes of multiple languages;

[0033] Extracting phoneme information of the second training sample using the multilingual phoneme detection model, where the phoneme information of the second training sample includes phoneme information of the teacher's pronunciation audio and phoneme information of the student's pronunciation audio in the second training sample;

[0034] Based on the phoneme information of the second training sample, obtaining the similarity between the teacher's pronunciation audio and the student's pronunciation audio;

[0035] The scoring model is fitted based on the similarity and the pronunciation score of the student's pronunciation audio, and the scoring model is used to score the audio to be evaluated.

[0036] By training a multilingual phoneme detection model using sample audio labeled with universal phoneme information, the model can accurately detect the universal phoneme information corresponding to the teacher's and student's pronunciations of the same passage. The model then calculates the similarity between the two and fits a regression model to describe the statistical relationship between the score label carried by the student's pronunciation and the similarity. Finally, the pronunciation evaluation system is trained to complete, expanding the application scope of the model and enabling multilingual user pronunciation evaluation. This effectively reduces the complexity of pronunciation evaluation while saving storage resources.

[0037] In one possible design, training the multilingual phoneme detection model based on the first training sample set includes:

[0038] Training an initialized multilingual phoneme detection model based on the first training sample set, where the initialized multilingual phoneme detection model is a pre-trained model for detecting universal phonemes;

[0039] For a target language, obtaining a first training sample of the target language from the first training sample set;

[0040] Adjusting model parameters of the initialized multilingual phoneme detection model using a first training sample of the target language to obtain model parameters corresponding to the target language;

[0041] The multilingual phoneme detection model is obtained based on the model parameters corresponding to the multiple languages.

[0042] Through the above method, the model training process is simplified and the accuracy of the trained model is guaranteed.

[0043] In one possible design, adjusting the model parameters of the initialized multilingual phoneme detection model using the first training sample of the target language to obtain model parameters corresponding to the target language includes:

[0044] Fixing the parameters of the middle layer in the initialized multilingual phoneme detection model;

[0045] Replacing the full-pass layers at the input and output ends of the initialized multilingual phoneme detection model with fully connected layers, wherein the full-pass layers do not process any data and the fully connected layers are language-related layers used to mine language features;

[0046] The model parameters of the fully connected layer are adjusted using the first training sample of the target language to obtain model parameters corresponding to the target language.

[0047] Through the above method, only the model parameters at the input and output ends of the pre-trained model are fine-tuned, which improves the accuracy of model detection while simplifying the training process.

[0048] In one possible design, obtaining a first training sample set includes:

[0049] Obtaining original label data of the teacher's pronunciation audio of each language, wherein the original label data includes the specific phonemes of each pronunciation in the teacher's pronunciation audio, wherein the specific phonemes are phonemes specific to a language;

[0050] Acquire a universal phoneme set, where the universal phoneme set is a set consisting of the universal phonemes;

[0051] Based on the universal phoneme set, the proprietary phonemes corresponding to the respective pronunciations in the audio frames of the teacher's pronunciation audio are replaced to obtain the label data of the first training sample.

[0052] Through the above method, training samples with universal phonemes as label data were obtained, providing sample support for model training.

[0053] In one possible design, obtaining the universal phoneme set includes:

[0054] Based on the teacher pronunciation audio of each language and the original label data of the teacher pronunciation audio of each language, respectively train the phoneme recognizer of each language;

[0055] Sending the teacher's pronunciation audio of each language to the phoneme recognizer of each language to determine similar phonemes in each language, wherein the similar phonemes refer to proprietary phonemes with similar pronunciations in each language;

[0056] The similar phonemes are bound and merged into the universal phonemes to obtain the universal phoneme set.

[0057] Through the above method, a universal phoneme set applicable to multiple languages can be obtained, which provides a basis for simultaneously evaluating the pronunciation quality of multiple languages.

[0058] In one possible design, the step of feeding the teacher's pronunciation audio of each language into a phoneme recognizer of each language to determine similar phonemes in each language includes:

[0059] For the target language, the teacher's pronunciation audio of the target language is fed into the phoneme recognizer of the target language to obtain the probability value of each audio frame in the teacher's pronunciation audio of the target language under each specific phoneme of the target language;

[0060] Determining the specific phonemes corresponding to the audio frames based on the probability values of the audio frames under the specific phonemes of the target language;

[0061] inputting an audio frame corresponding to a first specific phoneme in a first language into a phoneme recognizer in a second language to obtain a probability value of the audio frame corresponding to the first specific phoneme under each specific phoneme in the second language, where the first language and the second language are different languages;

[0062] Using the probability value of the audio frame corresponding to the first specific phoneme under each specific phoneme of the second language as the inverse of the distance between the first specific phoneme and each specific phoneme of the second language;

[0063] Based on the distances, similar phonemes in the languages are determined using a clustering algorithm.

[0064] In this way, phonemes with the same or similar pronunciations can be accurately determined from the specific phonemes of each language.

[0065] In one possible design, obtaining the original label data of the teacher's pronunciation audio of each language includes:

[0066] Obtaining a phoneme set of each language;

[0067] Based on the phoneme sets of the various languages, the teacher pronunciation audios of the various languages are aligned to obtain original label data of the teacher pronunciation audios of the various languages.

[0068] Through the above method, the effect of automatically adding labels to training samples is achieved.

[0069] According to one aspect of an embodiment of the present application, a pronunciation evaluation device is provided, the device comprising:

[0070] Audio acquisition module, used to obtain the pronunciation audio to be evaluated;

[0071] A feature extraction module is used to extract phoneme information of the pronunciation audio to be evaluated, wherein the phoneme information includes common phonemes of each pronunciation in the pronunciation audio, and the common phonemes are phonemes obtained by integrating proprietary phonemes of multiple languages;

[0072] A similarity determination module, configured to obtain a similarity between the pronunciation audio to be evaluated and the teacher pronunciation audio based on the phoneme information of the pronunciation audio to be evaluated and the phoneme information of the teacher pronunciation audio corresponding to the pronunciation audio to be evaluated;

[0073] The scoring determination module is used to determine the pronunciation score of the pronunciation audio to be evaluated based on the similarity.

[0074] According to one aspect of an embodiment of the present application, a training device for a pronunciation evaluation system is provided, the device comprising:

[0075] A sample acquisition module is configured to acquire a first training sample set and a second training sample set, wherein the first training sample set includes at least one first training sample, the sample data of the first training sample includes a teacher's pronunciation audio in any language, and the label data of the first training sample includes common phonemes of each pronunciation in the teacher's pronunciation audio; the second training sample set includes at least one second training sample, the sample data of the second training sample includes a teacher's pronunciation audio and a student's pronunciation audio for the same paragraph, and the label data of the second training sample includes a pronunciation score of the student's pronunciation audio;

[0076] a model training module, configured to train the multilingual phoneme detection model based on the first training sample set, wherein the multilingual phoneme detection model is configured to extract phoneme information from the pronunciation audio, wherein the phoneme information includes common phonemes for each pronunciation in the pronunciation audio, wherein the common phonemes are phonemes obtained by integrating specific phonemes of multiple languages;

[0077] a feature extraction module, configured to extract phoneme information of the second training sample using the multilingual phoneme detection model, wherein the phoneme information of the second training sample includes phoneme information of the teacher's pronunciation audio and phoneme information of the student's pronunciation audio in the second training sample;

[0078] A similarity determination module, configured to obtain a similarity between the teacher's pronunciation audio and the student's pronunciation audio based on the phoneme information of the second training sample;

[0079] The model fitting module is used to fit the scoring model based on the similarity and the pronunciation score of the student's pronunciation audio, and the scoring model is used to score the audio to be evaluated.

[0080] According to one aspect of an embodiment of the present application, a computer device is provided, comprising a processor and a memory, wherein a computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the above-mentioned pronunciation evaluation method.

[0081] According to one aspect of an embodiment of the present application, a computer device is provided, comprising a processor and a memory, wherein a computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the training method of the above-mentioned pronunciation evaluation system.

[0082] According to one aspect of an embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored. The computer program is loaded and executed by a processor to implement the above-mentioned pronunciation evaluation method.

[0083] According to one aspect of an embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored. The computer program is loaded and executed by a processor to implement the training method of the above-mentioned pronunciation evaluation system.

[0084] According to one aspect of an embodiment of the present application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the above-described pronunciation evaluation method.

[0085] According to one aspect of an embodiment of the present application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the training method of the pronunciation evaluation system described above.

[0086] The technical solutions provided in the embodiments of the present application can bring the following beneficial effects:

[0087] By comparing the student's pronunciation in any language with the teacher's pronunciation using universal phonemes applicable to multiple languages, the similarity between the student's pronunciation and the teacher's pronunciation is determined, and a score for the student's pronunciation is then derived. This eliminates the need for multiple models to implement pronunciation assessment in multiple languages, effectively reducing the complexity of pronunciation assessment while saving storage resources.

[0088] By training a multilingual phoneme detection model using sample audio labeled with universal phoneme information, the model can accurately detect the universal phoneme information corresponding to the teacher's and student's pronunciations of the same passage. The model then calculates the similarity between the two and fits a regression model to describe the statistical relationship between the score label carried by the student's pronunciation and the similarity. Finally, the pronunciation evaluation system is trained to complete, expanding the application scope of the model and enabling multilingual user pronunciation evaluation. This effectively reduces the complexity of pronunciation evaluation while saving storage resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0090] Figure 1is a schematic diagram of an application program operating environment provided by an embodiment of the present application;

[0091] Figure 2 This is a flowchart of a pronunciation evaluation method provided by one embodiment of the present application;

[0092] Figure 3 is a flowchart of a pronunciation evaluation method provided by another embodiment of the present application;

[0093] Figure 4 A schematic diagram showing an exemplary timing matching result;

[0094] Figure 5 A schematic diagram of a pronunciation evaluation system is shown as an example;

[0095] Figure 6 This is a flowchart of a training method for a pronunciation evaluation system provided by one embodiment of the present application;

[0096] Figure 7 A schematic diagram of a multilingual phoneme detection model is shown as an example;

[0097] Figure 8 This is a flowchart of a method for obtaining a first training sample set provided by an embodiment of the present application;

[0098] Figure 9 is a block diagram of a pronunciation evaluation device provided by one embodiment of the present application;

[0099] Figure 10 is a block diagram of a pronunciation evaluation device provided by another embodiment of the present application;

[0100] Figure 11 This is a block diagram of a training device for a pronunciation evaluation system provided by one embodiment of the present application;

[0101] Figure 12 is a block diagram of a training device for a pronunciation evaluation system provided in another embodiment of the present application;

[0102] Figure 13 This is a structural block diagram of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0103] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0104] First, some terms in this application are explained to facilitate understanding by those skilled in the art.

[0105] Speech recognition technology (Automatic Speech Recognition, ASR), also known as automatic speech recognition, aims to convert the vocabulary content in human speech into computer-readable input, such as keystrokes, binary codes or character sequences.

[0106] In the field of sound processing, Mel-Frequency Cepstrum Coefficients (MFCCs) are a linear transformation of the logarithmic energy spectrum based on the nonlinear mel scale of sound frequencies. MFCCs are widely used in speech recognition.

[0107] KL divergence (Kullback-Leibler Divergence), also known as relative entropy, is a measure of the asymmetry between two probability distributions. In information theory, relative entropy is equivalent to the difference in the Shannon entropy of two probability distributions. The aforementioned phoneme information can be considered the probability distribution of each audio frame on each universal phoneme. Relative entropy can measure the distance between two probability distributions. When two probability distributions are identical, their relative entropy is zero. As the difference between the two probability distributions increases, their relative entropy also increases. Therefore, relative entropy can be used to compare text similarity.

[0108] Neural Networks (NN) is a complex network system formed by a large number of simple processing units (called neurons) that are widely interconnected. It reflects many basic characteristics of human brain functions and is a highly complex nonlinear dynamic learning system.

[0109] A neural network can be composed of neural units (abbreviated as "neurons"), which can be x s The output of the operation unit with the intercept 1 as input can be:

[0110]

[0111] Where, s = 1, 2, ... n, n is a natural number greater than 1, W s is x sThe weight of the neural unit, b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer. The activation function can be a sigmoid function. A neural network is a network formed by connecting many of the above-mentioned single neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.

[0112] A convolutional neural network (CNN) is a deep neural network with a convolutional architecture. A CNN consists of a feature extractor consisting of convolutional layers and subsampling layers. This feature extractor can be viewed as a filter, and the convolution process can be thought of as convolving an input image or feature map with a trainable filter. A convolutional layer is the layer of neurons in a CNN that performs convolution on the input signal. Within a convolutional layer, a neuron can only connect to a subset of neurons in adjacent layers. A convolutional layer typically contains several feature planes, each composed of a rectangular arrangement of neurons. Neurons within the same feature plane share weights, referred to as the convolution kernel. Shared weights can be understood as ensuring that the method for extracting image information is independent of position. The underlying principle is that the statistical information of one part of the image is the same as that of another. This means that image information learned in one part can also be applied to other parts. Therefore, the same learned image information can be used at all positions in the image. In the same convolutional layer, multiple convolution kernels can be used to extract different image information. Generally speaking, the more convolution kernels there are, the richer the image information reflected by the convolution operation.

[0113] Convolution kernels can be initialized as matrices of random size, and during the training process of the convolutional neural network, the convolution kernels can be learned to obtain reasonable weights. In addition, the direct benefit of shared weights is that they reduce the number of connections between the layers of the convolutional neural network, while also reducing the risk of overfitting.

[0114] LSTM (Long Short Term Memory) is a specific form of RNN (Recurrent Neural Network), a general term for a family of neural networks capable of processing sequential data. RNNs have many variations, such as bidirectional RNNs. However, RNNs face significant difficulties in handling long-term dependencies (nodes that are distant in time series). Computing the connections between distant nodes involves multiple multiplications of the Jacobian matrix, which can lead to the problems of vanishing gradients (common) or dilation (less common). To address this problem, the most widely used is the gated RNN, of which the LSTM is the most famous. While leaky units allow RNNs to accumulate long-term connections between distant nodes by designing the weight coefficients between connections, gated RNNs generalize this idea, allowing these coefficients to change at different times and allowing the network to forget previously accumulated information. LSTM is one such gated RNN. LSTM increases the input threshold, forgetting threshold, and output threshold to make the weight of the self-loop variable. In this way, when the model parameters are fixed, the integral scale at different times can be changed dynamically, thus avoiding the problem of gradient vanishing or gradient expansion.

[0115] Loss function. During the training of a deep neural network, the goal is to ensure that the output of the network is as close as possible to the desired predicted value. This is done by comparing the network's predicted value with the desired target value, and then updating the weight vectors of each layer of the neural network based on the difference between the two. (Of course, this initial update typically involves a process that pre-configures parameters for each layer of the deep neural network.) For example, if the network's predicted value is too high, the weight vectors are adjusted to predict a lower value. This adjustment is repeated until the deep neural network can predict the desired target value or a value very close to it. Therefore, it is necessary to predefine how to compare the difference between the predicted and target values. This is the loss function, or objective function, which is a crucial equation used to measure the difference between the predicted and target values. For example, a higher loss function output (loss) indicates a greater difference, so training a deep neural network becomes a process of minimizing this loss.

[0116] Neural networks can use the backpropagation (BP) algorithm to correct the size of the parameters in the initial neural network model during training, reducing the reconstruction error loss of the neural network model. Specifically, forward propagation of the input signal to the output generates error loss. This error loss information is then backpropagated to update the parameters in the initial neural network model, thereby converging the error loss. The BP algorithm is a backpropagation movement dominated by error loss, aiming to obtain the optimal parameters of the neural network model, such as the weight matrix.

[0117] Please refer to Figure 1 , which shows a schematic diagram of an implementation environment of a solution provided by an embodiment of the present application. The implementation environment of the solution can be implemented as a pronunciation evaluation system. The implementation environment of the solution may include: a terminal 10 and a server 20.

[0118] The terminal 10 can be an electronic device such as a mobile phone, a tablet computer, a multimedia player, a wearable device, a PC (Personal Computer), a language learning terminal, an intelligent teaching machine, etc. The terminal 10 can be configured with or connected to a microphone to collect audio. A client having an application program can be installed and run in the terminal 10, and the application program can include a pronunciation evaluation function. In the embodiment of the present application, the type of the application program is not limited, and the application program can be a language learning application program, an educational tutoring application program, an instant messaging application program, a language evaluation application program, an educational examination application program, etc.

[0119] The server 20 may be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The server 20 may be a background server for the above application, used to provide background services for the application.

[0120] The terminal 10 and the server 20 can communicate via a network, which is not limited in this application.

[0121] In the pronunciation evaluation method provided in the embodiments of the present application, each step may be performed by the server 20, the terminal 10 (e.g., a client of an application running on the terminal 10), or the terminal 10 and the server 20 in cooperation with each other. For ease of description, the following method embodiments illustrate each step as being performed by a computer device, but this is not intended to be limiting.

[0122] Please refer to Figure 2 , which shows a flow chart of a pronunciation evaluation method provided by an embodiment of the present application. The method may include the following steps (210-240):

[0123] Step 210: Obtain the pronunciation audio to be evaluated.

[0124] The above-mentioned pronunciation audio to be evaluated refers to the audio of the speech whose pronunciation quality is to be evaluated. The audio data format includes but is not limited to the Moving Picture Experts Group Audio Layer III (MP3) format, the Moving Picture Experts Group (MPEG) format, the Audio Interchange File Format (AIFF) format, and the Microsoft Audio (Windows Media Audio, WMA) format, which is not limited in the present embodiment.

[0125] Optionally, in a language learning scenario, the pronunciation audio to be evaluated is student pronunciation audio, for example, a voice recorded by a student imitating the teacher's pronunciation based on a piece of language content.

[0126] Step 220: extract the phoneme information of the pronunciation audio to be evaluated.

[0127] The phoneme information includes common phonemes of each pronunciation in the pronunciation audio. The common phonemes are phonemes obtained by integrating the specific phonemes of multiple languages.

[0128] A phoneme is the smallest unit of speech defined by the natural properties of audio. Audio can contain multiple phonemes. From an acoustic perspective, a phoneme is the smallest unit of speech defined by sound quality. From a physiological perspective, a phoneme is formed by a single pronunciation action. For example, the word "ma" contains two pronunciation actions: "m" and "a," thus comprising two phonemes.

[0129] Each language has its own corresponding phonemes, such as Chinese phonemes and English phonemes. This application refers to the phonemes belonging to each language as proprietary phonemes. The universal phonemes mentioned above, which are obtained by integrating the proprietary phonemes of multiple languages, refer to the integration of the proprietary phonemes of multiple languages based on the principle of similar pronunciation. The proprietary phonemes with the same or similar pronunciation in different languages are integrated into a new element, which is called the universal phoneme.

[0130] The above-mentioned phoneme information reflects the information of the phonemes corresponding to each pronunciation in the pronunciation audio. The phonemes here can be proprietary phonemes or universal phonemes.

[0131] Extracting phoneme information from the audio to be evaluated refers to determining the phoneme corresponding to each pronunciation based on its pronunciation characteristics. These pronunciation characteristics may include characteristics of human speech, such as frequency, pitch, and loudness. The detailed process of extracting phoneme information is described in detail in the examples below; this section provides only a general overview.

[0132] Step 230 : obtaining the similarity between the pronunciation audio to be evaluated and the teacher pronunciation audio based on the phoneme information of the pronunciation audio to be evaluated and the phoneme information of the teacher pronunciation audio corresponding to the pronunciation audio to be evaluated.

[0133] Extract the phoneme information from the teacher's pronunciation audio. The process of extracting phoneme information from the teacher's pronunciation audio is the same as that of extracting phoneme information from the audio to be evaluated, with only the extraction object being different.

[0134] The phoneme information of the evaluation pronunciation audio is compared with the phoneme information of the teacher's pronunciation audio corresponding to the pronunciation audio to be evaluated, and the similarity between the pronunciation audio to be evaluated and the teacher's pronunciation audio is evaluated based on the degree of difference. Optionally, the smaller the degree of difference, the higher the similarity, and vice versa.

[0135] Step 240: Determine the pronunciation score of the pronunciation audio to be evaluated based on the similarity.

[0136] The similarity between the pronunciation audio to be evaluated and the teacher's pronunciation audio is used as a judgment basis to score the pronunciation audio to be evaluated. The above pronunciation score is an indicator for quantitatively evaluating the pronunciation quality of the pronunciation audio to be evaluated. Optionally, the higher the similarity, the higher the pronunciation score, indicating that the pronunciation quality of the pronunciation audio to be evaluated is higher and the pronunciation is more standard. Optionally, the above pronunciation score can adopt a full score system of 10 or a full score system of 100, which is not limited in the embodiments of the present application.

[0137] To sum up, the technical solution provided in the embodiment of the present application determines the similarity between the student's pronunciation and the teacher's pronunciation by comparing the universal phonemes applicable to multiple languages corresponding to the student's pronunciation in any language and the teacher's pronunciation, and then derives a score for the student's pronunciation. It does not require multiple models to realize pronunciation evaluation in multiple languages, effectively reducing the complexity of pronunciation evaluation while saving storage resources.

[0138] Please refer to Figure 3 , which shows a flow chart of a pronunciation evaluation method provided by another embodiment of the present application. The method may include the following steps (310-370):

[0139] Step 310: Obtain the pronunciation audio to be evaluated.

[0140] Optionally, after obtaining the pronunciation audio to be evaluated, the pronunciation audio to be evaluated is preprocessed. The above preprocessing refers to the pre-processing of the input pronunciation audio data by cutting off the silent areas at the beginning and end of the speech to reduce interference with subsequent steps. This is generally called Voice Activity Detection (VAD).

[0141] Step 320: Divide the pronunciation audio to be evaluated into at least one audio frame.

[0142] Since the audio data to be identified has time-varying characteristics, but its characteristics remain essentially stable within a short timeframe (e.g., 10 milliseconds to 30 milliseconds), the audio data to be identified can be segmented to analyze its characteristics. A segment here can be understood as an audio frame. It should be understood that the duration of an audio frame in this application can be 20 to 25 milliseconds. This is merely an illustration; in practice, other values are possible and are not intended to be limiting. The concept of an audio frame is not as clear as that of a video frame. In almost all video encoding formats, a frame is simply considered to be an encoded image. However, audio frames are dependent on the encoding format and are implemented by each encoding standard. For example, PCM (unencoded audio data) does not require the concept of frames; playback can be performed based on the sampling rate and sampling accuracy. For example, for audio with a sampling rate of 44.1 kHz and a sampling accuracy of 16 bits, the bitrate can be calculated as 44100 * 16 kbps, meaning the audio data per second is a fixed 44100 * 16 / 8 bytes. When decoding AAC (Advanced Audio Coding) audio with a sampling rate of 44.1kHz, the decoding time for one frame must be controlled within 23.22 milliseconds. Typically, one frame is sampled at 1024 points. The sampling frequency refers to the number of sound wave amplitude samples taken per second when digitizing an analog sound waveform. According to the Nyquist sampling theorem, to ensure distortion-free sound, the sampling frequency should be around 40kHz. Common audio sampling frequencies include 8kHz, 11.025kHz, 22.05kHz, 16kHz, 37.8kHz, 44.1kHz, and 48kHz. Using higher sampling frequencies can achieve DVD-quality sound. When decoding AAC audio with a sampling rate of 44.1kHz, the decoding time for one frame must be controlled within 23.22 milliseconds. Alternatively, one frame is sampled at 1024 points.

[0143] Optionally, before dividing the pronunciation audio to be evaluated into at least one audio frame, the pronunciation audio to be evaluated is pre-emphasized. The above pre-emphasis refers to enhancing the high-frequency part of the audio data to be recognized, and pre-emphasis can usually be achieved by using a digital filter.

[0144] Step 330: Obtain frequency domain features of each audio frame in the pronunciation audio to be evaluated.

[0145] Optionally, after the pronunciation audio to be evaluated is subjected to framing processing, a windowing process can also be performed. The purpose of windowing can be considered to be to emphasize the speech waveform near the sampling and to weaken the rest of the waveform. Processing each short segment of the pronunciation audio to be evaluated is to transform each segment, for example, using three window functions: rectangular window, Hamming window (Hamming) and Hanning window (Hanning).

[0146] By performing frequency domain conversion on each audio frame in the audio of the pronunciation to be evaluated. Frequency domain conversion refers to converting the data of each audio frame in the audio of the pronunciation to be evaluated in the time domain to the frequency domain. For voice companion separation, since the audio data is the superposition of different frequencies at the same time, it is difficult to reflect the difference between different frequencies in the time domain. Therefore, when analyzing audio, it is necessary to convert the audio data in the time domain to the frequency domain for analysis, and it is easier to separate. Frequency domain conversion methods include but are not limited to Fast Fourier Transform (FFT) and Discrete Fourier Transform (DFT).

[0147] After converting the data of each audio frame in the pronunciation audio to be evaluated from the time domain to the frequency domain, the frequency domain data of each audio frame can be obtained. At this time, the characteristics of each audio frame in the frequency domain can be extracted to obtain the frequency domain characteristics of each audio frame. Optionally, the frequency domain feature is Mel Frequency Band Energy (MFBE). The above Mel frequency band energy can be obtained by using a Mel filter to obtain a Mel spectrum. The obtained spectrum is then subjected to cepstrum analysis to obtain Mel frequency cepstrum coefficients (MFCC), thereby obtaining the Mel frequency band energy of each audio frame. Optionally, the frequency domain feature is used as the basis for identifying the phonemes corresponding to each audio frame.

[0148] Step 340 : Process the frequency domain features of each audio frame in the audio to be evaluated using a multilingual phoneme detection model to obtain phoneme information of the audio to be evaluated.

[0149] Optionally, the phoneme information of the pronunciation audio to be evaluated includes the phoneme information of each audio frame in the pronunciation audio to be evaluated. For example, the pronunciation audio to be evaluated includes M audio frames. At this time, the phoneme information of the pronunciation audio to be evaluated has M probability distributions. Optionally, there is a one-to-one correspondence between the audio frames and the probability distributions, and one audio frame corresponds to one probability distribution. Optionally, the probability distribution reflects the probability of the pronunciation of the audio frame on each universal phoneme. For example, if the number of universal phonemes is N, the dimension of the probability distribution is N, and the value of each dimension in the probability distribution is the probability that the audio frame is the universal phoneme corresponding to the dimension.

[0150] The multilingual phoneme detection model is a machine learning model used to determine the universal phoneme corresponding to each audio frame in the spoken audio. Optionally, the multilingual phoneme detection model is a machine learning model built using a long short-term memory neural network and a convolutional neural network, including LSTM layers and CNN layers, to analyze and identify the phonemes corresponding to each pronunciation in the spoken audio.

[0151] In an exemplary embodiment, the above step 340 includes the following sub-steps:

[0152] Step 341: Obtain the target language of the pronunciation audio to be evaluated.

[0153] Optionally, the target language is any language. Optionally, the target language of the audio to be evaluated is determined based on the language type selected by the user. Optionally, the target language of the audio to be evaluated is determined using language recognition technology. This embodiment of the application does not limit the method for obtaining the target language of the audio to be evaluated, and the method can be adjusted and selected based on actual circumstances.

[0154] Step 342: Select target model parameters corresponding to the target language.

[0155] Optionally, the multilingual phoneme detection model has multiple sets of model parameters, and different model parameters correspond to different languages, so that it has stronger pertinence when detecting common phonemes for pronunciation audio of different languages, thereby making the detection results more accurate.

[0156] In step 343 , the frequency domain features of each audio frame in the pronunciation audio to be evaluated are processed by the multilingual detection model according to the target model parameters to obtain the phoneme information of the pronunciation audio to be evaluated.

[0157] For example, if the language of the audio to be evaluated is English, the multilingual detection model will call the model parameters corresponding to English in the model to process the frequency domain features of each audio frame in the pronunciation audio to be evaluated, thereby detecting the universal phonemes corresponding to each audio frame in the pronunciation audio to be evaluated. The reason is that universal phonemes are new phonemes that are formed by integrating proprietary phonemes in various languages based on the principle of similar pronunciation. There is a corresponding relationship between universal phonemes and proprietary phonemes. There may be slight differences between the pronunciation features of the corresponding universal phonemes and the pronunciation features of the proprietary phonemes. The above-mentioned model parameters corresponding to the language can eliminate these slight differences, making the detection results more accurate.

[0158] Optionally, the above target model parameters are some parameters in a multilingual detection model.

[0159] In step 350 , a dynamic time normalization process is performed on the distribution probability vector of each audio frame in the audio to be evaluated and the distribution probability vector of each audio frame in the audio of the teacher's pronunciation to obtain a timing matching result.

[0160] Optionally, before executing step 350, the following steps need to be executed:

[0161] Based on the probability distribution of each audio frame in the audio of the pronunciation to be evaluated, a distribution probability vector of each audio frame in the audio of the pronunciation to be evaluated is generated.

[0162] Based on the distribution probability vector of each audio frame in the pronunciation audio to be evaluated, a distribution probability vector sequence of the pronunciation audio to be evaluated is generated.

[0163] The above distribution probability vector is a vector composed of the probability of the pronunciation of the audio frame on each universal phoneme. For example, the probability of an audio frame on each universal phoneme is P1, P2, P3, P4, ..., P n , then the distribution probability vector of the audio frame is (P1, P2, P3, P4, ..., P n ).

[0164] Optionally, the distribution probability vectors of each audio frame in the audio to be evaluated constitute a distribution probability vector sequence of the audio to be evaluated. Optionally, the distribution probability vectors of each audio frame are arranged in the distribution probability vector sequence according to the time sequence of each audio frame.

[0165] Optionally, through the Dynamic Time Warping (DTW) algorithm, based on the cosine distance between the distribution probability vector of each audio frame in the audio to be evaluated and the distribution probability vector of each audio frame in the teacher's pronunciation audio, the distribution probability vectors of the two are aligned in time sequence, and the two distribution probability vectors of the distribution probability vector of each audio frame in the audio to be evaluated and the distribution probability vector of each audio frame in the teacher's pronunciation audio that are aligned are matched distribution probability vectors.

[0166] Optionally, the timing matching result is a combination of matching distribution probability vectors obtained by aligning the pronunciation audio to be evaluated with the teacher's pronunciation audio according to the same common phonemes in timing, maintaining the temporal order of the matching distribution probability vectors.

[0167] In one example, if Figure 4 As shown, it exemplifies a schematic diagram of a timing matching result. The distribution probability vector 411 of the first audio frame in the distribution probability vector sequence 41 of the teacher's pronunciation audio matches the distribution probability vector 421 of the first audio frame in the phoneme information vector sequence 42 of the student's pronunciation audio. The distribution probability vector 412 of the third audio frame in the distribution probability vector sequence 41 of the teacher's pronunciation audio matches the distribution probability vector 422 of the second audio frame in the distribution probability vector sequence 42 of the student's pronunciation audio.

[0168] Step 360: Based on the similarity between the matching distribution probability vectors determined by the time series matching result, the similarity between the pronunciation audio to be evaluated and the teacher's pronunciation audio is obtained.

[0169] Optionally, the similarity is KL divergence.

[0170] Optionally, the average value of the similarity between the distribution probability vectors of each audio frame in the audio to be evaluated and the matching distribution probability vectors in the distribution probability vectors of each audio frame in the teacher's pronunciation audio is taken as the similarity between the audio to be evaluated and the teacher's pronunciation audio.

[0171] Step 370: Determine the pronunciation score of the pronunciation audio to be evaluated based on the similarity using the scoring model.

[0172] The above-mentioned scoring model is a regression model used to quantitatively describe the statistical relationship between similarity and pronunciation score.

[0173] Optionally, the similarity between the pronunciation audio to be evaluated and the teacher's pronunciation audio is input into the scoring model, and the scoring model outputs the pronunciation score of the pronunciation audio to be evaluated based on the statistical relationship between the similarity and the pronunciation score.

[0174] For an example, see Figure 5 , which exemplifies a schematic diagram of a pronunciation evaluation system. The teacher recording 50 and the student recording 51 are input into the multilingual phoneme detection model 52, and the probability of each frame in the teacher recording 50 being mapped to each universal phoneme in the universal phoneme set and the probability of each frame in the student recording 51 being mapped to each universal phoneme in the universal phoneme set are output. The probability of each frame in the teacher recording 50 being mapped to each universal phoneme in the universal phoneme set is used to generate a distribution probability vector sequence 53 of the teacher recording 50, and the probability of each frame in the student recording 51 being mapped to each universal phoneme in the universal phoneme set is used to generate a distribution probability vector sequence 54 of the student recording 51. The distribution probability vector sequence 53 of the teacher recording 50 and the distribution probability vector sequence 54 of the student recording 51 are aligned by the DTW algorithm, and the similarity between the two is calculated. The similarity is input into the regression model 55 for correction, and finally the pronunciation score 56 of the student recording 51 is output.

[0175] In summary, the technical solution provided by the embodiments of the present application sets model parameters corresponding to each language in a multilingual phoneme detection model, detects the probability distribution of student pronunciation and teacher pronunciation of any language under each common phoneme according to the model parameters corresponding to the language, and determines the similarity between the student pronunciation and the teacher pronunciation based on the probability distribution of the two, and then outputs the score of the student pronunciation through a regression model, thereby reducing the number of models required for multilingual pronunciation detection, and also ensuring the accuracy of pronunciation evaluation in multilingual cases, effectively reducing the complexity of pronunciation evaluation while saving storage resources.

[0176] Please refer to Figure 6 , which shows a flow chart of a method for training a pronunciation evaluation system provided by one embodiment of the present application. The pronunciation evaluation system includes a multilingual phoneme detection model and a scoring model. The method may include the following steps (610-650):

[0177] Step 610: Obtain a first training sample set and a second training sample set.

[0178] The first training sample set includes at least one first training sample, the sample data of the first training sample includes a teacher's pronunciation audio in any language, and the label data of the first training sample includes common phonemes of each pronunciation in the teacher's pronunciation audio.

[0179] Optionally, the first sample training set may be referred to as an ARAT (Audio Recordings Along with Transcriptions) dataset. Optionally, the first sample training set may also include student pronunciation audio in any language, and the label data of the first training sample may also include common phonemes of each pronunciation in the student pronunciation audio.

[0180] Optionally, the first training sample set includes the teacher's pronunciation audio and the transcription text corresponding to the student's pronunciation audio. The above transcription text is the text content corresponding to the teacher's pronunciation audio.

[0181] Optionally, the first training sample is stored in the form of a label file, such as an LBL file.

[0182] The second training sample set includes at least one second training sample, the sample data of the second training sample includes the teacher's pronunciation audio and the student's pronunciation audio for the same paragraph, and the label data of the second training sample includes the pronunciation score of the student's pronunciation audio.

[0183] Optionally, the second sample training set may be referred to as a TSR (Teacher Student Recording) data set.

[0184] It is easy to understand that the "teachers" and "students" mentioned in the embodiments of the present application are not limited to the concepts of teachers and students understood in people's daily lives. The teachers here refer to native speakers of various languages with standard pronunciation. The students here refer to users whose pronunciation quality is to be evaluated, and they are not necessarily students.

[0185] Step 620: training a multilingual phoneme detection model based on the first training sample set.

[0186] The multilingual phoneme detection model is used to extract phoneme information from pronunciation audio. The phoneme information includes the common phonemes of each pronunciation in the pronunciation audio. The common phonemes are phonemes obtained by integrating the proprietary phonemes of multiple languages.

[0187] The first training sample, i.e., the first training sample in the training data, is sent to the above-mentioned multilingual phoneme detection model, and the phoneme information of the pronunciation audio (teacher's pronunciation audio or student's pronunciation audio) is extracted through the multilingual phoneme detection model, that is, the common phonemes of each pronunciation in the predicted pronunciation audio are obtained.

[0188] The multilingual phoneme detection model is modified based on the labeled data in the first training sample, i.e., the differences between the common phonemes of each pronunciation in the labeled teacher pronunciation audio and the common phonemes of each pronunciation in the predicted pronunciation audio. Optionally, the multilingual phoneme detection model is modified using a loss function to ensure that the common phonemes of each pronunciation in the predicted pronunciation audio are consistent with the common phonemes of each pronunciation in the labeled teacher pronunciation audio, thereby ensuring the accuracy of the multilingual phoneme detection model.

[0189] In an exemplary embodiment, step 620 includes:

[0190] Step 621: Train an initialized multilingual phoneme detection model based on the first training sample set.

[0191] The initialized multilingual phoneme detection model is a pre-trained model for detecting universal phonemes.

[0192] A pre-trained model generally refers to the use of large-scale training samples (such as the first training sample set constructed in this application), designing a general phoneme detection training task, and training a large-scale neural network algorithm structure to learn and implement it. The final large-scale neural network algorithm structure is the pre-trained model. Other tasks can subsequently perform feature extraction or task fine-tuning based on this model to achieve specific task objectives. The idea of pre-training is to first train a task to obtain a set of model parameters, and then use this set of model parameters to initialize the network model parameters, and then use the initialized network model to train other tasks to obtain models adapted to other tasks. By pre-training on large-scale training samples, the above-mentioned initialized multilingual phoneme detection model can learn powerful phoneme detection capabilities and can extract rich phoneme information from audio. By fine-tuning downstream tasks directly on the pre-trained model, a downstream-specific model can be obtained quickly and easily.

[0193] Fine tuning refers to performing small-scale training on the basis of a pre-trained model using specific task objectives (downstream tasks) and task data (downstream data) to achieve slight adjustments to the pre-trained model parameters, ultimately obtaining a model adapted to specific data and tasks.

[0194] Step 622: For the target language, obtain a first training sample of the target language from the first training sample set.

[0195] For example, for English, a first training sample in English is obtained from the first training sample set.

[0196] Step 623 : Using the first training sample of the target language, adjust the model parameters of the initialized multilingual phoneme detection model to obtain model parameters corresponding to the target language.

[0197] Adjusting the model parameters of the initialized multilingual phoneme detection model using the first training sample in the target language can be understood as the aforementioned fine-tuning process, a more specific fine-tuning process in the embodiments of this application. Based on language classification, the initialized multilingual phoneme detection model is trained on a small scale using first training samples in different languages as different task data to obtain model parameters corresponding to each language. This allows for more targeted universal phoneme detection of audio pronunciations in different languages.

[0198] In an exemplary embodiment, the above step 623 includes the following sub-steps.

[0199] Step 623a: fix the parameters of the middle layer in the initialized multilingual phoneme detection model.

[0200] The initialized multilingual phoneme detection model includes a full-pass layer at the input, a full-pass layer at the output, and an intermediate layer consisting of an LSTM and a CNN. These intermediate layers refer to the intermediate layers in the initialized multilingual phoneme detection model. These intermediate layers already have the ability to extract phoneme information. To reduce training complexity, there is no need to adjust the model parameters of these intermediate layers.

[0201] Step 623b: Replace the full-pass layers at the input and output ends of the initialized multilingual phoneme detection model with fully connected layers.

[0202] The full-pass layer does not process the data. It can be understood that the output of the full-pass layer is the same as the output without any processing. The role of the above-mentioned full-pass layer is to occupy a position in the initialized multilingual phoneme detection model, making it convenient to add the fully connected layer to the initialized multilingual phoneme detection model.

[0203] The fully connected layers (FC) are language-related layers used to mine language features.

[0204] Fully connected layers can be used to map fixed-dimensional feature vectors (such as frequency-domain feature vectors) into different language categories for identification. Each node in a fully connected layer is connected to all nodes in the previous layer, integrating previously extracted features. Due to its fully connected nature, fully connected layers generally have the most parameters.

[0205] In a CNN architecture, after multiple convolutional and pooling layers, one or more fully connected layers are connected. Similar to an MLP, each neuron in a fully connected layer is fully connected to all neurons in the previous layer. Fully connected layers can integrate local information with class distinctions from the convolutional or pooling layers, such as the language of each audio recording. To improve CNN network performance, the activation function for each neuron in the fully connected layer generally uses the ReLU function. The output value of the final fully connected layer is passed to an output, which can be classified using softmax logistic regression. This layer is also called a softmax layer.

[0206] Step 623c: Use the first training sample of the target language to adjust the model parameters of the fully connected layer to obtain model parameters corresponding to the target language.

[0207] The above-mentioned fine-tuning process can be specifically reflected in adjusting the model parameters of the fully connected layer, using training samples of one language to generate a set of model parameters corresponding to the language, and finally obtaining multiple sets of model parameters corresponding to each language in the fully connected layer. When the model is used, it is only necessary to call the model parameters of the fully connected layer corresponding to the language of the audio to be evaluated for processing. There is no need to train a phoneme detection model for the pronunciation audio of each language. It is only necessary to set some model parameters for different languages in one model to correct the deviation between the proprietary phonemes and common phonemes of different languages. This greatly saves the training cost of the model and expands the application scope of the model. One model can be used to perform phoneme detection on audio in multiple languages at the same time.

[0208] Step 624: Obtain a multilingual phoneme detection model based on the model parameters corresponding to the multiple languages.

[0209] The above step 623c is performed on the first training sample of each language to obtain the model parameters in the fully connected layer corresponding to each language. After the model parameters corresponding to each language are trained and set, a multilingual phoneme detection model can be obtained.

[0210] In one example, if Figure 7 FIGURE 1 illustrates a schematic diagram of a multilingual phoneme detection model. In the figure, an audio frame 70 of spoken audio passes through a feature extraction layer 71 to generate a frequency domain feature vector (not shown). This frequency domain feature vector is then input into a multilingual phoneme detection model 72 to obtain a probability value 74 for each universal phoneme 73 of the audio frame 70.

[0211] Step 630: Extract phoneme information of the second training sample using a multilingual phoneme detection model.

[0212] The phoneme information of the second training sample includes the phoneme information of the teacher's pronunciation audio and the phoneme information of the student's pronunciation audio in the second training sample.

[0213] For the explanation of the phoneme information, please refer to the above embodiment and will not be repeated here.

[0214] Step 640: Based on the phoneme information of the second training sample, obtain the similarity between the teacher's pronunciation audio and the student's pronunciation audio.

[0215] For the description of the similarity between the teacher's pronunciation audio and the student's pronunciation audio, please refer to the above embodiment and will not be repeated here.

[0216] Step 650: Fit a scoring model based on the similarity and the pronunciation score of the student's pronunciation audio. The scoring model is used to score the audio to be evaluated.

[0217] Optionally, the similarity between the student pronunciation audio and the teacher pronunciation audio in the second training sample set is used as the input of the regression model, and the pronunciation score given by the teacher to the student pronunciation audio is used as the output of the regression model. The mathematical relationship between the above similarity and the pronunciation score is statistically analyzed to fit a regression model as the scoring model in the embodiment of the present application.

[0218] In summary, the technical solution provided by the embodiment of the present application trains a multilingual phoneme detection model by using sample audio with universal phoneme information as labels, so that the multilingual phoneme detection model can accurately detect the universal phoneme information corresponding to the teacher's pronunciation and the student's pronunciation of the same paragraph, and on this basis calculate the similarity between the two, and then fit a regression model to describe the statistical relationship between the scoring label carried by the student's pronunciation and the similarity. Finally, the pronunciation evaluation system is trained to complete the application scope of the model, realize the multilingual user pronunciation evaluation function, effectively reduce the complexity of pronunciation evaluation, and save storage resources.

[0219] In addition, for different languages, multiple sets of corresponding model parameters are set in the multilingual phoneme detection model to ensure the accuracy of pronunciation evaluation in multilingual cases.

[0220] Please refer to Figure 8 , which shows a flow chart of a method for obtaining a first training sample set provided by one embodiment of the present application. The pronunciation evaluation system includes a multilingual phoneme detection model and a scoring model. The method may include the following steps (810-830):

[0221] Step 810: Obtain original label data of the teacher's pronunciation audio in each language.

[0222] The original label data includes the unique phonemes of each pronunciation in the teacher's pronunciation audio. Unique phonemes are phonemes that are unique to a language. The difference between the original label data and the label data of the first training sample mentioned above is only that the phonemes corresponding to each pronunciation are different.

[0223] In an exemplary embodiment, the above step 810 includes the following sub-steps:

[0224] Step 811: Acquire phoneme sets of various languages.

[0225] The phone set (PS) of each language mentioned above refers to a set consisting of the exclusive phonemes of each language.

[0226] Step 812: Based on the phoneme sets of each language, the teacher pronunciation audios of each language are aligned to obtain the original label data of the teacher pronunciation audios of each language.

[0227] Optionally, the pronunciation audio and transcript are aligned at the phoneme level using the open-source Montreal Force Alignment tool.

[0228] The inputs to the Montreal Force Alignment tool described above include:

[0229] Prepare a file to store the correspondence between text and phonemes. In one possible implementation, text and phonemes can be mapped by constructing a lexicon. The lexicon serves as a bridge between the acoustic model and the language model, enabling a mapping from words to phonemes. The lexicon contains words that can be recognized by the speech recognition system and indicates their pronunciation. English words use words and phonetic symbols, while Chinese uses Chinese characters and pinyin for mapping, thus connecting the acoustic model and the language model to form a state space for search and preparing for decoding.

[0230] Prepare a lab file with the same name as the audio file to store the phonemes corresponding to the audio. With the above-mentioned file corresponding to the text and phonemes, there is no need to separate the phonemes here. The phonemes of the same word can be written directly together without spaces.

[0231] Prepare the pronunciation audio file. The official requirement is that the sampling rate should be greater than 16k.

[0232] The Montreal forced alignment tool can output the phonemes corresponding to each audio frame, the start and end times of each phoneme in each audio, and serve as the original label data for the teacher's pronunciation audio of each language.

[0233] Step 820: Obtain a universal phoneme set.

[0234] The universal phoneme set is a collection of universal phonemes.

[0235] In an exemplary embodiment, the above step 820 includes the following sub-steps:

[0236] Step 821 : Based on the teacher pronunciation audio of each language and the original label data of the teacher pronunciation audio of each language, respectively train the phoneme recognizer of each language.

[0237] The structure of the phoneme recognizers (Phone Recogization, PR) for each of the above-mentioned languages is the same as that of the multilingual phoneme detection model initialized above, and the training process is similar. The difference lies in the different label data in the training samples, and therefore the function of extracting phoneme information is different. The label data in the training samples of the initialized multilingual phoneme detection model are the universal phonemes corresponding to each audio frame, and the label data in the training samples of the phoneme recognizers of each language are the specific phonemes corresponding to each audio frame. For example, the label data of the English phoneme recognizer are the English phonemes corresponding to each audio frame. The phoneme information extracted by the initialized multilingual phoneme detection model is the universal phoneme corresponding to each audio frame, and the phoneme information extracted by the phoneme recognizers of each language is the specific phoneme corresponding to each audio frame.

[0238] Optionally, the phoneme recognizers of the above-mentioned languages have different numbers of parameters from the initialized multilingual phoneme detection model. The initialized multilingual phoneme detection model needs to add appropriate parameters, and the number of parameters of the initialized multilingual phoneme detection model is controlled within twice the number of parameters of the phoneme recognizer of a certain language.

[0239] Step 822: Send the teacher's pronunciation audio of each language to the phoneme recognizer of each language to determine similar phonemes in each language.

[0240] Similar phonemes refer to the proprietary phonemes with similar or identical pronunciations in different languages.

[0241] In a possible implementation, the specific implementation process of step 822 is as follows:

[0242] Step 822a: For the target language, the target language teacher's pronunciation audio is sent to the target language phoneme recognizer to obtain the probability value of each audio frame in the target language teacher's pronunciation audio under each specific phoneme of the target language.

[0243] For example, a recording of English content is fed into an English phoneme recognizer to obtain the probability value of each audio frame in the recording under each English phoneme, that is, a probability distribution is obtained to describe the possibility that the phoneme corresponding to the audio frame is each phoneme.

[0244] Step 822b: Determine the specific phonemes corresponding to each audio frame based on the probability values of each audio frame under each specific phoneme of the target language.

[0245] The target language's specific phonemes are then used to determine the corresponding specific phonemes for each audio frame. For example, if the probability of the phoneme "a" is the highest among the specific phonemes for a particular audio frame, then the phoneme corresponding to that audio frame is determined to be "a."

[0246] Step 822c: Input the audio frame corresponding to the first specific phoneme in the first language into a phoneme recognizer in the second language to obtain a probability value of the audio frame corresponding to the first specific phoneme under each specific phoneme in the second language.

[0247] The first language and the second language are different languages. The first proper phoneme is a proper phoneme in the first language. The similarity between the pronunciation of each proper phoneme in the second language and the first proper phoneme in the first language is quantitatively reflected by calculating the probability value of the audio frame corresponding to the first proper phoneme under each proper phoneme in the second language.

[0248] Step 822d: The probability value of the audio frame corresponding to the first specific phoneme under each specific phoneme of the second language is used as the inverse of the distance from the first specific phoneme to each specific phoneme of the second language.

[0249] Here, the distance between the first unique phoneme and each unique phoneme in the second language is used to measure the pronunciation similarity between each unique phoneme in the second language and the first unique phoneme in the first language. For example, if the probability value of phoneme 1 in language 1 is high under phoneme 2 in language 2, then the smaller the distance between phoneme 1 in language 1 and phoneme 2 in language 2, the more similar the pronunciation of phoneme 1 in language 1 and phoneme 2 in language 2 is.

[0250] Step 822e: Based on the distance, similar phonemes in each language are determined by a clustering algorithm.

[0251] Optionally, if the distance between the proprietary phonemes in each language meets the threshold condition, they can be determined as similar phonemes. The above threshold is a preset value used to distinguish the distance, and the embodiment of the present application does not limit the setting of the threshold.

[0252] The above clustering algorithm refers to cluster analysis, also known as group analysis, which is a statistical analysis method for studying classification problems and is also an important algorithm for data mining. Cluster analysis is composed of several patterns. Usually, a pattern is a vector of measurement or a point in a multidimensional space. Cluster analysis is based on similarity. Patterns in a cluster have more similarities than patterns in different clusters. Optionally, this application uses the k-means clustering algorithm.

[0253] Step 823: Bind and merge similar phonemes into common phonemes to obtain a common phoneme set.

[0254] Optionally, multiple similar phonemes are bound and merged into a common phoneme, and the name of the common phoneme can be determined according to actual conditions.

[0255] Alternatively, a specific phoneme having no similar phonemes becomes a universal phoneme alone.

[0256] Step 830 : Based on the universal phoneme set, replace the specific phonemes corresponding to each pronunciation in each audio frame of the teacher's pronunciation audio to obtain label data of the first training sample.

[0257] Optionally, based on the correspondence between similar phonemes and universal phonemes, the universal phonemes are used to replace the specific phonemes corresponding to each pronunciation in each audio frame in the teacher's pronunciation audio.

[0258] In summary, the technical solution provided in the embodiments of the present application uses a clustering algorithm to bind phonemes with similar pronunciations in various languages into universal phonemes, generate a universal phoneme set, and then replace the proprietary phonemes of each language, laying the foundation for realizing the user pronunciation evaluation function in multiple languages.

[0259] In an exemplary embodiment, the first training sample in the first training sample set must satisfy at least one of the following conditions.

[0260] 1. The recording is noise-free, or the signal-to-noise ratio (SNR) is greater than 30dB.

[0261] The signal-to-noise ratio (SNR or S / N) is the ratio of an amplifier's output signal power to the noise power it produces, often expressed in decibels. A higher SNR indicates less noise generated by a device. Generally speaking, a higher SNR indicates less noise mixed in with the signal, resulting in higher-quality playback. Otherwise, the opposite is true.

[0262] The signal-to-noise ratio (SNR) is the ratio of the signal level to the noise level. It's expressed in dB. For impulse noise, it's usually expressed as peak value, while for random noise, it's expressed as root mean square (RMS). It's abbreviated as S / N or SNR. The normalized SNR is the ratio of the signal energy per bit to the power spectral density of the noise, expressed in dB. Alternatively, it refers to the ratio of the signal power to the noise power at a point in the communication system. Different systems use different expressions depending on their needs. In analog communication systems, it often refers to the ratio of the average useful signal power to the average noise power at the demodulator output of a communication terminal. In digital communication systems, it often refers to the ratio of the average signal energy per symbol (bit) to the noise power per unit frequency band at the output of the terminal's digital demodulator and decoder. The SNR is an important parameter that measures the impact of noise on the signal. The SNR can be improved by improving transmission methods and enhancing equipment capabilities.

[0263] A simple definition of "noise" is "signals generated by the device during processing," unrelated to the input signal. For MP3 players, the signal-to-noise ratio (SNR) is a crucial parameter. It refers to the ratio of the maximum undistorted sound signal intensity produced by the source to the intensity of the noise emitted simultaneously. This ratio, often expressed as S / N and measured in decibels (dB), is the ratio of the maximum undistorted sound signal intensity produced by the source to the noise intensity. For players, a higher S / N ratio is ideal.

[0264] Second, the recording needs to be free of reverberation, or the reverberation time of RT60 should be less than 0.3 seconds.

[0265] RT60 refers to the reverberation time of a room. Specifically, it's defined as the time it takes for the sound field to decay by 60 dB, measured in seconds. The larger the RT60, the longer it takes for the sound produced in the room to fade away. The generally accepted definition of reverberation time is: the time it takes for the sound energy density to drop to 1 / 10^6 of its original value, equivalent to a 60dB decrease in the sound pressure level. The reverberation time at a given frequency is the time it takes for the sound in the room to reach a steady state. After the sound source ceases, the residual sound is repeatedly absorbed by the room's sound-absorbing materials, and the average sound energy density decays from its original value to one millionth (a 60dB decrease in sound energy density). This is expressed as T60 or RT. Too short a reverberation time results in a dry, dull, and unnatural sound; too long a reverberation time results in a mixed sound; at the appropriate level, the sound is mellow and pleasant.

[0266] 3. The duration of the audio recording should be less than 10 minutes, and preferably there should be one speaker in each audio segment. Optionally, the speakers should be evenly distributed in gender and age range from 15-60 years old.

[0267] 4. The total duration of recordings for a language in the ARAT dataset should exceed 200 hours, representing more than 200 speakers.

[0268] 5. The transcribed text does not need to be accurate to the time point of each word.

[0269] 6. Audio recordings should be made using a microphone that represents a flat response in the range of 200-500 Hz.

[0270] 7. The speech signal is considered to be wideband (bandwidth = 8kHz) and saved in PCM format at a sampling rate of 16kHz with a bit depth of 16 bits.

[0271] Pulse Code Modulation (PCM) is a coding method used in digital communications. It samples analog signals such as voice and images at regular intervals to discretize them. The sampled values are rounded off and quantized to the nearest integer. The sampled values are then represented by a set of binary codes to represent the amplitude of the sampled pulses.

[0272] 8. The spoken words in the language database should represent a wide variety of words so that the model learns a balanced representation of the acoustic model for the underlying phonemes.

[0273] In an exemplary embodiment, the second training sample in the second training sample set must satisfy at least one of the following conditions.

[0274] 1. The recording is noise-free, or the signal-to-noise ratio (SNR) is greater than 30dB.

[0275] Second, the recording needs to be free of reverberation, or the reverberation time of RT60 should be less than 0.3 seconds.

[0276] 3. The duration of the audio recording should be less than 10 seconds, and there should be one speaker (teacher or student) in one audio segment. Optionally, the speakers should be evenly distributed in the gender and age range of 15-60 years old.

[0277] 4. Audio recordings should be made using a microphone that represents a flat response in the range of 200-500 Hz.

[0278] Fifth, the speech signal is considered to be wideband (bandwidth = 8 kHz) and is saved in PCM format at a sampling rate of 16 kHz with a bit depth of 16 bits.

[0279] 6. Each language has at least 30 sentences.

[0280] 7. Each sentence must be recorded by at least 10 native speakers (teachers). This means at least 300 teacher recordings are required.

[0281] 8. Each teacher's recording will be imitated by 10 students, generating at least 3,000 student recordings.

[0282] 9. Each teacher will score 300 student recordings based on whether their pronunciation is standard.

[0283] In summary, the technical solution provided in the embodiments of the present application collects and obtains training samples according to various conditions, obtains high-quality training samples, and ensures the detection accuracy of the trained multilingual phoneme detection model.

[0284] The following are device embodiments of the present application, which can be used to implement the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0285] Please refer to Figure 9 , which shows a block diagram of a pronunciation evaluation device provided by one embodiment of the present application. The device has the function of implementing the above-mentioned pronunciation evaluation method. The device 900 may include: an audio acquisition module 910, a feature extraction module 920, a similarity determination module 930, and a score determination module 940.

[0286] The audio acquisition module 910 is used to acquire the pronunciation audio to be evaluated.

[0287] The feature extraction module 920 is used to extract phoneme information of the pronunciation audio to be evaluated. The phoneme information includes common phonemes of each pronunciation in the pronunciation audio. The common phonemes are phonemes obtained by integrating specific phonemes of multiple languages.

[0288] The similarity determination module 930 is configured to obtain the similarity between the pronunciation audio to be evaluated and the teacher pronunciation audio based on the phoneme information of the pronunciation audio to be evaluated and the phoneme information of the teacher pronunciation audio corresponding to the pronunciation audio to be evaluated.

[0289] The score determination module 940 is configured to determine a pronunciation score of the pronunciation audio to be evaluated based on the similarity.

[0290] In the exemplary embodiment, please refer to Figure 10 The feature extraction module 920 includes: an audio frame division unit 921, a frequency domain conversion unit 922 and a phoneme prediction unit 923.

[0291] The audio frame division unit 921 is configured to divide the pronunciation audio to be evaluated into at least one audio frame.

[0292] The frequency domain conversion unit 922 is used to obtain the frequency domain features of each audio frame in the pronunciation audio to be evaluated.

[0293] a phoneme prediction unit 923 configured to process the frequency domain features of each audio frame in the audio to be evaluated using a multilingual phoneme detection model to obtain phoneme information of the audio to be evaluated, wherein the phoneme information of the audio to be evaluated includes a probability distribution of each audio frame in the audio to be evaluated, wherein the probability distribution reflects the probability of the pronunciation of the audio frame at each universal phoneme;

[0294] The multilingual phoneme detection model is a machine learning model used to determine the universal phonemes corresponding to each audio frame in the pronunciation audio.

[0295] In an exemplary embodiment, the phoneme prediction unit 923 is configured to:

[0296] Obtaining the target language of the pronunciation audio to be evaluated;

[0297] Selecting target model parameters corresponding to the target language; wherein the multilingual phoneme detection model has multiple sets of model parameters, and different model parameters correspond to different languages;

[0298] The frequency domain features of each audio frame in the pronunciation audio to be evaluated are processed by the multilingual detection model according to the target model parameters to obtain the phoneme information of the pronunciation audio to be evaluated.

[0299] In an exemplary embodiment, the similarity determination module 930 is configured to:

[0300] Performing dynamic time normalization processing on the distribution probability vector of each audio frame in the audio of the pronunciation to be evaluated and the distribution probability vector of each audio frame in the audio of the teacher's pronunciation to obtain a timing matching result, wherein the distribution probability vector is a vector composed of the probability of the pronunciation of the audio frame on each universal phoneme, and the timing matching result is a combination of the matching distribution probability vectors obtained by aligning the audio of the pronunciation to be evaluated and the audio of the teacher's pronunciation according to the same universal phoneme in time sequence;

[0301] Based on the similarity between the matched distribution probability vectors determined based on the time series matching result, the similarity between the pronunciation audio to be evaluated and the teacher's pronunciation audio is obtained.

[0302] In an exemplary embodiment, the score determination module 940 is configured to:

[0303] The pronunciation score of the pronunciation audio to be evaluated is determined according to the similarity through a scoring model, and the scoring model is a regression model for quantitatively describing the statistical relationship between the similarity and the pronunciation score.

[0304] To sum up, the technical solution provided in the embodiment of the present application determines the similarity between the student's pronunciation and the teacher's pronunciation by comparing the universal phonemes applicable to multiple languages corresponding to the student's pronunciation in any language and the teacher's pronunciation, and then derives a score for the student's pronunciation. It does not require multiple models to realize pronunciation evaluation in multiple languages, effectively reducing the complexity of pronunciation evaluation while saving storage resources.

[0305] In addition, by setting the model parameters corresponding to each language in the multilingual phoneme detection model, the probability distribution of the student pronunciation and the teacher pronunciation of any language under each common phoneme is detected according to the model parameters corresponding to the language, and the similarity between the student pronunciation and the teacher pronunciation is determined based on the probability distribution of the two. The student pronunciation score is then output through the regression model, which reduces the number of models required for multilingual pronunciation detection and also ensures the accuracy of pronunciation evaluation in multilingual cases.

[0306] Please refer to Figure 11, which shows a block diagram of a training device for a pronunciation evaluation system provided in one embodiment of the present application. The pronunciation evaluation system includes a multilingual phoneme detection model and a scoring model. The device has the function of implementing the training method for the pronunciation evaluation system described above. The device 1100 may include: a sample acquisition module 1110, a model training module 1120, a feature extraction module 1130, a similarity determination module 1140, and a model fitting module 1150.

[0307] The sample acquisition module 1110 is used to obtain a first training sample set and a second training sample set, wherein the first training sample set includes at least one first training sample, the sample data of the first training sample includes the teacher's pronunciation audio in any language, and the label data of the first training sample includes the common phonemes of each pronunciation in the teacher's pronunciation audio. The second training sample set includes at least one second training sample, the sample data of the second training sample includes the teacher's pronunciation audio and the student's pronunciation audio for the same paragraph, and the label data of the second training sample includes the pronunciation score of the student's pronunciation audio.

[0308] The model training module 1120 is used to train the multilingual phoneme detection model based on the first training sample set. The multilingual phoneme detection model is used to extract phoneme information from the pronunciation audio. The phoneme information includes common phonemes for each pronunciation in the pronunciation audio. The common phonemes are phonemes obtained by integrating the proprietary phonemes of multiple languages.

[0309] The feature extraction module 1130 is used to extract the phoneme information of the second training sample through the multilingual phoneme detection model, where the phoneme information of the second training sample includes the phoneme information of the teacher's pronunciation audio and the phoneme information of the student's pronunciation audio in the second training sample.

[0310] The similarity determination module 1140 is configured to obtain the similarity between the teacher's pronunciation audio and the student's pronunciation audio based on the phoneme information of the second training sample.

[0311] The model fitting module 1150 is used to fit the scoring model based on the similarity and the pronunciation score of the student's pronunciation audio, and the scoring model is used to score the audio to be evaluated.

[0312] In the exemplary embodiment, please refer to Figure 12 The model training module 1120 includes: a model pre-training unit 1121, a sample selection unit 1122, a parameter adjustment unit 1123 and a model generation unit 1124.

[0313] The model pre-training unit 1121 is configured to train an initialized multilingual phoneme detection model based on the first training sample set, where the initialized multilingual phoneme detection model is a pre-trained model for detecting universal phonemes.

[0314] The sample selection unit 1122 is configured to obtain, for a target language, a first training sample of the target language from the first training sample set.

[0315] The parameter adjustment unit 1123 is configured to adjust the model parameters of the initialized multilingual phoneme detection model using the first training sample of the target language to obtain model parameters corresponding to the target language.

[0316] The model generation unit 1124 is configured to obtain the multilingual phoneme detection model based on the model parameters corresponding to the multiple languages.

[0317] In an exemplary embodiment, the parameter adjustment unit 1123 is configured to:

[0318] Fixing the parameters of the middle layer in the initialized multilingual phoneme detection model;

[0319] Replacing the full-pass layers at the input and output ends of the initialized multilingual phoneme detection model with fully connected layers, wherein the full-pass layers do not process data and the fully connected layers are language-related layers used to mine language features;

[0320] The model parameters of the fully connected layer are adjusted using the first training sample of the target language to obtain model parameters corresponding to the target language.

[0321] In the exemplary embodiment, please refer to Figure 12 The sample acquisition module 1110 includes: an original label acquisition unit 1111, a universal phoneme acquisition unit 1112 and a phoneme replacement unit 1113.

[0322] The original label acquisition unit 1111 is used to acquire original label data of the teacher's pronunciation audio in each language. The original label data includes the specific phonemes of each pronunciation in the teacher's pronunciation audio. The specific phonemes are phonemes that are specific to a language.

[0323] The universal phoneme acquisition unit 1112 is configured to acquire a universal phoneme set, where the universal phoneme set is a set of the universal phonemes.

[0324] The phoneme replacement unit 1113 is used to replace the specific phonemes corresponding to each pronunciation in each audio frame in the teacher's pronunciation audio based on the universal phoneme set to obtain the label data of the first training sample.

[0325] In an exemplary embodiment, the universal phoneme acquisition unit 1112 is configured to:

[0326] Based on the teacher pronunciation audio of each language and the original label data of the teacher pronunciation audio of each language, respectively train the phoneme recognizer of each language;

[0327] Sending the teacher's pronunciation audio of each language to the phoneme recognizer of each language to determine similar phonemes in each language, wherein the similar phonemes refer to proprietary phonemes with similar pronunciations in each language;

[0328] The similar phonemes are bound and merged into the universal phonemes to obtain the universal phoneme set.

[0329] In an exemplary embodiment, the universal phoneme acquiring unit 1112 is further configured to:

[0330] For the target language, the teacher's pronunciation audio of the target language is fed into the phoneme recognizer of the target language to obtain the probability value of each audio frame in the teacher's pronunciation audio of the target language under each specific phoneme of the target language;

[0331] Determining the specific phonemes corresponding to the audio frames based on the probability values of the audio frames under the specific phonemes of the target language;

[0332] inputting an audio frame corresponding to a first specific phoneme in a first language into a phoneme recognizer in a second language to obtain a probability value of the audio frame corresponding to the first specific phoneme under each specific phoneme in the second language, where the first language and the second language are different languages;

[0333] Using the probability value of the audio frame corresponding to the first specific phoneme under each specific phoneme of the second language as the inverse of the distance between the first specific phoneme and each specific phoneme of the second language;

[0334] Based on the distances, similar phonemes in the languages are determined using a clustering algorithm.

[0335] In an exemplary embodiment, the original label acquiring unit 1111 is configured to:

[0336] Obtaining a phoneme set of each language;

[0337] Based on the phoneme sets of the various languages, the teacher pronunciation audios of the various languages are aligned to obtain original label data of the teacher pronunciation audios of the various languages.

[0338] In summary, the technical solution provided by the embodiment of the present application trains a multilingual phoneme detection model by using sample audio with universal phoneme information as labels, so that the multilingual phoneme detection model can accurately detect the universal phoneme information corresponding to the teacher's pronunciation and the student's pronunciation of the same paragraph, and on this basis calculate the similarity between the two, and then fit a regression model to describe the statistical relationship between the scoring label carried by the student's pronunciation and the similarity. Finally, the pronunciation evaluation system is trained to complete the application scope of the model, realize the multilingual user pronunciation evaluation function, effectively reduce the complexity of pronunciation evaluation, and save storage resources.

[0339] Furthermore, a clustering algorithm binds similar-sounding phonemes across languages into universal phonemes, generating a universal phoneme set that replaces the specific phonemes of each language, laying the foundation for multilingual user pronunciation assessment. Furthermore, the multilingual phoneme detection model uses multiple sets of corresponding model parameters for different languages to ensure accurate pronunciation assessment across multiple languages.

[0340] Please refer to Figure 13 , which shows a block diagram of a computer device 1300 provided in one embodiment of the present application. The computer device 1300 may be an electronic device such as a mobile phone, a tablet computer, a multimedia player, a wearable device, a PC (Personal Computer), a language learning terminal, an intelligent teaching machine, etc. The computer device is used to implement the pronunciation evaluation method or the training method of the pronunciation evaluation system provided in the above embodiment. The computer device may be Figure 1 The terminal 10 or server 20 in the application execution environment is shown.

[0341] Typically, the computer device 1300 includes a processor 1301 and a memory 1302 .

[0342] The processor 1301 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 1301 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), or PLA (Programmable Logic Array). The processor 1301 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 1301 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1301 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0343] The memory 1302 may include one or more computer-readable storage media, which may be non-transitory. The memory 1302 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1302 is used to store at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is configured to be executed by one or more processors to implement the above-mentioned pronunciation evaluation method or the training method of the pronunciation evaluation system.

[0344] In some embodiments, computer device 1300 may optionally include a peripheral device interface 1303 and at least one peripheral device. Processor 1301, memory 1302, and peripheral device interface 1303 may be connected via a bus or signal lines. Each peripheral device may be connected to peripheral device interface 1303 via a bus, signal lines, or circuit boards.

[0345] Those skilled in the art will understand that Figure 13 The structure shown in the figure does not constitute a limitation on the computer device 1300, and the computer device 1300 may include more or fewer components than shown in the figure, or combine some components, or adopt a different component arrangement.

[0346] In an exemplary embodiment, a computer-readable storage medium is further provided, wherein a computer program is stored in the storage medium. When the computer program is executed by a processor, the computer program implements the above-mentioned pronunciation evaluation method.

[0347] In an exemplary embodiment, a computer-readable storage medium is further provided, wherein a computer program is stored in the storage medium. When the computer program is executed by a processor, the computer program implements the training method of the pronunciation evaluation system.

[0348] Optionally, the computer-readable storage medium may include: ROM (Read Only Memory), RAM (Random Access Memory), SSD (Solid State Drives), or an optical disk, etc. Among them, the random access memory may include ReRAM (Resistance Random Access Memory) and DRAM (Dynamic Random Access Memory).

[0349] In an exemplary embodiment, a computer program product or computer program is also provided. The computer program product or computer program includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the above-described pronunciation evaluation method.

[0350] In an exemplary embodiment, a computer program product or computer program is also provided. The computer program product or computer program includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the training method of the pronunciation evaluation system described above.

[0351] It should be understood that the "multiple" mentioned in this article refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent three situations: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previous and subsequent associated objects are in an "or" relationship. In addition, the step numbers described in this article only illustrate a possible execution sequence between the steps. In some other embodiments, the above steps may not be executed in the order of the numbers, such as two steps with different numbers are executed at the same time, or two steps with different numbers are executed in the opposite order to the diagram. The embodiments of the present application do not limit this.

[0352] The above description is merely an exemplary embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A pronunciation evaluation method, characterized in that: The method comprises: Get the pronunciation audio to be evaluated; Dividing the pronunciation audio to be evaluated into at least one audio frame; Obtaining frequency domain features of each audio frame in the pronunciation audio to be evaluated; Obtaining the target language of the pronunciation audio to be evaluated; selecting target model parameters corresponding to the target language; The frequency domain features of each audio frame in the audio of the pronunciation to be evaluated are processed according to the target model parameters by a multilingual phoneme detection model to obtain phoneme information of the audio of the pronunciation to be evaluated. The multilingual phoneme detection model includes an intermediate layer, a fully connected layer at the input end, and a fully connected layer at the output end. The fully connected layer in the multilingual phoneme detection model is a language-related layer for mining language features. The intermediate layer in the multilingual phoneme detection model has a set of parameters. The fully connected layer in the multilingual phoneme detection model has multiple sets of model parameters. Different sets of model parameters in the multiple sets of model parameters correspond to different languages. The target model parameters are one set of the multiple sets of model parameters. The phoneme information of the audio of the pronunciation to be evaluated includes a probability distribution of each audio frame in the audio of the pronunciation to be evaluated. The probability distribution reflects the probability of the pronunciation of the audio frame on each universal phoneme. The universal phoneme is a phoneme obtained by integrating the proprietary phonemes of multiple languages. Based on the phoneme information of the pronunciation audio to be evaluated and the phoneme information of the teacher pronunciation audio corresponding to the pronunciation audio to be evaluated, obtaining the similarity between the pronunciation audio to be evaluated and the teacher pronunciation audio; Based on the similarity, a pronunciation score of the pronunciation audio to be evaluated is determined.

2. The method according to claim 1, characterized in that The obtaining of the similarity between the pronunciation audio to be evaluated and the teacher pronunciation audio based on the phoneme information of the pronunciation audio to be evaluated and the phoneme information of the teacher pronunciation audio corresponding to the pronunciation audio to be evaluated includes: Performing dynamic time normalization processing on the distribution probability vector of each audio frame in the audio of the pronunciation to be evaluated and the distribution probability vector of each audio frame in the audio of the teacher's pronunciation to obtain a timing matching result, wherein the distribution probability vector is a vector composed of the probability of the pronunciation of the audio frame on each universal phoneme, and the timing matching result is a combination of the matching distribution probability vectors obtained by aligning the audio of the pronunciation to be evaluated and the audio of the teacher's pronunciation according to the same universal phoneme in time sequence; Based on the similarity between the matched distribution probability vectors determined based on the time series matching result, the similarity between the pronunciation audio to be evaluated and the teacher's pronunciation audio is obtained.

3. The method according to claim 1 or 2, characterized in that Determining the pronunciation score of the pronunciation audio to be evaluated based on the similarity includes: The pronunciation score of the pronunciation audio to be evaluated is determined according to the similarity through a scoring model, and the scoring model is a regression model for quantitatively describing the statistical relationship between the similarity and the pronunciation score.

4. A training method for a pronunciation evaluation system, characterized in that: The pronunciation evaluation system includes a multilingual phoneme detection model and a scoring model; the method includes: Obtain a first training sample set and a second training sample set, wherein the first training sample set includes at least one first training sample, sample data of the first training sample includes teacher pronunciation audio in any language, and label data of the first training sample includes common phonemes of each pronunciation in the teacher pronunciation audio; the second training sample set includes at least one second training sample, sample data of the second training sample includes teacher pronunciation audio and student pronunciation audio for the same paragraph, and label data of the second training sample includes pronunciation scores of the student pronunciation audio; Training the multilingual phoneme detection model based on the first training sample set, wherein the multilingual phoneme detection model is used to extract phoneme information from the pronunciation audio, wherein the phoneme information includes common phonemes of each pronunciation in the pronunciation audio, wherein the common phonemes are phonemes obtained by integrating specific phonemes of multiple languages; Extracting phoneme information of the second training sample using the multilingual phoneme detection model, where the phoneme information of the second training sample includes phoneme information of the teacher's pronunciation audio and phoneme information of the student's pronunciation audio in the second training sample; Based on the phoneme information of the second training sample, obtaining the similarity between the teacher's pronunciation audio and the student's pronunciation audio; The scoring model is fitted based on the similarity and the pronunciation score of the student's pronunciation audio, and the scoring model is used to score the audio to be evaluated.

5. The method according to claim 4, characterized in that The step of training the multilingual phoneme detection model based on the first training sample set includes: Training an initialized multilingual phoneme detection model based on the first training sample set, where the initialized multilingual phoneme detection model is a pre-trained model for detecting universal phonemes; For a target language, obtaining a first training sample of the target language from the first training sample set; Adjusting model parameters of the initialized multilingual phoneme detection model using a first training sample of the target language to obtain model parameters corresponding to the target language; The multilingual phoneme detection model is obtained based on the model parameters corresponding to the multiple languages.

6. The method according to claim 5, characterized in that The adjusting the model parameters of the initialized multilingual phoneme detection model using the first training sample of the target language to obtain model parameters corresponding to the target language includes: Fixing the parameters of the middle layer in the initialized multilingual phoneme detection model; Replacing the full-pass layers at the input and output ends of the initialized multilingual phoneme detection model with fully connected layers, wherein the full-pass layers do not process data and the fully connected layers are language-related layers used to mine language features; The model parameters of the fully connected layer are adjusted using the first training sample of the target language to obtain model parameters corresponding to the target language.

7. The method according to any one of claims 4 to 6, characterized in that The obtaining of the first training sample set includes: Obtaining original label data of the teacher's pronunciation audio of each language, wherein the original label data includes the specific phonemes of each pronunciation in the teacher's pronunciation audio, wherein the specific phonemes are phonemes specific to a language; Acquire a universal phoneme set, where the universal phoneme set is a set consisting of the universal phonemes; Based on the universal phoneme set, the proprietary phonemes corresponding to the respective pronunciations in the audio frames of the teacher's pronunciation audio are replaced to obtain the label data of the first training sample.

8. The method according to claim 7, characterized in that The obtaining of the universal phoneme set includes: Based on the teacher pronunciation audio of each language and the original label data of the teacher pronunciation audio of each language, respectively train the phoneme recognizer of each language; Sending the teacher's pronunciation audio of each language to the phoneme recognizer of each language to determine similar phonemes in each language, wherein the similar phonemes refer to proprietary phonemes with similar pronunciations in each language; The similar phonemes are bound and merged into the universal phonemes to obtain the universal phoneme set.

9. The method according to claim 8, characterized in that The step of sending the teacher's pronunciation audio of each language into the phoneme recognizer of each language to determine similar phonemes in each language includes: For the target language, the teacher's pronunciation audio of the target language is fed into the phoneme recognizer of the target language to obtain the probability value of each audio frame in the teacher's pronunciation audio of the target language under each specific phoneme of the target language; Determining the specific phonemes corresponding to the audio frames based on the probability values of the audio frames under the specific phonemes of the target language; inputting an audio frame corresponding to a first specific phoneme in a first language into a phoneme recognizer in a second language to obtain a probability value of the audio frame corresponding to the first specific phoneme under each specific phoneme in the second language, where the first language and the second language are different languages; Using the probability value of the audio frame corresponding to the first specific phoneme under each specific phoneme of the second language as the inverse of the distance between the first specific phoneme and each specific phoneme of the second language; Based on the distances, similar phonemes in the languages are determined using a clustering algorithm.

10. The method according to claim 7, characterized in that The original label data of the teacher's pronunciation audio of each language is obtained, including: Obtaining a phoneme set of each language; Based on the phoneme sets of the various languages, the teacher pronunciation audios of the various languages are aligned to obtain original label data of the teacher pronunciation audios of the various languages.

11. A pronunciation evaluation device, characterized in that: The device comprises: Audio acquisition module, used to obtain the pronunciation audio to be evaluated; The feature extraction module is used to divide the pronunciation audio to be evaluated into at least one audio frame; obtain the frequency domain features of each audio frame in the pronunciation audio to be evaluated; obtain the target language of the pronunciation audio to be evaluated; select the target model parameters corresponding to the target language; process the frequency domain features of each audio frame in the pronunciation audio to be evaluated according to the target model parameters through a multilingual phoneme detection model to obtain the phoneme information of the pronunciation audio to be evaluated; wherein the multilingual phoneme detection model includes an intermediate layer, a fully connected layer at the input end and a fully connected layer at the output end, and the multilingual phoneme detection model includes a plurality of layers, a plurality of layers, and a plurality of layers. The fully connected layer in the multilingual phoneme detection model is a language-related layer for mining language features. The intermediate layer in the multilingual phoneme detection model has a set of parameters. The fully connected layer in the multilingual phoneme detection model has multiple sets of model parameters, with different sets of model parameters corresponding to different languages. The target model parameters are one of the multiple sets of model parameters. The phoneme information of the pronunciation audio to be evaluated includes a probability distribution of each audio frame in the pronunciation audio to be evaluated. The probability distribution reflects the probability of the pronunciation of the audio frame on each universal phoneme. The universal phoneme is a phoneme obtained by integrating the proprietary phonemes of multiple languages. A similarity determination module, configured to obtain a similarity between the pronunciation audio to be evaluated and the teacher pronunciation audio based on the phoneme information of the pronunciation audio to be evaluated and the phoneme information of the teacher pronunciation audio corresponding to the pronunciation audio to be evaluated; The scoring determination module is used to determine the pronunciation score of the pronunciation audio to be evaluated based on the similarity.

12. The device according to claim 11, characterized in that The similarity determination module is configured to: Performing dynamic time normalization processing on the distribution probability vector of each audio frame in the audio of the pronunciation to be evaluated and the distribution probability vector of each audio frame in the audio of the teacher's pronunciation to obtain a timing matching result, wherein the distribution probability vector is a vector composed of the probability of the pronunciation of the audio frame on each universal phoneme, and the timing matching result is a combination of the matching distribution probability vectors obtained by aligning the audio of the pronunciation to be evaluated and the audio of the teacher's pronunciation according to the same universal phoneme in time sequence; Based on the similarity between the matched distribution probability vectors determined based on the time series matching result, the similarity between the pronunciation audio to be evaluated and the teacher's pronunciation audio is obtained.

13. The device according to claim 11 or 12, characterized in that The scoring determination module is used to: The pronunciation score of the pronunciation audio to be evaluated is determined according to the similarity through a scoring model, and the scoring model is a regression model for quantitatively describing the statistical relationship between the similarity and the pronunciation score.

14. A training device for a pronunciation evaluation system, characterized in that: The pronunciation evaluation system includes a multilingual phoneme detection model and a scoring model; the device includes: A sample acquisition module is configured to acquire a first training sample set and a second training sample set, wherein the first training sample set includes at least one first training sample, the sample data of the first training sample includes a teacher's pronunciation audio in any language, and the label data of the first training sample includes common phonemes of each pronunciation in the teacher's pronunciation audio; the second training sample set includes at least one second training sample, the sample data of the second training sample includes a teacher's pronunciation audio and a student's pronunciation audio for the same paragraph, and the label data of the second training sample includes a pronunciation score of the student's pronunciation audio; a model training module, configured to train the multilingual phoneme detection model based on the first training sample set, wherein the multilingual phoneme detection model is configured to extract phoneme information from the pronunciation audio, wherein the phoneme information includes common phonemes for each pronunciation in the pronunciation audio, wherein the common phonemes are phonemes obtained by integrating specific phonemes of multiple languages; a feature extraction module, configured to extract phoneme information of the second training sample using the multilingual phoneme detection model, wherein the phoneme information of the second training sample includes phoneme information of the teacher's pronunciation audio and phoneme information of the student's pronunciation audio in the second training sample; A similarity determination module, configured to obtain a similarity between the teacher's pronunciation audio and the student's pronunciation audio based on the phoneme information of the second training sample; The model fitting module is used to fit the scoring model based on the similarity and the pronunciation score of the student's pronunciation audio, and the scoring model is used to score the audio to be evaluated.

15. The device according to claim 14, characterized in that The model training module includes: a model pre-training unit, configured to train an initialized multilingual phoneme detection model based on the first training sample set, wherein the initialized multilingual phoneme detection model is a pre-trained model for detecting universal phonemes; a sample selection unit, configured to obtain, for a target language, a first training sample of the target language from the first training sample set; a parameter adjustment unit, configured to adjust model parameters of the initialized multilingual phoneme detection model using a first training sample of the target language to obtain model parameters corresponding to the target language; The model generation unit is used to obtain the multilingual phoneme detection model based on the model parameters corresponding to the multiple languages.

16. The device according to claim 15, characterized in that The parameter adjustment unit is used to: Fixing the parameters of the middle layer in the initialized multilingual phoneme detection model; Replacing the full-pass layers at the input and output ends of the initialized multilingual phoneme detection model with fully connected layers, wherein the full-pass layers do not process data and the fully connected layers are language-related layers used to mine language features; The model parameters of the fully connected layer are adjusted using the first training sample of the target language to obtain model parameters corresponding to the target language.

17. The device according to any one of claims 14 to 16, characterized in that The sample acquisition module includes: an original label acquisition unit, configured to acquire original label data of the teacher's pronunciation audio of each language, wherein the original label data includes the specific phonemes of each pronunciation in the teacher's pronunciation audio, wherein the specific phonemes are phonemes specific to a language; a universal phoneme acquisition unit, configured to acquire a universal phoneme set, wherein the universal phoneme set is a set consisting of the universal phonemes; The phoneme replacement unit is used to replace the specific phonemes corresponding to each pronunciation in each audio frame in the teacher's pronunciation audio based on the universal phoneme set to obtain the label data of the first training sample.

18. The device according to claim 17, characterized in that The universal phoneme acquisition unit is used to: Based on the teacher pronunciation audio of each language and the original label data of the teacher pronunciation audio of each language, respectively train the phoneme recognizer of each language; Sending the teacher's pronunciation audio of each language to the phoneme recognizer of each language to determine similar phonemes in each language, wherein the similar phonemes refer to proprietary phonemes with similar pronunciations in each language; The similar phonemes are bound and merged into the universal phonemes to obtain the universal phoneme set.

19. The device according to claim 18, characterized in that The universal phoneme acquisition unit is further configured to: For the target language, the teacher's pronunciation audio of the target language is fed into the phoneme recognizer of the target language to obtain the probability value of each audio frame in the teacher's pronunciation audio of the target language under each specific phoneme of the target language; Determining the specific phonemes corresponding to the audio frames based on the probability values of the audio frames under the specific phonemes of the target language; inputting an audio frame corresponding to a first specific phoneme in a first language into a phoneme recognizer in a second language to obtain a probability value of the audio frame corresponding to the first specific phoneme under each specific phoneme in the second language, where the first language and the second language are different languages; Using the probability value of the audio frame corresponding to the first specific phoneme under each specific phoneme of the second language as the inverse of the distance between the first specific phoneme and each specific phoneme of the second language; Based on the distances, similar phonemes in the languages are determined using a clustering algorithm.

20. The device according to claim 19, characterized in that The original label acquisition unit is configured to: Obtaining a phoneme set of each language; Based on the phoneme sets of the various languages, the teacher pronunciation audios of the various languages are aligned to obtain original label data of the teacher pronunciation audios of the various languages.

21. A computer device, characterized in that: The computer device includes a processor and a memory, wherein a computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the method according to any one of claims 1 to 3, or to implement the method according to any one of claims 4 to 10.

22. A computer-readable storage medium, characterized in that The storage medium stores a computer program, which is loaded and executed by a processor to implement the method according to any one of claims 1 to 3, or to implement the method according to any one of claims 4 to 10.

Images