Pronunciation evaluation model optimization method based on multi-scale fusion of neural networks
Through a multi-scale fusion pronunciation evaluation model based on neural networks, combined with speech recognition and convolutional neural network, the problem of insufficient evaluation of pronunciation information in automatic English pronunciation evaluation is solved, and the multi-scale fusion of pronunciation characteristics is realized, which improves the evaluation Meticulousness and accuracy enhance the naturalness and intelligibility of the language.
Patent Information
- Application Number
- CN202210772121.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-06-30
- Publication Date
- 2024-12-31
- Estimated Expiration
- 2042-06-30
AI Technical Summary
The existing technology lacks detailed evaluation of rhythm information in the automatic evaluation of English pronunciation, especially the evaluation of sub-items such as stress and rhythm. It is difficult to fully reflect the ups and downs of the speaker's pronunciation, tone and tone, and affect the naturalness and ability of the language. Understand.
Using a multi-scale fusion pronunciation evaluation model based on neural networks, combining end-to-end speech recognition model and multi-scale convolutional neural network, pronunciation features of different scales are extracted, and features are fused through attention mechanisms, combined with traditional pronunciation quality evaluation algorithms. Achieve the global and local evaluation of rhythm information.
It realizes a more detailed and accurate evaluation of pronunciation quality, takes into account the pronunciation-related characteristics of different particle sizes, improves the naturalness and comprehensibility of the language, and provides more objective and efficient pronunciation correction suggestions.
Smart Images
Figure CN115240710B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of speech pronunciation evaluation, and specifically to a pronunciation evaluation model optimization method based on multi-scale fusion of neural networks, and optimizes a pronunciation evaluation system by using neural networks and multi-scale fusion technology. Background Art
[0002] Automatic English pronunciation assessment is a technology in which the subject pronounces the specified English text, and the computer gives a score based on the subject's pronunciation quality. The computer conducts a fair, objective and efficient automatic assessment of the subject's English pronunciation level, assisting English language learners to correct pronunciation errors and improve their English speaking level. With the rapid development of the global economy, exchanges and cooperation between different countries in various aspects such as politics, economy, culture and education are becoming more frequent. More and more people are beginning to learn a second language other than their mother tongue. To master a language of communication, oral learning is essential. However, one-on-one learning and face-to-face interactive communication between teachers and students are often limited by time, space and economic conditions. Therefore, online education is becoming more and more popular, and the pronunciation assessment technology and application of automatically assessing learners' pronunciation and correcting accent errors by computers are also favored by learners.
[0003] At present, in domestic and foreign research, the automatic evaluation of prosodic pronunciation quality is mostly evaluated from the perspective of overall listening quality, while the evaluation of pronunciation quality for specific sub-items, such as the evaluation of stress pronunciation quality and rhythm pronunciation quality, is still relatively rare. When people communicate verbally, they transmit not only language and text information to each other, but also rich prosodic information. Prosodic information belongs to suprasegmental information, which mainly reflects the intonation (rhythm), emphasis (stress), intonation and tone of the speaker's pronunciation. On the one hand, prosodic information helps the speaker to express the information to be expressed more clearly and accurately, and improves the naturalness and comprehensibility of the language; on the other hand, prosodic information helps the listener to understand the information heard more clearly and accurately, and even includes the grasp and understanding of the speaker's intention, emotion, attitude, tone and other aspects. In the task of automatic pronunciation quality evaluation, it is very necessary and very important to evaluate the prosodic pronunciation quality.
[0004] In recent years, deep learning, as a new machine learning method, has been widely used in various fields of artificial intelligence. In this context, this patent proposes a multi-scale fusion pronunciation evaluation model based on neural networks to address the shortcomings of the above-mentioned pronunciation evaluation. The speech recognition model based on neural networks is used as the acoustic model, and multiple CNN networks of different scales are set to convolve the evaluation features to mine the rhythmic information of different scales. In addition, the attention mechanism model is used to Summary of the invention
[0005] In order to solve the technical problem raised in the background technology, the present invention adopts a pronunciation evaluation model optimization method based on multi-scale fusion of neural networks.
[0006] The technical solution of the present invention is a pronunciation evaluation model optimization method based on multi-scale fusion of neural networks, comprising the following steps:
[0007] Step 1: Design and select the acoustic model: Select an end-to-end speech recognition model as the acoustic model to calculate the GOP score of the audio to be evaluated. In addition, it is necessary to design and construct a pronunciation evaluation training dataset for subsequent model training.
[0008] Step 2, calculation of GOP score based on neural network: After completing the speech recognition model described in step 1, the speech recognition model of step 1 is used to recognize the speech to be evaluated, and the GOP score is calculated using the output of the neural network:
[0009]
[0010] Formula (1) uses the average frame-level posterior probability constructed by the neural network output as the GOP score; here P(s t ,O t ) is the output of the last softmax layer of the neural network model, where O refers to the observation sequence of speech, O t is the observation sequence corresponding to the speech frame at time t, t s and t e Respectively represent the start frame and end frame of phoneme P, s t is the state label of frame t after forced alignment;
[0011] Step 3: Construction of multi-scale convolutional neural network
[0012] 3) Extraction of prosody-related features: Extract the above-mentioned relevant prosody acoustic features of each frame and use them as the input of the convolutional neural network. Assuming that the speech to be evaluated is divided into N frames, and each frame contains the above-mentioned M-dimensional relevant prosody-related features, the input is an N*M matrix;
[0013] 4) Multi-scale neural network construction, using one-dimensional convolution to analyze and extract the original convolution features:
[0014] Set T (1, 2, ..., T) convolutional neural networks of different scales, and the convolution kernel size of each convolutional network is C1*1, C2*1, ..., C T *1, where the number of each convolution kernel is M;
[0015] Step 4: Multi-feature fusion based on attention mechanism:
[0016] 1) For the T prosodic features of different scales learned in step 3, assume that these T features are represented by S = [s1, s2, ..., s t ], according to the attention mechanism of the following formula (2), the final feature representation E can be calculated:
[0017] Q=Q'W q ,K=SW k ,V=SW v
[0018]
[0019] Among them, Q' is the vector randomly initialized by the neural network, W q ,W k , W v is the random initial matrix of the neural network used to perform linear transformation on Q′ and S. After linear transformation, the query vector Q, comparison vector K and content vector V will be obtained. f refers to the reduction of the dimension of the vector, d m Refers to the dimension of the vector, A refers to the scoring function of the attention mechanism. The present invention adopts softmax activation as the scoring function to fix the result within the range of 0-1. As the neural network continues to learn and update, it can eventually achieve the fusion of features of different scales;
[0020] 2) Calculate the score obtained by fusion features e The GOP score is further integrated as shown in the following formula (3):
[0021] score final =α*score e +(1-α)*GOP
[0022] α=sigmod(W α s t+1 +b α ) (3)
[0023] Among them, s t+1 is the output before softmax based on the neural network recognition model used in step 1, W α and b α The random initial matrix is also used for linear transformation in the evaluation model, α is the score e The corresponding weight, (1-α) is the weight corresponding to the GOP score, and the final score is final It comprehensively considers different rhythmic acoustic features and the final evaluation results of GOP. Sigmod is an activation function used to ensure that α is a weight value between 0 and 1.
[0024] Further evaluation steps:
[0025] 1) Receive the audio to be evaluated, and calculate the GOP score and the output s before the softmax of the recognition model through step 1 t+1 ;
[0026] 2) Extract rhythm-related features and extract corresponding deep features through CNNs of different scales;
[0027] 3) Fusion of features of different scales through attention mechanism;
[0028] 4) The fused feature score and the original GOP score are combined to obtain the final score.
[0029] Beneficial effects:
[0030] The technical solution of the present invention can achieve:
[0031] 1) The traditional quality of pronunciation (GOP) algorithm is combined with multiple prosody-related pronunciation features to implement a neural network-based pronunciation evaluation model.
[0032] 2) Considering the globality and locality of prosodic information, CNN networks of different scales are used to mine prosodic-related pronunciation features of different granularities.
[0033] 3) An attention mechanism model is adopted to fuse pronunciation features of different scales and features related to posterior probability to achieve multi-scale fused pronunciation features. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] Figure 1 Schematic diagram of a one-dimensional convolutional neural network;
[0035] Figure 2 Flowchart of the pronunciation assessment system. DETAILED DESCRIPTION
[0036] The present invention is further described below in conjunction with the accompanying drawings.
[0037] In order to solve the technical problems raised in the background technology, the present invention adopts a pronunciation evaluation model optimization method based on multi-scale fusion of neural networks, and mainly designs the following three aspects:
[0038] 1) A speech recognition model based on a neural network is used as the acoustic model, and the output of the neural network is used as the basis for evaluating the correctness of pronunciation.
[0039] 2) Convolutional neural networks of different scales are used to mine rhythm-related features of different scales, taking into account rhythmic information such as emphasis, tone, and stress.
[0040] 3) The attention mechanism is used to learn the weights of features related to different scales and GOPs, and to realize a pronunciation evaluation system that integrates multiple information.
[0041] Step 1: Acoustic model design and selection.
[0042] 1) Acoustic Model Model Design and Selection, The present invention selects a general end-to-end speech recognition model as an acoustic model to calculate the GOP score of the audio to be evaluated. Among them, the end-to-end acoustic model based on a neural network needs to be pre-trained.
[0043] 2) Construct a pronunciation evaluation training data set, and invite three experienced English teachers to score the overall pronunciation quality of these voices from 0 to 5 in terms of pronunciation accuracy, fluency and completeness, with 0 being the lowest and 5 being the highest. Finally, the average of the scores of the three teachers will be used to manually score each voice data.
[0044] Step 2: Calculation of GOP score based on neural network.
[0045] After completing the speech recognition model described in step 1, the speech recognition model used in step 1 can be used to recognize the speech to be evaluated, and the output of the neural network can be used to calculate the GOP score:
[0046]
[0047] Formula (1) uses the average frame-level posterior probability constructed by the neural network output as the GOP score; here P(s t ,O t ) is the output of the last softmax layer of the neural network model, where O refers to the observation sequence of speech, O t is the observation sequence corresponding to the speech frame at time t, t s and t e Respectively represent the start frame and end frame of phoneme P, s t is the state label of frame t after forced alignment;
[0048] Step 3: Construction of multi-scale convolutional neural network.
[0049] 1) Extraction of rhythm-related features
[0050] The three most commonly used acoustic features related to rhythm perception are pitch, intensity and duration, as well as their corresponding statistical and dynamic features. Therefore, the above-mentioned related rhythm acoustic features of each frame are first extracted and used as the input of the convolutional neural network. Assuming that the speech to be evaluated is divided into N frames, and each frame contains the above-mentioned M-dimensional related rhythm-related features, the input is an N*M matrix.
[0051] 2) Multi-scale neural network construction
[0052] Because different acoustic features have different performance characteristics at different time scales, if the acoustic features are analyzed only from one granularity, some local information may be ignored. For example, the intonation or intonation of the entire sentence often requires analysis over a longer time window to discover the corresponding statistical characteristics and rules. However, for a convolutional network with a fixed convolution kernel, features can only be analyzed from a time window of one scale, ignoring the rhythmic information of other scales.
[0053] The study of convolutional neural networks originated from the study of the visual system in biology. In 1962, Hubel and Wiesel discovered a type of cell that was sensitive to local areas of the visual input space when studying the visual cortex of the cat brain, and defined it as a "receptive field". The receptive field covers the entire visual domain in a certain way, which can better obtain the local spatial correlation in the image. Therefore, scholars have expanded this structural feature and applied it to neural networks to extract local features of the input layer. Convolutional neural networks include structures such as input layer (Input Layer), convolutional layer (Convolutional layer), pooling layer (Pooling layer), fully connected layer (Fully-Connected Layer) and output layer (Output Layer). The most core layer structures in CNN are convolutional layers and pooling. The present invention uses one-dimensional convolution as follows Figure 1 As shown below, the original convolution features are analyzed and extracted:
[0054] This patent sets T (1, 2, ..., T) convolutional neural networks of different scales, and the convolution kernel size of each convolutional network is C1*1, C2*1, ..., C T *1, where the number of each convolution kernel is M.
[0055] Step 4: Multi-feature fusion based on attention mechanism
[0056] 1) For the T prosodic features of different scales learned in step 3, assume that these T features are represented by S = [s1, s2, ..., s t ], the final feature representation E calculated by the attention mechanism and the calculation process are shown in the following formula:
[0057] Q=Q'W q ,K=SW k ,V=SW v
[0058]
[0059] Among them, Q' is the vector randomly initialized by the neural network, Wq ,W k , W v is the random initial matrix of the neural network used to perform linear transformation on Q′ and S. After linear transformation, the query vector Q, comparison vector K and content vector V will be obtained. f refers to the reduction of the dimension of the vector, d m Refers to the dimension of the vector, A refers to the scoring function of the attention mechanism. The present invention adopts softmax activation as the scoring function to fix the result within the range of 0-1. As the neural network continues to learn and update, it can eventually achieve the fusion of features of different scales;
[0060] 2) Although the fused feature E can be used to calculate the final evaluation score final However, considering the authority of GOP score, this patent considers the score calculated by fusion feature. e ) and GOP score are further integrated as shown in the following formula:
[0061] score final =α*score e +(1-α)*GOP
[0062] α=sigmod(W α s t+1 +b α ) (3)
[0063] Among them, s t+1 is the output before softmax based on the neural network recognition model used in step 1, W α and b α The random initial weights in the evaluation model are also used to obtain the final score. final It comprehensively considers different rhythmic acoustic characteristics and the final evaluation results of GOP.
[0064] Evaluation steps:
[0065] 1) Receive the audio to be evaluated, and calculate the GOP score and the output s before the softmax of the recognition model through step 1 t+1 ;
[0066] 2) Extract rhythm-related features and extract corresponding deep features through CNNs of different scales;
[0067] 3) Fusion of features of different scales through attention mechanism;
[0068] 4) The fused feature score and the original GOP score are combined to obtain the final score.
Claims
1. A pronunciation evaluation model optimization method based on multi-scale fusion of neural networks, characterized in that: The steps include: Step 1: Design and select the acoustic model: Select the end-to-end speech recognition model as the acoustic model to calculate the GOP score of the audio to be evaluated; In addition, it is necessary to design and construct a pronunciation evaluation training dataset for subsequent model training; Step 2, calculation of GOP score based on neural network: After completing the speech recognition model described in step 1, the speech to be evaluated is recognized by the speech recognition model in step 1, and the GOP score is calculated using the output of the neural network: Formula (1) uses the average frame-level posterior probability constructed by the neural network output as the GOP score; here P(s t ,O t ) is the output of the last softmax layer of the neural network model, where O refers to the observation sequence of speech, O t is the observation sequence corresponding to the speech frame at time t, t s and t e Respectively represent the start frame and end frame of phoneme P, s t is the state label of frame t after forced alignment; Step 3: Construction of multi-scale convolutional neural network 1) Extraction of prosody-related features: Extract the above-mentioned relevant prosody acoustic features of each frame and use them as the input of the convolutional neural network. Assuming that the speech to be evaluated is divided into N frames, and each frame contains M-dimensional relevant prosody-related features, the input is an N*M matrix; 2) Multi-scale neural network construction, using one-dimensional convolution to analyze and extract the original convolution features: Set up T convolutional neural networks of different scales, and the convolution kernel size of each convolutional network is C1*1, C2*1, …,C T *1, where the number of each convolution kernel is M; Step 4: Multi-feature fusion based on attention mechanism: 1) For the T prosodic features of different scales learned in step 3, assume that these T features are represented by S = [s1, s2, ..., s t ], according to the attention mechanism of the following formula (2), the final feature representation E can be calculated: Q=Q′W q ,K=SW k ,V=SW v Among them, Q' is the vector randomly initialized by the neural network, W q ,W k , W v is the random initial matrix of the neural network used to perform linear transformation on Q′ and S. After linear transformation, the query vector Q, the comparison vector K and the content vector V will be obtained. f refers to the reduction of the dimension of the vector, d m Refers to the dimension of the vector, A refers to the scoring function of the attention mechanism, and softmax activation is used as the scoring function to fix the result to the range of 0-1. As the neural network continues to learn and update, it can eventually achieve the fusion of features of different scales; 2) Calculate the score obtained by fusion features e The GOP score is further integrated as shown in the following formula (3): score final =α*score e +(1-α)*GOP α=sigmod(W α s t+1 +b α ) (3) Among them, s t+1 is the output before softmax based on the neural network recognition model used in step 1, W α and b α The random initial matrix is also used for linear transformation in the evaluation model, α is the score e The corresponding weight, (1-α) is the weight corresponding to the GOP score, and the final score is final It comprehensively considers different rhythmic acoustic features and the final evaluation results of GOP. Sigmod is an activation function used to ensure that α is a weight value between 0 and 1.
2. The optimization method according to claim 1, characterized in that: Evaluation steps: 1) Receive the audio to be evaluated, and calculate the GOP score and the output s before the softmax of the recognition model through step 1 t+1 ; 2) Extract rhythm-related features and extract corresponding deep features through CNNs of different scales; 3) Fusion of features of different scales through attention mechanism; 4) The fused feature score and the original GOP score are combined to obtain the final score.