English pronunciation evaluation method, network parameter updating method, device and electronic equipment

Abstract

The embodiments of the present invention provide an English pronunciation evaluation method, a network parameter updating method, a device and an electronic device, and relate to the technical field of speech evaluation. The speech feature vector sequence of the speech to be evaluated is calculated, and the phoneme label sequence of the reference text corresponding to the speech to be evaluated is obtained through a phoneme dictionary, the phoneme label sequence and the speech feature vector sequence are input into a pre-trained phoneme feature alignment model, and the aligned phoneme label sequence is calculated, and the speech feature vector sequence is input into an acoustic model to obtain a first acoustic feature vector sequence, and the speech feature vector sequence is input into a bypass acoustic network to obtain a second acoustic feature vector sequence, and the first acoustic feature vector sequence and the second acoustic feature vector sequence are spliced ​​to obtain an acoustic splicing feature vector sequence, and the aligned phoneme label sequence and the acoustic splicing feature vector sequence are input into a scoring network to obtain a comprehensive score of the speech to be evaluated. Thus, the evaluation of the speech is more accurate.

Description

Technical Field

[0001] The present invention relates to the technical field of speech evaluation, and in particular to an English pronunciation evaluation method, a network parameter updating method, a device and an electronic device. Background Art

[0002] English pronunciation assessment is a language assessment technique that uses a systematic approach to measure and analyze the accuracy, fluency and naturalness of an individual's speech when speaking English. This assessment usually includes examinations of phonemes, pitch, intonation, stress and connected speech, aiming to provide an objective indicator that reflects the English pronunciation level of a learner or user.

[0003] Traditional pronunciation scoring techniques, such as Goodness of Pronunciation (GOP)-based evaluation, use Gaussian Mixture Model (GMM) to align phonemes with audio frames, extract features from the audio with the help of acoustic models, and then score based on these features. However, in this method, the distribution of a single phoneme across multiple frames will distort the accuracy of the score, the feature extraction process cannot fully capture the important acoustic characteristics in the audio frame, and this type of technology often ignores the contextual dependency of pronunciation, resulting in inaccurate scoring. Summary of the invention

[0004] In view of this, an object of the embodiments of the present invention is to provide an English pronunciation evaluation method, a network parameter updating method, an apparatus and an electronic device to at least partially improve the above-mentioned problems.

[0005] In order to achieve the above purpose, the technical solution adopted by the embodiment of the present invention is as follows:

[0006] In a first aspect, an embodiment of the present invention provides an English pronunciation evaluation method, the method comprising:

[0007] Calculate the speech feature vector sequence of the speech to be evaluated, and obtain the phoneme label sequence of the reference text corresponding to the speech to be evaluated through the phoneme dictionary;

[0008] Inputting the phoneme label sequence and the speech feature vector sequence into a pre-trained phoneme feature alignment model to calculate an aligned phoneme label sequence;

[0009] Inputting the speech feature vector sequence into an acoustic model to obtain a first acoustic feature vector sequence; inputting the speech feature vector sequence into a bypass acoustic network to obtain a second acoustic feature vector sequence; wherein the acoustic model is used to extract complex features, and the bypass acoustic network is used to extract basic features;

[0010] Concatenating the first acoustic feature vector sequence and the second acoustic feature vector sequence to obtain an acoustic concatenated feature vector sequence;

[0011] The aligned phoneme label sequence and the acoustic concatenation feature vector sequence are input into a scoring network to obtain a comprehensive score of the speech to be evaluated.

[0012] Optionally, the acoustic model includes a feature transformation layer and an acoustic coding network, and the step of inputting the speech feature vector sequence into the acoustic model to obtain a first acoustic feature vector sequence includes:

[0013] Inputting the speech feature vector sequence into the feature conversion layer, performing feature mapping on the speech feature vector sequence using a conversion matrix, increasing the feature information content of the speech feature vector sequence, and obtaining a speech mapping feature sequence;

[0014] The speech mapping feature sequence is input into the acoustic coding network, and attention processing is performed to obtain a first acoustic feature vector sequence.

[0015] Optionally, the acoustic coding network includes multiple coding layers, and the coding layers include an attention network layer, a feedforward neural network layer, a residual network layer, and an inter-layer normalization layer; wherein the calculation formula of the attention mechanism of the attention network layer is:

[0016]

[0017] in, is the function for calculating attention, is the normalized exponential function, is the query vector, is the key vector, is a value vector, is the dimension of the query vector;

[0018] The attention mechanism includes a window autoregressive attention mechanism and a global autoregressive attention mechanism; wherein, for a non-last speech frame, the window autoregressive attention mechanism is adopted; and for the last speech frame, the global autoregressive attention mechanism is adopted.

[0019] Optionally, the scoring network includes a phoneme-level scoring network, a word-level scoring network, and a sentence-level scoring network, the comprehensive score includes a phoneme-level score, a word-level score, and a sentence-level score, and the step of inputting the aligned phoneme label sequence and the acoustic concatenation feature vector sequence into the scoring network to obtain a comprehensive score of the speech to be evaluated includes:

[0020] According to the aligned phoneme label sequence, the acoustic concatenation feature vector sequence is split at the phoneme level and the word level to obtain a phoneme level acoustic concatenation feature vector sequence and a word level acoustic concatenation feature vector sequence;

[0021] Inputting the phoneme-level acoustic concatenation feature vector sequence into the phoneme-level scoring network to obtain the phoneme-level score;

[0022] Inputting the word-level acoustic concatenation feature vector sequence into the word-level scoring network to obtain the word-level score;

[0023] The acoustic concatenation feature vector sequence is input into the sentence level scoring network to obtain the sentence level score.

[0024] Optionally, the phoneme-level scoring network and the word-level scoring network are both single closed continuous-time neural networks; wherein the projection dimension of the phoneme-level scoring network is 1, and the projection dimension of the word-level scoring network is 3.

[0025] Optionally, the sentence-level scoring network includes a linear layer, a nonlinear layer, a residual layer, and a linear scoring layer, and inputting the acoustic concatenation feature vector sequence into the sentence-level scoring network to obtain the sentence-level score includes:

[0026] Inputting the acoustic concatenation feature vector sequence into the linear layer to obtain a linear feature vector;

[0027] Inputting the linear feature vector into the nonlinear layer to obtain a nonlinear feature vector;

[0028] Inputting the nonlinear feature vector and the tail frame of the acoustic concatenation feature vector sequence into the residual layer to obtain a residual feature vector;

[0029] The residual feature vector is input into the linear scoring layer to obtain the sentence level score.

[0030] Optionally, calculating the speech feature vector sequence of the speech to be evaluated includes:

[0031] Processing the speech to be evaluated by using at least one Mel filter to obtain a Mel frequency spectrum feature vector sequence;

[0032] Perform logarithmic processing on the Mel frequency spectrum feature vector sequence to obtain a logarithmic result;

[0033] The logarithmic result is truncated at a lower bound so that all values ​​in the obtained truncated result are greater than or equal to the maximum value in the logarithmic result minus 8;

[0034] The truncation result is scaled, and all values ​​of the truncation result are scaled to a preset interval to obtain a speech feature vector sequence; the preset interval is:

[0035]

[0036] in, is the maximum value among the logarithmic results.

[0037] In a second aspect, an embodiment of the present invention provides a network parameter updating method, the method comprising:

[0038] Using the acoustic model training data to train the acoustic model to obtain a trained acoustic model, and locking the parameters of the acoustic model;

[0039] Input the scoring training data into the acoustic model, the bypass acoustic network and the scoring network using the method described in any one of the first aspects above for forward propagation;

[0040] Use the MSE loss function to calculate the loss value between the predicted comprehensive score output by the scoring network and the manual score;

[0041] According to the loss value, the parameters of the bypass acoustic network and the scoring network are updated.

[0042] In a third aspect, an embodiment of the present invention provides an English pronunciation evaluation device, the device comprising:

[0043] A speech processing unit, used to calculate a speech feature vector sequence of the speech to be evaluated, and obtain a phoneme label sequence of a reference text corresponding to the speech to be evaluated through a phoneme dictionary;

[0044] A sequence alignment unit, used for inputting the phoneme label sequence and the speech feature vector sequence into a pre-trained phoneme feature alignment model to calculate an aligned phoneme label sequence;

[0045] An acoustic feature vector sequence calculation unit, used for inputting the speech feature vector sequence into an acoustic model to obtain a first acoustic feature vector sequence; inputting the speech feature vector sequence into a bypass acoustic network to obtain a second acoustic feature vector sequence; wherein the acoustic model is used to extract complex features, and the bypass acoustic network is used to extract basic features;

[0046] an acoustic feature vector sequence concatenation unit, configured to concatenate the first acoustic feature vector sequence and the second acoustic feature vector sequence to obtain an acoustic concatenated feature vector sequence;

[0047] The speech scoring unit is used to input the aligned phoneme label sequence and the acoustic concatenation feature vector sequence into a scoring network to obtain a comprehensive score of the speech to be evaluated.

[0048] In a fourth aspect, an embodiment of the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the methods described above when executing the program.

[0049] An English pronunciation evaluation method, a network parameter updating method, a device and an electronic device provided by an embodiment of the present invention use a phoneme feature alignment model to align a speech feature vector sequence of a speech to be evaluated with a phoneme label sequence of a corresponding reference text to obtain an aligned phoneme label sequence, input the speech feature vector sequence into an acoustic model and a bypass acoustic network respectively to obtain a first acoustic feature vector sequence and a second acoustic feature vector sequence respectively, and concatenate the first acoustic feature vector sequence and the second acoustic feature vector sequence to obtain an acoustic concatenated feature vector sequence, input the acoustic concatenated feature vector sequence and the aligned phoneme label sequence into a scoring network to obtain a comprehensive score of the speech to be evaluated, extract the basic features of the speech feature vector sequence and the complex features extracted by the acoustic model through an acoustic bypass network, thereby integrating the characteristics of the basic features and the complex features extracted by the acoustic model, and thus achieving a more accurate evaluation effect.

[0050] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0052] Figure 1 A schematic structural block diagram of an electronic device provided by an embodiment of the present invention;

[0053] Figure 2 A schematic structural block diagram of an English pronunciation evaluation system provided by an embodiment of the present invention;

[0054] Figure 3 A flowchart of an English pronunciation evaluation method provided by an embodiment of the present invention;

[0055] Figure 4 A schematic diagram of calculating an aligned phoneme label sequence provided by an embodiment of the present invention;

[0056] Figure 5A schematic diagram of a concatenated acoustic feature vector sequence provided by an embodiment of the present invention;

[0057] Figure 6 Another schematic diagram of a flow chart of an English pronunciation evaluation method provided by an embodiment of the present invention;

[0058] Figure 7 A schematic diagram of an autoregressive attention mechanism provided by an embodiment of the present invention;

[0059] Figure 8 A schematic diagram of a coding layer of an acoustic model provided by an embodiment of the present invention;

[0060] Fig. 9 Another schematic structural block diagram of an English pronunciation evaluation system provided by an embodiment of the present invention;

[0061] Fig.10 A schematic diagram of a processing flow of a scoring network provided by an embodiment of the present invention;

[0062] Fig.11 A schematic diagram of splitting an acoustic concatenation feature vector sequence provided by an embodiment of the present invention;

[0063] Fig.12 A schematic diagram of a sentence-level scoring process provided by an embodiment of the present invention;

[0064] Fig.13 A schematic diagram of the structure of scoring training data provided by an embodiment of the present invention;

[0065] Fig.14 A schematic structural block diagram of an English pronunciation assessment device provided in an embodiment of the present invention.

[0066] Icons: 100-electronic device; 101-memory; 102-communication interface; 103-processor; 104-bus; 300-English pronunciation evaluation system; 310-phoneme feature alignment model; 320-acoustic model; 330-bypass acoustic network; 340-scoring network; 341-phoneme level scoring network; 342-word level scoring network; 343-sentence level scoring network; 400-English pronunciation evaluation device; 410-speech processing unit; 420-sequence alignment unit; 430-acoustic feature vector sequence calculation unit; 440-acoustic feature vector sequence splicing unit; 450-speech scoring unit. DETAILED DESCRIPTION

[0067] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0068] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0069] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings. At the same time, in the description of the present invention, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0070] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0071] English pronunciation assessment is a language assessment technique that uses a systematic approach to measure and analyze the accuracy, fluency and naturalness of an individual's speech when speaking English. This assessment usually includes examinations of phonemes, pitch, intonation, stress and connected speech, aiming to provide an objective indicator that reflects the English pronunciation level of a learner or user.

[0072] Traditional pronunciation scoring techniques, such as GOP-based evaluation, use GMM to align phonemes with audio frames and extract features from the audio with the help of acoustic models, and then score based on these features. Such methods face multiple problems: the distribution of a single phoneme across multiple frames may distort the accuracy of the score; the feature extraction process may not fully capture the critical acoustic characteristics within the audio frame; and such techniques often ignore the contextual dependence of pronunciation. These limitations together weaken the precision and reliability of the scoring results, resulting in inaccurate scoring methods.

[0073] Based on the above situation, the embodiments of the present invention provide an English pronunciation evaluation method, a network parameter updating method, a device and an electronic device, which perform alignment through a trained phoneme feature alignment model, use an acoustic model to extract complex features, bypass the acoustic network to extract basic features, and splice the complex features with the basic features, thereby integrating the characteristics of the basic features and the complex features extracted by the acoustic model, thereby achieving a more accurate evaluation effect.

[0074] To implement the process steps and functions of each example of the present invention, please refer to Figure 1 , Figure 1 A schematic structural block diagram of an electronic device provided by an embodiment of the present invention. The electronic device 100 includes a memory 101 and a processor 103, and the memory 101 and the processor 103 are electrically connected to each other directly or indirectly to realize data transmission or interaction. For example, these elements can be electrically connected to each other through one or more communication buses 104 or signal lines. The memory 101 can be used to store software programs and modules, and the processor 103 executes the software programs and modules stored in the memory 101 to perform various functional applications and data processing.

[0075] The electronic device 100 may be, but is not limited to, a personal computer (PC), a server, a distributed computer, etc. It is understandable that the electronic device 100 is not limited to a physical server, but may also be a virtual machine on a physical server, a virtual machine built on a cloud platform, etc., which can provide the same functions as the server or virtual machine. The operating system of the electronic device 100 may be, but is not limited to, a Windows system, a Linux system, etc.

[0076] Among them, the memory 101 can be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), etc.

[0077] The communication connection between the electronic device 100 and an external device is achieved through at least one communication interface 102 (which can be wired or wireless).

[0078] The processor 103 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the embodiment of the present invention can be completed by the hardware integrated logic circuit in the processor 103 or the instructions in the form of software. The processor 103 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0079] Understandably, Figure 1 The structure shown is for illustration only. The electronic device 100 may also include Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown. Figure 1 Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0080] The following is an exemplary description of the English pronunciation evaluation method provided by the present invention. Specifically, Figure 2 A schematic structural block diagram of an English pronunciation evaluation system provided by an embodiment of the present invention, Figure 3 A flowchart of an English pronunciation evaluation method provided by an embodiment of the present invention is shown in FIG. Figure 2 , Figure 3The English pronunciation evaluation system 300 includes a phoneme feature alignment model 310, an acoustic model 320, a bypass acoustic network 330, and a scoring network 340. The English pronunciation evaluation system 300 can be deployed in the above Figure 1 The electronic device 100 shown in the figure can be executed by the above-mentioned Figure 1 The electronic device 100 shown in FIG. 1 includes the following steps: Figure 3 The following steps are described:

[0081] S210: Calculate a speech feature vector sequence of the speech to be evaluated, and obtain a phoneme label sequence of a reference text corresponding to the speech to be evaluated through a phoneme dictionary.

[0082] In the pronunciation assessment, the English speech uttered by the evaluator and the reference text corresponding to the speech are collected. In a simple assessment method, for example, an English text "WE CALL IT BEAR" is given, and the evaluator reads aloud according to the text, and the speech uttered by the evaluator is collected to obtain the speech to be evaluated.

[0083] There are many ways to calculate the speech feature vector sequence. Preferably, the Mel Frequency Cepstral Coefficients (MFCCs) can be calculated. Mel Frequency Cepstral Coefficents (MFCCs) are widely used features that simulate the sensitivity of the human auditory system to different frequencies and can well capture the spectral information of the speech signal.

[0084] Among them, the phoneme dictionary is a database that maps words to their phoneme sequences. Phonemes are the smallest units in phonetics and the basic elements that make up language sounds. They can be combined into syllables and words. Therefore, the phoneme label sequence of the reference text corresponding to the speech to be evaluated can be obtained through the phoneme dictionary. For example, converting "WE CALL IT BEAR" into a phoneme label sequence will result in "W IY0 K AO0 L IH0 TB EH0 R".

[0085] Phoneme dictionaries can be handcrafted or learned from large amounts of speech data using automatic learning algorithms.

[0086] S220: Input the phoneme label sequence and the speech feature vector sequence into a pre-trained phoneme feature alignment model to calculate an aligned phoneme label sequence.

[0087] A phoneme label sequence includes multiple phonemes, and a speech includes multiple speech frames. Since pronunciation can be fast or slow, and phonemes can be long or short, a phoneme often corresponds to multiple speech frames, not one-to-one. Therefore, it is necessary to align the phoneme label sequence and the speech feature vector sequence. To put it in a way that is easy to understand, the speech feature vector sequence of a speech frame corresponds to the previous phoneme, and a phoneme corresponds to the speech feature vector sequence of at least one speech frame.

[0088] The phoneme feature alignment model may be implemented by using dynamic time warping (DTW), hidden Markov model (HMM), connectionist temporal classification (CTC), etc. Preferably, this embodiment uses a hidden Markov model - Gaussian mixture model (HMM-GMM).

[0089] For an intuitive understanding of the process of computing aligned phoneme label sequences, see Figure 4 , Figure 4 A schematic diagram of calculating an aligned phoneme label sequence is provided for an embodiment of the present invention. After aligning the phoneme label sequence and the speech feature vector sequence through HMM-GMM, it can be seen that the phoneme "W" corresponds to the speech feature vector sequence of two speech frames, and the phoneme "IY0" corresponds to the speech feature vector sequence of three speech frames.

[0090] S230: Inputting the speech feature vector sequence into the acoustic model to obtain a first acoustic feature vector sequence; inputting the speech feature vector sequence into the bypass acoustic network to obtain a second acoustic feature vector sequence.

[0091] The acoustic model is used to extract complex features, and the bypass acoustic network is used to extract basic features. The acoustic bypass network has the same structure as the acoustic model, but the network depth and intermediate hidden layer dimensions are much smaller than the acoustic model. This is because the role of the acoustic bypass network is to learn basic features, such as the frequency components of speech signals. These basic features and the complex features of the acoustic model will be jointly applied to the input of the scoring network to achieve a more comprehensive and accurate scoring.

[0092] S240: Concatenate the first acoustic feature vector sequence and the second acoustic feature vector sequence to obtain an acoustic concatenated feature vector sequence.

[0093] After obtaining the first acoustic feature vector sequence and the second acoustic feature vector sequence, since two models with the same structure are used to extract features of the same speech, there are the same number of speech frames, and each speech frame corresponds to a feature. The features corresponding to each speech frame can be directly concatenated. For details, see Figure 5 , Figure 5 A schematic diagram of a concatenated acoustic feature vector sequence provided by an embodiment of the present invention, wherein the feature vector corresponding to each speech frame of the concatenated acoustic feature vector sequence is richer and more comprehensive.

[0094] S250: Inputting the aligned phoneme label sequence and the acoustic concatenation feature vector sequence into a scoring network to obtain a comprehensive score of the speech to be evaluated.

[0095] The English pronunciation evaluation method uses a phoneme feature alignment model to align the speech feature vector sequence of the speech to be evaluated and the phoneme label sequence of its corresponding reference text to obtain an aligned phoneme label sequence, inputs the speech feature vector sequence into the acoustic model and the bypass acoustic network respectively, obtains a first acoustic feature vector sequence and a second acoustic feature vector sequence respectively, concatenates the first acoustic feature vector sequence and the second acoustic feature vector sequence to obtain an acoustic concatenated feature vector sequence, inputs the acoustic concatenated feature vector sequence and the aligned phoneme label sequence into a scoring network, obtains a comprehensive score of the speech to be evaluated, extracts the basic features of the speech feature vector sequence and the complex features extracted by the acoustic model through the acoustic bypass network, thereby integrating the characteristics of the basic features and the complex features extracted by the acoustic model, and thus achieving a more accurate evaluation effect.

[0096] In a possible implementation, calculating the speech feature vector sequence of the speech to be evaluated may include the following steps:

[0097] S211: Process the speech to be evaluated using at least one Mel filter to obtain a Mel frequency spectrum feature vector sequence.

[0098] In this embodiment, 40 Mel filters may be used to filter the speech to be evaluated, and the signal may be mapped to a Mel frequency scale to simulate the human ear's perception of different frequencies, thereby obtaining a Mel frequency spectrum feature vector sequence.

[0099] S212: Perform logarithmic processing on the Mel frequency spectrum feature vector sequence to obtain a logarithmic result.

[0100] To perform a logarithmic transformation on the Mel spectrum, you can use log 10 or ln function, which helps compress the dynamic range and enhance the characteristics of the speech signal.

[0101] S213: The logarithmic result is truncated at a lower bound so that all values ​​in the obtained truncated result are greater than or equal to the maximum value in the logarithmic result minus 8.

[0102] Process the logarithm of the result to ensure that the minimum value of all values ​​is not less than the maximum value in the previous step minus 8. For example, if the maximum value is , then all values All become .

[0103] S214: Scaling the truncated result, scaling all values ​​of the truncated result to a preset interval, and obtaining a speech feature vector sequence. The preset interval is:

[0104]

[0105] in, is the maximum value among the logarithmic results.

[0106] Through the above steps S211 to S214, the speech to be evaluated can be converted into a speech feature vector sequence to facilitate subsequent processing.

[0107] In one possible implementation, both the acoustic model and the bypass acoustic network may include a feature transformation layer and an acoustic encoding network, see Figure 6 , Figure 6 Another flow chart of an English pronunciation evaluation method provided by an embodiment of the present invention, wherein inputting a speech feature vector sequence into an acoustic model to obtain a first acoustic feature vector sequence may include the following steps:

[0108] S231: Input the speech feature vector sequence into the feature transformation layer, perform feature mapping on the speech feature vector sequence using the transformation matrix, increase the feature information content of the speech feature vector sequence, and obtain a speech mapping feature sequence.

[0109] The feature transformation layer is a component in the deep learning model that is used to enhance the feature representation capability. In speech processing, the feature transformation layer can map the input speech feature vector through a transformation matrix to extract richer feature information, thereby improving the model's ability to understand speech signals.

[0110] S232: Input the speech mapping feature sequence into the acoustic coding network, perform attention processing, and obtain a first acoustic feature vector sequence.

[0111] The acoustic coding network may include multiple coding layers, each of which includes an attention network layer, a feedforward neural network layer, a residual network layer, and an inter-layer normalization layer. The calculation formula of the attention mechanism of the attention network layer is:

[0112]

[0113] in, is the function for calculating attention, is the normalized exponential function, is the query vector, is the key vector, is a value vector, is the dimension of the query vector.

[0114] The conventional multiplicative combination attention mechanism aims to capture the interaction between features. Different from the conventional calculation formula, this attention mechanism calculation formula introduces the additive combination of Q and K, which can retain the independent features of Q and K. In speech evaluation, the features of individual core speech frames can directly affect the evaluation results, which can expand the representation ability of attention and achieve better attention calculation effects.

[0115] The attention mechanism includes a window autoregressive attention mechanism and a global autoregressive attention mechanism. For a non-last speech frame, the window autoregressive attention mechanism is used; for the last speech frame, the global autoregressive attention mechanism is used.

[0116] See also Figure 7 , Figure 7 A schematic diagram of an autoregressive attention mechanism provided for an embodiment of the present invention, wherein the solid line portion is a schematic diagram of the window autoregression of a single speech frame, and the dotted line portion is a schematic diagram of the global autoregression of the last speech frame.

[0117] Through this autoregressive attention mechanism, the local autoregressive attention mechanism is conducive to the accuracy of evaluation at the phoneme and word level, while the global attention mechanism at the end frame is conducive to the accuracy of evaluation at the sentence level. This attention mechanism is more in line with the characteristics of audio signals, which can reduce the interference of subsequent features on the evaluation of previous phonemes and words, and improve accuracy.

[0118] Further, see Figure 8 , Figure 8 A schematic diagram of the encoding layer of an acoustic model provided for an embodiment of the present invention, wherein the feedforward neural network layer, the residual network layer and the inter-layer normalization layer are referred to as the linear part, wherein the function of the feedforward neural network layer is nonlinear transformation and feature reorganization, and the residual network is used to alleviate the gradient disappearance / explosion problem.

[0119] Similarly, the bypass acoustic network has the same structure as the acoustic model, and the second acoustic feature vector sequence can be obtained in the same way as above, which will not be repeated here.

[0120] Although the bypass acoustic network has the same structure as the acoustic model, the depth and intermediate hidden layer dimensions of the network are much smaller than those of the acoustic model. This is because the role of the network is to learn basic features, such as the frequency components of the speech signal. These basic features and the complex features of the acoustic model will be applied together to the input of the scoring network to achieve a more comprehensive and accurate score.

[0121] Specifically, in this embodiment, the specific parameters of the acoustic model are as follows:

[0122] Feature transformation layer: The transformation matrix is ​​40x1280.

[0123] Acoustic coding network: It consists of 24 coding layers with a hidden state dimension of 1280. The feedforward neural network layer consists of a 1280x4800 layer, a relu activation function, and a 4800x1280 linear layer.

[0124] The specific parameters of the bypass acoustic network are as follows:

[0125] Feature transformation layer: The transformation matrix is ​​40x128.

[0126] Acoustic coding network: It consists of 8 coding layers with a hidden state dimension of 128. The feedforward neural network layer consists of a 128x256 layer, a relu activation function, and a 256x128 linear layer.

[0127] Furthermore, in order to obtain more dimensional scores, the speech to be evaluated can be scored through multiple networks. For details, see Fig. 9 , Fig.10 , Fig. 9 Another schematic structural block diagram of an English pronunciation evaluation system provided by an embodiment of the present invention is shown in FIG. Fig.10 A processing flow diagram of a scoring network provided in an embodiment of the present invention is shown, wherein the scoring network includes a phoneme-level scoring network 341, a word-level scoring network 342, and a sentence-level scoring network 343, and the comprehensive scoring includes a phoneme-level scoring, a word-level scoring, and a sentence-level scoring. The above step S250 includes the following steps: Fig.10 The following steps are shown:

[0128] S251: According to the aligned phoneme label sequence, the acoustic concatenation feature vector sequence is split into a phoneme layer and a word layer to obtain a phoneme layer acoustic concatenation feature vector sequence and a word layer acoustic concatenation feature vector sequence.

[0129] The acoustic concatenation feature vector sequence includes all phonemes, which needs to be split into the acoustic concatenation feature vector sequences at the phoneme level and the word level, see Fig.11 , Fig.11A schematic diagram of a split acoustic concatenation feature vector sequence provided for an embodiment of the present invention. For example, the feature vector corresponding to a single phoneme "W" is put into a phoneme-level acoustic concatenation feature vector sequence. For the word "WE", the feature vectors of all phonemes corresponding to the word "WE" need to be put into a word-level acoustic concatenation feature vector sequence. The word "WE" includes the phonemes "W" and "IY0", that is, the feature vectors corresponding to "W" and "IY0" are combined into a concatenation feature vector sequence of "WE". All words are processed in this way and all are put into a word-level acoustic concatenation feature vector sequence.

[0130] S252: Input the phoneme-level acoustic concatenation feature vector sequence into a phoneme-level scoring network to obtain a phoneme-level score.

[0131] S253: Input the word-level acoustic concatenation feature vector sequence into the word-level scoring network to obtain the word-level score.

[0132] Among them, both the phoneme-level scoring network and the word-level scoring network are single closed-form continuous-time neural networks (CfC). CfC is a new type of neural network architecture designed to solve the problem of modeling continuous-time dynamic systems, especially in the field of neuroscience. This network architecture uses a mathematical closed-form approximation to replace the traditional numerical integration method to describe the dynamic interactions between neurons. Compared with traditional recurrent neural network (RNN) solutions, CfC shows more obvious advantages in efficiency, scalability, stability and interpretability in processing time series data.

[0133] Using CfC as a scoring tool at the phoneme and word level, it can more accurately capture the temporal characteristics of speech signals and grasp the speech features of key frames.

[0134] The projection dimension of the phoneme-level scoring network is 1, which is used to predict the score of phoneme pronunciation. The projection dimension of the word-level scoring network is 3, which is used to predict the accuracy, stress and total score of word pronunciation.

[0135] S254: Input the acoustic concatenation feature vector sequence into the sentence-level scoring network to obtain a sentence-level score.

[0136] In a possible implementation, the sentence-level scoring network may include a linear layer, a nonlinear layer, a residual layer, and a linear scoring layer, and step S254 may include the following steps:

[0137] S2541: Input the acoustic concatenation feature vector sequence into the linear layer to obtain a linear feature vector.

[0138] S2542: Input the linear feature vector into the nonlinear layer to obtain a nonlinear feature vector.

[0139] S2543: Input the nonlinear feature vector and the tail frame of the acoustic concatenation feature vector sequence into the residual layer to obtain a residual feature vector.

[0140] S2544: Input the residual feature vector into the linear scoring layer to obtain a sentence-level score.

[0141] See also Fig.12 , Fig.12 A flow chart of sentence-level scoring provided in an embodiment of the present invention is provided, in which a sequence of acoustic concatenated feature vectors is input into a linear layer and a nonlinear layer to obtain a nonlinear feature vector, and the nonlinear feature vector and the tail frame of the acoustic concatenated feature vector sequence are input into a residual layer and a linear scoring layer to obtain a sentence-level score.

[0142] Through this sentence-level scoring network, the last frame is used as the input of the residual layer, providing contextual information at the end of the sequence, which helps the model better understand the completeness of the sequence and makes the sentence-level scoring more accurate.

[0143] For the phoneme feature alignment model 310, the acoustic model 320, the bypass acoustic network 330 and the scoring network 340 in the English pronunciation evaluation system 300, the phoneme feature alignment model may be HMM-GMM, and the acoustic model 320, the bypass acoustic network 330 and the scoring network 340 need to be trained in advance. Specifically, the present invention further provides a network parameter updating method, which includes:

[0144] S510: Train the acoustic model using the acoustic model training data to obtain a trained acoustic model, and lock the parameters of the acoustic model.

[0145] For acoustic model training, the acoustic model training data includes: audio data and corresponding text transcription. The acoustic model training data is further preprocessed, including removing silent segments, noise reduction, and standardizing audio formats and sampling rates.

[0146] The acoustic model is initialized randomly and trained using the batch gradient descent method. This includes:

[0147] Extract a batch of data from the acoustic model training data for model training in this iteration. This data includes input features and corresponding labels. Input this batch of data into the acoustic model and calculate the output of the acoustic model through forward propagation. Use the Connectionist Temporal Classification (CTC) loss function to calculate the loss value between the model output and the true label. Based on the calculated loss value, use the back propagation algorithm to calculate the gradient of the model parameters. Update the model parameters based on the calculated gradient to minimize the loss function. Repeat the above process until the preset conditions are met.

[0148] The acoustic model training data in this part is the collected English speech data and its corresponding standard text. In the training data processing, these standard texts will be converted into phoneme label sequences according to the phoneme vocabulary as the real label data. The speech data of this example is 2000 hours and is trained through the pytorch and deepspeed distributed training framework.

[0149] The training parameter details include:

[0150] Optimizer: AdamW optimizer, specific parameters include learning rate: 0.0001, weight decay: 0.01, learning rate adjustment: warm up the learning rate in the first 500 steps, from 10 -7 Warm up to 0.0001 and then decay linearly to 0.00001 with a period of 5000 steps.

[0151] Batch size: 256.

[0152] Number of training steps: 6000. That is, the training is completed after 6000 steps; and the final network weight is selected as the used network weight.

[0153] The parameters of the trained acoustic model are locked and will not be updated in subsequent training.

[0154] S520: Using the above-mentioned English pronunciation evaluation method, the scoring training data is input into the acoustic model, the bypass acoustic network and the scoring network for forward propagation.

[0155] The scoring training data is the evaluation speech, the standard text corresponding to the evaluation speech, and the manual scoring corresponding to the evaluation speech. The manual scoring can include three levels, namely the phoneme level, word level and sentence level, see Fig.13 , Fig.13A schematic diagram of the structure of scoring training data provided by an embodiment of the present invention, wherein text is a standard text, sentence is a sentence-level score, including four dimensional scores, namely accuracy, completeness, fluency and total; word is a word-level score, including three dimensional scores, namely accuracy, stress and total word score; phonemes is a phoneme-level score, including one dimensional score.

[0156] The scoring training data will be processed as follows:

[0157] The evaluation speech in the scoring training data will be pre-calculated to obtain the corresponding speech feature vector sequence.

[0158] According to the phoneme dictionary, the standard text is converted into a phoneme label sequence.

[0159] The speech feature vector sequence and the phoneme label sequence are input into the pre-trained HMM-GMM model to obtain the aligned phoneme label sequence corresponding to the speech feature vector sequence.

[0160] After the data processing is completed, the acoustic bypass network and the scoring network are randomly initialized, and the model is trained using the batch gradient descent method, including:

[0161] A batch of data is extracted from the scoring training data for model training in this iteration. These data include speech feature vector sequences, corresponding aligned phoneme label sequences, and manual scores.

[0162] The prediction scores at the phoneme level, the word level, and the sentence level are obtained by forward propagation calculation of the above-mentioned English pronunciation evaluation method.

[0163] S530: Use the MSE loss function to calculate the loss value between the predicted comprehensive score output by the scoring network and the manual score.

[0164] S540: Update the parameters of the bypass acoustic network and the scoring network according to the loss value.

[0165] In this embodiment, specific parameters of the bypass acoustic network are as follows:

[0166] Feature transformation layer: The transformation matrix is ​​40x128.

[0167] Acoustic coding network: It consists of 8 coding layers with a hidden state dimension of 128. The feedforward neural network layer consists of a 128x256 layer, a relu activation function, and a 256x128 linear layer.

[0168] The phoneme-level scoring network is specifically a single closed-loop continuous-time neural network with the following parameters: input feature size 1408, 128 hidden units, and a projection dimension of 1.

[0169] The word-level scoring network is specifically a single closed-form continuous-time neural network with the following parameters: input feature size 1408, 128 hidden units, and a projection dimension of 3.

[0170] The sentence-level scoring network is specifically:

[0171] Linear layer: is a 1408x32 linear layer.

[0172] Non-linear layer: 32x128 non-linear layer with relu activation.

[0173] Linear Scoring: 128x4 linear scoring layer with softmax.

[0174] In this embodiment, the scoring training data is further divided into training data and verification data, and the best network weight is selected through the verification data, and the network weight with the smallest MSE loss value on the verification data is used as the final network weight used.

[0175] The training parameter details include:

[0176] Optimizer: AdamW optimizer, specific parameters include learning rate: 0.001, weight decay: 0.001, learning rate adjustment: warm up the learning rate in the first 200 steps, from 10 -6 Warm up to 0.001 and then decay linearly to 0.00001 with a period of 2000 steps.

[0177] Batch size: 64.

[0178] Number of training steps: 3000. That is, training is completed after 3000 steps.

[0179] Verification steps: Verification is performed on the validation dataset every 300 steps.

[0180] Furthermore, the present invention also provides an English pronunciation evaluation device 400, see Fig.14 , Fig.14 A schematic structural block diagram of an English pronunciation evaluation device provided by an embodiment of the present invention, the English pronunciation evaluation device 400 comprises:

[0181] The speech processing unit 410 is used to calculate the speech feature vector sequence of the speech to be evaluated, and obtain the phoneme label sequence of the reference text corresponding to the speech to be evaluated through the phoneme dictionary;

[0182] A sequence alignment unit 420 is used to input the phoneme label sequence and the speech feature vector sequence into a pre-trained phoneme feature alignment model to calculate an aligned phoneme label sequence;

[0183] The acoustic feature vector sequence calculation unit 430 is used to input the speech feature vector sequence into the acoustic model to obtain a first acoustic feature vector sequence; input the speech feature vector sequence into the bypass acoustic network to obtain a second acoustic feature vector sequence; wherein the acoustic model is used to extract complex features, and the bypass acoustic network is used to extract basic features;

[0184] An acoustic feature vector sequence concatenation unit 440, configured to concatenate the first acoustic feature vector sequence and the second acoustic feature vector sequence to obtain an acoustic concatenated feature vector sequence;

[0185] The speech scoring unit 450 is used to input the aligned phoneme label sequence and the acoustic concatenation feature vector sequence into the scoring network to obtain a comprehensive score of the speech to be evaluated.

[0186] In summary, an English pronunciation evaluation method, a network parameter updating method, a device and an electronic device provided by an embodiment of the present invention use a phoneme feature alignment model to align the speech feature vector sequence of the speech to be evaluated with the phoneme label sequence of its corresponding reference text to obtain an aligned phoneme label sequence, input the speech feature vector sequence into the acoustic model and the bypass acoustic network respectively, obtain the first acoustic feature vector sequence and the second acoustic feature vector sequence respectively, and splice the first acoustic feature vector sequence and the second acoustic feature vector sequence to obtain an acoustic splicing feature vector sequence, input the acoustic splicing feature vector sequence and the aligned phoneme label sequence into the scoring network to obtain a comprehensive score of the speech to be evaluated, extract the basic features of the speech feature vector sequence and the complex features extracted by the acoustic model through the acoustic bypass network, thereby integrating the characteristics of the basic features and the complex features extracted by the acoustic model, and thus achieving a more accurate evaluation effect. Specifically, the following beneficial effects are included:

[0187] 1. The local autoregressive attention mechanism is beneficial to the accuracy of evaluation at the phoneme and word level, while the global attention mechanism at the tail frame is beneficial to the accuracy of evaluation at the sentence level.

[0188] 2. This attention mechanism is more consistent with the characteristics of audio signals, which can reduce the interference of subsequent features on the evaluation of previous phonemes and words, and improve accuracy.

[0189] 3. Closed continuous-time neural networks have obvious advantages in terms of efficiency, scalability, stability, and interpretability in processing audio features. As a scoring tool at the phoneme and word level, it can more accurately capture the temporal characteristics of speech signals and grasp the speech features of key frames.

[0190] 4. The design of the acoustic bypass network is conducive to learning basic features. These basic features and the complex features of the acoustic model will be jointly applied to the input of the scoring network to achieve more comprehensive and accurate scoring.

[0191] 5. The design of the sentence-level network fully integrates the characteristics of basic features and complex features extracted by the acoustic model, thereby achieving more accurate evaluation results.

[0192] In the embodiments provided by the present invention, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and a module, a program segment or a part of the code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0193] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0194] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a computer-readable storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0195] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

[0196] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A method for evaluating English pronunciation, characterized in that: The method comprises: Calculate the speech feature vector sequence of the speech to be evaluated, and obtain the phoneme label sequence of the reference text corresponding to the speech to be evaluated through the phoneme dictionary; Inputting the phoneme label sequence and the speech feature vector sequence into a pre-trained phoneme feature alignment model to calculate an aligned phoneme label sequence; Inputting the speech feature vector sequence into an acoustic model to obtain a first acoustic feature vector sequence; inputting the speech feature vector sequence into a bypass acoustic network to obtain a second acoustic feature vector sequence; wherein the acoustic model is used to extract complex features, and the bypass acoustic network is used to extract basic features; Concatenating the first acoustic feature vector sequence and the second acoustic feature vector sequence to obtain an acoustic concatenated feature vector sequence; The aligned phoneme label sequence and the acoustic concatenation feature vector sequence are input into a scoring network to obtain a comprehensive score of the speech to be evaluated.

2. The method according to claim 1, characterized in that The acoustic model includes a feature transformation layer and an acoustic coding network, and the speech feature vector sequence is input into the acoustic model to obtain a first acoustic feature vector sequence, including: Inputting the speech feature vector sequence into the feature conversion layer, performing feature mapping on the speech feature vector sequence using a conversion matrix, increasing the feature information content of the speech feature vector sequence, and obtaining a speech mapping feature sequence; The speech mapping feature sequence is input into the acoustic coding network, and attention processing is performed to obtain a first acoustic feature vector sequence.

3. The method according to claim 2, characterized in that The acoustic coding network includes multiple coding layers, and the coding layers include an attention network layer, a feedforward neural network layer, a residual network layer, and an inter-layer normalization layer; wherein the calculation formula of the attention mechanism of the attention network layer is: in, is the function for calculating attention, is the normalized exponential function, is the query vector, is the key vector, is a value vector, is the dimension of the query vector; The attention mechanism includes a window autoregressive attention mechanism and a global autoregressive attention mechanism; wherein, for a non-last speech frame, the window autoregressive attention mechanism is adopted; and for the last speech frame, the global autoregressive attention mechanism is adopted.

4. The method according to claim 1, characterized in that: The scoring network includes a phoneme-level scoring network, a word-level scoring network, and a sentence-level scoring network, the comprehensive score includes a phoneme-level score, a word-level score, and a sentence-level score, and the aligned phoneme label sequence and the acoustic concatenation feature vector sequence are input into the scoring network to obtain a comprehensive score of the speech to be evaluated, including: According to the aligned phoneme label sequence, the acoustic concatenation feature vector sequence is split at the phoneme level and the word level to obtain a phoneme level acoustic concatenation feature vector sequence and a word level acoustic concatenation feature vector sequence; Inputting the phoneme-level acoustic concatenation feature vector sequence into the phoneme-level scoring network to obtain the phoneme-level score; Inputting the word-level acoustic concatenation feature vector sequence into the word-level scoring network to obtain the word-level score; The acoustic concatenation feature vector sequence is input into the sentence level scoring network to obtain the sentence level score.

5. The method according to claim 4, characterized in that The phoneme-level scoring network and the word-level scoring network are both single closed continuous-time neural networks; wherein the projection dimension of the phoneme-level scoring network is 1, and the projection dimension of the word-level scoring network is 3.

6. The method according to claim 4, characterized in that The sentence-level scoring network includes a linear layer, a nonlinear layer, a residual layer, and a linear scoring layer. The acoustic concatenation feature vector sequence is input into the sentence-level scoring network to obtain the sentence-level score, including: Inputting the acoustic concatenation feature vector sequence into the linear layer to obtain a linear feature vector; Inputting the linear feature vector into the nonlinear layer to obtain a nonlinear feature vector; Inputting the nonlinear feature vector and the tail frame of the acoustic concatenation feature vector sequence into the residual layer to obtain a residual feature vector; The residual feature vector is input into the linear scoring layer to obtain the sentence level score.

7. The method according to claim 1, characterized in that The step of calculating a speech feature vector sequence of the speech to be evaluated includes: Processing the speech to be evaluated by using at least one Mel filter to obtain a Mel frequency spectrum feature vector sequence; Perform logarithmic processing on the Mel frequency spectrum feature vector sequence to obtain a logarithmic result; The logarithmic result is truncated at a lower bound so that all values ​​in the obtained truncated result are greater than or equal to the maximum value in the logarithmic result minus 8; The truncation result is scaled, and all values ​​of the truncation result are scaled to a preset interval to obtain a speech feature vector sequence; the preset interval is: in, is the maximum value among the logarithmic results.

8. A network parameter updating method, characterized in that: The method comprises: Using the acoustic model training data to train the acoustic model to obtain a trained acoustic model, and locking the parameters of the acoustic model; Inputting the scoring training data into the acoustic model, the bypass acoustic network and the scoring network using the method described in any one of claims 1 to 7 for forward propagation; Use the MSE loss function to calculate the loss value between the predicted comprehensive score output by the scoring network and the manual score; According to the loss value, the parameters of the bypass acoustic network and the scoring network are updated.

9. An English pronunciation assessment device, characterized in that: The device comprises: A speech processing unit, used to calculate a speech feature vector sequence of the speech to be evaluated, and obtain a phoneme label sequence of a reference text corresponding to the speech to be evaluated through a phoneme dictionary; A sequence alignment unit, used for inputting the phoneme label sequence and the speech feature vector sequence into a pre-trained phoneme feature alignment model to calculate an aligned phoneme label sequence; An acoustic feature vector sequence calculation unit, used for inputting the speech feature vector sequence into an acoustic model to obtain a first acoustic feature vector sequence; inputting the speech feature vector sequence into a bypass acoustic network to obtain a second acoustic feature vector sequence; wherein the acoustic model is used to extract complex features, and the bypass acoustic network is used to extract basic features; an acoustic feature vector sequence concatenation unit, configured to concatenate the first acoustic feature vector sequence and the second acoustic feature vector sequence to obtain an acoustic concatenated feature vector sequence; The speech scoring unit is used to input the aligned phoneme label sequence and the acoustic concatenation feature vector sequence into a scoring network to obtain a comprehensive score of the speech to be evaluated.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

Images