Pronunciation scoring method, device, electronic device and storage medium

Abstract

The present disclosure provides a pronunciation scoring method, device, electronic device and storage medium, which relate to the field of speech technology. The above method includes: obtaining a speech feature vector sequence corresponding to the speech to be scored; converting the text of the speech to be scored into a phoneme sequence according to the pronunciation; time-aligning the speech feature vector sequence and the phoneme sequence to obtain an aligned phoneme sequence, determining the pronunciation feature vector of each phoneme in the aligned phoneme sequence, and generating a pronunciation feature vector sequence, wherein the pronunciation feature vector includes a pitch feature, a speech speed feature and a stress feature; scoring the pronunciation feature vector sequence through a scoring model to obtain a pronunciation score. The present disclosure improves the accuracy of scoring by time-aligning the speech feature vector and the phoneme in view of the dynamic change characteristics of phonemes in time.

Description

Technical Field

[0001] The present disclosure relates to the technical field of speech processing, and in particular to a pronunciation scoring method and device. Background Art

[0002] Speech pronunciation scoring is a technical means to quantitatively evaluate an individual's level of pronunciation clarity, natural intonation, and language fluency. This technology is widely used in many fields such as language learning, speech recognition, and human-computer interaction. Speech pronunciation scoring can not only help learners understand the advantages and disadvantages of their own pronunciation in a timely manner and provide direction for subsequent learning, but also has important significance for improving the accuracy of speech recognition systems and enhancing human-computer interaction experience.

[0003] In the related art, although speech pronunciation scoring can be realized, the accuracy of current pronunciation scoring is low. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a pronunciation scoring method and device, which can solve the existing problems in a targeted manner.

[0005] Based on the above purpose, in a first aspect, the present disclosure proposes a pronunciation scoring method, comprising: obtaining a speech feature vector sequence corresponding to a speech to be scored; converting the text of the speech to be scored into a phoneme sequence according to the pronunciation; time-aligning the speech feature vector sequence and the phoneme sequence to obtain an aligned phoneme sequence, determining a pronunciation feature vector for each phoneme in the aligned phoneme sequence, and generating a pronunciation feature vector sequence, wherein the pronunciation feature vector includes a pitch feature, a speaking speed feature, and a stress feature; and scoring the pronunciation feature vector sequence through a scoring model to obtain a pronunciation score.

[0006] In a second aspect, a pronunciation scoring device is also provided, comprising: an acquisition unit, configured to acquire a speech feature vector sequence corresponding to a speech to be scored; a conversion unit, configured to convert the text of the speech to be scored into a phoneme sequence according to the pronunciation; an alignment unit, configured to time-align the speech feature vector sequence and the phoneme sequence to obtain an aligned phoneme sequence, determine a pronunciation feature vector for each phoneme in the aligned phoneme sequence, and generate a pronunciation feature vector sequence, wherein the pronunciation feature vector includes a pitch feature, a speech speed feature, and a stress feature; and a scoring unit, configured to score the pronunciation feature vector sequence through a scoring model to obtain a pronunciation score.

[0007] According to a third aspect, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect.

[0008] According to a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, and the computer program is executed by a processor to implement any method described in the first aspect.

[0009] In a fifth aspect, a computer program product is also provided, including a computer program, wherein the computer program is executed by a processor to implement any method described in the first aspect.

[0010] In general, the present disclosure has at least the following beneficial effects: in view of the dynamic change characteristics of phonemes in time, the accuracy of scoring is improved by aligning speech feature vectors and phonemes in time. In addition, the interpretability and accuracy of scoring are improved through the scoring model. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments disclosed in accordance with the present disclosure and should not be regarded as limiting the scope of the present disclosure.

[0012] Figure 1 A flowchart of a pronunciation scoring method according to an embodiment of the present disclosure is shown;

[0013] Figure 2 The structure of the scoring model according to the embodiment of the present disclosure is shown;

[0014] Figure 3 a shows the phoneme probability distribution feature sequence corresponding to each time frame, b shows the alignment path corresponding to the time frame, and c shows the waveform information of the aligned phoneme sequence;

[0015] Figure 4a A schematic diagram of a pronunciation scoring device according to an embodiment of the present disclosure is shown;

[0016] Figure 4b Another schematic diagram of a pronunciation scoring device according to an embodiment of the present disclosure is shown;

[0017] Figure 5 A schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure is shown;

[0018] Figure 6 A schematic diagram of a storage medium provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0019] The present disclosure is further described in detail below in conjunction with the accompanying drawings and embodiments. It is understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It is also necessary to explain that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0020] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0021] Figure 1 The pronunciation scoring method of the present disclosure is shown. In an embodiment of the present disclosure, the method includes:

[0022] Step S101, obtaining a speech feature vector sequence corresponding to the speech to be rated.

[0023] Step S102: converting the text of the speech to be rated into a phoneme sequence according to the pronunciation.

[0024] Step S103, time-aligning the speech feature vector sequence and the phoneme sequence to obtain an aligned phoneme sequence, determining the pronunciation feature vector of each phoneme in the aligned phoneme sequence, and generating a pronunciation feature vector sequence, wherein the pronunciation feature vector includes a pitch feature, a speech rate feature, and a stress feature.

[0025] Step S104, scoring the pronunciation feature vector sequence through a scoring model to obtain a pronunciation score.

[0026] In this embodiment, various methods may be used to determine the speech feature vector sequence corresponding to the speech to be scored. For example, the speech to be scored is input into a feature vector generation model to obtain the speech feature vector sequence.

[0027] The text of the speech to be rated can be a text obtained by semantic conversion of the speech to be rated, or a standard text corresponding to the text. The speech to be rated and the standard text corresponding to the speech to be rated can be first obtained, and the standard text can be converted into a phoneme sequence according to the pronunciation.

[0028] The pronunciation here can be represented as a pronunciation dictionary or a preset pronunciation model. The pronunciation dictionary is a mapping table in which each Chinese character or word corresponds to a subphoneme sequence. Phoneme is the basic unit of speech and represents the smallest pronunciation unit.

[0029] The scoring model can be a formula or a pre-trained deep neural network.

[0030] Optionally, if a standard text is used, the following word segmentation of the standard text may be performed: first, the standard text is word segmented to separate it into individual Chinese characters or words. For example, for Chinese text, a word segmentation tool may be used to decompose a sentence into individual Chinese characters or words.

[0031] After that, search for the phoneme sequence: for each Chinese character or word after word segmentation, search the pronunciation dictionary to find its corresponding subphoneme sequence. And connect the subphoneme sequence found with blank phonemes as intervals to form a phoneme sequence. Encode the phonemes: map each phoneme to a unique integer to get the phoneme sequence.

[0032] In view of the dynamic change characteristics of phonemes in time, this embodiment improves the accuracy of scoring by aligning speech feature vectors and phonemes in time. In addition, this embodiment improves the interpretability and accuracy of scoring through a scoring model.

[0033] In some optional implementations of any embodiment of the present disclosure, the time alignment of the speech feature vector sequence and the phoneme sequence includes: using an acoustic model to process the speech feature vector sequence and the phoneme sequence to obtain a predicted phoneme probability distribution feature sequence corresponding to each time frame in the speech to be scored; constructing a probability matrix composed of the alignment probabilities between each time frame and the phoneme sequence; starting from the end of the probability matrix, tracing back the maximum alignment probability corresponding to each time frame in the probability matrix to obtain an alignment path with the maximum alignment probability; and calculating the aligned phoneme sequence on the time frame according to the alignment path.

[0034] In these optional implementations, a dynamic programming algorithm can be used to construct a probability matrix for aligning a time frame with a phoneme sequence. Then, starting from the end, a greedy algorithm is used to backtrack from the alignment probability matrix to obtain an alignment path with the highest probability. Based on the path, the aligned phoneme sequence on the time frame is calculated.

[0035] In some optional implementations of any embodiment of the present disclosure, the pitch feature includes at least one of a phoneme posterior probability feature and a phoneme posterior ratio feature; the phoneme posterior probability feature is the logarithmic value of the posterior probability of the occurrence of a phoneme when a phoneme probability distribution feature sequence and its speech frame range are set, the phoneme posterior probability feature is used to indicate the probability intensity of the occurrence of a phoneme within a target time period, and the phoneme probability distribution feature sequence corresponds to each speech frame of the speech to be scored.

[0036] The phoneme posterior ratio feature is to set the phoneme probability distribution feature sequence and its speech frame range In the case of , the logarithmic value of the posterior given probability ratio of the true phoneme to the predicted phoneme, where the true phoneme is the phoneme in the aligned phoneme sequence, and the predicted phoneme is the phoneme corresponding to the phoneme probability distribution feature sequence.

[0037] The specific definition of the phoneme posterior probability feature is the logarithm of the posterior probability of the phoneme appearing given the predicted phoneme probability distribution feature sequence and its speech frame range. This feature reflects the probability intensity of the phoneme occurring in a specific time period. The calculation formula is as follows:

[0038]

[0039] In the above formula represents the i-th phoneme, Represents the posterior probability feature of the i-th phoneme. Indicates the start of alignment and consecutive start times, represents the corresponding end time. p represents the predicted probability of the acoustic model. Represents the audio signal at time t.

[0040] The specific definition of the phoneme posterior ratio feature is given by the predicted phoneme probability distribution feature sequence and its speech frame range. Relative to phonemes The logarithmic value of the posterior probability ratio. This feature is used to distinguish the deviation between the predicted phoneme and the true phoneme. The specific calculation formula is:

[0041]

[0042] In the above formula Represents the posterior ratio feature of the i-th phoneme.

[0043] The above-mentioned phoneme posterior ratio feature and phoneme posterior ratio feature are finally combined in the following form to form the intonation feature:

[0044] .

[0045] In some optional implementations of any embodiment of the present disclosure, the speech rate feature includes pronunciation duration, and the pronunciation duration is a ternary feature vector, and the ternary feature vector includes the phoneme frame size, the mean and standard deviation of the phoneme frame size of the phoneme in a preset data set.

[0046] In some optional implementations of any embodiment of the present disclosure, the stress feature includes a relative energy feature, a relative duration feature and a stress identification feature; the relative energy feature is the ratio of the energy of each phoneme in a word to the average energy of all phonemes in the entire word in an aligned phoneme sequence; the relative duration feature is the ratio of the number of phoneme frames of each phoneme in a word to the number of phoneme frames of the entire word in an aligned phoneme sequence; the stress identification feature is used to indicate whether a phoneme needs to be stressed.

[0047] The speech rate feature is formed by analyzing the pronunciation duration (expressed in frames) of each phoneme in the tested speech and combining these measured values ​​with the mean and standard deviation in the reference data set to form a ternary feature vector. The specific expression is:

[0048]

[0049] Here Description The phoneme frame size of the phone, and Description The mean and standard deviation of the phoneme frame sizes in the reference dataset.

[0050] Here, the reference dataset refers to the audio data and its corresponding text that are additionally collected and expected to score well in the evaluation scenario;

[0051] and The specific calculation method includes:

[0052] Calculate the speech feature vector sequence MFCC feature sequence of the audio data in the reference data set according to the above method;

[0053] Calculate the time-aligned aligned phoneme sequence of the reference data set according to the aforementioned method;

[0054] The mean and standard deviation of the speech frame size of each phoneme are calculated, and we get and ;

[0055] Rereading features: Contains three features, namely:

[0056] Relative energy feature: Based on the phoneme sequence, calculate the ratio of the energy of each phoneme in the word to the average energy of all phonemes in the whole word. The formula is as follows:

[0057]

[0058] in:

[0059] in represents the relative energy characteristics of phoneme i, Indicates the start of alignment and consecutive start times, Indicates the corresponding end time. represents the energy sound energy on j phoneme frame;

[0060] Relative duration feature: Based on the phoneme sequence, calculate the ratio of the phoneme frame of each phoneme in the word to the phoneme frame of the entire word. The formula is as follows:

[0061]

[0062] in:

[0063] in Indicates the relative duration characteristics of phoneme i, Indicates the start of alignment and consecutive start times, Indicates the corresponding end time.

[0064] Stress identification feature: This feature indicates whether the phoneme needs to be stressed, represented by two values, 0 for no stress and 1 for stress;

[0065] In summary, the reread feature can be further expressed as: .

[0066] In some optional implementations of any embodiment of the present disclosure, the scoring model includes a feature network, a coding network and a scoring network; the scoring of the pronunciation feature vector sequence by the scoring model includes: using the feature network to process the pronunciation feature vector sequence to obtain a cross feature vector sequence; using the coding network to process the cross feature vector sequence to obtain a target coding feature vector; using the scoring network to process the target coding feature vector to obtain a scoring value of the speech to be scored in each speech attribute.

[0067] Optionally, the feature network is composed of multiple layers of nonlinear layers; the use of the feature network to process the pronunciation feature vector sequence to obtain a cross feature vector sequence includes: generating a new pronunciation feature vector corresponding to a new feature cross pattern for the pronunciation feature vector sequence through the multiple layers of nonlinear layers to obtain a new pronunciation feature vector sequence.

[0068] Optionally, the encoding network includes multiple transformer layers, each transformer layer has a single-head self-attention mechanism module integrated therein; the step of using the encoding network to process the cross-feature vector sequence to obtain a target encoding feature vector includes: using the encoding network to process the cross-feature vector sequence to increase the richness of the cross-feature vector sequence to obtain a head encoding feature vector, and the head encoding feature vector is the target encoding feature vector.

[0069] The scoring network can specifically include a feature network, an encoding network, and a scoring network. The feature network consists of multiple nonlinear layers, each of which uses a ReLU activation function. Its core task is to capture and construct new feature interaction patterns based on the input pronunciation feature vector sequence through the learning ability of these nonlinear layers, and finally output a processed cross feature vector sequence.

[0070] Encoding network: Built on the feature network, this part uses multiple Transformer layers, each of which integrates a single-head self-attention mechanism. The self-attention mechanism allows the model to focus on information at different positions in the sequence when processing sequence data, thereby improving the model's ability to understand the input sequence. The encoding network receives the cross feature vector sequence from the feature network as input, and through this series of transformation operations, generates a richer encoding feature vector sequence as output.

[0071] Scoring network: This is a relatively simple network structure, which includes a nonlinear layer and a linear layer. The nonlinear layer uses the tanh (hyperbolic tangent) activation function to introduce nonlinear characteristics, so that the scoring network of the model can learn more complex mapping relationships. The main function of this network is to evaluate the head encoding feature vector output by the encoding network. The output result is a vector representing the score value in each dimension. The head encoding feature vector is the first vector in the encoding feature vector sequence.

[0072] The scoring model is obtained as follows:

[0073] Data preparation: Collect historical pronunciation evaluation data, including audio data, standard text to which the audio belongs, and manual scores in each dimension.

[0074] Preprocessing: The collected data is processed according to the above-mentioned calculation to obtain a sequence of pronunciation feature vectors.

[0075] Model training: The model is trained using the stochastic gradient descent method as follows:

[0076] Model initialization: Build the model according to the aforementioned model structure and randomly initialize all weight parameters of the model.

[0077] Forward propagation: Randomly sample the pronunciation feature vector sequence of the training sample, perform forward propagation through the model, and calculate the score of each dimension of the model's prediction.

[0078] Loss calculation: The mean square error loss function is used to quantify the difference between the predicted scores of each dimension and the actual manual scores. The specific loss function is defined as follows:

[0079]

[0080] Where N represents the number of samples, and M represents the number of dimensions of the rating. represents the manual rating of the j dimension of the i sample, Represents the model prediction score of the j dimension of the i sample.

[0081] Gradient calculation: The gradient of the loss function with respect to the model parameters is calculated through the back propagation algorithm. This process starts from the last layer of the model and calculates the gradient of each layer forward layer by layer until the first layer.

[0082] Weight update: Update the model parameters using gradient descent.

[0083] Iterative training: Repeat the process of forward propagation, loss calculation, gradient calculation, and weight update until the performance of the model on the training set meets the predetermined criteria or the maximum number of iterations is reached.

[0084] Weight confirmation: Use the validation set to calculate the loss value of the model weights saved at each stage of the training process. By comparing these loss values, the model weight with the smallest loss value is selected as the final model weight used.

[0085] like Figure 2 As shown, the scoring model in this application may specifically include:

[0086] The feature network consists of two nonlinear layers with relu activation of 90x128 and 128x256, and the size of the output cross feature vector sequence is 256;

[0087] Encoding network: It consists of 3 transformer layers. The hidden layer dimension of the transformer is 256, and the dimension of the output encoding feature vector is also 256.

[0088] Scoring network: It consists of a 256x64 non-linear layer and a 64x3 linear layer, with an output size dimension of 3, corresponding to the scoring dimension.

[0089] During the model training process, we selected the distributed training method, saved the temporary weights every 100 steps, and calculated the loss value on the test set. After the training was completed, we compared the loss values ​​of each temporary weight value on the test set and selected the temporary weight with the smallest loss value as the final weight of the scoring model.

[0090] In some optional implementations of any embodiment of the present disclosure, obtaining a speech feature vector sequence corresponding to the speech to be scored includes: calculating the Mel-frequency cepstrum of the speech signal to obtain a Mel-frequency cepstrum feature vector sequence; taking the logarithm of the Mel-frequency cepstrum feature vector sequence to obtain a logarithmically transformed feature vector; and normalizing the feature vector to a target numerical range to obtain a speech feature vector sequence.

[0091] In some optional implementations of any embodiment of the present disclosure, the step of calculating the pronunciation feature vector sequence includes:

[0092] 1. Take the natural logarithm of the Mel frequency cepstrum feature vector sequence. Normalize the feature vector after logarithmic transformation to a specific range, specifically using the minimum-maximum normalization method, and the specific calculation formula is:

[0093]

[0094] Represents the standardized value, that is, the speech feature vector sequence, represents the value before normalization, and They represent the maximum and minimum values ​​of all samples in the dimension to which x belongs.

[0095] 2. Phoneme sequence calculation

[0096] Taking "I have an apple" as an example, the encoding is introduced in detail:

[0097] a. Segment the sentence and get four words: I, Have, an, and apple.

[0098] b. Find the subphone sequence of each word in CMUdict:

[0099] I: In CMUdict, the subphone sequence corresponding to "I" is "AY1"

[0100] have: In CMUdict, the subphone sequence corresponding to "have" is "HH AE1 V".

[0101] In CMUdict, the subphone sequence of "an" is usually represented as "AH0".

[0102] apple: In CMUdict, the subphone sequence corresponding to "apple" is "AE1 P AH0 L".

[0103] c. Combined phoneme sequence: connect the subphoneme sequence of each word in order, using blank phonemes as intervals, to form a complete phoneme sequence of the sentence, that is, "AY1 SIL HH AE1 V SIL AH0 SIL AE1 P AH0 L", where SIL represents the blank phoneme.

[0104] d. Encode the phonemes: that is, encode "AY1 SIL HH AE1 V SIL AH0 SIL AE1 P AH0 L". According to the CMUdict encoding order, the encoding result is "6 0 16 2 35 0 3 0 2 27 3 21".

[0105] 3. Alignment calculation

[0106] It is necessary to first input the speech feature vector sequence into the acoustic model, calculate the predicted phoneme probability distribution feature sequence corresponding to each speech frame, and then align the predicted phoneme probability distribution feature sequence and the phoneme sequence using the aforementioned method. Figure 3 Where a, b, and c are the results of aligning the sentence "this is a test":

[0107] like Figure 3 As shown in a, the figure shows the phoneme probability distribution feature sequence corresponding to each time frame.

[0108] like Figure 3 As shown in b, the figure shows the alignment path corresponding to the time frame and the maximum phoneme probability of each phoneme in the above CMUdict. For example, the maximum phoneme probabilities of phonemes DH, IH1, and S are 0.95, 0.88, and 0.41, respectively.

[0109] like Figure 3 As shown in c, the figure shows the waveform information of the aligned phoneme sequence. This speech is 2.1875 seconds in total. The maximum phoneme probability of the subphoneme sequence DH IH1 S is 0.66, the maximum phoneme probability of the subphoneme sequence IH1 Z is 0.94, the maximum phoneme probability of the subphoneme sequence AH0 is 0.44, and the maximum phoneme probability of the subphoneme sequence T EH1 ST is 0.50.

[0110] 4. Pronunciation feature vector sequence

[0111] According to the above method, the pronunciation feature vector sequence can be calculated. It should be noted here that the pronunciation feature vector sequence is a sequence composed of pronunciation feature vectors, and the pronunciation feature vector is composed of the following parts:

[0112]

[0113] In this embodiment, because CMUdict has 42 phonemes, the feature vector dimension in the pronunciation feature vector sequence is 42+42+3+3=90 features.

[0114] Furthermore, in this embodiment, the reference data set used to calculate the speech rate feature includes about one hundred evaluation voices with very good pronunciation and their corresponding standard texts. The speech signal and the standard text are aligned, and then the mean and standard deviation of the phoneme frame size of each phoneme are counted to calculate the speech rate feature part.

[0115] The present disclosure provides a pronunciation scoring device 400, which is used to execute the pronunciation scoring method described in the above embodiment. Figure 4a As shown, the device includes: an acquisition unit 401, configured to acquire a speech feature vector sequence corresponding to a speech to be scored; a conversion unit 402, configured to convert the text of the speech to be scored into a phoneme sequence according to the pronunciation; an alignment unit 403, configured to perform time alignment on the speech feature vector sequence and the phoneme sequence to obtain an aligned phoneme sequence, determine a pronunciation feature vector of each phoneme in the aligned phoneme sequence, and generate a pronunciation feature vector sequence, wherein the pronunciation feature vector includes a pitch feature, a speech speed feature, and a stress feature; and a scoring unit 404, configured to score the pronunciation feature vector sequence through a scoring model to obtain a pronunciation score.

[0116] like Figure 4b As shown, another schematic diagram of the pronunciation scoring device is shown, wherein the input module receives the speech to be scored and the standard text from the client. The scoring module calculates the scores of each dimension according to the above method. The output module returns the calculated scores of each dimension to the client.

[0117] The pronunciation scoring device provided in the above-mentioned embodiment of the present disclosure and the pronunciation scoring method provided in the embodiment of the present disclosure are based on the same inventive concept and have the same beneficial effects as the method adopted, run or implemented by the application program stored therein.

[0118] The embodiment of the present disclosure also provides an electronic device corresponding to the pronunciation scoring method provided in the above embodiment, so as to execute the above pronunciation scoring method. The embodiment of the present disclosure is not limited.

[0119] Please refer to Figure 5 , which shows a schematic diagram of an electronic device provided by some embodiments of the present disclosure. Figure 5As shown, the electronic device 50 includes: a processor 500, a memory 501, a bus 502 and a communication interface 503, and the processor 500, the communication interface 503 and the memory 501 are connected via the bus 502; the memory 501 stores a computer program that can be run on the processor 500, and the processor 500 executes the method provided in any of the aforementioned embodiments of the present disclosure when running the computer program.

[0120] The memory 501 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 503 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0121] The bus 502 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory 501 is used to store programs, and the processor 500 executes the programs after receiving execution instructions. The pronunciation scoring method disclosed in any implementation of the aforementioned embodiment of the present disclosure may be applied to the processor 500, or implemented by the processor 500.

[0122] The processor 500 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 500 or the instruction in the form of software. The above processor 500 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present disclosure can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in conjunction with the embodiments of the present disclosure can be directly embodied as a hardware decoding processor to be executed, or the hardware and software modules in the decoding processor can be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 501, and the processor 500 reads the information in the memory 501 and completes the steps of the above method in combination with its hardware.

[0123] The electronic device provided by the embodiment of the present disclosure and the pronunciation scoring method provided by the embodiment of the present disclosure are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by them.

[0124] The present disclosure also provides a computer-readable storage medium corresponding to the pronunciation scoring method provided in the above embodiment. Figure 6 The computer-readable storage medium shown is a CD 60 on which a computer program (ie, a program product) is stored. When the computer program is executed by a processor, the pronunciation scoring method provided by any of the aforementioned embodiments will be executed.

[0125] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

[0126] The computer-readable storage medium provided by the above-mentioned embodiment of the present disclosure and the pronunciation scoring method provided by the embodiment of the present disclosure are based on the same inventive concept, and have the same beneficial effects as the method adopted, run or implemented by the application program stored therein.

[0127] It should be noted that:

[0128] In the above text, the term "comprises", "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be noted that the scope of the method and device in the embodiments of the present disclosure is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0129] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present disclosure, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in the various embodiments of the present disclosure.

[0130] The embodiments of the present disclosure are described above in conjunction with the accompanying drawings, which are only specific implementation modes of the present disclosure. However, the present disclosure is not limited to the above-mentioned specific implementation modes. The above-mentioned specific implementation modes are merely illustrative and not restrictive. Under the guidance of the present disclosure, ordinary technicians in this field can also make many forms without departing from the purpose of the present disclosure and the scope of protection of the claims, all of which are within the protection of the present disclosure.

Claims

1. A pronunciation scoring method, characterized in that: include: Obtain a speech feature vector sequence corresponding to the speech to be rated; According to the pronunciation, converting the text of the speech to be scored into a phoneme sequence; Performing time alignment on the speech feature vector sequence and the phoneme sequence to obtain an aligned phoneme sequence, determining a pronunciation feature vector of each phoneme in the aligned phoneme sequence, and generating a pronunciation feature vector sequence, wherein the pronunciation feature vector includes a pitch feature, a speech rate feature, and a stress feature; The pronunciation feature vector sequence is scored through the scoring model to obtain a pronunciation score; The scoring model includes a feature network, an encoding network and a scoring network; The scoring of the pronunciation feature vector sequence by the scoring model includes: Processing the pronunciation feature vector sequence using a feature network to obtain a cross feature vector sequence; Processing the cross feature vector sequence using a coding network to obtain a target coding feature vector; The target encoding feature vector is processed by the scoring network to obtain the scoring value of each speech attribute of the speech to be scored.

2. The method according to claim 1, characterized in that The time-aligning the speech feature vector sequence and the phoneme sequence comprises: Using an acoustic model, processing the speech feature vector sequence and the phoneme sequence to predict the phoneme probability distribution feature sequence corresponding to each time frame in the speech to be scored; Constructing a probability matrix composed of alignment probabilities between each time frame in the phoneme probability distribution feature sequence and the phoneme sequence; Starting from the end of the probability matrix, tracing back the maximum alignment probability corresponding to each time frame in the probability matrix to obtain an alignment path with the maximum alignment probability; According to the alignment path, an aligned phoneme sequence on a time frame is calculated.

3. The method according to claim 2, characterized in that The intonation feature includes at least one of a phoneme posterior probability feature and a phoneme posterior ratio feature; The phoneme posterior probability feature is the logarithmic value of the posterior probability of the occurrence of a phoneme when the phoneme probability distribution feature sequence and its speech frame range are set. The phoneme posterior probability feature is used to indicate the probability intensity of the occurrence of a phoneme within a target time period. The phoneme probability distribution feature sequence corresponds to each speech frame of the speech to be scored; The phoneme posterior ratio feature is the logarithmic value of the posterior given probability ratio of the real phoneme to the predicted phoneme when the phoneme probability distribution feature sequence and the speech frame range are set.

4. The method according to claim 1, characterized in that: The speech rate feature includes pronunciation duration, which is a ternary feature vector. The ternary feature vector includes the phoneme frame size, the mean and standard deviation of the phoneme frame size of the phoneme in a preset data set.

5. The method according to claim 1, characterized in that: The rereading features include relative energy features, relative duration features and rereading identification features; The relative energy feature is the ratio of the energy of each phoneme in the word to the average energy of all phonemes in the whole word in the aligned phoneme sequence; The relative duration feature is the ratio of the number of phoneme frames of each phoneme in the word to the number of phoneme frames of the entire word in the aligned phoneme sequence; The stress identification feature is used to indicate whether a phoneme needs to be stressed.

6. The method according to claim 1, characterized in that The feature network is composed of multiple nonlinear layers; the feature network is used to process the pronunciation feature vector sequence to obtain a cross feature vector sequence, including: The pronunciation feature vector sequence is generated by the multi-layer nonlinear layer to correspond to a new feature cross pattern, thereby obtaining a new pronunciation feature vector sequence.

7. The method according to claim 1, characterized in that The encoding network includes multiple transformer layers, each of which has a single-head self-attention mechanism module integrated inside; The adopting a coding network to process the cross feature vector sequence to obtain a target coding feature vector includes: The cross feature vector sequence is processed by using a coding network to increase the richness of the cross feature vector sequence, and a head coding feature vector is obtained, where the head coding feature vector is the target coding feature vector.

8. The method according to claim 1, characterized in that The step of obtaining a speech feature vector sequence corresponding to the speech to be rated includes: Calculate the Mel frequency cepstrum of the speech signal to obtain a Mel frequency cepstrum feature vector sequence; Taking the logarithm of the Mel-frequency cepstrum feature vector sequence to obtain a feature vector after logarithmic transformation; The feature vector is normalized to a target numerical range to obtain a speech feature vector sequence.

9. A pronunciation scoring device, characterized in that: include: An acquisition unit, configured to acquire a speech feature vector sequence corresponding to the speech to be rated; A conversion unit, configured to convert the text of the speech to be scored into a phoneme sequence according to the pronunciation; an alignment unit configured to perform time alignment on the speech feature vector sequence and the phoneme sequence to obtain an aligned phoneme sequence, determine a pronunciation feature vector of each phoneme in the aligned phoneme sequence, and generate a pronunciation feature vector sequence, wherein the pronunciation feature vector includes a pitch feature, a speech rate feature, and a stress feature; A scoring unit is configured to score the pronunciation feature vector sequence through a scoring model to obtain a pronunciation score; The scoring model includes a feature network, an encoding network and a scoring network; The scoring unit is further configured to: Processing the pronunciation feature vector sequence using a feature network to obtain a cross feature vector sequence; Processing the cross feature vector sequence using a coding network to obtain a target coding feature vector; The target encoding feature vector is processed by the scoring network to obtain the scoring value of each speech attribute of the speech to be scored.

Images