Speech processing method, apparatus, device, and storage medium

Abstract

The application provides a speech processing method, device and equipment and a storage medium. The method comprises the following steps: determining evaluation text corresponding to evaluation speech and pinyin recognition results of each character in the evaluation text. The pinyin recognition results comprise evaluation pinyin of a character, at least one pinyin phoneme in the evaluation pinyin and pronunciation confidence of the pinyin phoneme. Based on the pinyin phonemes in the evaluation pinyin of the character, the phoneme categories to which the pinyin phonemes belong and the pronunciation confidence of the pinyin phonemes, phoneme scores of each pinyin phoneme in the character are determined. For each character in the evaluation text, character scores of the character are determined according to the phoneme scores of the pinyin phonemes in the evaluation pinyin of the character. Comprehensive pronunciation scores of the evaluation speech are determined according to the character scores of each character in the evaluation text. Evaluation results of the evaluation speech are output to a terminal. The scheme can more comprehensively and truly reflect pronunciation conditions of Putonghua evaluation of a Putonghua evaluator, and is conducive to timely discovery and correction of pronunciation errors.

Description

Technical Field

[0001] This application relates to the field of speech recognition technology, and in particular to a speech processing method, apparatus, device, and storage medium. Background Technology

[0002] Chinese learners or Mandarin practitioners can use oral assessment apps to test their pronunciation and receive oral scores.

[0003] However, the current oral assessment methods cannot accurately and comprehensively reflect the pronunciation of Mandarin assessors, which is not conducive to assessors identifying and correcting pronunciation errors in a timely manner and improving their oral proficiency. Summary of the Invention

[0004] In view of this, this application provides a speech processing method, apparatus, device, and storage medium that can more comprehensively and realistically reflect the pronunciation of Mandarin assessors, which is conducive to timely detection and correction of pronunciation errors.

[0005] To achieve the above objectives, this application provides the following technical solution:

[0006] On the one hand, this application provides a speech processing method, including:

[0007] Obtain the user's evaluation voice input based on the reference text used for testing;

[0008] The evaluation speech is subjected to speech recognition to obtain speech recognition results. The speech recognition results include the evaluation text corresponding to the evaluation speech and the pinyin recognition results of each character in the evaluation text. The pinyin recognition results of the characters include: the evaluation pinyin of the character, at least one pinyin phoneme in the evaluation pinyin, and the pronunciation confidence of the pinyin phoneme.

[0009] For each character in the evaluation text, based on each phoneme in the evaluation pinyin of the character, the phoneme category to which the phoneme belongs, and the pronunciation confidence of the phoneme, a phoneme score is determined for each phoneme in the character. The phoneme score represents the accuracy of the pronunciation of the phoneme. The phoneme category to which the phoneme belongs is one of the initial consonant and the final vowel.

[0010] For each character in the evaluation text, the character score is determined based on the phoneme score of each phoneme in the evaluation pinyin of the character, wherein the character score is used to characterize the accuracy of the pronunciation of the character;

[0011] The overall pronunciation score of the evaluated speech is determined based on the character score of each character in the evaluation text.

[0012] The evaluation results of the evaluation speech are output to the terminal. The evaluation results include: the evaluation text, the character score of each character in the evaluation text, and the comprehensive pronunciation score.

[0013] In one possible implementation, the pinyin recognition result of the character further includes: the evaluated tone of the evaluated pinyin of the character;

[0014] Before determining the character score, the method further includes:

[0015] Determine the reference tone of the reference pinyin for each reference character in the reference text;

[0016] Based on the reference tone corresponding to each reference character in the reference text, the tone evaluation result of the evaluation tone corresponding to each character in the evaluation text is determined respectively. The tone evaluation result of the evaluation tone is used to characterize whether the evaluation tone is correct or incorrect.

[0017] The determination of a character score based on the phoneme scores of each phoneme in the evaluation pinyin of the character includes:

[0018] The character score is determined based on the phoneme scores of each phoneme in the evaluated pinyin of the character and the tone evaluation results of the evaluated tone corresponding to the evaluated pinyin.

[0019] In another possible implementation, based on the phonemes in the evaluated pinyin of the character, the phoneme category to which the phonemes belong, and the pronunciation confidence of the phonemes, a phoneme score for each phoneme in the character is determined, including:

[0020] Based on the phonetic features corresponding to different phonetic phonemes, the phonetic features of each phonetic phoneme in the evaluation pinyin of the character are determined;

[0021] Determine the category features corresponding to the phoneme categories of each phoneme in the evaluated pinyin of the character;

[0022] For each phoneme in the evaluation of the character, the phoneme features, category features and pronunciation confidence of the phoneme are input into the phoneme recognition layer of the scoring model to obtain the phoneme score of the phoneme output by the phoneme recognition layer.

[0023] The character score is determined based on the phoneme scores of each phoneme in the evaluated pinyin of the character and the tone evaluation results of the corresponding evaluated tone of the evaluated pinyin, including:

[0024] The tone evaluation result of the tone corresponding to the character and the phoneme score of each phoneme of the character output by the phoneme recognition layer of the scoring model are input into the character scoring layer of the scoring model to obtain the character score of the character output by the character scoring layer. The scoring model is trained using the phoneme features, category features and tone evaluation results of the phonemes corresponding to each character in multiple text samples labeled with comprehensive pronunciation scores.

[0025] In one possible implementation, the step of outputting the evaluation result of the evaluation speech to the terminal includes:

[0026] The evaluation result interface is output to the terminal. The evaluation result page displays the evaluation text and the comprehensive pronunciation score. The evaluation text also indicates abnormal characters whose character scores are lower than a first threshold, as well as the character scores of the abnormal characters.

[0027] Another possible implementation includes:

[0028] The evaluation result interface of the terminal displays the evaluation pitch and reference pitch of the words whose pitch evaluation result is incorrect.

[0029] In another aspect, this application also provides a voice processing apparatus, including:

[0030] The speech acquisition unit is used to acquire the evaluation speech input by the user in response to the reference text used for testing;

[0031] A speech recognition unit is used to perform speech recognition on the evaluation speech to obtain a speech recognition result. The speech recognition result includes the evaluation text corresponding to the evaluation speech and the pinyin recognition result of each character in the evaluation text. The pinyin recognition result of the character includes: the evaluation pinyin of the character, at least one pinyin phoneme in the evaluation pinyin, and the pronunciation confidence of the pinyin phoneme.

[0032] The phoneme scoring unit is used to determine the phoneme score of each phoneme in the character in the evaluation text based on each phoneme in the evaluation pinyin of the character, the phoneme category to which the phoneme belongs, and the pronunciation confidence of the phoneme. The phoneme score represents the accuracy of the pronunciation of the phoneme. The phoneme category to which the phoneme belongs is one of the initial consonant and the final vowel.

[0033] The character scoring unit is used to determine the character score for each character in the evaluation text based on the phoneme scores of each phoneme in the evaluation pinyin of the character, wherein the character score is used to characterize the accuracy of the pronunciation of the character;

[0034] The speech scoring unit is used to determine the overall pronunciation score of the speech based on the character score of each character in the evaluation text.

[0035] The result output unit is used to output the evaluation result of the evaluation speech to the terminal. The evaluation result includes: the evaluation text, the character score of each character in the evaluation text, and the comprehensive pronunciation score.

[0036] In another aspect, this application also provides a computer device, the computer device including a processor and a memory, the memory storing at least one instruction, at least one program, code set or instruction set, the at least one instruction, the at least one program, the code set or instruction set being loaded and executed by the processor to implement the speech processing method as described above.

[0037] In another aspect, this application also provides a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the speech processing method described above.

[0038] As described above, after performing speech recognition on the user-input evaluation speech, this application combines the phonemes and their categories contained in the evaluation pinyin of each character in the evaluation text, as well as the pronunciation confidence of each phoneme, to determine the phoneme score of each phoneme for each character. Since the phoneme score reflects the accuracy of the pronunciation of the phonemes, for each character in the evaluation text, a character score that can characterize the overall pronunciation accuracy of the character can be determined according to the phoneme scores of each phoneme in the evaluation pinyin of that character in the evaluation text. Based on this, the comprehensive pronunciation score determined by combining the character scores of each character in the evaluation text can accurately and comprehensively reflect the overall pronunciation accuracy of the evaluation speech, thus more realistically reflecting the user's spoken language pronunciation.

[0039] Furthermore, since this application analyzes not only the overall pronunciation of the evaluation speech, but also the pronunciation accuracy of each word in the evaluation text corresponding to the evaluation speech, after outputting the word scores and comprehensive pronunciation scores of each word corresponding to the evaluation speech to the terminal, users can not only understand the overall speech evaluation situation, but also understand the pronunciation of each word. This allows for timely and accurate identification of words with abnormal pronunciation, which is beneficial for timely correction of users' pronunciation errors. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0041] Figure 1 A schematic diagram of the architecture of the speech evaluation system to which the solution of this application is applicable is shown;

[0042] Figure 2 This paper shows a flowchart of an embodiment of a speech processing method provided in this application;

[0043] Figure 3 A flowchart illustrating yet another embodiment of the speech method provided in this application is shown;

[0044] Figure 4 This paper illustrates a schematic diagram of the principle framework for determining a comprehensive pronunciation score based on a scoring model in this application.

[0045] Figure 5 This paper shows a schematic flowchart of yet another embodiment of a speech processing method provided in this application;

[0046] Figure 6 A schematic diagram of the flow interaction of the speech processing method of this application in an application scenario is shown;

[0047] Figure 7 This invention provides a schematic diagram illustrating the structural composition of one embodiment of a voice processing device.

[0048] Figure 8 A schematic diagram of the component architecture of a computer device according to this application is shown. Detailed Implementation

[0049] The solution proposed in this application is applicable to Mandarin proficiency testing scenarios, such as when a Mandarin test taker inputs speech based on a reference text provided by the testing application, and the speech testing system analyzes the test taker's speech based on the reference text to provide a test score for the Mandarin proficiency test.

[0050] To facilitate understanding, the architecture of a voice assessment system to which the solution in this application applies will be described below. Figure 1 As shown, it illustrates a schematic diagram of the composition structure of a voice evaluation system to which this application applies.

[0051] like Figure 1 As shown, the voice assessment system may include: terminal 101 and server 102.

[0052] The terminal 101 and the server 102 can communicate via a network. For example, the terminal 101 can install and run an assessment application for evaluating Mandarin proficiency, and the server is the application server in the Mandarin assessment platform corresponding to the assessment application. Accordingly, a communication connection is established between the assessment application and the server 102.

[0053] Terminal 101 can display reference text. For example, after terminal 101 detects a Mandarin proficiency test request through the testing application, it displays reference text so that the user can read it aloud.

[0054] Simultaneously, terminal 101 can also collect the evaluation voice input by the user in response to the reference text. For example, while displaying the reference text, an evaluation start button can also be displayed on the display interface. After detecting that the user clicks the evaluation start button, the user's voice is collected through a microphone or other voice acquisition device, and the collected voice is identified as the evaluation voice of the reference text.

[0055] Based on the above, the terminal can send the information of the reference text and the evaluation voice to the server, such as sending the reference text or the identifier of the reference text to the server.

[0056] Server 102 can determine the evaluation score of the test voice based on the reference text and the evaluation voice, and feed the evaluation score back to the terminal so that the terminal can display the evaluation score.

[0057] In this application, the terminal can be an electronic device with a display screen and voice acquisition device, such as a mobile phone, tablet computer, or personal computer.

[0058] This server can be a single, independent server.

[0059] This server can also be a server within a cloud service platform. Accordingly, this cloud server can be a cloud server capable of providing basic cloud computing services such as cloud services and cloud computing.

[0060] Cloud computing services are computing services based on cloud technology. Cloud technology refers to a hosting technology that unifies hardware, software, network, and other resources within a wide area network (WAN) or local area network (LAN) to achieve data computing, storage, processing, and sharing.

[0061] Cloud technology is a collective term for network technologies, information technologies, integration technologies, management platform technologies, and application technologies applied to the cloud computing business model. It can form resource pools, providing flexible and convenient on-demand access. Cloud computing technology will become a crucial support. Backend services of technical network systems require substantial computing and storage resources, such as video websites, image websites, and many portal websites. With the rapid development and application of the internet industry, every item may have its own identification mark in the future, requiring transmission to backend systems for logical processing. Data at different levels will be processed separately, and various industry data will all require robust system support, which can only be achieved through cloud computing.

[0062] It should be noted that, Figure 1 This is merely an example of one possible architecture for a voice processing system. In practical applications, if the terminal has strong data processing capabilities, the solution in this application can also be completed on the terminal side, in which case a server is not required.

[0063] The speech processing method of this application is described below with reference to the flowchart.

[0064] First, the solution proposed in this application will be introduced from the server side.

[0065] like Figure 2 The diagram illustrates a flowchart of an embodiment of a voice processing method according to this application. The method of this embodiment is applied to the aforementioned server or terminal, and may include:

[0066] S201, Obtain the evaluation voice input by the user for the reference text used in the test.

[0067] The evaluation voice is the user's actual voice input based on the reference text.

[0068] For example, in the case of a server in this embodiment, after the terminal outputs the reference text for testing, it sends the collected evaluation voice to the server.

[0069] S202, perform speech recognition on the evaluation speech to obtain the speech recognition result.

[0070] The speech recognition results include the evaluation text corresponding to the evaluation speech and the pinyin recognition results of each character in the evaluation text.

[0071] Evaluation text refers to the text parsed from the evaluation speech. In order to distinguish it from the reference text, the text parsed from the evaluation speech is called evaluation text.

[0072] The pinyin recognition result of a character includes the evaluated pinyin of the character, at least one pinyin phoneme in the evaluated pinyin, and the pronunciation confidence of the pinyin phoneme.

[0073] Among them, the evaluated pinyin of a character is the pinyin string of each character in the evaluated text obtained by parsing the evaluated speech. For example, assuming there is the character "你" in the evaluated text, the pinyin of "你" is "ni".

[0074] Among them, pinyin phonemes include each initial and final that make up the pinyin of a Chinese character. It can be understood that the pinyin of most Chinese characters is composed of an initial and a final, and there are also some Chinese characters whose pinyin consists of only one final. Therefore, the pinyin phonemes of the pinyin of a character are at least one initial in the pinyin of the character. If the pinyin of the character contains an initial, the pinyin phonemes also include the initial that makes up the character. For example, the pinyin of "你" is "ni", and correspondingly, the pinyin phonemes include the initial "n" and the final "i".

[0075] The pronunciation confidence of a pinyin phoneme, also known as the pronunciation quality (Goodness of Pronunciation, GOP), can be calculated using, for example, the GOP algorithm. The pronunciation confidence of a pinyin phoneme reflects the accuracy of the pinyin phoneme belonging to the accurate pronunciation phoneme. The higher the confidence, the more standard the pronunciation of the pinyin phoneme.

[0076] It can be understood that during the process of speech recognition of the evaluated speech, the speech recognition can be combined with the reference text. For example, based on each character and the reference pinyin of each character in the reference text (for the sake of distinction, the pinyin of the characters in the reference text is called the reference pinyin), etc., to determine the speech recognition result. The specific process of the present application for determining the speech recognition result is not limited.

[0077] S203. For each character in the evaluated text, based on each pinyin phoneme in the evaluated pinyin of the character, the phoneme category to which the pinyin phoneme belongs, and the pronunciation confidence of the pinyin phoneme, determine the phoneme score of each pinyin phoneme in the character.

[0078] Phoneme categories can be divided into two categories: initials and finals. Among them, for the evaluated pinyin of any character in the evaluated text, the phoneme category of a certain pinyin phoneme in the evaluated pinyin is an initial or a final.

[0079] Among them, the phoneme score characterizes the accuracy of phoneme pronunciation.

[0080] For example, in one possible implementation, a phoneme scoring model can be pre-trained. For each phoneme in the evaluated pinyin, the pinyin phoneme, its phoneme category, and its pronunciation confidence can be input into the phoneme scoring model to obtain the phoneme score output by the model. In practical applications, each pinyin phoneme in the evaluated pinyin of each character can be input into the phoneme scoring model simultaneously, and the model can output the phoneme score of each pinyin phoneme for each character sequentially.

[0081] The phoneme scoring model is trained using the phonemes, phoneme categories, and pronunciation confidence of each character in multiple text samples labeled with comprehensive pronunciation scores.

[0082] Of course, during the training of this phoneme scoring model, it is necessary to combine the phoneme scores of the pinyin phonemes involved in each character in the text sample with the phoneme scoring model in training, to calculate the character score of each character in the text sample, and finally determine the calculated comprehensive pronunciation score of the text sample. On this basis, by combining the loss function value between the calculated comprehensive pronunciation score of each text sample and the actual annotated comprehensive pronunciation score of each text sample, the phoneme scoring model can be continuously adjusted until the accuracy of the finally calculated comprehensive pronunciation score meets the training requirements.

[0083] It is understandable that there may be other ways to determine the phoneme scoring of Pinyin phonemes, and there are no restrictions on this.

[0084] Understandably, this application, after determining the evaluation pinyin for each Chinese character in the evaluation text, analyzes the accuracy of the pronunciation of the Chinese character in terms of the initials and finals that make up the evaluation pinyin, thereby achieving a more granular pronunciation analysis.

[0085] S204. For each character in the evaluation text, determine the character score based on the phoneme scores of each phoneme in the character's evaluation pinyin.

[0086] Among them, the character score is used to characterize the accuracy of the pronunciation of the character.

[0087] For example, in one possible implementation, a character scoring model can be pre-trained. This model can be trained using the phoneme scores of the phonemes involved in each character in multiple text samples labeled with comprehensive pronunciation scores. To improve the accuracy of the character scoring model in determining character scores, it can be trained together with the aforementioned phoneme scoring model. This will be illustrated with an example later, without imposing any restrictions.

[0088] Correspondingly, each phoneme of the pinyin of a character can be input into the phoneme scoring model to obtain the character score.

[0089] Understandably, for each character in the evaluation text, the pronunciation of the character can be reflected more precisely from its pinyin spelling. Therefore, by combining the phoneme scores of each pinyin phoneme in the character's evaluation pinyin, an accurate score of the character's pronunciation accuracy can be obtained.

[0090] S205. Determine the overall pronunciation score of the evaluated speech based on the character scores of each character in the evaluation text.

[0091] In practical applications, there are various ways to determine the overall pronunciation score by combining the character score with the overall pronunciation score. For example, the average score of each character in the evaluation text can be calculated, and this average value can be used as the overall pronunciation score of the evaluated speech. Of course, there are other ways to determine the overall pronunciation score, and this application does not limit this.

[0092] The overall pronunciation score reflects the overall pronunciation of each word in the evaluation text, and thus reflects the overall pronunciation accuracy of the evaluation speech relative to the reference text.

[0093] S206, output the evaluation results of the evaluation voice to the terminal.

[0094] The evaluation results include at least: the evaluation text, the character scores of each character in the evaluation text, and the overall pronunciation score.

[0095] For example, when this embodiment is applied to a terminal, the terminal can output the evaluation results.

[0096] For example, in the case where the solution in this embodiment is executed by the server side, the server can send the evaluation results to the terminal, and the terminal user can not only view the overall pronunciation score, but also view the score of each character.

[0097] It is understandable that the evaluation results can be output to the terminal at once, or in batches. For example, partial evaluation results can be output to the terminal, and the remaining evaluation results can be displayed when the user needs them, so that the user can ultimately obtain evaluation results from different dimensions.

[0098] In one scenario, the evaluation results page is output to the terminal, displaying the evaluation text, the overall pronunciation score, and the individual character scores. In this case, the user can visually view the evaluation text, individual character scores, and the overall pronunciation score on this results page.

[0099] In another scenario, an evaluation results page containing the evaluation text and a comprehensive pronunciation score can be output to the terminal, along with the word scores associated with each character in the evaluation text on the results page. Correspondingly, on the evaluation results page displayed on the terminal, if a user wishes to view the word score of a specific character, they can click on that character. If the terminal detects that a character on the evaluation results page has been clicked, it can display the word score of that character above it on the evaluation results page.

[0100] The above are two examples. This application does not impose any restrictions on the specific method of displaying the evaluation results on the terminal output.

[0101] It is understood that, in this embodiment, in order to enable terminal users to understand the specific pronunciation of each character, the evaluation result in this application may also include the evaluation pinyin of each character in the evaluation text and the phoneme score of each pinyin phoneme in the evaluation pinyin.

[0102] For example, the evaluation pinyin of each character and the phoneme score of each pinyin phoneme can be output to the evaluation results page of the terminal.

[0103] For example, the evaluation results page of the terminal can output the evaluation pinyin of each character, and mark the pinyin phonemes whose phoneme scores are lower than a set phoneme score threshold, for example, by highlighting abnormal pinyin phonemes through bolding or color differentiation. Alternatively, it could display the evaluation pinyin of a character only when its character score is lower than a set threshold, and mark the pinyin phonemes in that evaluation pinyin whose phoneme scores are lower than the set phoneme score threshold.

[0104] For example, after returning the evaluated pinyin of each character and the phoneme scores of each pinyin phoneme to the terminal, if the user requests to display the pronunciation of a certain character (e.g., if the user clicks on a character in the displayed evaluation text), the evaluated pinyin of that character is displayed, and the pinyin phonemes with phoneme scores lower than a set phoneme score threshold are marked.

[0105] This application, after performing speech recognition on the user-input evaluation speech, combines the phonemes and their categories contained in the evaluation pinyin of each character in the evaluation text, as well as the pronunciation confidence of each phoneme, to determine the phoneme score of each phoneme for each character. Since the phoneme score reflects the accuracy of pronunciation, for each character in the evaluation text, based on the phoneme scores of each phoneme in the evaluation pinyin of that character, a character score that characterizes the overall pronunciation accuracy of that character can be determined. Based on this, the comprehensive pronunciation score determined by combining the character scores of each character in the evaluation text can accurately and comprehensively reflect the overall pronunciation accuracy of the evaluation speech, thus providing a more realistic reflection of the user's spoken language pronunciation.

[0106] Meanwhile, since this application not only analyzes the overall pronunciation of the evaluated speech, but also analyzes the pronunciation accuracy of each character in the evaluation text corresponding to the evaluated speech. Therefore, after outputting the character scores and comprehensive pronunciation scores of each character corresponding to the evaluated speech to the terminal, it not only enables the user to understand the overall speech evaluation situation, but also enables the user to understand the pronunciation of each character, so as to timely and accurately discover the characters with abnormal pronunciation, which is beneficial to timely correcting the pronunciation errors of each character of the user.

[0107] It can be understood that the tones of each character in the evaluated speech will also affect the pronunciation level of Putonghua. Therefore, in this application, in addition to analyzing the character scores by combining the pronunciation of each phoneme in the evaluation pinyin of each character in the evaluated speech, the tones of each character in the evaluated speech, that is, the tones of the evaluation pinyin corresponding to the characters, can also be combined to analyze the character scores.

[0108] As Figure 3 shown, it shows a schematic flowchart of another embodiment of a speech processing method of this application. The method of this embodiment applied to a server or a terminal may include:

[0109] S301, obtain the evaluated speech input by the user for the reference text for testing.

[0110] S302, perform speech recognition on the evaluated speech to obtain a speech recognition result.

[0111] Among them, the speech recognition result includes the evaluation text corresponding to the evaluated speech and the pinyin recognition result of each character in the evaluation text.

[0112] The pinyin recognition result of a character includes: the evaluation pinyin of the character, at least one phoneme in the evaluation pinyin, the pronunciation confidence of each phoneme, and the evaluation tone of the evaluation pinyin. For example, the evaluation pinyin of "你" is "ni", and the evaluation tone is the third tone.

[0113] S303, for each character in the evaluation text, determine the phoneme score of each phoneme in the character based on each phoneme in the evaluation pinyin of the character, the phoneme category to which the phoneme belongs, and the pronunciation confidence of the phoneme.

[0114] S304, determine the reference tone of the reference pinyin of each reference character in the reference text.

[0115] Among them, for the sake of distinction, each character in the reference text is called a reference character, the pinyin string of each reference character is called a reference pinyin, and the tone of the pinyin of the reference character is called a reference tone.

[0116] Among them, the tones of pinyin can also be called tones, which can be divided into five types: light tone, first tone, second tone, third tone, and fourth tone. The reference tone of a reference character is the correct tone of the reference character.

[0117] Among them, when the information of the reference text returned by the terminal to the server includes the reference pinyin and reference tone of each character in the reference text, the server can directly obtain the reference pinyin and reference tone of each character in the reference text.

[0118] When the identifier of the reference text returned by the terminal to the server, the server can obtain the reference text saved in the server and the reference pinyin and reference tone of each reference character in the reference text based on the identifier of the reference text.

[0119] S305, according to the reference tones corresponding to each reference character in the reference text, respectively determine the tone evaluation results of the evaluation tones corresponding to the evaluation pinyin of each character in the evaluation text.

[0120] Among them, the tone evaluation result of the evaluation tone is used to represent whether the evaluation tone is correct or incorrect.

[0121] For example, the character "你" in the reference text corresponds to "ni" with the third tone, and the recognized evaluation text in the evaluation speech is also "你" with the pinyin "ni", but the recognized evaluation tone belongs to an inaccurate tone, such as the evaluation tone is the second tone, then the tone evaluation result of this evaluation tone is incorrect.

[0122] S306, for each character in the evaluation text, determine the character score of the character according to the phoneme scores of each phoneme in the evaluation pinyin of the character and the tone evaluation result of the evaluation tone corresponding to the evaluation pinyin.

[0123] For example, a trained character score model can be used, and combined with the phoneme scores of each phoneme in the evaluation pinyin of the character and the tone evaluation result of the evaluation tone, the character score of the character can be determined.

[0124] For another example, the phoneme comprehensive score corresponding to the evaluation pinyin of the character can be determined first according to the phoneme scores of each phoneme in the evaluation pinyin of the character. Correspondingly, the phoneme comprehensive score and the tone evaluation result can both be used as the character scores of two different dimensions of the character; or, the phoneme comprehensive score and the tone evaluation result are input into the character score model to obtain the character score of the character.

[0125] Of course, there can be other ways to determine the character score by combining the phoneme scores of pinyin phonemes and the tone evaluation results of evaluation tones, and there is no restriction on this.

[0126] S307, output the evaluation result of the evaluation speech to the terminal.

[0127] The evaluation results include: the evaluated text, the character scores of each character in the evaluated text, and the comprehensive pronunciation score.

[0128] In an optional manner, the evaluation result may further include the tone evaluation result of the evaluated tone of each character in the evaluated text.

[0129] Alternatively, the evaluation result includes: the evaluated tone of the character with an incorrect tone evaluation result. Of course, for the convenience of the user to compare the tone they pronounced with the correct tone, the server may simultaneously return the evaluated tone of the character with an incorrect tone evaluation result and the corresponding reference tone to the terminal.

[0130] Of course, the evaluation result may also include the phoneme score of the corresponding phoneme of each character in the evaluated text.

[0131] In this embodiment, the pronunciation situation of each character in the evaluated speech is comprehensively determined by combining the evaluated tone of the evaluated pinyin of each character recognized by the evaluated speech and the phonemes in each evaluated pinyin, which is beneficial to more accurately determine the pronunciation situation of each character, and further beneficial to more accurately and comprehensively obtain the comprehensive pronunciation situation of the evaluated speech according to the pronunciation situation of each character.

[0132] It can be understood that in the process of determining the scores of phonemes, characters, and the entire speech in this application, corresponding models can be used for scoring for each dimension respectively.

[0133] In an optional implementation manner, in order to improve the accuracy of scoring, this application can train an overall scoring model, and output the phoneme score of the pinyin phoneme, the character score, and the comprehensive pronunciation score through different layers of the scoring model.

[0134] As Figure 4 shown, it shows a schematic diagram of a composition structure of the scoring model in this application. Taking the evaluated text corresponding to the input evaluated speech as "How are you" as an example for illustration in Figure 4 .

[0135] From Figure 4 it can be seen that after recognizing the evaluated text "How are you" of the evaluated speech and the pinyin recognition results of each character, the feature 401 of each pinyin phoneme, the category feature 402 of the phoneme category of the pinyin phoneme, and the pronunciation confidence 403 in the evaluated pinyin of each character can be input into the convolutional neural network layer 404 serving as the phoneme evaluation layer, and the phoneme score 405 of each pinyin phoneme of each character output by this convolutional neural network layer can be obtained.

[0136] Correspondingly, the phoneme score of the pinyin phoneme of each character and the tone evaluation result 406 of the evaluated tone of the evaluated pinyin corresponding to each character are input into the character scoring model 407 of the evaluation model, and the character score of the character can be obtained.

[0137] Based on this, the character score of each character in the evaluation text can be calculated through the sentence scoring layer 408 in the scoring model to obtain the comprehensive pronunciation score 409.

[0138] For ease of understanding, combined with Figure 4 The structure of the scoring model shown introduces the speech processing method of this application.

[0139] like Figure 5 The diagram illustrates a flowchart of another embodiment of a speech processing method according to this application. This embodiment's method is applied to a server and may include:

[0140] S501, obtains the evaluation speech input by the user for the reference text used in the test.

[0141] S502, determine the speech recognition result of the evaluation speech.

[0142] The speech recognition results include the evaluation text corresponding to the evaluation speech and the pinyin recognition results of each character in the evaluation text.

[0143] The pinyin recognition results for the characters include: the evaluated pinyin of the character, at least one pinyin phoneme in the evaluated pinyin, the pronunciation confidence of each pinyin phoneme, and the evaluated tone of the evaluated pinyin.

[0144] Among them, the evaluation tone of the evaluation pinyin is the actual tone of the evaluation pinyin of the character obtained through evaluation speech analysis.

[0145] The pronunciation confidence score GOP(p) of the phoneme p in Pinyin can be calculated using the following formula:

[0146]

[0147] Where P(p|0) (p) ) represents the probability of the phoneme p occurring during the corresponding pronunciation. Q represents all phonemes, NF(p) represents the number of frames in which the phoneme p is pronounced. P(p) represents the probability of the phoneme p occurring, P(0) represents the probability of the phoneme p occurring. (p) |p) represents the probability that the phoneme p is pronounced as the corresponding phonological segment, which can be obtained during speech recognition. P(q) represents the probability that the phoneme q appears.

[0148] S503, for each character in the evaluation text, determine the category features corresponding to the phoneme category of each pinyin phoneme in the evaluation category of the character, and determine the phoneme features of each pinyin phoneme in the evaluation pinyin of the character based on the phoneme features corresponding to different pinyin phonemes.

[0149] For a given phoneme in Pinyin, its phoneme category is either an initial consonant or a final vowel. The category feature corresponding to the phoneme category is the feature representation of that phoneme category. In practical applications, feature vectors can be set for the initial consonant category and the final vowel category. For example, the category feature for the initial consonant category can be represented as [1,0], and the category feature for the final vowel category can be represented as [0,1].

[0150] Similarly, the phonetic features of Pinyin phonemes are the feature representations of Pinyin phonemes. The phonetic features of different Pinyin phonemes are different. This application can learn and save the phonetic features of different Pinyin phonemes through pre-training. For example, since each initial consonant and each final vowel is a Pinyin phoneme, an initial vector representation can be pre-assigned to each initial consonant and final vowel. Simultaneously, during the training of the scoring model, the vector representations of various initial consonants and final vowels are continuously adjusted until the final vector representation of each initial consonant and each final vowel is determined and saved.

[0151] Based on this, according to the pre-set phonetic features of different phonetic phonemes, the phonetic features of each phonetic phoneme in the evaluation pinyin of the character can be queried.

[0152] S504, for each phoneme in the evaluation of the character, the phoneme features, category features and pronunciation confidence of the corresponding phoneme are input into the phoneme recognition layer of the scoring model to obtain the phoneme score of the phoneme output by the phoneme recognition layer.

[0153] The scoring model is trained using the phoneme features, category features, and tone evaluation results of the pinyin phonemes corresponding to each character in multiple text samples labeled with comprehensive pronunciation scores. In this application, the scoring model may include at least a phoneme recognition layer and a character scoring layer. The scoring model may also include a nonlinear transformation layer and a sentence scoring layer located after the character scoring layer, which will be described in detail later.

[0154] In one implementation, the phoneme features, category features, and pronunciation confidence of a pinyin phoneme can be concatenated into a feature vector of the pinyin phoneme, and this feature vector can be input into the phoneme recognition layer of the scoring model.

[0155] For example, suppose the category feature of the initial consonant can be represented as [0,1], while the category feature of the final vowel can be represented as [1,0]. Based on this, for Figure 4For "you" in Chinese, the pinyin for "you" is "ni". Assume that the GOP of "n" can be expressed as 0.1, and assume that the phoneme features (specifically, the phoneme feature vector) of the pinyin phoneme "n" obtained through query can be expressed as [0.1, 0.3, 0.2]. Then, the combined representation of these three parts of features is called the feature vector 410 of "n" as [0.1, 0, 1, 0.1, 0.3, 0.2]. Based on this, the feature vector of "n" can be input into the phoneme recognition layer of the scoring model, such as Figure 4 the convolutional neural network layer in

[0156] Correspondingly, the convolutional neural network model can output the phoneme score of "n", such as Figure 4 shown as the phoneme score of "n" being 0.1. Similarly, the phoneme score of another pinyin phoneme "i" of "you" is "0.3". Similarly, the phoneme score of "h" is 0.1, the phoneme score of "ao" is 0.2, the phoneme score of "m" is 0.5, and the phoneme score of "a" is 0.7.

[0157] S505, determine the reference tones of the reference pinyins of each reference character in the reference text.

[0158] S506, based on the reference tones corresponding to each reference character in the reference text, respectively determine the tone evaluation results of the evaluated tones corresponding to the evaluated pinyins of each character in the evaluated text.

[0159] Among them, the tone evaluation result of the evaluated tone is used to represent whether the evaluated tone is correct or incorrect.

[0160] S507, for each character in the evaluated text, input the phoneme scores of each pinyin phoneme of the character output by the phoneme recognition layer into the attention model layer of the scoring model to obtain the comprehensive phoneme score of the character output by the attention model layer.

[0161] It can be understood that after training the scoring model, the attention weights of the attention model are also determined.

[0162] For example, in the attention model, for each pinyin phoneme, it is necessary to calculate its attention weight coefficient U p and α p, as shown in Formula 2 and Formula 3:

[0163] U p = tanh(w * O p + b) (Formula 2);

[0164]

[0165] Among them, O p is the phoneme score of the pinyin phoneme p. Among them, w and b are respectively the internal parameters determined through training in the attention model, and Uw is an internal vector in the attention model. After the attention model is trained, this U w is the fixed context vector. q belongs to the pinyin phonemes in C, and C is the set of all pinyin phonemes of the characters.

[0166] Correspondingly, after determining the attention weight coefficient of the pinyin phoneme, the comprehensive phoneme score S corresponding to the character can be calculated using the following formula 4 C :

[0167]

[0168] On this basis, the character score can be represented by these two dimensions: the score of the first dimension and the tone evaluation result.

[0169] S508, input the comprehensive phoneme score and the tone evaluation result of the evaluated tone corresponding to the character into the non-linear transformation layer of the scoring model to obtain the character score of the character.

[0170] Among them, the attention model layer and the non-linear transformation layer constitute the character scoring layer.

[0171] For example, the non-linear transformation layer can be composed of the activation function sigmoid.

[0172] In a possible implementation manner, the comprehensive phoneme score and the tone evaluation result of the evaluated tone can be used to form the character feature of the character, and then the character feature is input into the non-thread transformation layer to obtain the character score.

[0173] For example, it can be set that if the tone evaluation result of the evaluated tone is correct, it is represented as 1; if it is incorrect, it is represented as 0. Still combined with Figure 4 , taking the character "你" as an example, assuming that the comprehensive phoneme score of "你" is 0.5 and the tone evaluation result is incorrect, represented as 0. Correspondingly, the scoring feature of the character score can be represented as [0.6, 0]. Then, through the non-linear transformation layer, the character score of you can be obtained.

[0174] Correspondingly, the scoring feature of the character is non-linearly transformed through S508 to obtain the final character score.

[0175] S509, determine the comprehensive pronunciation score of the evaluated speech according to the character scores of each character in the evaluated text.

[0176] For example, the average value of the character scores of each character can be calculated, and this average value is determined as the comprehensive pronunciation score of the evaluated speech.

[0177] For example Figure 4 as shown, the context vector U wis [0.1, 0.3, 0.5]. After passing each pinyin phoneme of each character through the attention layer, the comprehensive phoneme score of each character is obtained. Correspondingly, after the eigenvalue of the phoneme score of the pinyin phoneme and the tone score result of each character's pinyin tone forms the character feature, it can be transformed through the non-linear transformation layer. Then, the average value of the character scores of each character is obtained through the sentence scoring layer to obtain the comprehensive pronunciation score.

[0178] For example, the tone score results of each character in "How are you?" are: you: 0; hao: 1, ma: 0; and the comprehensive phoneme scores of each character are: you: 0.5, hao: 0.8; ma: 0.3. Based on this, the final comprehensive pronunciation score is obtained as 0.7.

[0179] S510, output the evaluation result interface to the terminal.

[0180] For example, the evaluation result interface displays the evaluation text and the comprehensive pronunciation score.

[0181] In one possible implementation, the evaluation tone and the reference tone corresponding to the character with an incorrect tone evaluation result can also be displayed in the evaluation result interface. The reference tone can be the correctly marked character corresponding to the target character.

[0182] In another possible implementation, the abnormal characters with a character score lower than the first threshold and the character scores of the abnormal characters can also be marked in the evaluation text displayed in the evaluation result interface. The first threshold can be set as needed.

[0183] S511, after obtaining the viewing instruction of the target abnormal character of the terminal, display the evaluation pinyin of the target abnormal character in the evaluation result interface, and mark the pinyin phoneme with a phoneme score lower than the second threshold.

[0184] Among them, the viewing instruction is generated after the target abnormal character displayed in the evaluation result interface is detected and clicked by the terminal. The target abnormal character belongs to the abnormal characters marked in the evaluation text.

[0185] The second threshold can be different from the first threshold or the same, and can be specifically set as needed.

[0186] It can be understood that this step S511 is an optional step, and its purpose is to enable the user to view the abnormal information of each abnormal character in more detail.

[0187] For the sake of easy understanding, the solution of this application will be introduced from the perspective of the interaction between the terminal and the server. As Figure 6 shown, it shows a flow interaction schematic diagram of an application scenario of the voice processing method of this application.

[0188] By Figure 6It can be seen that on the terminal, the evaluation start interface 601 can be presented through the evaluation application, and a reference text for the user to read is displayed in this evaluation interface. After the user clicks the "Start Reading Aloud" button in the evaluation start interface 601, the terminal can collect the user's voice. At the same time, the evaluation start interface 601 of the terminal will change to the evaluation intermediate interface 602. A "Stop Reading Aloud" button is displayed in this evaluation intermediate interface 602.

[0189] If the user finishes reading aloud, the user can click the "Stop Reading Aloud" button in the evaluation intermediate interface 602. Correspondingly, the terminal can end the user's voice collection and send the collected voice as the evaluation voice to the server.

[0190] After the server obtains this evaluation voice, it can analyze this evaluation voice in the manner of any one of the foregoing embodiments of the voice processing method, and finally obtain an evaluation result. As described above, this evaluation result can include a comprehensive pronunciation score, and can also include the character scores of each character in the evaluation text corresponding to the evaluation voice, the evaluation pinyin of each character, and the phoneme scores of each phoneme in the evaluation pinyin.

[0191] After the terminal obtains the evaluation result returned by the server, it can display the evaluation result interface 603. Figure 6 It can be seen that the evaluation result interface at least displays the evaluation text and the comprehensive pronunciation score corresponding to the evaluation text. For Figure 6 example, the comprehensive pronunciation score is 80.

[0192] At the same time, in the evaluation result interface 603, abnormal characters with abnormal pronunciations (such as characters with a character score lower than the first threshold) are also highlighted in bold, such as "gua" shown in the evaluation result interface 603.

[0193] If the user hopes to view the character score of a certain character, the user can also click or move the cursor to that character, and then the character score of that character will be displayed above the character. For Figure 6 example, the arrow in [Figure] points to "bian" indicating that this character is selected, and the score of this character, which is 70, is displayed above "bian".

[0194] It can be understood that the user can also request to view whether there are abnormal pronunciations of the initial consonants and finals in the evaluation pinyin of any character in the evaluation result interface. For example, the user can click the "Evaluation Result" button in the evaluation result interface to display the evaluation pinyin corresponding to all characters and the phoneme scores of relevant initial consonants, finals, and vowels. Or, after selecting a certain character, a menu bar can be displayed. The menu bar can present a pinyin details option. Clicking this option can view the phoneme scores of the initial consonants and finals in the specific evaluation pinyin, as well as one or more of the information such as the phoneme scores of the initial consonants or finals with phoneme scores lower than the set value. For example, a specific mark can be used to represent such abnormal pronunciation phoneme scores and display them, etc.

[0195] Corresponding to the speech processing method of this application, this application also provides a speech processing device.

[0196] like Figure 7 The diagram illustrates a flowchart of an embodiment of a voice processing device according to this application. This device can be applied to a server or a terminal. The device may include:

[0197] The speech acquisition unit 701 is used to acquire the evaluation speech input by the user in response to the reference text used for testing;

[0198] The speech recognition unit 702 is used to perform speech recognition on the evaluation speech and obtain a speech recognition result. The speech recognition result includes the evaluation text corresponding to the evaluation speech and the pinyin recognition result of each character in the evaluation text. The pinyin recognition result of the character includes: the evaluation pinyin of the character, at least one pinyin phoneme in the evaluation pinyin, and the pronunciation confidence of the pinyin phoneme.

[0199] Phoneme scoring unit 703 is used to determine the phoneme score of each phoneme in the evaluation text for each character, based on the phonemes in the evaluation pinyin of the character, the phoneme category to which the phoneme belongs, and the pronunciation confidence of the phoneme. The phoneme score represents the accuracy of the pronunciation of the phoneme. The phoneme category to which the pinyin phoneme belongs is one of the initial consonant and the final vowel.

[0200] The character scoring unit 704 is used to determine the character score for each character in the evaluation text based on the phoneme scores of each phoneme in the evaluation pinyin of the character. The character score is used to characterize the accuracy of the pronunciation of the character.

[0201] The speech scoring unit 705 is used to determine the overall pronunciation score of the speech based on the character score of each character in the evaluation text.

[0202] The result output unit 706 is used to output the evaluation result of the evaluation speech to the terminal. The evaluation result includes: the evaluation text, the character score of each character in the evaluation text, and the comprehensive pronunciation score.

[0203] In one possible implementation, the pinyin recognition result of the character identified by the speech recognition unit also includes: the evaluation tone of the character's pinyin;

[0204] The device may also include:

[0205] The tone determination unit is used to determine the reference tone of the reference pinyin of each reference character in the reference text before the character scoring unit determines the character score of the character.

[0206] The tone evaluation unit is used to determine the tone evaluation result of the evaluation tone corresponding to each evaluation tone of each character in the evaluation text based on the reference tone corresponding to each reference character in the reference text. The tone evaluation result of the evaluation tone is used to indicate whether the evaluation tone is correct or incorrect.

[0207] The character scoring unit is specifically used to determine the character score based on the phoneme scores of each phoneme in the evaluated pinyin of the character and the tone evaluation results of the corresponding evaluated tone of the evaluated pinyin.

[0208] In one alternative approach, the phoneme scoring unit includes:

[0209] The phoneme feature determination unit is used to determine the phoneme features of each phoneme in the evaluation pinyin of the character based on the phoneme features corresponding to different pinyin phonemes.

[0210] The category feature determination unit is used to determine the category features corresponding to the phoneme categories of each phoneme in the evaluation pinyin of the character;

[0211] The phoneme scoring processing unit is used to input the phoneme features, category features and pronunciation confidence of each phoneme in the evaluation of the character into the phoneme recognition layer of the scoring model, and obtain the phoneme score of the phoneme output by the phoneme recognition layer.

[0212] The character scoring unit is specifically used to input the tone evaluation result of the character's corresponding tone and the phoneme score of each phoneme of the character output by the phoneme recognition layer of the scoring model into the character scoring layer of the scoring model to obtain the character score of the character output by the character scoring layer. The scoring model is trained using the phoneme features, category features and tone evaluation results of the phonemes of each character in multiple text samples labeled with comprehensive pronunciation scores.

[0213] In one alternative approach, the word scoring unit includes:

[0214] The phoneme synthesis subunit is used to input the phoneme scores of each phoneme of the character output by the phoneme recognition layer into the attention model layer of the scoring model to obtain the phoneme synthesis score of the character output by the attention model layer.

[0215] The character scoring subunit is used to input the comprehensive phoneme score and the tone evaluation result of the corresponding tone of the character into the nonlinear transformation layer of the scoring model to obtain the character score. The attention model layer and the nonlinear transformation layer constitute the character scoring layer.

[0216] In one possible implementation, the result output unit is specifically used to output an evaluation result interface to the terminal. The evaluation result page displays the evaluation text and the comprehensive pronunciation score, and the evaluation text indicates abnormal characters whose word scores are lower than a first threshold and the word scores of the abnormal characters.

[0217] The result output unit is also used to display the evaluation pitch and reference pitch corresponding to the words whose pitch evaluation result is incorrect in the evaluation result interface of the terminal.

[0218] In one alternative embodiment, the device further includes:

[0219] The anomaly details processing unit is used to display the evaluation pinyin of the target abnormal character on the evaluation result interface of the terminal after receiving the viewing instruction of the target abnormal character sent by the terminal, and to mark the pinyin phonemes whose phoneme scores are lower than the second threshold in the evaluation pinyin. The viewing instruction is generated by the terminal after detecting that the target abnormal character displayed in the evaluation result interface has been clicked. The target abnormal character belongs to the abnormal characters marked in the evaluation text.

[0220] Furthermore, this application also provides a computer device, which can be the aforementioned terminal or server. For example... Figure 8 This illustrates a schematic diagram of the architectural composition of the computer device provided in this application. Figure 8 The computer device 800 may include a processor 801 and a memory 802.

[0221] Optionally, the computer device may also include: a communication interface 803, an input unit 804, a display 805, and a communication bus 806.

[0222] The processor 801, memory 802, communication interface 803, input unit 804 and display 805 communicate with each other through communication bus 806.

[0223] In this embodiment of the application, the processor 801 may be a central processing unit, an application-specific integrated circuit, etc.

[0224] The memory stores at least one instruction, at least one program, code set, or instruction set, which is loaded and executed by the processor to implement the speech processing method mentioned in the above embodiment.

[0225] In one possible implementation, the memory 802 may include a program storage area and a data storage area, wherein the program storage area may store the operating system, the programs mentioned above, etc.; and the data storage area may store data created during the use of the computer device.

[0226] The communication interface 803 can be used as an interface for a communication module.

[0227] This application may also include an input unit 804, which may include a touch sensing unit, a keyboard, etc.

[0228] The display 805 includes a display panel, such as a touch display panel.

[0229] certainly, Figure 8 The computer device structure shown does not constitute a limitation on the computer device in the embodiments of this application. In practical applications, the computer device may include more than [other components]. Figure 8 More or fewer components as shown, or combinations of certain components.

[0230] On the other hand, this application also provides a computer-readable storage medium, characterized in that the computer-readable storage medium stores at least one instruction, at least one program, code set or instruction set, wherein the at least one instruction, the at least one program, the code set or instruction set is loaded and executed by a processor to implement the speech processing method as described in any of the above embodiments.

[0231] This application also proposes a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various optional implementations of the above-described speech processing methods or speech processing apparatus. Specific implementation processes can be referred to the descriptions of the corresponding embodiments above, and will not be repeated here.

[0232] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. Furthermore, the features described in the various embodiments of this specification can be substituted or combined with each other, enabling those skilled in the art to implement or use this application. For apparatus embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0233] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0234] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0235] The above are merely preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A speech processing method, characterized in that, include: Obtain the user's evaluation voice input based on the reference text used for testing; The evaluation speech is subjected to speech recognition to obtain speech recognition results. The speech recognition results include the evaluation text corresponding to the evaluation speech and the pinyin recognition results of each character in the evaluation text. The pinyin recognition results of the characters include: the evaluation pinyin of the character, at least one pinyin phoneme in the evaluation pinyin, the pronunciation confidence of the pinyin phoneme, and the evaluation tone of the evaluation pinyin. For each character in the evaluation text, a scoring model is used to determine the phoneme score of each phoneme in the character's evaluation pinyin, based on the phoneme category to which the phoneme belongs and the pronunciation confidence of the phoneme. This includes: Based on the pre-set phoneme features corresponding to different pinyin phonemes, the phoneme features of each pinyin phoneme in the evaluation pinyin of the character are retrieved; the category features corresponding to the phoneme category of each pinyin phoneme in the evaluation pinyin of the character are determined; for each pinyin phoneme in the evaluation pinyin of the character, the phoneme features, category features, and pronunciation confidence of the pinyin phoneme are input into the phoneme recognition layer of the scoring model to obtain the phoneme score of the pinyin phoneme output by the phoneme recognition layer; wherein, the phoneme score represents the accuracy of the pronunciation of the phoneme, and the phoneme category to which the pinyin phoneme belongs is one of the initials and finals; Determine the reference tone of the reference pinyin for each reference character in the reference text; Based on the reference tone corresponding to each reference character in the reference text, the tone evaluation result of the evaluation tone corresponding to each character in the evaluation text is determined respectively. The tone evaluation result of the evaluation tone is used to characterize whether the evaluation tone is correct or incorrect. For each character in the evaluation text, the phoneme scores of each phoneme in the evaluation pinyin of the character output by the phoneme recognition layer are input into the attention model layer of the scoring model to obtain the comprehensive phoneme score of the character output by the attention model layer; wherein, through , and Determine the phoneme comprehensive score of the character. , and Representing the attention weight coefficients respectively, Characterizing Pinyin phonemes Phoneme scoring, and Each characterizes the determined intrinsic parameters trained in the model. The fixed context vector obtained after the representation model is trained. belong Pinyin phonemes in The set of all phonemes representing the character; The comprehensive phoneme score and the tone evaluation result of the evaluation tone corresponding to the evaluation pinyin are input into the nonlinear transformation layer of the scoring model to determine the character score. The character score is used to characterize the accuracy of the character's pronunciation. The scoring model is trained using the phoneme features, category features, and tone evaluation results of the pinyin phonemes corresponding to each character in multiple text samples labeled with comprehensive pronunciation scores. The scoring model includes at least a phoneme recognition layer and a character scoring layer in sequence. The attention model layer and the nonlinear transformation layer constitute the character scoring layer. The overall pronunciation score of the evaluated speech is determined based on the character score of each character in the evaluation text. The evaluation results of the evaluation speech are output to the terminal in batches. The evaluation results include: the evaluation text, the character score of each character in the evaluation text, and the comprehensive pronunciation score.

2. The method according to claim 1, characterized in that, The step of outputting the evaluation result of the evaluation speech to the terminal includes: The evaluation result interface is output to the terminal. The evaluation result page displays the evaluation text and the comprehensive pronunciation score. The evaluation text also indicates abnormal characters whose character scores are lower than a first threshold, as well as the character scores of the abnormal characters.

3. The method according to claim 2, characterized in that, Also includes: The evaluation result interface of the terminal displays the evaluation pitch and reference pitch of the words whose pitch evaluation result is incorrect.

4. The method according to claim 3, characterized in that, Also includes: After receiving a viewing instruction for the target abnormal character sent by the terminal, the evaluation pinyin of the target abnormal character is displayed on the evaluation result interface of the terminal, and the pinyin phonemes with phoneme scores lower than the second threshold are marked. The viewing instruction is generated by the terminal after detecting that the target abnormal character displayed in the evaluation result interface has been clicked. The target abnormal character belongs to the abnormal characters marked in the evaluation text.

5. A voice processing device, characterized in that, include: The speech acquisition unit is used to acquire the evaluation speech input by the user in response to the reference text used for testing; A speech recognition unit is used to perform speech recognition on the evaluation speech to obtain a speech recognition result. The speech recognition result includes the evaluation text corresponding to the evaluation speech and the pinyin recognition result of each character in the evaluation text. The pinyin recognition result of the character includes: the evaluation pinyin of the character, at least one pinyin phoneme in the evaluation pinyin, the pronunciation confidence of the pinyin phoneme, and the evaluation tone of the evaluation pinyin. A phoneme scoring unit is used to determine the phoneme score of each phoneme in the character in the evaluation text, based on a scoring model, the phoneme category to which the phoneme belongs, and the pronunciation confidence of the phoneme, including: Based on the pre-set phoneme features corresponding to different pinyin phonemes, the phoneme features of each pinyin phoneme in the evaluation pinyin of the character are retrieved; the category features corresponding to the phoneme category of each pinyin phoneme in the evaluation pinyin of the character are determined; for each pinyin phoneme in the evaluation pinyin of the character, the phoneme features, category features, and pronunciation confidence of the pinyin phoneme are input into the phoneme recognition layer of the scoring model to obtain the phoneme score of the pinyin phoneme output by the phoneme recognition layer; wherein, the phoneme score represents the accuracy of the pronunciation of the phoneme, and the phoneme category to which the pinyin phoneme belongs is one of the initials and finals; A tone determination unit is used to determine the reference tone of the reference pinyin for each reference character in the reference text; The pitch evaluation unit is used to determine the pitch evaluation result of the evaluation pitch corresponding to the evaluation pinyin of each character in the evaluation text based on the reference pitch corresponding to each reference character in the reference text. The pitch evaluation result of the evaluation pitch is used to characterize whether the evaluation pitch is correct or incorrect. The character scoring unit is used to input the phoneme scores of each phoneme in the evaluation pinyin of each character in the evaluation text (output by the phoneme recognition layer) into the attention model layer of the scoring model to obtain the comprehensive phoneme score of the character output by the attention model layer; wherein, through , and Determine the phoneme comprehensive score of the character. , and Representing the attention weight coefficients respectively, Characterizing Pinyin phonemes Phoneme scoring, and Each characterizes the determined intrinsic parameters trained in the model. The fixed context vector obtained after the representation model is trained. belong Pinyin phonemes in The set of all phonemes representing the character; The comprehensive phoneme score and the tone evaluation result of the evaluation tone corresponding to the evaluation pinyin are input into the nonlinear transformation layer of the scoring model to determine the character score. The character score is used to characterize the accuracy of the character's pronunciation. The scoring model is trained using the phoneme features, category features, and tone evaluation results of the pinyin phonemes corresponding to each character in multiple text samples labeled with comprehensive pronunciation scores. The scoring model includes at least a phoneme recognition layer and a character scoring layer in sequence. The attention model layer and the nonlinear transformation layer constitute the character scoring layer. The speech scoring unit is used to determine the overall pronunciation score of the speech based on the character score of each character in the evaluation text. The result output unit is used to output the evaluation results of the evaluation speech to the terminal in batches. The evaluation results include: the evaluation text, the character score of each character in the evaluation text, and the comprehensive pronunciation score.

6. The apparatus according to claim 5, characterized in that, The result output unit is specifically used to output an evaluation result interface to the terminal. The evaluation result page displays the evaluation text and the comprehensive pronunciation score, and the evaluation text indicates abnormal characters whose word scores are lower than a first threshold and the word scores of the abnormal characters.

7. The apparatus according to claim 6, characterized in that, The result output unit is further configured to display, in the evaluation result interface of the terminal, the evaluation pitch and reference pitch corresponding to the word whose pitch evaluation result is incorrect.

8. The apparatus according to claim 6, characterized in that, The device further includes: An anomaly details processing unit is used to display the evaluation pinyin of the target anomaly character on the evaluation result interface of the terminal after receiving a viewing instruction for the target anomaly character sent by the terminal, and to mark the pinyin phonemes whose phoneme scores are lower than a second threshold in the evaluation pinyin. The viewing instruction is generated by the terminal after detecting that the target anomaly character displayed in the evaluation result interface has been clicked. The target anomaly character belongs to the anomaly characters marked in the evaluation text.

9. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one instruction, at least one program, a code set, or an instruction set, the at least one instruction, the at least one program, the code set, or the instruction set being loaded and executed by the processor to implement the speech processing method as described in any one of claims 1-4.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the speech processing method as described in any one of claims 1-4.

11. A computer program product comprising computer instructions stored in a computer-readable storage medium, wherein a processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions to cause the computer device to perform the speech processing method as described in any one of claims 1-4.

Images