A word pronunciation scoring method and device, electronic equipment and storage medium

Abstract

The application provides a word pronunciation scoring method and device, electronic equipment and storage medium. The method comprises: inputting the audio features of each audio frame in the pronunciation audio into a phoneme detection model to obtain the pronunciation probability of each standard pronunciation phoneme of the word for each audio frame; constructing a two-dimensional table; taking the cell in the first row and the first column of the two-dimensional table as a starting point, moving directly from the latest starting point to the cell to the right of the latest starting point, taking the current cell as a starting point, and repeating the above steps until the movement cannot continue, taking the finally located cell as an ending point, and determining the movement path between the initial starting point and the ending point; and determining the preliminary pronunciation score based on the pronunciation probability written in each cell passed by the target movement path and the number of standard pronunciation phonemes. The application can score the pronunciation of a user for a word at the phoneme level, thereby correcting the mispronunciation of the user more accurately.

Description

Technical Field

[0001] This application relates to the field of natural language processing, and more specifically, to a word pronunciation scoring method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the continuous development and advancement of technology, it is now possible to score users' pronunciation of words and sentences to reflect the accuracy of their pronunciation, thereby enabling users to learn languages ​​better.

[0003] However, existing scoring methods are generally coarse and lack fine granularity. They can only reflect the accuracy of a user's pronunciation of sentences relatively accurately, but cannot accurately reflect the accuracy of a user's pronunciation of words. In other words, existing methods cannot accurately judge the quality of a user's pronunciation of words at the phoneme level. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a word pronunciation scoring method, device, electronic device and storage medium that can enable users to score word pronunciation at the phoneme level, thereby more accurately correcting users' mispronunciations.

[0005] In a first aspect, embodiments of this application provide a word pronunciation scoring method, the method comprising:

[0006] Receive the user's audio pronunciation of the target word;

[0007] The audio features of each audio frame in the pronunciation audio are input into the trained phoneme detection model for phoneme recognition, so as to obtain the pronunciation probability of each standard pronunciation phoneme of the target word for each audio frame.

[0008] Construct a target two-dimensional table, wherein the target two-dimensional table includes n×m cells, where n is the number of currently existing audio frames, m is the number of standard pronunciation phonemes, and the cell located in the i-th row and j-th column of the target two-dimensional table is written with the pronunciation probability of the j-th standard pronunciation phoneme of the target word for the i-th currently existing audio frame arranged in chronological order, 1≤i≤n, 1≤j≤m;

[0009] Starting from the cell in the first row and first column of the target two-dimensional table, move directly from the latest starting point to the cell immediately to the right of the latest starting point, or the cell immediately below the latest starting point, or the cell immediately to the lower right of the latest starting point, and take the current cell as the starting point. Repeat the steps of moving directly from the latest starting point to the cell immediately to the right of the latest starting point, or the cell immediately below the latest starting point, or the cell immediately to the lower right of the latest starting point, and take the current cell as the starting point, until it is impossible to move any further. Take the final cell as the ending point and determine the movement path between the initial starting point and the ending point.

[0010] Based on the pronunciation probability written in each cell traversed by the target movement path and the number of standard pronunciation phonemes, a preliminary pronunciation score is determined to characterize the accuracy of the user's pronunciation of the target word. The target movement path is the movement path with the highest probability sum among all movement paths. For each movement path, the probability sum is the sum of the pronunciation probabilities written in each cell traversed by the movement path. The higher the preliminary pronunciation score, the higher the accuracy of the user's pronunciation of the target word.

[0011] In one possible implementation, the phoneme detection model is a CTC model, which sequentially includes: a batch normalization layer, a zero-padding layer, a convolutional neural network layer, a max pooling layer, a batch normalization layer, a gated recurrent unit layer, a time-distributed dense layer, and a CTC output layer.

[0012] In one possible implementation, the audio features of each audio frame in the pronunciation audio are input into a trained phoneme detection model for phoneme recognition, to obtain the pronunciation probability of each audio frame for each standard pronunciation phoneme of the target word, including:

[0013] The audio features of each audio frame in the audio recording are input into the phoneme detection model for phoneme recognition, so as to obtain the pronunciation probability of each audio frame for each phoneme in the whole set of phonemes.

[0014] After determining a preliminary pronunciation score to characterize the accuracy of the user's pronunciation of the target word based on the pronunciation probability written in each cell traversed by the target movement path and the number of the standard pronunciation phonemes, the method further includes:

[0015] For each existing audio frame, the phoneme with the highest pronunciation probability among all phonemes in that audio frame is determined as the actual pronunciation phoneme of that audio frame.

[0016] Based on the actual phonemes of each existing audio frame arranged in chronological order and the standard phonemes of the target word, the user's pronunciation status of the target word is determined, wherein the pronunciation status includes: omission, mispronunciation, multiple pronunciations, and consistency.

[0017] The standard pronunciation phonemes of the target word, the preliminary pronunciation score, the pronunciation probability written in each grid traversed by the target movement path, and the pronunciation status are input into a trained word pronunciation scoring model to obtain the user's final pronunciation score for the target word. The word pronunciation scoring model is an XGBoost regression model.

[0018] In one possible implementation, after inputting the audio features of each audio frame in the pronunciation audio into a trained phoneme detection model for phoneme recognition to obtain the pronunciation probability of each audio frame for each standard pronunciation phoneme of the target word, the method further includes:

[0019] If at least two consecutive reference audio frames have the same actual phoneme, then remove every reference audio frame after the first reference audio frame.

[0020] For each audio frame, if the actual phoneme of the audio frame is an empty phoneme, then the audio frame is removed.

[0021] Secondly, embodiments of this application also provide a word pronunciation scoring device, the device comprising:

[0022] The receiving module is used to receive the user's pronunciation audio of the target word;

[0023] The input module is used to input the audio features of each audio frame in the pronunciation audio into the trained phoneme detection model for phoneme recognition, so as to obtain the pronunciation probability of each audio frame for each standard pronunciation phoneme of the target word.

[0024] A table construction module is used to construct a target two-dimensional table, wherein the target two-dimensional table includes n×m cells, where n is the number of currently existing audio frames and m is the number of standard pronunciation phonemes. The cell located in the i-th row and j-th column of the target two-dimensional table contains the pronunciation probability of the j-th standard pronunciation phoneme of the target word for the i-th currently existing audio frame arranged in chronological order, 1≤i≤n, 1≤j≤m;

[0025] The path determination module is used to start from the cell in the first row and first column of the target two-dimensional table, move directly from the latest starting point to the cell immediately to the right of the latest starting point, the cell immediately below the latest starting point, or the cell immediately to the lower right of the latest starting point, and take the current cell as the starting point. The steps are repeated to move directly from the latest starting point to the cell immediately to the right of the latest starting point, the cell immediately below the latest starting point, or the cell immediately to the lower right of the latest starting point, and take the current cell as the starting point, until no further movement is possible, so that the final cell is taken as the ending point, and the movement path between the initial starting point and the ending point is determined.

[0026] The preliminary scoring module is used to determine a preliminary pronunciation score, which characterizes the accuracy of the user's pronunciation of the target word, based on the pronunciation probability written in each cell traversed by the target movement path and the number of standard pronunciation phonemes. The target movement path is the movement path with the highest probability sum among all movement paths. For each movement path, the probability sum is the sum of the pronunciation probabilities written in each cell traversed by the movement path. The higher the preliminary pronunciation score, the higher the accuracy of the user's pronunciation of the target word.

[0027] In one possible implementation, the phoneme detection model is a CTC model, which sequentially includes: a batch normalization layer, a zero-padding layer, a convolutional neural network layer, a max pooling layer, a batch normalization layer, a gated recurrent unit layer, a time-distributed dense layer, and a CTC output layer.

[0028] In one possible implementation, the input module is specifically used for:

[0029] The audio features of each audio frame in the audio recording are input into the phoneme detection model for phoneme recognition, so as to obtain the pronunciation probability of each audio frame for each phoneme in the whole set of phonemes.

[0030] The device further includes:

[0031] The phoneme determination module is used to determine the initial pronunciation score, which characterizes the accuracy of the user's pronunciation of the target word, based on the pronunciation probability written in each grid passed by the target movement path and the number of standard pronunciation phonemes, for each existing audio frame, the phoneme with the highest pronunciation probability among all phonemes in the audio frame is determined as the actual pronunciation phoneme of the audio frame.

[0032] The pronunciation status determination module is used to determine the user's pronunciation status for the target word based on the actual pronunciation phonemes of each currently existing audio frame arranged in chronological order and the standard pronunciation phonemes of the target word. The pronunciation status includes: omission, mispronunciation, multiple pronunciations, and consistency.

[0033] The final scoring module is used to input the standard pronunciation phonemes of the target word, the preliminary pronunciation score, the pronunciation probability written in each grid passed by the target movement path, and the pronunciation status into the trained word pronunciation scoring model to obtain the user's final pronunciation score for the target word, wherein the word pronunciation scoring model is an XGBoost regression model.

[0034] In one possible implementation, the device further includes:

[0035] The first removal module is used to input the audio features of each audio frame in the pronunciation audio into the trained phoneme detection model for phoneme recognition, and obtain the pronunciation probability of each audio frame for each standard pronunciation phoneme of the target word. If at least two consecutive reference audio frames with the same actual pronunciation phoneme appear, then remove each reference audio frame after the first reference audio frame.

[0036] The second removal module is used to remove an audio frame if the actual phoneme of the audio frame is an empty phoneme.

[0037] Thirdly, embodiments of this application also provide an electronic device, including: a processor, a storage medium, and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of the word pronunciation scoring method described in any of the first aspects.

[0038] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the word pronunciation scoring method as described in any of the first aspects.

[0039] The embodiments of this application provide a word pronunciation scoring method, device, electronic device, and storage medium that can enable users to score word pronunciation at the phoneme level, thereby more accurately correcting users' mispronunciations. Attached Figure Description

[0040] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 A flowchart of a word pronunciation scoring method provided in an embodiment of this application is shown;

[0042] Figure 2 An example of a target two-dimensional table provided in this application is shown;

[0043] Figure 3 This application illustrates a movement path provided by an embodiment of the present application;

[0044] Figure 4 This application illustrates a target movement path provided by an embodiment of the present application;

[0045] Figure 5 This illustration shows a structural schematic diagram of a word pronunciation scoring device provided in an embodiment of this application;

[0046] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0048] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0049] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0050] To facilitate understanding of this embodiment, a detailed description of a word pronunciation scoring method, apparatus, electronic device, and storage medium provided in this application embodiment will be given.

[0051] Reference Figure 1 The diagram shown is a flowchart of a word pronunciation scoring method provided in an embodiment of this application. The method includes:

[0052] S101, Receive the user's audio pronunciation of the target word;

[0053] For example, the target word can be a word in any language, such as an English word.

[0054] S102. Input the audio features of each audio frame in the pronunciation audio into the trained phoneme detection model to perform phoneme recognition, and obtain the pronunciation probability of each audio frame for each standard pronunciation phoneme of the target word.

[0055] For example, the fbank algorithm can be used to perform feature transformation on the phonological audio (e.g., to obtain the audio features of each audio frame in the phonological audio).

[0056] For example, the duration of each audio frame can be 25ms, so that at the normal speaking speed, one audio frame corresponds to one phoneme (that is, each audio frame can completely produce one phoneme).

[0057] Let's take an English word as an example: For instance, if the target word is "architecture", the standard pronunciation phonemes of "architecture" include the following eight in sequence: aa, k, ih, t, eh, k, ch, ax.

[0058] S103. Construct a target two-dimensional table, wherein the target two-dimensional table includes n×m cells, n is the number of currently existing audio frames, m is the number of standard pronunciation phonemes, and the cell located in the i-th row and j-th column of the target two-dimensional table is written with the pronunciation probability of the j-th standard pronunciation phoneme of the target word for the i-th currently existing audio frame arranged in chronological order, 1≤i≤n, 1≤j≤m;

[0059] Reference Figure 2 As shown, this is a target two-dimensional table provided in an embodiment of this application, wherein the audio frame includes 7 elements, i.e., n is 7; the target word is architecture, i.e., m is 8.

[0060] For example, Figure 2 The 0.476 written in the first cell of the first row and first column means that the probability of the first standard phoneme aa of the architecture in the first audio frame (currently existing) is 0.476.

[0061] For example, Figure 2 The 0 written in the cell in the third row and second column means that the probability of the pronunciation of the second standard phoneme ih of architecture in the third audio frame (currently existing) is 0.

[0062] For example, Figure 2 The cell in the fifth row and eighth column shows 0.023, which means that the probability of the fifth audio frame pronouncing the eighth standard phoneme ax in the architecture is 0.023.

[0063] S104. Starting from the cell in the first row and first column of the target two-dimensional table, move directly from the latest starting point to the cell immediately to the right of the latest starting point, or the cell immediately below the latest starting point, or the cell immediately to the lower right of the latest starting point, and take the current cell as the starting point. Repeat the steps of moving directly from the latest starting point to the cell immediately to the right of the latest starting point, or the cell immediately below the latest starting point, or the cell immediately to the lower right of the latest starting point, and take the current cell as the starting point, until it is impossible to move any further, so that the final cell is taken as the endpoint, and determine the movement path between the initial starting point and the endpoint.

[0064] Reference Figure 3 As shown, this is a movement path provided in an embodiment of this application. Figure 2 Based on the provided two-dimensional table, Figure 3 The darker grid is Figure 3 The shown movement path traverses the grid cells, specifically... Figure 3The shown movement path passes through the following cells in sequence: the cell in the first row and first column (i.e., the initial starting point), the cell in the second row and second column, the cell in the second row and third column, the cell in the third row and fourth column, the cell in the fourth row and fifth column, the cell in the fifth row and sixth column, the cell in the sixth row and seventh column, and the cell in the seventh row and eighth column (i.e., the ending point).

[0065] S105. Based on the pronunciation probability written in each cell of the target movement path and the number of standard pronunciation phonemes, determine a preliminary pronunciation score to characterize the accuracy of the user's pronunciation of the target word. The target movement path is the movement path with the highest probability sum among all movement paths. For each movement path, the probability sum is the sum of the pronunciation probabilities written in each cell of the movement path. The higher the preliminary pronunciation score, the higher the accuracy of the user's pronunciation of the target word.

[0066] Reference Figure 4 As shown, this is a target movement path provided in an embodiment of this application. Figure 2 Based on the provided two-dimensional table, Figure 5 The darker grid is Figure 5 The grid cells traversed by the target's movement path are shown in the diagram. Figure 5 The cells that the target movement path passes through are, in order: the cell in the first row and first column (i.e., the initial starting point), the cell in the second row and second column, the cell in the third row and third column, the cell in the fourth row and fourth column, the cell in the fifth row and fifth column, the cell in the sixth row and sixth column, the cell in the seventh row and seventh column, and the cell in the seventh row and eighth column.

[0067] So, Figure 3 The sum of probabilities corresponding to the shown movement paths is: 0.476 + 0.947 + 0 + 0 + 0 + 0 + 0 = 1.423;

[0068] Figure 4 The sum of probabilities corresponding to the target movement path shown is: 0.476+0.947+0.983+0.638+0.268+0.918+0+0=4.23.

[0069] Based on probability and the principle of maximization, Figure 4 compared to Figure 3 To provide a more reasonable target movement path.

[0070] Specifically, a preliminary pronunciation score, representing the accuracy of a user's pronunciation of a target word, can be determined using the following methods:

[0071] For each column in the target 2D table, obtain the target cell in that column that is on the target's movement path;

[0072] If the column contains only one target cell, then the formula min(100, 120 + 15 × (log ... 10 o) Obtain the sub-score corresponding to the column, where o is the probability of the pronunciation written in the target cell in the column;

[0073] If the column contains multiple target cells, then use the formula min(100, 120 + 15 × (log ... 10 p) obtains the sub-score corresponding to the column, where p is the arithmetic mean of the pronunciation probabilities written in all target cells in the column;

[0074] Sum the sub-scores for each column to get the total score;

[0075] Divide the total score by the number of standard pronunciation phonemes of the target word to obtain a preliminary pronunciation score.

[0076] In one possible implementation, the phoneme detection model is a CTC model, which sequentially includes: a batch normalization layer, a zero-padding layer, a convolutional neural network layer, a max pooling layer, a batch normalization layer, a gated recurrent unit layer, a time-distributed dense layer, and a CTC output layer.

[0077] Preferably, the number of layers in the convolutional neural network is 4;

[0078] Preferably, the number of maximum pooling layers is 2;

[0079] In this application, a gated recurrent unit layer is used instead of the long short-term memory layer used in the prior art, thereby enabling the phoneme detection model to complete training (i.e. converge faster).

[0080] In one possible implementation, the audio features of each audio frame in the pronunciation audio are input into a trained phoneme detection model for phoneme recognition, to obtain the pronunciation probability of each audio frame for each standard pronunciation phoneme of the target word, including:

[0081] The audio features of each audio frame in the audio recording are input into the phoneme detection model for phoneme recognition, so as to obtain the pronunciation probability of each audio frame for each phoneme in the whole set of phonemes.

[0082] Assuming the target word is an English word, then the total number of phonemes is 49 (48+1, including one empty phoneme).

[0083] For a given audio frame, the sum of the pronunciation probabilities of each phoneme in the entire audio frame is 1.

[0084] After determining a preliminary pronunciation score to characterize the accuracy of the user's pronunciation of the target word based on the pronunciation probability written in each cell traversed by the target movement path and the number of the standard pronunciation phonemes, the method further includes:

[0085] For each existing audio frame, (by using the GreedySearch method), the phoneme with the highest pronunciation probability among all phonemes in that audio frame is determined as the actual pronunciation phoneme of that audio frame.

[0086] For example, if the probability of pronunciation of aa in this audio frame is 51% (greater than 50%, therefore aa must be the phoneme with the highest probability of pronunciation among all phonemes in this audio frame), then the actual phoneme pronounced in this audio frame is aa.

[0087] In one possible implementation, r (where r is a preset positive integer, for example, r can be 3) can also be determined based on the BeamSearch method, and the actual phoneme sequence with the highest sum of pronunciation probabilities in the first r groups can be retained, thereby obtaining a more accurate target phoneme than the locally optimal GreedySearch method.

[0088] Based on the actual phonemes of each existing audio frame arranged in chronological order and the standard phonemes of the target word, the user's pronunciation status of the target word is determined, wherein the pronunciation status includes: omission, mispronunciation, multiple pronunciations, and consistency.

[0089] When pronunciation conditions do not include consistency, pronunciation conditions include at least one of omission, mispronunciation, and extra pronunciation, for example, both omission and mispronunciation.

[0090] When pronunciation conditions include consonance, then pronunciation conditions only include consonance.

[0091] The standard pronunciation phonemes of the target word, the preliminary pronunciation score, the pronunciation probability written in each grid traversed by the target movement path, and the pronunciation status are input into a trained word pronunciation scoring model to obtain the user's final pronunciation score for the target word. The word pronunciation scoring model is an XGBoost regression model.

[0092] In one possible implementation, after inputting the audio features of each audio frame in the pronunciation audio into a trained phoneme detection model for phoneme recognition to obtain the pronunciation probability of each audio frame for each standard pronunciation phoneme of the target word, the method further includes:

[0093] If at least two consecutive reference audio frames have the same actual phoneme, then remove every reference audio frame after the first reference audio frame.

[0094] For example, if the actual phoneme of the first audio frame is aa, the actual phoneme of the second audio frame is aa, the actual phoneme of the third audio frame is also aa, and the actual phoneme of the fourth audio frame is ih, then the second and third audio frames are removed, and the original fourth audio frame is taken as the current second audio frame.

[0095] For each audio frame, if the actual phoneme of the audio frame is an empty phoneme, then the audio frame is removed.

[0096] For example, if the actual phoneme of the first audio frame is aa, the actual phoneme of the second audio frame is k, the actual phoneme of the third audio frame is the empty phoneme, and the actual phoneme of the fourth audio frame is ih, then the third audio frame is removed, and the original fourth audio frame is taken as the current third audio frame.

[0097] The word pronunciation scoring method provided in this application embodiment can enable users to score word pronunciation at the phoneme level, thereby more accurately correcting users' mispronunciations.

[0098] Reference Figure 5 The diagram shown is a structural schematic of a word pronunciation scoring device provided in an embodiment of this application. The device includes:

[0099] The receiving module 501 is used to receive the audio of the user's pronunciation of the target word;

[0100] The input module 502 is used to input the audio features of each audio frame in the pronunciation audio into the trained phoneme detection model for phoneme recognition, so as to obtain the pronunciation probability of each audio frame for each standard pronunciation phoneme of the target word.

[0101] The table construction module 503 is used to construct a target two-dimensional table, wherein the target two-dimensional table includes n×m cells, where n is the number of currently existing audio frames and m is the number of standard pronunciation phonemes. The cell located in the i-th row and j-th column of the target two-dimensional table is written with the pronunciation probability of the j-th standard pronunciation phoneme of the target word for the i-th currently existing audio frame arranged in chronological order, 1≤i≤n, 1≤j≤m;

[0102] The path determination module 504 is used to take the cell in the first row and first column of the target two-dimensional table as the starting point, move directly from the latest starting point to the cell immediately to the right of the latest starting point, or the cell immediately below the latest starting point, or the cell immediately to the lower right of the latest starting point, and take the current cell as the starting point, and repeat the steps of moving directly from the latest starting point to the cell immediately to the right of the latest starting point, or the cell immediately below the latest starting point, or the cell immediately to the lower right of the latest starting point, and take the current cell as the starting point, until it is impossible to move further, so as to take the final cell as the endpoint, and determine the movement path between the initial starting point and the endpoint;

[0103] The preliminary scoring module 505 is used to determine a preliminary pronunciation score that characterizes the accuracy of the user's pronunciation of the target word based on the pronunciation probability written in each grid passed by the target movement path and the number of standard pronunciation phonemes. The target movement path is the movement path with the highest probability sum among all movement paths. For each movement path, the probability sum corresponding to the movement path is the sum of the pronunciation probabilities written in each grid passed by the movement path. The higher the preliminary pronunciation score, the higher the accuracy of the user's pronunciation of the target word.

[0104] In one possible implementation, the phoneme detection model is a CTC model, which sequentially includes: a batch normalization layer, a zero-padding layer, a convolutional neural network layer, a max pooling layer, a batch normalization layer, a gated recurrent unit layer, a time-distributed dense layer, and a CTC output layer.

[0105] In one possible implementation, the input module 502 is specifically used for:

[0106] The audio features of each audio frame in the audio recording are input into the phoneme detection model for phoneme recognition, so as to obtain the pronunciation probability of each audio frame for each phoneme in the whole set of phonemes.

[0107] The device further includes:

[0108] The phoneme determination module is used to determine the phoneme with the highest pronunciation probability among all phonemes in each existing audio frame after the preliminary scoring module 505 determines the preliminary pronunciation score that characterizes the accuracy of the user's pronunciation of the target word based on the pronunciation probability written in each grid passed by the target movement path and the number of standard pronunciation phonemes.

[0109] The pronunciation status determination module is used to determine the user's pronunciation status for the target word based on the actual pronunciation phonemes of each currently existing audio frame arranged in chronological order and the standard pronunciation phonemes of the target word. The pronunciation status includes: omission, mispronunciation, multiple pronunciations, and consistency.

[0110] The final scoring module is used to input the standard pronunciation phonemes of the target word, the preliminary pronunciation score, the pronunciation probability written in each grid passed by the target movement path, and the pronunciation status into the trained word pronunciation scoring model to obtain the user's final pronunciation score for the target word, wherein the word pronunciation scoring model is an XGBoost regression model.

[0111] In one possible implementation, the device further includes:

[0112] The first removal module is used to input the audio features of each audio frame in the pronunciation audio into the trained phoneme detection model for phoneme recognition in the input module 502, and obtain the pronunciation probability of each audio frame for each standard pronunciation phoneme of the target word. If at least two consecutive reference audio frames with the same actual pronunciation phoneme appear, then remove each reference audio frame after the first reference audio frame.

[0113] The second removal module is used to remove an audio frame if the actual phoneme of the audio frame is an empty phoneme.

[0114] This application provides a word pronunciation scoring device that enables users to score word pronunciation at the phoneme level, thereby more accurately correcting users' mispronunciations.

[0115] Reference Figure 6 As shown in the embodiment of this application, an electronic device 600 includes a processor 601, a memory 602, and a bus. The memory 602 stores machine-readable instructions executable by the processor 601. When the electronic device is running, the processor 601 communicates with the memory 602 via the bus, and the processor 601 executes the machine-readable instructions to perform the steps of the word pronunciation scoring method described above.

[0116] Specifically, the memory 602 and processor 601 mentioned above can be general-purpose memory and processor, without any specific limitations. When the processor 601 runs the computer program stored in the memory 602, it can execute the above-mentioned word pronunciation scoring method.

[0117] Corresponding to the above-described word pronunciation scoring method, this application also provides a computer-readable storage medium storing a computer program, which, when run by a processor, executes the steps of the above-described word pronunciation scoring method.

[0118] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the method embodiments, and will not be repeated here. In the several embodiments provided in this application, it should be understood that the disclosed systems, methods, and procedures can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection may be through some communication interfaces; the indirect coupling or communication connection of devices or modules may be electrical, mechanical, or other forms.

[0119] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0120] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0121] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0122] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for scoring word pronunciation, characterized in that, The method includes: Receive the user's audio pronunciation of the target word; The audio features of each audio frame in the pronunciation audio are input into the trained phoneme detection model for phoneme recognition, so as to obtain the pronunciation probability of each standard pronunciation phoneme of the target word for each audio frame. Construct a target two-dimensional table, wherein the target two-dimensional table includes n×m cells, where n is the number of currently existing audio frames, m is the number of standard pronunciation phonemes, and the cell located in the i-th row and j-th column of the target two-dimensional table is written with the pronunciation probability of the j-th standard pronunciation phoneme of the target word for the i-th currently existing audio frame arranged in chronological order, 1≤i≤n, 1≤j≤m; Starting from the cell in the first row and first column of the target two-dimensional table, move directly from the latest starting point to the cell immediately to the right of the latest starting point, or the cell immediately below the latest starting point, or the cell immediately to the lower right of the latest starting point, and take the current cell as the starting point. Repeat the steps of moving directly from the latest starting point to the cell immediately to the right of the latest starting point, or the cell immediately below the latest starting point, or the cell immediately to the lower right of the latest starting point, and take the current cell as the starting point, until it is impossible to move any further. Take the final cell as the ending point and determine the movement path between the initial starting point and the ending point. Based on the pronunciation probability written in each cell traversed by the target movement path and the number of standard pronunciation phonemes, a preliminary pronunciation score is determined to characterize the accuracy of the user's pronunciation of the target word. The target movement path is the movement path with the highest probability sum among all movement paths. For each movement path, the probability sum is the sum of the pronunciation probabilities written in each cell traversed by the movement path. The higher the preliminary pronunciation score, the higher the accuracy of the user's pronunciation of the target word.

2. The word pronunciation scoring method according to claim 1, characterized in that, The phoneme detection model is a CTC model, which includes, in sequence: a batch normalization layer, a zero-padding layer, a convolutional neural network layer, a max pooling layer, a batch normalization layer, a gated recurrent unit layer, a time-distributed dense layer, and a CTC output layer.

3. The word pronunciation scoring method according to claim 1, characterized in that, The audio features of each audio frame in the pronunciation audio are input into a trained phoneme detection model for phoneme recognition, to obtain the pronunciation probability of each audio frame for each standard pronunciation phoneme of the target word, including: The audio features of each audio frame in the audio recording are input into the phoneme detection model for phoneme recognition, so as to obtain the pronunciation probability of each audio frame for each phoneme in the whole set of phonemes. After determining a preliminary pronunciation score to characterize the accuracy of the user's pronunciation of the target word based on the pronunciation probability written in each cell traversed by the target movement path and the number of the standard pronunciation phonemes, the method further includes: For each existing audio frame, the phoneme with the highest pronunciation probability among all phonemes in that audio frame is determined as the actual pronunciation phoneme of that audio frame. Based on the actual phonemes of each existing audio frame arranged in chronological order and the standard phonemes of the target word, the user's pronunciation status of the target word is determined, wherein the pronunciation status includes: omission, mispronunciation, multiple pronunciations, and consistency. The standard pronunciation phonemes of the target word, the preliminary pronunciation score, the pronunciation probability written in each grid traversed by the target movement path, and the pronunciation status are input into a trained word pronunciation scoring model to obtain the user's final pronunciation score for the target word. The word pronunciation scoring model is an XGBoost regression model.

4. The word pronunciation scoring method according to claim 3, characterized in that, After inputting the audio features of each audio frame in the pronunciation audio into a trained phoneme detection model for phoneme recognition, and obtaining the pronunciation probability of each audio frame for each standard pronunciation phoneme of the target word, the method further includes: For each audio frame, if the actual phoneme of the audio frame is an empty phoneme, then the audio frame is removed. If at least two consecutive reference audio frames have the same actual phoneme, then remove every reference audio frame after the first reference audio frame. For each audio frame, if the actual phoneme of the audio frame is an empty phoneme, then the audio frame is removed.

5. A word pronunciation scoring device, characterized in that, The device includes: The receiving module is used to receive the user's pronunciation audio of the target word; The input module is used to input the audio features of each audio frame in the pronunciation audio into the trained phoneme detection model for phoneme recognition, so as to obtain the pronunciation probability of each audio frame for each standard pronunciation phoneme of the target word. A table construction module is used to construct a target two-dimensional table, wherein the target two-dimensional table includes n×m cells, where n is the number of currently existing audio frames and m is the number of standard pronunciation phonemes. The cell located in the i-th row and j-th column of the target two-dimensional table contains the pronunciation probability of the j-th standard pronunciation phoneme of the target word for the i-th currently existing audio frame arranged in chronological order, 1≤i≤n, 1≤j≤m; The path determination module is used to start from the cell in the first row and first column of the target two-dimensional table, move directly from the latest starting point to the cell immediately to the right of the latest starting point, the cell immediately below the latest starting point, or the cell immediately to the lower right of the latest starting point, and take the current cell as the starting point. The steps are repeated to move directly from the latest starting point to the cell immediately to the right of the latest starting point, the cell immediately below the latest starting point, or the cell immediately to the lower right of the latest starting point, and take the current cell as the starting point, until no further movement is possible, so that the final cell is taken as the ending point, and the movement path between the initial starting point and the ending point is determined. The preliminary scoring module is used to determine a preliminary pronunciation score, which characterizes the accuracy of the user's pronunciation of the target word, based on the pronunciation probability written in each cell traversed by the target movement path and the number of standard pronunciation phonemes. The target movement path is the movement path with the highest probability sum among all movement paths. For each movement path, the probability sum is the sum of the pronunciation probabilities written in each cell traversed by the movement path. The higher the preliminary pronunciation score, the higher the accuracy of the user's pronunciation of the target word.

6. The word pronunciation scoring device according to claim 5, characterized in that, The phoneme detection model is a CTC model, which includes, in sequence: a batch normalization layer, a zero-padding layer, a convolutional neural network layer, a max pooling layer, a batch normalization layer, a gated recurrent unit layer, a time-distributed dense layer, and a CTC output layer.

7. The word pronunciation scoring device according to claim 5, characterized in that, The input module is specifically used for: The audio features of each audio frame in the audio recording are input into the phoneme detection model for phoneme recognition, so as to obtain the pronunciation probability of each audio frame for each phoneme in the whole set of phonemes. The device further includes: The phoneme determination module is used to determine the initial pronunciation score, which characterizes the accuracy of the user's pronunciation of the target word, based on the pronunciation probability written in each grid passed by the target movement path and the number of standard pronunciation phonemes, for each existing audio frame, the phoneme with the highest pronunciation probability among all phonemes in the audio frame is determined as the actual pronunciation phoneme of the audio frame. The pronunciation status determination module is used to determine the user's pronunciation status for the target word based on the actual pronunciation phonemes of each currently existing audio frame arranged in chronological order and the standard pronunciation phonemes of the target word. The pronunciation status includes: omission, mispronunciation, multiple pronunciations, and consistency. The final scoring module is used to input the standard pronunciation phonemes of the target word, the preliminary pronunciation score, the pronunciation probability written in each grid passed by the target movement path, and the pronunciation status into the trained word pronunciation scoring model to obtain the user's final pronunciation score for the target word, wherein the word pronunciation scoring model is an XGBoost regression model.

8. The word pronunciation scoring device according to claim 7, characterized in that, The device further includes: The first removal module is used to input the audio features of each audio frame in the pronunciation audio into the trained phoneme detection model for phoneme recognition, and obtain the pronunciation probability of each audio frame for each standard pronunciation phoneme of the target word. If at least two consecutive reference audio frames with the same actual pronunciation phoneme appear, then remove each reference audio frame after the first reference audio frame. The second removal module is used to remove an audio frame if the actual phoneme of the audio frame is an empty phoneme.

9. An electronic device, characterized in that, include: The device includes a processor, a storage medium, and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is in operation, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of the word pronunciation scoring method as described in any one of claims 1 to 4.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the word pronunciation scoring method as described in any one of claims 1 to 4.

Images