Spoken language evaluation method and device, equipment and storage medium

Abstract

The invention relates to the technical field of audio processing, and discloses a spoken language evaluation method, the method is applied to spoken language evaluation equipment integrated with a speech speed prediction model, and the method comprises the following steps: obtaining a frame level acoustic feature of a to-be-detected audio, and calculating a speech speed value of the to-be-detected audio according to the frame level acoustic feature; inputting the frame-level acoustic features of the to-be-tested audio and the speech speed value of the to-be-tested audio into a speech speed prediction model to obtain a speech speed adjustment value of the to-be-tested audio; adjusting the speech speed of the to-be-tested audio according to the speech speed adjustment value of the to-be-tested audio; and scoring based on the to-be-tested audio after the speech speed is adjusted. After the speech speed of the audio to be detected is adjusted, the speech speed is more consistent with the normal speech speed, so that the start frame and the end frame of each audio clip corresponding to each phoneme of the reference text in the audio to be detected can be more accurately detected in the alignment process. Therefore, the alignment effect of the to-be-tested audio and the reference text is improved, and the scoring accuracy is improved. The invention further discloses a spoken language evaluation device and equipment and a storage medium.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of audio processing, in particular to a spoken language evaluation method and device, equipment and storage medium. BACKGROUND

[0002] At present, the spoken language evaluation is performed by using the artificial evaluation method, which cannot avoid the individual bias of the evaluator, so that the uniformity of the scoring standard cannot be guaranteed, and sometimes the real level of the evaluated person cannot be accurately reflected. For large-scale spoken language test, a large amount of manpower, material resources and financial resources are required, which limits the regular and large-scale evaluation test. Therefore, it is necessary to realize the automatic evaluation of spoken language. The automatic evaluation of spoken language is to use a computer to automatically score the spoken language of a user.

[0003] In the related art, the automatic evaluation of spoken language usually includes an alignment process and a scoring process. In the alignment process, an acoustic model is used to determine, based on the to-be-evaluated audio and the reference text serving as the evaluation standard, which audio frames in the to-be-evaluated audio correspond to each phoneme in the reference text, that is, to determine the time boundary information (the start frame and the end frame of the audio segment) of each audio segment in the to-be-evaluated audio corresponding to each phoneme in the reference text. According to the correspondence between each phoneme in the reference text and each audio segment in the spoken language audio, the reference text and the spoken language audio are aligned to obtain an alignment result. In the scoring process, the pronunciation of each phoneme in the to-be-evaluated audio is scored based on the alignment result, so as to realize the automatic evaluation of the pronunciation of the spoken language in the to-be-evaluated audio. SUMMARY

[0004] The following presents a simplified summary of some aspects of the disclosed embodiments in order to provide a basic understanding of such embodiments. This summary is not an extensive overview of the disclosure, and is not intended to identify key / critical elements or to delineate the scope of these embodiments. Its sole purpose is to present some aspects of these embodiments in a simplified form as a prelude to the more detailed description that is presented later.

[0005] The embodiments of the present application provide a spoken language evaluation method and device, equipment and storage medium, which can improve the accuracy of scoring when performing automatic evaluation of spoken language.

[0006] In a first aspect, the embodiments of the present application provide a spoken language evaluation method, applied to a spoken language evaluation device, wherein the spoken language evaluation device is integrated with a speech rate preset model; the method comprises the following steps:

[0007] Acoustic features of each audio frame in the to-be-evaluated audio are obtained, and a speech rate value of the to-be-evaluated audio is calculated according to the acoustic features of each audio frame;

[0008] The acoustic features of each audio frame in the to-be-evaluated audio and the speech rate value of the to-be-evaluated audio are input into the speech rate prediction model, so that the speech rate prediction model outputs a speech rate adjustment value of the to-be-evaluated audio;

[0009] adjust the speech rate of the to-be-tested audio according to the speech rate adjustment value of the to-be-tested audio;

[0010] score the to-be-tested audio after the speech rate is adjusted.

[0011] Optionally, the spoken language evaluation device further integrates a clustering model; the speech rate value of the to-be-tested audio is calculated according to the acoustic features of each audio frame, including: inputting the acoustic features of each audio frame into the clustering model, so that the clustering model classifies the acoustic features of each audio frame to output a plurality of clustering clusters; and calculating the ratio of the number of audio frames in the to-be-tested audio to the number of clustering clusters to obtain the speech rate value of the to-be-tested audio.

[0012] Optionally, inputting the acoustic features of each audio frame into the clustering model so that the clustering model classifies the acoustic features of each audio frame to output a plurality of clustering clusters includes: inputting a target audio frame in each audio frame into the clustering model; wherein the target audio frame is any one of the audio frames in the to-be-tested audio; causing the clustering model to calculate the distance between the acoustic features of the target audio frame and the clustering centers of each clustering cluster to obtain a plurality of distance data; and causing the clustering model to filter out the minimum distance data from the plurality of distance data, and taking the clustering cluster corresponding to the minimum distance data as the clustering cluster to which the acoustic features of the target audio frame belong.

[0013] Optionally, adjusting the speech rate of the to-be-tested audio according to the speech rate adjustment value of the to-be-tested audio includes: determining the speech rate adjustment mode of the to-be-tested audio according to the speech rate adjustment value of the to-be-tested audio; in the case that the speech rate adjustment mode of the to-be-tested audio is to increase the speech rate, compressing the time length of the to-be-tested audio according to the speech rate adjustment value; and in the case that the speech rate adjustment mode of the to-be-tested audio is to decrease the speech rate, stretching the time length of the to-be-tested audio according to the speech rate adjustment value.

[0014] Optionally, the spoken language evaluation device further integrates an acoustic model; and scoring the to-be-tested audio after the speech rate is adjusted includes: inputting the to-be-tested audio after the speech rate is adjusted and the reference text into the acoustic model, so that the acoustic model performs forced alignment based on the to-be-tested audio after the speech rate is adjusted and the reference text to output an alignment sequence; and scoring the to-be-tested audio according to the alignment sequence.

[0015] Optionally, the spoken language evaluation device further integrates an audio feature model, which is trained by a self-supervised learning method; and the acoustic features of each audio frame in the to-be-tested audio are obtained by: receiving the spoken language audio from a user as the to-be-tested audio; framing the to-be-tested audio to obtain a plurality of audio frames; and inputting the plurality of audio frames into the audio feature model so that the audio feature model outputs the acoustic features of each audio frame.

[0016] Optionally, the training process of the speech rate prediction model comprises: inputting the audio of normal speech rate and the reference text into the acoustic model to make the acoustic model perform forced alignment based on the audio of normal speech rate and the reference text to obtain a standard alignment sequence; calculating a loss function value according to the standard alignment sequence and the alignment sequence; and updating parameters of the speech rate prediction model based on the loss function value.

[0017] In a second aspect, an embodiment of the present application provides a device for oral test, integrated in an oral test device, wherein the oral test device is integrated with a speech rate preset model; the device comprises:

[0018] The speech rate calculation module is configured to obtain acoustic features of each audio frame in the audio to be tested, and calculate a speech rate value of the audio to be tested according to the acoustic features of each audio frame.

[0019] The speech rate prediction module is configured to input the acoustic features of each audio frame in the audio to be tested and the speech rate value of the audio to be tested into the speech rate prediction model, so that the speech rate prediction model outputs a speech rate adjustment value of the audio to be tested.

[0020] The speech rate adjustment module is configured to adjust the speech rate of the audio to be tested according to the speech rate adjustment value of the audio to be tested.

[0021] The oral test scoring module is configured to score the audio to be tested after the speech rate is adjusted.

[0022] In a third aspect, an embodiment of the present application provides an oral test device, wherein the oral test device is integrated with an audio representation model, a clustering model, a speech rate prediction model and an acoustic model; the oral test device comprises:

[0023] The memory stores program instructions;

[0024] The processor is configured to execute the method for oral test according to the first aspect when the program instructions are executed.

[0025] In a fourth aspect, an embodiment of the present application provides a storage medium, which stores program instructions, wherein the program instructions are executed to perform the method for oral test according to the first aspect.

[0026] The method and device for oral test, the equipment and the storage medium provided by the embodiments of the present application can achieve the following technical effects:

[0027] In the implementation of the present application, when automatically evaluating the spoken language of a user by using a spoken language evaluation device, for the to-be-evaluated audio corresponding to the spoken language, an actual speech rate value of the to-be-evaluated audio is calculated according to the acoustic features of each audio frame in the to-be-evaluated audio. A speech rate prediction model integrated in the spoken language evaluation device can calculate a speech rate adjustment value of the to-be-evaluated audio based on the actual speech rate value of the to-be-evaluated audio and the acoustic features of each audio frame in the to-be-evaluated audio. After adjusting the speech rate of the to-be-evaluated audio with reference to the speech rate adjustment value of the to-be-evaluated audio, the speech rate of the to-be-evaluated audio can be more in line with the normal speech rate. In this way, when scoring based on the to-be-evaluated audio after adjusting the speech rate, that is, when performing the alignment process and the scoring process based on the to-be-evaluated audio after adjusting the speech rate, because the speech rate of the to-be-evaluated audio is adjusted to be more in line with the normal speech rate, in the alignment process, the time boundary information of each audio segment corresponding to each phoneme in the to-be-evaluated audio and the reference text can be more accurately detected, that is, the start frame and the end frame of each audio segment corresponding to each phoneme in the to-be-evaluated audio and the reference text can be more accurately detected. Thus, the alignment effect of the to-be-evaluated audio and the reference text is improved, so that the alignment result can more accurately reflect the actual pronunciation of each phoneme in the to-be-evaluated audio, and the situation that the alignment result cannot accurately represent the pronunciation of each phoneme due to inaccurate detection of the time boundary information in the alignment process is reduced. Further, when scoring based on the alignment result of the to-be-evaluated audio and the reference text, the accuracy of the scoring can be improved.

[0028] The foregoing general description and the following description are merely exemplary and explanatory and are not intended to limit the present application. BRIEF DESCRIPTION OF DRAWINGS

[0029] One or more embodiments are illustrated by way of example with reference to the accompanying drawings, which are schematic and not intended to be limiting of the embodiments, in which like reference numerals denote like elements in the figures, and in which:

[0030] Figure 1 is a schematic diagram of an application environment to which a spoken language evaluation method provided by an embodiment of the present application is applicable;

[0031] Figure 2 is a schematic diagram of another application environment to which a spoken language evaluation method provided by an embodiment of the present application is applicable;

[0032] Figure 3 is a flowchart of a spoken language evaluation method provided by an embodiment of the present application;

[0033] Figure 4 is a schematic diagram of a to-be-evaluated audio speech rate adjustment process and an alignment process provided by an embodiment of the present application;

[0034] Figure 5 is a schematic diagram of a speech rate prediction model training process provided by an embodiment of the present application;

[0035] Figure 6 is a schematic diagram of a spoken language evaluation device provided by an embodiment of the present application;

[0036] Figure 7 is a schematic diagram of a spoken language evaluation device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0037] In order to enable a more detailed understanding of the features and technical content of the embodiments of the present application, the implementation of the embodiments of the present application is described in detail below in conjunction with the drawings, which are only used for reference and do not limit the embodiments of the present application. In the following technical description, in order to facilitate explanation, a plurality of details are provided to provide a full understanding of the disclosed embodiments. However, one or more embodiments can still be implemented without these details. In other cases, well-known structures and devices can be simplified to facilitate the drawings.

[0038] The terms "first", "second", and the like in the specification and claims of the embodiments of the present application and the above drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the embodiments of the present application described herein can be implemented. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion.

[0039] Unless otherwise specified, the term "a plurality of" means two or more, and the term "a plurality of frames" means two or more audio frames.

[0040] In the embodiments of the present application, the character " / " represents an "or" relationship between the objects before and after it. For example, A / B means: A or B.

[0041] The term "and / or" is a description of the association between objects, which means that there can be three relationships. For example, A and / or B means: A or B, or, A and B, three relationships.

[0042] The term "corresponding" can refer to an association or binding relationship, A corresponding to B means that there is an association or binding relationship between A and B.

[0043] In the related art, automatic oral test generally includes an alignment process and a scoring process. In the alignment process, based on the audio to be tested and the reference text as the test standard, an acoustic model is used to determine which audio frames in the audio to be tested correspond to each phoneme in the reference text, i.e., to determine the time boundary information (start frame and end frame of the audio segment) of each audio segment in the audio to be tested corresponding to each phoneme in the reference text. According to the correspondence between each phoneme in the reference text and each audio segment in the oral audio, the reference text and the oral audio are aligned to obtain an alignment result. In the scoring process, based on the alignment result, the pronunciation of each phoneme in the audio to be tested is scored to realize automatic oral test of the pronunciation in the audio to be tested.

[0044] Specifically, in the alignment process based on the audio to be tested and the reference text, whether the start frame and the end frame of each audio segment corresponding to each phoneme in the reference text can be accurately detected from the audio to be tested is a key factor affecting whether the pronunciation features of each phoneme in the alignment result can accurately reflect the actual pronunciation. Accurate acquisition of the time boundary information of each audio segment can ensure the accuracy of the alignment result, and further ensure the accuracy of the subsequent scoring process. However, when a user performs automatic oral test, the user's speaking speed may be different from the normal speed, i.e., the user's oral audio speed is inconsistent with the normal speed. The normal speed is a moderate, smooth and natural speaking speed, which does not make the listener feel hurried or slow. Thus, in the alignment process of the oral test, due to the influence of the oral audio speed, the acoustic model cannot accurately detect the start frame and the end frame of each audio segment corresponding to each phoneme in the reference text in the oral audio. Thus, the alignment result is affected, and further the accuracy of the scoring is affected.

[0045] In view of this, the embodiments of the present application provide a scheme for oral test, which takes into account the situation that the speaking speed of the oral audio provided by the user is different from the normal speed, affecting the pronunciation features of each phoneme in the alignment result in the alignment process. In the scheme, before performing alignment based on the audio to be tested and the reference text, the speaking speed of the audio to be tested is adjusted to make the speaking speed of the audio to be tested consistent with the normal speed as much as possible. Thus, under the premise that the speaking speed of the audio to be tested is basically the same as the normal speed, performing alignment based on the audio to be tested and the reference text can improve the accuracy of the alignment result, so that the pronunciation features of each phoneme in the alignment result can accurately reflect the actual pronunciation. Thus, based on the alignment result, the accuracy of the scoring can be improved.

[0046] In combination with Figure 1 As shown in FIG. 1, the embodiments of the present application provide an application environment to which a method for oral test is applicable, which includes a user terminal 11 and an oral test device 12. A network connection is established between the user terminal 11 and the oral test device 12. In the application environment, the user terminal 11 and the oral test device 12 are connected through the network connection.

[0047] The user terminal 11 can collect the user's spoken language audio. For example, the user terminal 11 receives a user's spoken language evaluation instruction, executes the spoken language evaluation instruction to collect the user's spoken language to generate spoken language audio. Specifically, after the user opens the spoken language evaluation software on the user terminal 11, the user selects the corresponding evaluation text from various evaluation texts (such as Chinese evaluation text, English evaluation text, etc.) displayed by the user terminal 11 to read, at which time the user terminal 11 can collect the user's reading to generate spoken language audio.

[0048] After the user terminal 11 collects the user's spoken language audio data as the audio to be tested, the user terminal 11 sends the audio to be tested to the spoken language evaluation device 12 through a network connection. The spoken language evaluation device 12 automatically evaluates the audio to be tested based on the audio to be tested and the reference text as the evaluation standard, and obtains the spoken language evaluation result. The spoken language evaluation device 12 feeds back the spoken language evaluation result to the user terminal 11 through the network connection, so as to display the evaluation result to the user on the user terminal 11, thereby realizing automatic evaluation of the user's spoken language. In addition, in the embodiment of the present application, the form of the network connection between the user terminal 11 and the spoken language evaluation device 12 can be wired connection or wireless connection. The form of the network connection between the user terminal 11 and the spoken language evaluation device 12 is not specifically limited in the embodiment of the present application, and can be other connection forms in the prior art, as long as the transmission of the spoken language audio to be tested and the spoken language evaluation result can be realized.

[0049] In the embodiment of the present application, the user terminal 11 refers to an electronic device capable of collecting the user's spoken language audio, for example, the user terminal 11 can be a desktop device or a mobile terminal, specifically, a desktop computer, a tablet computer, a smart phone, etc. The spoken language evaluation device 12 refers to an electronic device or a server capable of performing the spoken language evaluation function according to the user's spoken language audio, for example, the spoken language evaluation device 12 can be a computer device, and can also be a stand-alone server, a cloud server, a server cluster or a virtual server.

[0050] Optionally, in the embodiments of the present application, the spoken language evaluation device 12 is integrated with an audio feature model, a clustering model, a speech rate prediction model and an acoustic model. The audio feature model can be a Wav2vec model. The speech rate prediction model can be a Transformer model. The acoustic model can be a TDNN (Time-Delay Neural Networks) model. Specifically, the audio feature model is used to extract acoustic features of each audio frame in the audio to be tested. The clustering model is used to perform a clustering algorithm (such as a K-means clustering algorithm) based on the acoustic features of each audio frame in the audio to be tested, and divide the acoustic features of each audio frame into each cluster. The speech rate prediction model is used to calculate the actual speech rate of the audio to be tested after the acoustic features of each audio frame are divided into each cluster, so that the spoken language evaluation device 12 adjusts the speech rate of the audio to be tested based on the actual speech rate of the audio to be tested. The acoustic model is used to generate an alignment result after alignment based on the audio to be tested after speech rate adjustment, so that the spoken language evaluation device scores based on the alignment result.

[0051] Optionally, in combination with Figure 2 As shown in the application environment shown in the application, the method for evaluating spoken language provided by the embodiments of the present application can not include the user terminal 11, and the functions performed by the user terminal 11 can be integrated into the spoken language evaluation device 12. The spoken language evaluation device 12 can perform the task of collecting the user's spoken language audio and the task of automatically scoring based on the spoken language audio.

[0052] Based on Figure 1 or Figure 2 As shown in the application environment shown in the application, the method for evaluating spoken language provided by the embodiments of the present application can not include the user terminal 11, and the functions performed by the user terminal 11 can be integrated into the spoken language evaluation device 12. The spoken language evaluation device 12 can perform the task of collecting the user's spoken language audio and the task of automatically scoring based on the spoken language audio. Figure 3 As shown in the application environment shown in the application, the method for evaluating spoken language provided by the embodiments of the present application can not include the user terminal 11, and the functions performed by the user terminal 11 can be integrated into the spoken language evaluation device 12. The spoken language evaluation device 12 can perform the task of collecting the user's spoken language audio and the task of automatically scoring based on the spoken language audio.

[0053] S31, acoustic features of each audio frame in the audio to be tested are obtained, and a speech rate value of the audio to be tested is calculated according to the acoustic features of each audio frame;

[0054] S32, the acoustic features of each audio frame in the audio to be tested and the speech rate value of the audio to be tested are input into the speech rate prediction model, so that the speech rate prediction model outputs a speech rate adjustment value of the audio to be tested;

[0055] S33, the speech rate of the audio to be tested is adjusted according to the speech rate adjustment value of the audio to be tested;

[0056] S34, the audio to be tested after adjusting the speech rate is scored.

[0057] In the method for oral test provided in the application, when automatically testing the oral of a user by using an oral test device, the actual speech rate value of the to-be-tested audio corresponding to the oral is calculated according to the acoustic features of each audio frame in the to-be-tested audio. The speech rate prediction model integrated in the oral test device can calculate the speech rate adjustment value of the to-be-tested audio based on the actual speech rate value of the to-be-tested audio and the acoustic features of each audio frame in the to-be-tested audio. After adjusting the speech rate of the to-be-tested audio with reference to the speech rate adjustment value of the to-be-tested audio, the speech rate of the to-be-tested audio can be more in line with the normal speech rate. In this way, when scoring based on the to-be-tested audio after adjusting the speech rate, that is, when performing the alignment process and the scoring process based on the to-be-tested audio after adjusting the speech rate, since the speech rate of the to-be-tested audio is adjusted to be more in line with the normal speech rate, in the alignment process, the time boundary information of each audio segment corresponding to each phoneme in the to-be-tested audio and the reference text can be more accurately detected, that is, the start frame and the end frame of each audio segment corresponding to each phoneme in the to-be-tested audio and the reference text can be more accurately detected. Thus, the alignment effect of the to-be-tested audio and the reference text is improved, so that the alignment result can more accurately reflect the actual pronunciation of each phoneme in the to-be-tested audio, and the situation that the alignment result cannot accurately represent the pronunciation of each phoneme due to inaccurate detection of the time boundary information in the alignment process is reduced. Further, when scoring based on the alignment result of the to-be-tested audio and the reference text, the accuracy of the scoring can be improved.

[0058] Optionally, in the step S31, the process of obtaining the acoustic features of each audio frame in the to-be-tested audio includes: receiving the oral audio from the user as the to-be-tested audio. The to-be-tested audio is framed to obtain a plurality of audio frames. The plurality of audio frames are input into the audio feature model so that the audio feature model outputs the acoustic features of each audio frame.

[0059] In this embodiment, after the to-be-tested audio is framed, the plurality of audio frames are input into the audio feature model by frame, so that the audio feature model outputs the acoustic features of each audio frame, that is, the frame-level acoustic features of the to-be-tested audio are obtained. In this way, the frame-level acoustic features can better represent the speech features of the to-be-tested audio, and the representation ability of the speech features corresponding to the to-be-tested audio is improved. Thus, the evaluation performance of the downstream evaluation task is improved.

[0060] In the embodiment of the application, the audio feature model is generated by training in a self-supervised learning manner. Specifically, in the training process of the audio feature model, the sample audio is pre-adjusted in speech rate to obtain audio with different speech rates as sample audio. The sample audio is input into the audio feature model for training to obtain the trained audio feature model. In this way, the audio feature model is generated by training in a self-supervised learning manner, without the need for artificially labeled sample data, thereby improving the generation efficiency of the model.

[0061] Optionally, in the step S31, the speech rate of the audio to be tested is calculated according to the acoustic features of each audio frame, including: inputting the acoustic features of each audio frame into a clustering model, and allowing the clustering model to classify the acoustic features of each audio frame to output a plurality of clustering clusters. The speech rate of the audio to be tested is calculated by dividing the number of audio frames in the audio to be tested by the number of clustering clusters.

[0062] In this embodiment, when calculating the speech rate of the audio to be tested, the acoustic features of each audio frame in the audio to be tested are input into the clustering model, and the clustering model classifies the acoustic features of each audio frame to output a plurality of clustering clusters. For each clustering cluster, the audio segments composed of the audio frames corresponding to the acoustic features in the same clustering cluster can represent the pronunciation of a phoneme. For example, assuming that the audio to be tested is a spoken audio of the word “you”, the spoken audio is divided into 5 audio frames, and the acoustic features of the 5 audio frames are obtained after feature extraction by frame. The acoustic features of the 5 audio frames are input into the clustering model, and the clustering model outputs two clustering clusters, wherein the first clustering cluster corresponds to the phoneme “n”, and the first clustering cluster includes the acoustic features of 3 audio frames; the second clustering cluster corresponds to the phoneme “i”, and the second clustering cluster includes the acoustic features of 2 audio frames. In the case where each clustering cluster corresponds to the acoustic features of each phoneme in the audio to be tested, the ratio of the number of audio frames in the audio to be tested to the number of clustering clusters can reflect how many audio frames are occupied by an average phoneme in the audio to be tested, and the speech rate of the audio to be tested is represented by the number of audio frames occupied by an average phoneme in the audio to be tested. Continuing with the previous example, the number of audio frames in the audio to be tested is 5, the number of clustering clusters corresponding to the audio to be tested is 2, and the calculated speech rate is 2.5, that is, an average phoneme in the audio to be tested occupies 2.5 audio frames.

[0063] In addition, in this embodiment, the number of clustering clusters in the clustering model can be determined by methods such as elbow method and silhouette coefficient method, and the embodiments of the present application do not make specific limitations thereto.

[0064] It should be noted that in the related art, when calculating the speech rate of an audio, the audio frame sequence of the audio needs to be text annotated to obtain a text sequence of the audio frame sequence. The text sequence is aligned with the audio frame sequence to obtain an alignment relationship between the text sequence and the audio frame sequence. According to the alignment relationship, the speech rate of the audio unit in the audio frame sequence is calculated. Specifically, the audio frame sequence can be divided into a plurality of audio units according to a preset rule. According to the alignment relationship between the text sequence and the audio frame sequence, the number of complete texts contained in each audio unit is calculated. The speech rate of each audio unit is calculated according to the number of complete texts contained in each audio unit and the number of audio frames corresponding to the number of complete texts. Each complete text corresponds to a complete phoneme, for example, “you” corresponds to “ni” as a complete phoneme.

[0065] Exemplarily, 1-20 frames of the audio frame sequence are divided into a target audio unit. Then, the number of audio frames contained in the target audio unit is 20, assuming that the target audio unit corresponds to the word "hello", the number of complete phonemes contained in the target audio unit is 2, and the labeling speech rate of the target audio unit is calculated by dividing the number of complete phonemes contained in each audio unit by the number of audio frames corresponding to the number of complete phonemes, which is 0.1.

[0066] However, the speech rate calculation manner provided in the embodiments of the present application directly performs clustering calculation on the acoustic features of each audio frame in the to-be-tested audio, and the correspondence between the audio frame and the phoneme in the to-be-tested audio can be obtained, and the number of audio frames or the audio duration occupied by each phoneme in the to-be-tested audio can be further calculated. Thus, the speech rate of the to-be-tested audio is calculated. The speech rate calculation manner provided in the embodiments of the present application does not need to perform text labeling on the audio frame sequence of the to-be-tested audio, realizes unsupervised clustering calculation, and improves the calculation efficiency of the speech rate of the to-be-tested audio.

[0067] Further, in the embodiments of the present application, the acoustic features of each audio frame are input into the clustering model, the clustering model classifies the acoustic features of each audio frame, and the process of outputting a plurality of clustering clusters includes: sequentially selecting a single audio from each audio frame as a target audio frame. The target audio frame is input into the clustering model. The clustering model calculates the distance between the acoustic features of the target audio frame and the clustering centers of each clustering cluster, to obtain a plurality of distance data. The clustering model selects the minimum distance data from the plurality of distance data, and takes the clustering cluster corresponding to the minimum distance data as the clustering cluster to which the acoustic features of the target audio frame belong.

[0068] Exemplarily, assuming that the acoustic features of the target audio frame are X t , the calculation formula of the clustering process is as follows:

[0069] i * = argmin i ||X t -C i || (1)

[0070] In formula (1), i * is the clustering cluster of the acoustic features of the target audio frame. C i is the clustering center of each clustering cluster. ||X t -C i || is the distance measurement function between the target audio features and the clustering center of each clustering cluster. The argmin function is used to select the minimum distance data from the plurality of distance data.

[0071] In the embodiment, after the clustering operation is performed on the acoustic features of each audio frame in the to-be-tested audio, a clustering index can be assigned to each clustering cluster. In this way, based on the clustering index of each clustering cluster, the acoustic features of the audio frames in each clustering cluster can be quickly searched, so as to quickly determine the clustering center, thereby improving the clustering efficiency of the clustering model.

[0072] Optionally, in combination with Figure 4 As shown in the above step S32, after the acoustic features of each audio frame in the to-be-tested audio and the speech rate value of the to-be-tested audio are input into the speech rate prediction model, the speech rate prediction model can learn the speech rate variation feature in the to-be-tested audio based on the normal speech rate or the expected speech rate in the user evaluation process, so as to output the speech rate adjustment value of the to-be-tested audio based on the speech rate variation feature.

[0073] Optionally, in combination with Figure 4 As shown in the above step S33, adjusting the speech rate of the to-be-tested audio according to the speech rate adjustment value of the to-be-tested audio includes: determining the speech rate adjustment mode of the to-be-tested audio according to the speech rate adjustment value of the to-be-tested audio. In the case where the speech rate adjustment mode of the to-be-tested audio is to increase the speech rate, the time length of the to-be-tested audio is compressed according to the speech rate adjustment value. In the case where the speech rate adjustment mode of the to-be-tested audio is to decrease the speech rate, the time length of the to-be-tested audio is stretched according to the speech rate adjustment value.

[0074] In the embodiment, the speech rate adjustment mode of the to-be-tested audio can be determined through the speech rate adjustment value of the to-be-tested audio. For example, if the speech rate adjustment value is positive, the speech rate of the to-be-tested audio needs to be increased. If the speech rate adjustment value is negative, the speech rate of the to-be-tested audio needs to be decreased. Accordingly, in the case where the speech rate of the to-be-tested audio needs to be increased, the time length of the to-be-tested audio is compressed according to the speech rate adjustment value, so as to increase the speech rate of the to-be-tested audio. In the case where the speech rate of the to-be-tested audio needs to be decreased, the time length of the to-be-tested audio is stretched according to the speech rate adjustment value, so as to decrease the speech rate of the to-be-tested audio. In this way, after the speech rate of the to-be-tested audio is adjusted according to the speech rate adjustment value, the speech rate of the to-be-tested audio can be more consistent with the normal speech rate. In the subsequent alignment process, the time boundary information of each audio segment corresponding to each phoneme in the to-be-tested audio and the reference text can be more accurately detected, that is, the start frame and the end frame of each audio segment corresponding to each phoneme in the to-be-tested audio and the reference text can be more accurately detected. Thus, the alignment effect of the to-be-tested audio and the reference text is improved, so that the alignment result can more accurately reflect the actual pronunciation of each phoneme in the to-be-tested audio, and the situation that the alignment result cannot accurately represent the pronunciation of each phoneme due to inaccurate detection of the time boundary information in the alignment process is reduced.

[0075] In addition, in the embodiment, when the speech rate of the to-be-tested audio is increased or decreased, a speech rate adjustment mode of "variable speed and constant tone" is adopted. For example, if the speech rate of the to-be-tested audio is increased, two adjacent pitch periods of the to-be-tested audio are combined into one pitch period. If the speech rate of the to-be-tested audio is decreased, two adjacent pitch periods of the to-be-tested audio are expanded into three pitch periods, so as to compress or stretch the to-be-tested audio in the time axis. In this way, during the speech rate adjustment of the to-be-tested audio, the pronunciation characteristics of the spoken language in the to-be-tested audio are not affected. Therefore, the influence on the subsequent alignment result and the scoring result is reduced, so as to ensure the accuracy of the alignment result and the scoring result.

[0076] Optionally, in combination with Figure 4 As shown in the figure, in the step S34, the process of scoring the to-be-tested audio based on the to-be-tested audio with the adjusted speech rate includes: inputting the to-be-tested audio with the adjusted speech rate and the reference text into an acoustic model, so that the acoustic model performs forced alignment based on the to-be-tested audio with the adjusted speech rate and the reference text, and outputs an alignment sequence. According to the alignment sequence, the to-be-tested audio is scored.

[0077] In the embodiment, the to-be-tested audio is scored based on the alignment sequence obtained by performing forced alignment on the to-be-tested audio and the reference text. For example, the forced alignment of the to-be-tested audio and the reference text can be performed in the following two ways: in the first way, the audio frame sequence of the to-be-tested audio is converted into a text sequence, and forced alignment is performed based on the text sequence of the to-be-tested audio and the reference text sequence, so as to obtain the alignment sequence. In the second way, the frame-level features of the to-be-tested audio are extracted, and the text features of the reference text are extracted, and forced alignment is performed based on the frame-level features of the to-be-tested audio and the text features of the reference text, so as to obtain the alignment sequence.

[0078] After obtaining the alignment sequence, the speech features of each phoneme in the to-be-tested audio can be extracted from the alignment sequence. The speech features of each phoneme are used to represent whether the pronunciation of each phoneme in the to-be-tested audio conforms to the standard, and the scoring result is obtained based on the speech features of each phoneme. In order to facilitate the user to show the scoring result of the to-be-tested audio, the speech features of each phoneme in the to-be-tested audio can be mapped into a score based on a preset mapping relationship. The score can adopt a percentage system, that is, the higher the score value, the more accurate the pronunciation of the phoneme.

[0079] With this implementation, when scoring based on the audio to be tested after adjusting the speech rate, i.e., when performing the alignment process and the scoring process based on the audio to be tested after adjusting the speech rate, because the speech rate of the audio to be tested is adjusted to be more in line with the normal speech rate, in the alignment process, the time boundary information of each audio segment corresponding to each phoneme in the audio to be tested and the reference text can be more accurately detected, i.e., the start frame and the end frame of each audio segment corresponding to each phoneme in the audio to be tested and the reference text can be more accurately detected. Thus, the alignment effect of the audio to be tested and the reference text is improved, so that the alignment result can more accurately reflect the actual pronunciation of each phoneme in the audio to be tested, and the situation that the alignment result cannot accurately represent the pronunciation of each phoneme due to inaccurate detection of the time boundary information in the alignment process is reduced. Further, when scoring based on the alignment result of the audio to be tested and the reference text, the accuracy of the scoring can be improved.

[0080] In combination Figure 5 As shown in FIG. 6, the embodiments of the present application provide a method for training a speech rate prediction model, which comprises the following steps:

[0081] inputting the audio with normal speech rate and the reference text into the acoustic model, so that the acoustic model performs forced alignment based on the audio with normal speech rate and the reference text to obtain a standard alignment sequence; calculating a loss function value according to the standard alignment sequence and the alignment sequence; and updating the parameters of the speech rate prediction model based on the loss function value. The alignment sequence is generated by the acoustic model when performing the oral test task indicated by the user.

[0082] In this implementation, the standard alignment sequence can be used as label data, and the cross entropy between the label data and the alignment sequence can be used as the loss function. The similarity between the alignment sequence and the label data can be evaluated by the loss function value, and in the case that the loss function value is large, the parameters of the speech rate prediction model can be adjusted based on the loss function value, so as to train the model. With this implementation for training the speech rate prediction model, the speech rate prediction model can be continuously adjusted during the application of the speech rate prediction model, so as to optimize the speech rate prediction model and improve the accuracy of the speech rate prediction model.

[0083] In combination Figure 6 As shown in FIG. 6, the embodiments of the present application provide a device for oral test, which is integrated into the oral test equipment in the above embodiments. The device comprises a speech rate calculation module 601, a speech rate prediction module 602, a speech rate adjustment module 603, an oral test scoring module 604, and a model training module 605. Wherein:

[0084] The speech rate calculation module 601 is configured to obtain the acoustic features of each audio frame in the audio to be tested, and calculate the speech rate value of the audio to be tested according to the acoustic features of each audio frame.

[0085] The speech rate prediction module 602 is configured to input the acoustic features of each audio frame in the to-be-tested audio and the speech rate value of the to-be-tested audio into a speech rate prediction model, so that the speech rate prediction model outputs a speech rate adjustment value of the to-be-tested audio.

[0086] The speech rate adjustment module 603 is configured to adjust the speech rate of the to-be-tested audio according to the speech rate adjustment value of the to-be-tested audio.

[0087] The oral test scoring module 604 is configured to score the to-be-tested audio after the speech rate is adjusted.

[0088] Optionally, the speech rate calculation module 601 is specifically configured to input the acoustic features of each audio frame into a clustering model, so that the clustering model classifies the acoustic features of each audio frame to output a plurality of clustering clusters. The ratio of the number of audio frames in the to-be-tested audio to the number of clustering clusters is calculated to obtain the speech rate value of the to-be-tested audio.

[0089] Optionally, the speech rate calculation module 601 is specifically configured to input a target audio frame in each audio frame into a clustering model; wherein the target audio frame is any one of the audio frames in the to-be-tested audio. The clustering model is caused to calculate the distance between the acoustic features of the target audio frame and the clustering centers of each clustering cluster to obtain a plurality of distance data. The clustering model is caused to select the minimum distance data from the plurality of distance data, and the clustering cluster corresponding to the minimum distance data is taken as the clustering cluster to which the acoustic features of the target audio frame belong.

[0090] Optionally, the speech rate adjustment module 603 is specifically configured to determine a speech rate adjustment mode of the to-be-tested audio according to the speech rate adjustment value of the to-be-tested audio. In the case that the speech rate adjustment mode of the to-be-tested audio is to increase the speech rate, the duration of the to-be-tested audio is compressed according to the speech rate adjustment value. In the case that the speech rate adjustment mode of the to-be-tested audio is to decrease the speech rate, the duration of the to-be-tested audio is stretched according to the speech rate adjustment value.

[0091] Optionally, the oral test scoring module 604 is specifically configured to input the to-be-tested audio after the speech rate is adjusted and a reference text into an acoustic model, so that the acoustic model performs forced alignment based on the to-be-tested audio after the speech rate is adjusted and the reference text to output an alignment sequence. The to-be-tested audio is scored according to the alignment sequence.

[0092] Optionally, the speech rate calculation module 601 is specifically configured to receive oral test audio from a user as the to-be-tested audio. The to-be-tested audio is framed to obtain a plurality of audio frames. The plurality of audio frames are input into an audio feature model, so that the audio feature model outputs acoustic features of each audio frame.

[0093] Optionally, the model training module 605 is specifically configured to: input the audio at the normal speech rate and the reference text into the acoustic model, so that the acoustic model performs forced alignment based on the audio at the normal speech rate and the reference text to obtain a standard alignment sequence. According to the standard alignment sequence and the alignment sequence, a loss function value is calculated. Based on the loss function value, parameters of the speech rate prediction model are updated.

[0094] The device for oral test provided in the embodiments of the present application can execute the method for oral test in the above embodiments, and has similar implementation principles and technical effects, which will not be described here.

[0095] In combination with Figure 7 As shown in the figure, the embodiments of the present application provide a device for oral test 12, which includes a processor 100 and a memory 101. Optionally, the device for oral test 12 can also include a communication interface 102 and a bus 103. The processor 100, the communication interface 102 and the memory 101 can complete mutual communication through the bus 103. The communication interface 102 can be used for information transmission. The processor 100 can call the logical instructions in the memory 101 to execute the method for oral test in the above embodiments.

[0096] In addition, the logical instructions in the memory 101 described above can be implemented in the form of a software function unit and sold or used as an independent product, which can be stored in a computer readable storage medium.

[0097] The memory 101 as a computer readable storage medium can be used to store software programs, computer executable programs, such as program instructions / modules corresponding to the method in the embodiments of the present application. The processor 100 executes the program instructions / modules stored in the memory 101, thereby executing function applications and data processing, that is, implementing the method for oral test in the above embodiments.

[0098] The memory 101 can include a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application required by a function; the data storage area can store data created according to the use of the terminal device, etc. In addition, the memory 101 can include a high-speed random access memory, and can also include a non-volatile memory.

[0099] The embodiments of the present application provide a storage medium, which stores computer executable instructions, and the computer executable instructions are configured to execute the method for oral test.

[0100] The computer storage medium described above can be a transitory computer readable storage medium, or a non-transitory computer readable storage medium.

[0101] The technical solutions of the embodiments of the present application can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes one or more instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method disclosed in the embodiments of the present application. The storage medium described above can be a non-transitory storage medium, including a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes, or can be a transitory storage medium.

[0102] The above description and drawings sufficiently illustrate the embodiments of the present disclosure to enable one skilled in the art to practice them. Other embodiments can include structural, logical, electrical, process, and other changes. The embodiments represent only a few of the possible variations. Individual components and functions are optional unless explicitly required, and the order of operations can be changed. Parts and features of some embodiments can be included in or replace parts and features of other embodiments. Also, the words used in this application are used only to describe the embodiments and not to limit the claims. As used in the description of the embodiments and the claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. Similarly, the term "and / or" as used in this application refers to any and all possible combinations of one or more associated listed items. In addition, when used in this application, the term "comprise" and its variants "comprises" and / or "comprising" and the like mean the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. Without more limitations, the element defined by the phrase "comprising a" does not exclude the presence of additional identical elements in the process, method, or device including the stated element. In this document, each embodiment focuses on the differences from other embodiments, and the same or similar parts between various embodiments can be referred to each other. For the method, product, etc. disclosed in the embodiments, if it corresponds to the method part disclosed in the embodiments, the relevant part can be referred to the description of the method part.

[0103] Those skilled in the art can understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods for each specific application to realize the described functions, but such implementation should not be considered beyond the scope of the embodiments of the present application. The skilled person can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described system, device and unit can refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

[0104] In the embodiments disclosed herein, the disclosed methods, products (including but not limited to devices, equipment, etc.) can be implemented in other ways. For example, the above-described device embodiments are only schematic, for example, the division of the units can only be a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms. The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to implement the embodiments. In addition, each functional unit in the embodiments of the present application can be integrated in one processing unit, or each unit can be a physically independent unit, or two or more units can be integrated in one unit.

[0105] The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other processing device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other processing device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

Claims

1. A method of spoken language assessment, characterized by, The method is applied to a speech evaluation device integrated with a speech rate preset model, and includes the following steps: Acquiring acoustic features of each audio frame in the to-be-tested audio, and calculating a speech rate value of the to-be-tested audio according to the acoustic features of each audio frame; Inputting the acoustic features of each audio frame in the to-be-tested audio and the speech rate value of the to-be-tested audio into a speech rate prediction model, so that the speech rate prediction model outputs a speech rate adjustment value of the to-be-tested audio; Adjusting the speech rate of the to-be-tested audio according to the speech rate adjustment value of the to-be-tested audio; Evaluating the to-be-tested audio after the speech rate is adjusted.

2. The method of claim 1, wherein, The speech evaluation device is also integrated with a clustering model; calculating the speech rate value of the to-be-tested audio according to the acoustic features of each audio frame includes: Inputting the acoustic features of each audio frame into a clustering model, so that the clustering model classifies the acoustic features of each audio frame to output a plurality of clustering clusters; Calculating the ratio of the number of audio frames to the number of clustering clusters in the to-be-tested audio to obtain the speech rate value of the to-be-tested audio.

3. The method of claim 2, wherein, Inputting the acoustic features of each audio frame into a clustering model, so that the clustering model classifies the acoustic features of each audio frame to output a plurality of clustering clusters, including: Inputting a target audio frame in the to-be-tested audio into the clustering model; wherein the target audio frame is any one of the audio frames in the to-be-tested audio; Making the clustering model calculate the distances between the acoustic features of the target audio frame and the clustering centers of each clustering cluster to obtain a plurality of distance data; Making the clustering model filter out the minimum distance data from the plurality of distance data, and taking the clustering cluster corresponding to the minimum distance data as the clustering cluster to which the acoustic features of the target audio frame belong.

4. The method of claim 1, wherein, According to the speech rate adjustment value of the to-be-tested audio, adjusting the speech rate of the to-be-tested audio includes: Determining the speech rate adjustment mode of the to-be-tested audio according to the speech rate adjustment value of the to-be-tested audio; In the case that the speech rate adjustment mode of the to-be-tested audio is to increase the speech rate, compressing the time length of the to-be-tested audio according to the speech rate adjustment value; In the case that the speech rate adjustment mode of the to-be-tested audio is to decrease the speech rate, stretching the time length of the to-be-tested audio according to the speech rate adjustment value.

5. The method of claim 1, wherein, The speech evaluation device is also integrated with an acoustic model; evaluating the to-be-tested audio after the speech rate is adjusted includes: Inputting the to-be-tested audio after the speech rate is adjusted and a reference text into the acoustic model, so that the acoustic model performs forced alignment based on the to-be-tested audio after the speech rate is adjusted and the reference text to output an alignment sequence; Evaluating the to-be-tested audio according to the alignment sequence.

6. The method of claim 1, wherein, The speech evaluation device is also integrated with an audio representation model, which is trained by a self-supervised learning method; acquiring the acoustic features of each audio frame in the to-be-tested audio includes: Receiving a spoken language audio from a user as the to-be-tested audio; Frameing the to-be-tested audio to obtain a plurality of audio frames; Inputting the plurality of audio frames into the audio representation model, so that the audio representation model outputs the acoustic features of each audio frame.

7. The method of claim 5, wherein, The training process of the speech rate prediction model includes: inputting the audio at the normal speech rate and the reference text into the acoustic model to force align the acoustic model based on the audio at the normal speech rate and the reference text to obtain a standard alignment sequence; calculating a loss function value according to the standard alignment sequence and the alignment sequence; updating parameters of the speech rate prediction model based on the loss function value.

8. An apparatus for oral speech evaluation, the apparatus comprising: The device is integrated into a spoken language evaluation device, and the spoken language evaluation device is integrated with a speech rate preset model. The device comprises: a speech rate calculation module configured to obtain acoustic features of each audio frame in a to-be-tested audio, and calculate a speech rate value of the to-be-tested audio according to the acoustic features of each audio frame; a speech rate prediction module configured to input the acoustic features of each audio frame in the to-be-tested audio and the speech rate value of the to-be-tested audio into the speech rate prediction model, so that the speech rate prediction model outputs a speech rate adjustment value of the to-be-tested audio; a speech rate adjustment module configured to adjust the speech rate of the to-be-tested audio according to the speech rate adjustment value of the to-be-tested audio; a spoken language scoring module configured to score the to-be-tested audio after the speech rate is adjusted.

9. A spoken language evaluation device, characterized by The spoken language evaluation device is integrated with an audio representation model, a clustering model, a speech rate prediction model and an acoustic model. The spoken language evaluation device comprises: a memory storing program instructions; a processor configured to execute the method for evaluating spoken language according to any one of claims 1 to 7 when running the program instructions.

10. A storage medium storing program instructions, characterized in that, The program instructions, when executed, perform the method for evaluating spoken language according to any one of claims 1 to 7.

Images