Audio evaluation methods, devices, readable media, and electronic devices

Abstract

This disclosure relates to an audio evaluation method, apparatus, readable medium, and electronic device. The method includes: acquiring audio data; extracting audio features of each audio frame from the audio data; performing clustering processing on the audio features of each audio frame to obtain a category for each audio frame; and determining an evaluation result of the audio data based on the category and audio features of each audio frame. This disclosure can improve the accuracy of audio evaluation.

Description

Technical Field

[0001] This disclosure relates to the field of audio processing technology, and more specifically, to an audio evaluation method, apparatus, readable medium, and electronic device. Background Technology

[0002] For language learners, besides standard pronunciation, fluency in spoken language also reflects their level of proficiency. Therefore, spoken fluency has become one of the important dimensions for evaluating a language learner's proficiency. Among related technologies, speech recognition technology is commonly used to determine spoken fluency; however, the accuracy of fluency determined by this method is not high. Summary of the Invention

[0003] This section is provided to briefly introduce the concepts, which will be described in detail in the Detailed Description section later. This section is not intended to identify key or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0004] In a first aspect, this disclosure provides an audio evaluation method, the method comprising:

[0005] Acquire audio data;

[0006] Extract the audio features of each audio frame from the audio data;

[0007] Clustering is performed on the audio features of each audio frame to obtain the category of each audio frame;

[0008] The evaluation result of the audio data is determined based on the category and audio characteristics of each audio frame.

[0009] Secondly, this disclosure provides an audio evaluation apparatus, the apparatus comprising:

[0010] The acquisition module is configured to acquire audio data.

[0011] The extraction module is configured to extract audio features from each audio frame in the audio data;

[0012] A clustering module is configured to perform clustering processing on the audio features of each audio frame to obtain the category of each audio frame;

[0013] The determining module is configured to determine the evaluation result of the audio data based on the category of each audio frame and the audio features of each audio frame.

[0014] Thirdly, this disclosure provides a computer-readable medium having a computer program stored thereon, which, when executed by a processing device, implements the steps of the method described in the first aspect.

[0015] Fourthly, this disclosure provides an electronic device, comprising:

[0016] A storage device having at least one computer program stored thereon;

[0017] At least one processing device is configured to execute the at least one computer program in the storage device to implement the steps of the method described in the first aspect.

[0018] The above technical solution clusters the audio features of each audio frame to obtain the category of each audio frame. This achieves frame-level clustering alignment on unsupervised features using unsupervised clustering technology, thereby aligning the category of each audio frame with the phonemes in the audio data. The alignment process does not require the use of text data, thus avoiding recognition or alignment errors that exist in speech recognition and improving the accuracy of the evaluation results of audio data.

[0019] Other features and advantages of this disclosure will be described in detail in the following detailed description section. Attached Figure Description

[0020] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale. In the drawings:

[0021] Figure 1 This is a flowchart illustrating an audio evaluation method according to an exemplary embodiment.

[0022] Figure 2 This is a flowchart illustrating the training process of a fluency recognition model according to an exemplary embodiment.

[0023] Figure 3 This is a block diagram illustrating an audio evaluation apparatus according to an exemplary embodiment.

[0024] Figure 4 This is a schematic diagram of the structure of an electronic device according to an exemplary embodiment. Detailed Implementation

[0025] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0026] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0027] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0028] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0029] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0030] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0031] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0032] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0033] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0034] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0035] Meanwhile, it is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0036] As mentioned in the background section, speech recognition technology is commonly used to determine spoken fluency. For example, spoken fluency can be determined using speech recognition technology through the following process: the learner's audio is recognized as text using a speech recognition model; the audio and corresponding text are aligned to obtain the time information of each phoneme in the text; the time information of each phoneme in the text is processed into features that can reflect spoken fluency, such as the average pronunciation duration of each phoneme in the text, the average pause time in the text, the number of pauses in a sentence, etc.; using sequence modeling methods, the phoneme time, phoneme information, and the acoustic features of the phonemes are time-series modeled to determine the spoken fluency of the audio.

[0037] However, this approach relies on speech recognition models. To achieve a high-performance speech recognition model, a large amount of text-annotated training data is required. Collecting this data is very difficult, especially for some less commonly used languages. At the same time, speech recognition models may have potential recognition or alignment errors, which can have a significant impact on subsequent fluency assessments, leading to low accuracy in determining spoken fluency.

[0038] Figure 1 This is a flowchart illustrating an audio evaluation method according to an exemplary embodiment. Figure 1 As shown, the method may include:

[0039] In step 11, audio data is acquired.

[0040] Audio data can be speech data generated by language learners through assessment language, which refers to the language whose spoken fluency is being assessed. The type of assessment language can be determined according to the specific circumstances; for example, the assessment language can be English or German, etc., and this disclosure does not impose any restrictions on it.

[0041] In a possible implementation, the audio data may be obtained from question-and-answer audio in an open scenario and / or audio for following along with text in a follow-along scenario.

[0042] For example, in an open context, two speakers can engage in a question-and-answer dialogue, with one speaker generating audio by asking a question and the other generating audio by answering. It's important to note that in an open context, the question-and-answer audio includes neither the question nor the answer audio with corresponding text content. In some embodiments, the language learner can be either of the two speakers; therefore, the audio data can be either the question audio or the answer audio from the question-and-answer audio.

[0043] In a shadowing scenario, language learners can generate a shadowing audio by reading aloud the content of a text. Accordingly, this shadowing audio can be used as audio data.

[0044] In step 12, the audio features of each audio frame in the audio data are extracted.

[0045] The audio features of each audio frame can be frame-level audio features. In a possible implementation, the audio features of each audio frame in the audio data can be extracted using a machine learning model. In some embodiments, extracting the audio features of each audio frame in the audio data includes: processing the audio data according to a convolutional neural network to obtain the encoded features of each audio frame; and processing the encoded features of each audio frame according to a context encoder to obtain the audio features of each audio frame.

[0046] Convolutional neural networks (CNNs) can obtain the encoded features of multiple audio segments of preset duration in audio data. The preset duration can be the duration of an audio frame, such as 25ms. The encoded features can be represented by feature vectors. For example, a CNN can encode a 25ms audio segment into a feature vector every 20ms from audio data with a sampling rate of 16kHz, thereby obtaining the encoded features of each audio frame.

[0047] A context encoder can consider information from other audio segments in the entire audio data besides the currently encoded audio segment. Therefore, the audio features of each audio frame output by the context encoder can consider information from other audio frames in the audio data besides the current audio frame. In other words, the audio features of each audio frame can consider the context information in the audio data.

[0048] In some embodiments, the convolutional neural network and the context encoder may belong to the same model, which can be an unsupervised learning model. For example, the convolutional neural network and the context encoder may be included in the wav2vec model. Accordingly, extracting audio features from each audio frame in the audio data may include: processing the audio data according to the wav2vec model to obtain the audio features of each audio frame. The wav2vec model is an unsupervised pre-trained speech model, which is learned and trained on a large number of unlabeled speech datasets, that is, it is pre-trained using large-scale unsupervised data. The unsupervised features output by the wav2vec model have the ability to distinguish phonemes; therefore, the extracted audio features can distinguish phonemes.

[0049] In possible implementations, audio features can also be F-bank (Fiterbank) features, MFCC (Mel Frequency Cepstrum Coefficient) features, or deep features. For methods of extracting F-bank and MFCC features, please refer to related technologies. Regarding deep features, they can be obtained by processing audio data using a pre-trained speech recognition model. For example, the audio features of each audio frame can be obtained from the output data of the penultimate or penultimate network layer of the speech recognition model. The speech recognition model can be determined according to actual needs; for example, the speech recognition model can be a Gaussian mixture model-Hidden Markov Model or a deep neural network-Hidden Markov Model, etc., and this disclosure does not impose any limitations on this.

[0050] In step 13, the audio features of each audio frame are clustered to obtain the category of each audio frame.

[0051] In a possible implementation, clustering the audio features of each audio frame to obtain the category of each audio frame can include: clustering the audio features of each audio frame according to a clustering model to obtain the category of each audio frame. The clustering model can be specifically determined according to actual needs; for example, the clustering model can include a K-means clustering model or a hierarchical clustering model, etc.

[0052] Clustering the audio features of each audio frame can classify multiple audio frames, allowing audio frames belonging to the same category to be grouped into a single class and thus obtaining the category of each audio frame.

[0053] In some embodiments, categories can be used to reflect the mapping relationship between audio frames and phonemes in audio data. Categories can be represented by category identifiers, for example, identifier 1 represents category 1, identifier 2 represents category 2, and so on. The total number of categories can be specifically set according to actual needs, for example, the total number of categories can be 30 or 40, etc.

[0054] For example, taking the audio data corresponding to "Good" as an example, "Good" can include three phonemes: [G, OW, D]. Assuming the audio data includes 10 audio frames, the sequence of audio frames is [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]. If the first audio frame corresponds to the silence phoneme SIL, the second to fourth frames correspond to the phoneme G, the fifth and sixth frames correspond to the phoneme OW, the seventh and eighth frames correspond to the phoneme D, and the ninth and tenth frames correspond to the silence phoneme SIL, then by clustering the 10 audio frames, the first, second to fourth, fifth to sixth, seventh to eighth, and ninth to tenth frames can each be assigned the same category label. It is worth noting that silence phonemes do not actually produce sound; they only represent silence, pauses, or meaningless sounds, and silence phonemes are also included in the audio data. Therefore, it can be seen that the category of each audio frame obtained by clustering can be aligned with the phonemes in the audio data. Thus, the category can reflect the mapping relationship between audio frames and phonemes.

[0055] This disclosure achieves frame-level clustering alignment on unsupervised features by clustering the audio features of each audio frame, thereby aligning the category of each audio frame with the phonemes in the audio data. The alignment process does not require the use of text data, thus avoiding recognition or alignment errors that exist in speech recognition and improving the accuracy of audio data evaluation results, such as improving the accuracy of evaluating the spoken fluency of audio data.

[0056] In step 14, the evaluation result of the audio data is determined based on the category and audio characteristics of each audio frame.

[0057] In a possible implementation, the evaluation results include spoken fluency, which reflects the speaker's fluency level corresponding to the audio data. The evaluation result is determined based on the category and audio features of each audio frame, including processing the category and audio features of each audio frame using a trained fluency recognition model to determine the spoken fluency of the audio data. The fluency recognition model can be specifically determined according to the actual situation; for example, the fluency recognition model can be an LSTM (Long Short-Term Memory) model, which can output sentence-level spoken fluency.

[0058] The trained fluency recognition model can be a pre-trained machine learning model. For details regarding the training process of the fluency recognition model, please refer to the following... Figure 2 The details and related descriptions will not be repeated here.

[0059] In determining the evaluation results of audio data, such as the spoken fluency evaluation results of audio data, this disclosure does not require the audio to be recognized as text. Since the fluency evaluation in open scenarios requires the speech recognition model to recognize the audio as text, this disclosure has a good fluency recognition effect for audio data in open scenarios, and the fluency recognition performance is not limited by the training data size of the speech recognition model.

[0060] Figure 2 This is a flowchart illustrating the training process of a fluency recognition model according to an exemplary embodiment. Figure 2 As shown, the training process may include:

[0061] In step 21, multiple sample audio files are obtained, and the sample audio files are labeled with spoken fluency tags.

[0062] In a possible implementation, multiple sample audio samples are obtained from question-and-answer audio in an open scenario and / or follow-up audio for the text in a follow-up scenario. Specific details regarding question-and-answer audio and follow-up audio can be found in the relevant description of step 11 above, and will not be repeated here.

[0063] In some embodiments, question-and-answer audio in open scenarios and follow-up audio in follow-up scenarios can be randomly collected from teaching resources, thereby obtaining rich sample audio, allowing the fluency recognition model to be fully trained and improving its recognition performance. By using question-and-answer audio in open scenarios and follow-up audio in follow-up scenarios to train the fluency recognition model, the trained fluency recognition model can achieve excellent fluency recognition performance on audio data obtained in open scenarios or follow-up scenarios.

[0064] In some embodiments, spoken fluency labels may be manually labeled, for example, based on spoken fluency scores given by language experts. In possible implementations, multiple sample audio files may be acquired via storage devices or by calling relevant interfaces.

[0065] In step 22, the sample audio features of each sample audio frame in each sample audio are extracted.

[0066] In step 23, for each audio sample, the audio features of each audio frame of that audio sample are clustered to obtain the category of each audio frame in each audio sample.

[0067] The specific details of steps 22 and 23 are similar to those of steps 12 and 13, and can be found in the relevant descriptions of steps 12 and 13 above, which will not be repeated here.

[0068] In step 24, for each audio sample, the initial fluency recognition model processes the category and audio features of each audio frame in the audio sample to determine the predicted spoken fluency of each audio sample.

[0069] In step 25, a loss function is determined based on the spoken fluency label and predicted spoken fluency of each audio sample, and the initial fluency recognition model is trained based on the loss function to obtain a trained fluency recognition model.

[0070] In a possible implementation, the loss function can be constructed based on the difference between the spoken fluency label and the predicted spoken fluency; for example, the loss function could be a cross-entropy loss function.

[0071] For example, the parameters of the initial fluency recognition model can be updated based on the loss value of the loss function until the loss value meets a preset condition. For example, the loss value converges, or the loss value is less than a preset value. When the loss value meets the preset condition, the fluency recognition model training is complete, and the trained initial fluency recognition model is obtained.

[0072] Figure 3 This is a block diagram illustrating an audio evaluation apparatus according to an exemplary embodiment. Figure 3 As shown, the audio evaluation device 300 may include:

[0073] Module 310 is configured to acquire audio data;

[0074] Extraction module 320 is configured to extract audio features from each audio frame in the audio data;

[0075] Clustering module 330 is configured to perform clustering processing on the audio features of each audio frame to obtain the category of each audio frame;

[0076] The determining module 340 is configured to determine the evaluation result of the audio data based on the category of each audio frame and the audio characteristics of each audio frame.

[0077] Optionally, the category is used to reflect the mapping relationship between the audio frame and the phonemes in the audio data.

[0078] Optionally, the clustering module 330 is further configured to:

[0079] The audio features of each audio frame are clustered according to the clustering model to obtain the category of each audio frame.

[0080] Optionally, the extraction module 320 is further configured to:

[0081] The audio data is processed using a convolutional neural network to obtain the encoded features of each audio frame;

[0082] The encoded features of each audio frame are processed according to the context encoder to obtain the audio features of each audio frame.

[0083] Optionally, the evaluation result includes spoken fluency, which reflects the speaker's fluency level corresponding to the audio data, and the determining module 340 is further configured to:

[0084] The spoken fluency of the audio data is determined by processing the category and audio features of each audio frame according to the trained fluency recognition model.

[0085] Optionally, the fluency recognition model is trained in the following manner:

[0086] Multiple sample audio files were acquired, and the sample audio files were labeled with spoken fluency tags;

[0087] Extract the sample audio features of each sample audio frame in each of the sample audio samples;

[0088] For each of the sample audios, the audio features of each of the sample audio frames are clustered to obtain the category of each of the sample audio frames in each of the sample audios;

[0089] For each of the sample audios, the category of each sample audio frame and the audio features of each sample audio frame in the sample audio are processed by the initial fluency recognition model to determine the predicted spoken fluency of each sample audio.

[0090] Based on the spoken fluency label and the predicted spoken fluency of each sample audio, a loss function is determined, and the initial fluency recognition model is trained according to the loss function to obtain a trained fluency recognition model.

[0091] Optionally, the sample audios are obtained from question-and-answer audio in an open scenario and / or audio for following along with text in a follow-along scenario.

[0092] The following is for reference. Figure 4This diagram illustrates a structural schematic of an electronic device 400 suitable for implementing embodiments of the present disclosure. The terminal devices in these embodiments may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 4 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0093] like Figure 4 As shown, electronic device 400 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 401, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 402 or a program loaded from storage device 408 into random access memory (RAM) 403. RAM 403 also stores various programs and data required for the operation of electronic device 400. Processing device 401, ROM 402, and RAM 403 are interconnected via bus 404. Input / output (I / O) interface 405 is also connected to bus 404.

[0094] Typically, the following devices can be connected to I / O interface 405: input devices 406 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 407 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 408 including, for example, magnetic tapes, hard disks, etc.; and communication devices 409. Communication device 409 allows electronic device 400 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 4 An electronic device 400 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0095] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 409, or installed from storage device 408, or installed from ROM 402. When the computer program is executed by processing device 401, it performs the functions defined in the methods of embodiments of this disclosure.

[0096] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0097] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0098] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0099] The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: acquire audio data; extract audio features of each audio frame in the audio data; perform clustering processing on the audio features of each audio frame to obtain the category of each audio frame; and determine the evaluation result of the audio data based on the category and audio features of each audio frame.

[0100] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0101] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0102] The modules described in the embodiments of this disclosure can be implemented in software or hardware. The names of the modules are not, in some cases, intended to limit the functionality of the module itself.

[0103] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0104] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0105] According to one or more embodiments of this disclosure, Example 1 provides an audio evaluation method, the method comprising:

[0106] Acquire audio data;

[0107] Extract the audio features of each audio frame from the audio data;

[0108] Clustering is performed on the audio features of each audio frame to obtain the category of each audio frame;

[0109] The evaluation result of the audio data is determined based on the category and audio characteristics of each audio frame.

[0110] According to one or more embodiments of this disclosure, Example 2 provides the audio evaluation method of Example 1, wherein the category is used to reflect the mapping relationship between the audio frame and the phonemes in the audio data.

[0111] According to one or more embodiments of this disclosure, Example 3 provides an audio evaluation method of Example 1 or 2, wherein clustering the audio features of each audio frame to obtain the category of each audio frame includes:

[0112] The audio features of each audio frame are clustered according to the clustering model to obtain the category of each audio frame.

[0113] According to one or more embodiments of this disclosure, Example 4 provides an audio evaluation method of Example 1, wherein extracting audio features from each audio frame of the audio data includes:

[0114] The audio data is processed using a convolutional neural network to obtain the encoded features of each audio frame;

[0115] The encoded features of each audio frame are processed according to the context encoder to obtain the audio features of each audio frame.

[0116] According to one or more embodiments of this disclosure, Example 5 provides an audio evaluation method of Example 1, wherein the evaluation result includes spoken fluency, the spoken fluency being used to reflect the degree of spoken fluency of the speaker corresponding to the audio data, and determining the evaluation result of the audio data based on the category of each audio frame and the audio features of each audio frame includes:

[0117] The spoken fluency of the audio data is determined by processing the category and audio features of each audio frame according to the trained fluency recognition model.

[0118] According to one or more embodiments of this disclosure, Example 6 provides the audio evaluation method of Example 5, wherein the fluency recognition model is trained based on the following:

[0119] Multiple sample audio files were acquired, and the sample audio files were labeled with spoken fluency tags;

[0120] Extract the sample audio features of each sample audio frame in each of the sample audio samples;

[0121] For each of the sample audios, the audio features of each of the sample audio frames are clustered to obtain the category of each of the sample audio frames in each of the sample audios;

[0122] For each of the sample audios, the category of each sample audio frame and the audio features of each sample audio frame in the sample audio are processed by the initial fluency recognition model to determine the predicted spoken fluency of each sample audio.

[0123] Based on the spoken fluency label and the predicted spoken fluency of each sample audio, a loss function is determined, and the initial fluency recognition model is trained according to the loss function to obtain a trained fluency recognition model.

[0124] According to one or more embodiments of this disclosure, Example 7 provides the audio evaluation method of Example 6, wherein the plurality of sample audios are obtained from question-and-answer audio in an open scenario and / or follow-up audio for follow-up text in a follow-up scenario.

[0125] According to one or more embodiments of this disclosure, Example 8 provides an audio evaluation apparatus, the apparatus comprising:

[0126] The acquisition module is configured to acquire audio data.

[0127] The extraction module is configured to extract audio features from each audio frame in the audio data;

[0128] A clustering module is configured to perform clustering processing on the audio features of each audio frame to obtain the category of each audio frame;

[0129] The determining module is configured to determine the evaluation result of the audio data based on the category of each audio frame and the audio features of each audio frame.

[0130] According to one or more embodiments of this disclosure, Example 9 provides the audio evaluation apparatus of Example 8, wherein the category is used to reflect the mapping relationship between the audio frame and the phonemes in the audio data.

[0131] According to one or more embodiments of this disclosure, Example 10 provides an audio evaluation apparatus of Example 8 or 9, wherein the clustering module is further configured to:

[0132] The audio features of each audio frame are clustered according to the clustering model to obtain the category of each audio frame.

[0133] According to one or more embodiments of this disclosure, Example 11 provides the audio evaluation apparatus of Example 8, wherein the extraction module is further configured to:

[0134] The audio data is processed using a convolutional neural network to obtain the encoded features of each audio frame;

[0135] The encoded features of each audio frame are processed according to the context encoder to obtain the audio features of each audio frame.

[0136] According to one or more embodiments of this disclosure, Example 12 provides the audio evaluation apparatus of Example 8, wherein the evaluation result includes spoken fluency, the spoken fluency being used to reflect the spoken fluency of the speaker corresponding to the audio data, and the determining module is further configured to:

[0137] The spoken fluency of the audio data is determined by processing the category and audio features of each audio frame according to the trained fluency recognition model.

[0138] According to one or more embodiments of this disclosure, Example 13 provides the audio evaluation apparatus of Example 12, wherein the fluency recognition model is trained based on the following:

[0139] Multiple sample audio files were acquired, and the sample audio files were labeled with spoken fluency tags;

[0140] Extract the sample audio features of each sample audio frame in each of the sample audio samples;

[0141] For each of the sample audios, the audio features of each of the sample audio frames are clustered to obtain the category of each of the sample audio frames in each of the sample audios;

[0142] For each of the sample audios, the category of each sample audio frame and the audio features of each sample audio frame in the sample audio are processed by the initial fluency recognition model to determine the predicted spoken fluency of each sample audio.

[0143] Based on the spoken fluency label and the predicted spoken fluency of each sample audio, a loss function is determined, and the initial fluency recognition model is trained according to the loss function to obtain a trained fluency recognition model.

[0144] According to one or more embodiments of this disclosure, Example 14 provides the audio evaluation apparatus of Example 13, wherein the plurality of sample audios are obtained from question-and-answer audio in an open scenario and / or follow-up audio for follow-up text in a follow-up scenario.

[0145] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0146] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0147] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative forms of implementing the claims. Regarding the apparatus in the above embodiments, the specific manner in which the various modules perform their operations has been described in detail in the embodiments relating to the method, and will not be elaborated upon here.

Claims

1. An audio evaluation method, characterized in that, The method includes: Acquire audio data; The audio data is processed using a convolutional neural network to obtain the encoded features of each audio frame; The encoded features of each audio frame are processed according to the context encoder to obtain the audio features of each audio frame; Clustering is performed on the audio features of each audio frame to obtain the category of each audio frame; The audio frame category and audio features of each audio frame are processed according to the trained fluency recognition model to determine the evaluation result of the audio data; the evaluation result includes spoken fluency, which reflects the degree of spoken fluency of the speaker corresponding to the audio data.

2. The audio evaluation method according to claim 1, characterized in that, The category is used to reflect the mapping relationship between the audio frame and the phonemes in the audio data.

3. The audio evaluation method according to claim 1 or 2, characterized in that, The process of clustering the audio features of each audio frame to obtain the category of each audio frame includes: The audio features of each audio frame are clustered according to the clustering model to obtain the category of each audio frame.

4. The audio evaluation method according to claim 1, characterized in that, The fluency recognition model was trained using the following method: Multiple sample audio files were acquired, and the sample audio files were labeled with spoken fluency tags; Extract the sample audio features of each sample audio frame in each of the sample audio samples; For each of the sample audios, the audio features of each of the sample audio frames are clustered to obtain the category of each of the sample audio frames in each of the sample audios; For each of the sample audios, the category of each sample audio frame and the audio features of each sample audio frame in the sample audio are processed by the initial fluency recognition model to determine the predicted spoken fluency of each sample audio. Based on the spoken fluency label and the predicted spoken fluency of each sample audio, a loss function is determined, and the initial fluency recognition model is trained according to the loss function to obtain a trained fluency recognition model.

5. The audio evaluation method according to claim 4, characterized in that, The sample audio samples were obtained from question-and-answer audio in open scenarios and / or audio for following along with text in follow-along scenarios.

6. An audio evaluation device, characterized in that, The device includes: The acquisition module is configured to acquire audio data. The extraction module is configured to process the audio data according to a convolutional neural network to obtain the encoded features of each audio frame; and to process the encoded features of each audio frame according to a context encoder to obtain the audio features of each audio frame. A clustering module is configured to perform clustering processing on the audio features of each audio frame to obtain the category of each audio frame; The determination module is configured to process the category and audio features of each audio frame according to a trained fluency recognition model to determine the evaluation result of the audio data; the evaluation result includes spoken fluency, which reflects the fluency of the speaker corresponding to the audio data.

7. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processing device, it implements the steps of the method described in any one of claims 1-5.

8. An electronic device, characterized in that, include: A storage device having at least one computer program stored thereon; At least one processing device is configured to execute the at least one computer program in the storage device to implement the steps of the method according to any one of claims 1-5.

Images