Pronunciation evaluation method and device, storage medium and electronic device

By displaying example sentence text and generating action videos of pronunciation organs, the problem of users' difficulty in judging their pronunciation status is solved, and accurate pronunciation evaluation is provided, which improves learning effect.

CN113077819BActive Publication Date: 2024-11-22BEIJING YOUZHUJU NETWORK TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110298227.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-03-19
Publication Date
2024-11-22
Estimated Expiration
2041-03-19

AI Technical Summary

Technical Problem

When learning pronunciation, it is difficult for users to accurately judge their pronunciation because they cannot observe the specific movement of other people's pronunciation organs, resulting in hindering pronunciation learning.

Method used

By presenting the example sentence text to the user, collecting the audio of their reading, generating the action video of the pronunciation organ, and comparing it with the standard action video, it provides pronunciation evaluation information so that the user can more accurately evaluate his pronunciation.

Benefits of technology

It realizes accurate evaluation of user pronunciations, helping users to understand and improve pronunciation more intuitively and improve learning effects.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113077819B_ABST
    Figure CN113077819B_ABST
Patent Text Reader

Abstract

The present disclosure relates to a pronunciation evaluation method and device, a storage medium and an electronic device, the method comprising: displaying an example sentence text to a user; collecting an audio to be evaluated that the user reads aloud based on the example sentence text; generating a pronunciation organ action video reflecting the action of the pronunciation organ when the user reads aloud the example sentence text; generating pronunciation evaluation information based on the pronunciation organ action video and the pronunciation organ standard action video corresponding to the example sentence text; and displaying the pronunciation evaluation information to the user. The present disclosure can accurately evaluate the pronunciation of the user and intuitively reflect whether the pronunciation of the user is accurate.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of education, and in particular, to a pronunciation evaluation method and device, a storage medium, and an electronic device. Background Art

[0002] When learning pronunciation, users can usually only imitate the pronunciation they hear, or imitate the lip movements of others. It is difficult for users to observe the specific movement of other people's specific vocal organs, and thus it is difficult for users to correctly judge their own pronunciation, which hinders pronunciation learning. Summary of the invention

[0003] This summary is provided to introduce concepts in a brief form that will be described in detail in the detailed description below. This summary is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0004] In a first aspect, the present disclosure provides a pronunciation evaluation method, comprising displaying an example sentence text to a user; collecting audio to be evaluated that is read aloud by the user based on the example sentence text; generating a pronunciation organ movement video based on the audio to be evaluated; generating pronunciation evaluation information based on the pronunciation organ movement video and the standard pronunciation organ movement video corresponding to the example sentence text; and displaying the pronunciation evaluation information to the user.

[0005] In a second aspect, the present disclosure provides a pronunciation evaluation device, comprising: an example sentence display module, used to display example sentence text to a user; an audio acquisition module, used to collect audio to be evaluated read by a user based on the example sentence text; a video generation module, used to generate a pronunciation organ movement video based on the audio to be evaluated; a pronunciation evaluation module, used to generate pronunciation evaluation information based on the pronunciation organ movement video and the pronunciation organ standard movement video corresponding to the example sentence text; and an evaluation display module, used to display the pronunciation evaluation information to the user.

[0006] In a third aspect, the present disclosure provides a computer-readable medium having a computer program stored thereon, which, when executed by a processing device, implements the steps of the method described in the first aspect of the present disclosure.

[0007] In a fourth aspect, the present disclosure provides an electronic device, comprising a storage device and a processing device, wherein a computer program is stored in the storage device; the processing device is used to execute the computer program in the storage device to implement the steps of the method described in the first aspect of the present disclosure.

[0008] Through the above technical solution, at least the following technical effects can be achieved:

[0009] By obtaining the audio to be evaluated that the user reads aloud based on the example sentence text, and generating pronunciation evaluation information by generating a pronunciation organ movement video based on the audio to be evaluated and the standard pronunciation organ movement video corresponding to the example sentence text, the user's pronunciation can be evaluated more accurately, thereby more intuitively reflecting whether the user's pronunciation is accurate.

[0010] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale. In the drawings:

[0012] Figure 1 The present invention is a flowchart of a pronunciation evaluation method according to an exemplary disclosed embodiment.

[0013] Figure 2 It is a flowchart showing another pronunciation evaluation method according to an exemplary disclosed embodiment of the present disclosure.

[0014] Figure 3 It is a block diagram of a pronunciation evaluation device according to an exemplary disclosed embodiment of the present disclosure.

[0015] Figure 4 It is a block diagram of an electronic device according to an exemplary disclosed embodiment of the present disclosure. DETAILED DESCRIPTION

[0016] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0017] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0018] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0019] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0020] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0021] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0022] Figure 1 is a flowchart of a pronunciation evaluation method according to an exemplary disclosed embodiment. Figure 1 As shown, the method includes:

[0023] S11. Displaying example sentence text to the user.

[0024] The example sentence text may be a text of any length such as a phrase, a sentence, a paragraph, an article, etc. The example sentence text may also refer to a clause of a longer text after being segmented.

[0025] For example, in a scenario where a user is learning pronunciation, if the user wants to test and practice pronunciation, the example sentence text can be displayed to the user in the form of text so that the user can perform a pronunciation test. If the user wants to learn pronunciation, the example sentence text can be displayed to the user in the form of audio so that the user can read along. In addition, the present disclosure is not limited to displaying the example sentence text to the user in the form of text and audio together.

[0026] The example text can be displayed in the form of text through the display device of the user terminal, and can also be displayed in the form of voice through the playback device of the user terminal. The voice corresponding to the example text can be pre-stored, and the text can be converted into voice for direct use when the voice needs to be displayed.

[0027] User terminals may include mobile phones, computers, learning machines, wearable devices, and any other devices with display functions.

[0028] In a possible implementation, example sentence audio is generated based on the example sentence text, the audio is synthesized with a video of standard pronunciation organ movements to form an example sentence demonstration video, and the example sentence text and the example sentence demonstration video are displayed to the user.

[0029] The standard motion video of the pronunciation organs is generated based on the example text, and the video features can be generated by a pre-trained video feature generation model. The example text is divided into unit text sequences, and the unit text sequences are input into the video feature generation model to obtain a video feature sequence, and the standard motion video of the pronunciation organs is generated based on the video feature sequence.

[0030] The unit text sequence is a sequence obtained by arranging the example text into small units for generating a video. In the present disclosure, the unit text can be a phoneme, a word, a single character, etc. After segmenting the example text, a more refined model input value can be obtained, so that the model can more efficiently generate an accurate video feature sequence based on the unit text. For example, in the case of the example text "How are you", the example text can be segmented into a unit text sequence of "how", "are", and "you" based on words, or segmented into a unit text sequence of "how", "are", and "you" based on phonemes. A unit text sequence of [ɑ:][ju:].

[0031] The video feature generation model is trained in the following way:

[0032] The sample text is divided into a sample unit text sequence, model training data is constructed according to the sample unit text sequence and a sample video feature sequence of a sample pronunciation organ action video corresponding to the sample unit text sequence, and the video feature generation model is obtained by training according to the model training data.

[0033] The sample pronunciation organ movement video is a demonstration video produced or recorded based on the sample text. The demonstration video can be an animation demonstration video of the oral cavity made using any animation production and rendering software, or it can be a head video of a person reading the sample text taken by an MRI machine.

[0034] By extracting the video features of the sample pronunciation organ action video frame by frame or frame by frame, feature information of multiple image frames of the sample pronunciation organ action video can be obtained, and the video feature information can be arranged in the order of the video frames to obtain a sample video feature sequence. It is worth noting that the present disclosure does not limit the form of the feature information of the image frame, and any form of feature information that can be extracted and restored to an image through processing can be used as feature information in the video feature sequence in the present disclosure.

[0035] In a possible implementation, the feature information is principal component information, and the principal component information of each video frame is obtained by performing principal component analysis on the sample pronunciation organ action video frame by frame, and the principal component information of each video frame is arranged in the order of the video frames to obtain the sample video feature sequence. By restoring the principal component information, a restored image can be obtained, and the restored image can be arranged and synthesized in the order of the sample video feature sequence to obtain a restored demonstration video. The sample unit text sequence and the sample video feature sequence corresponding to the sample unit text sequence are used as training samples to train the video feature generation model, so that the video feature generation model can generate corresponding video features or video feature sequences based on any unit text.

[0036] The video feature generation model can be a deep learning model, which generates training samples for input into the deep learning model by labeling each sample unit text in the sample unit text sequence. After multiple rounds of iterative training, the deep learning model can accurately generate video features based on the unit text.

[0037] The video feature generation model can also be an attention model, which includes an encoder and a decoder, wherein the encoder is used to generate an encoding result based on a unit text sequence, and the decoder is used to generate a video feature sequence based on the encoding result. The encoder and the decoder are trained in the form of end-to-end training from a unit text sequence to a video feature sequence, so that the attention model can accurately generate a video feature sequence based on a unit text sequence. It is worth noting that when the demonstration video to be generated is an MRI (Magnetic Resonance Imaging) video, considering that the recording cost of MRI videos is relatively high, recording a longer video at one time can reduce the recording cost. Therefore, the sample pronunciation organ action video can be segmented from a complete sample demonstration video, and correspondingly, the sample text is also segmented from a complete sample demonstration text.

[0038] The sample demonstration video may be an MRI video taken by a nuclear magnetic resonance apparatus of a user reciting a sample text. Multiple sample texts are obtained by sentence segmentation of the sample demonstration text, and the sample demonstration video is divided into sub-videos corresponding to each sample text based on the sentence segmentation results, thereby obtaining multiple sample pronunciation organ movement videos.

[0039] In one possible implementation, the sample demonstration text is divided into sentences to obtain multiple sample texts, and speech recognition is performed on the sample speech recorded synchronously with the sample demonstration video. Based on the speech recognition result, the speech segment corresponding to each sample text is determined, and based on the timeline information of each speech segment, the sample pronunciation organ action video corresponding to each speech segment is determined from the sample demonstration video. For example, by dividing the sample demonstration text "Howare you? I'm fine thank you, and you?" into sentences, four clauses of "Howare you", "I'm fine", "thank you", and "and you" can be obtained. By recognizing the sample speech with a duration of 6 seconds, the timeline information of the speech segment corresponding to "Howare you" can be obtained as "00:00:00 to 00:01:40", and the timeline information of the speech segment corresponding to "I'm fine" can be obtained as "00:01:40 to 00:02:50", and "thank you " The time axis information of the voice segment corresponding to "and you" is "00:02:50 to 00:04:40", and the time axis information of the voice segment corresponding to "and you" is "00:04:40 to 00:06:00", then the sample demonstration sub-video with a duration of 6 seconds can be divided into four video segments of "00:00:00 to 00:01:40", "00:01:40 to 00:02:50", "00:02:50 to 00:04:40", and "00:04:40 to 00:06:00" according to the time axis information, and each video segment is a sample demonstration sub-video of its corresponding sample clause text. The above sentence segmentation method is only shown as an example. Those skilled in the art may use other sentence segmentation methods to segment sentences, and the present disclosure does not limit this.

[0040] Considering that the recording instrument may not have a recording function when recording the MRI video, an additional recording device is required to record the sample voice, and the sample demonstration video and the sample voice may have a time difference due to different start times and end times during recording. Therefore, in a possible implementation, the timeline information of the sample voice and the sample demonstration video is aligned; the length of the sample voice or the sample demonstration video is adjusted to make the length of the sample voice and the sample demonstration video consistent.

[0041] Considering that people may change their postures when recording videos, the facial position in the recorded video is not fixed, which may affect the aesthetics of the video and the extraction of feature information of the video, increasing the training cost of the model. Therefore, in a possible implementation, the facial position in the sample demonstration video is adjusted frame by frame so that the same organs in each video frame are located at the same image position. The adjustment can be performed in the form of pixel tracking or optical flow tracking, or in the form of feature point extraction and alignment. The processing of the video frame includes but is not limited to rotation, translation, enlargement, and reduction. The screen size of the video frame can also be uniformly cropped to reduce interference information in the video.

[0042] S12: collecting the audio to be evaluated read aloud by the user based on the example sentence text.

[0043] The voice read aloud by the user can be collected through the voice collection device of the user terminal.

[0044] In a possible implementation, speech recognition may be performed on the collected audio to be evaluated, and the recognition result may be compared with the example text. When the text similarity is lower than a preset similarity threshold, a prompt message may be sent to the user to remind the user to read the example text again.

[0045] S13, generating a pronunciation organ movement video reflecting the movement of the pronunciation organs when the user reads the example sentence text.

[0046] In a possible implementation, the audio to be evaluated is converted into an audio feature vector to be processed; the audio feature vector to be processed is input into a video generation model to obtain a pronunciation organ movement video corresponding to the audio to be evaluated output by the video generation model.

[0047] An achievable implementation method converts the audio to be evaluated into an audio feature vector to be evaluated, which may specifically be: inputting the audio to be evaluated into a speech recognition model to obtain the audio feature vector to be evaluated, wherein the audio feature vector to be evaluated includes a phoneme posterior probability vector for each frame of audio in the audio to be evaluated, and the dimension of each phoneme posterior probability vector is the phoneme dimension included in the language type corresponding to the audio to be evaluated.

[0048] Phonemes are the smallest units of speech divided according to the natural properties of speech. Each human speech, animal speech, and instrument sound can be divided into a limited number of smallest units of speech based on its properties.

[0049] Each frame of audio in the audio to be evaluated may be the audio of a phoneme. A phoneme may be represented by a phoneme posterior probability vector. The dimension of each phoneme posterior probability vector is the phoneme dimension included in the language type corresponding to the audio to be evaluated. For example, assuming that the language type corresponding to the audio to be evaluated is English, since the number of phonemes in English is 48, the dimension of the English phoneme posterior probability vector is 48. That is to say, an English phoneme posterior probability vector includes 48 probability values ​​greater than or equal to 0 and less than 1, and the sum of the 48 probability values ​​is 1. The phoneme corresponding to the maximum value of the 48 probability values ​​is the English phoneme represented by the phoneme posterior probability vector. For another example, assuming that the language type corresponding to the audio to be evaluated is a language type that imitates a target musical instrument, if the target musical instrument has 50 phonemes, then the dimension of the phoneme posterior probability vector is also 50, specifically consisting of 50 probability values ​​whose sum is 1.

[0050] Each frame of audio in the audio to be evaluated can also be an audio of a word. Correspondingly, a word is represented by a posterior probability vector of the word. Therefore, it is worth noting that the audio frame playback time corresponding to each frame of audio in the audio to be evaluated can be freely set according to needs, so that each frame of audio is an audio of one or more phonemes / words.

[0051] Automatic Speech Recognition (ASR) is a model that converts sound into corresponding text or commands.

[0052] Since the number of characters or words in any language is huge while the number of phonemes is relatively small, and the pronunciation of each character or word is composed of one or more phonemes, therefore, in a preferred implementation, the speech recognition model can be trained in the following way: constructing a model training sample based on sample audio frames and the phonemes corresponding to the sample audio frames; and training the speech recognition model based on the model training samples.

[0053] Specifically, the sample audio frame is subjected to signal processing and knowledge mining, the speech feature parameters of the sample audio frame are analyzed, a speech template is created, and a speech parameter library is obtained. A mapping table of speech feature parameters and phonemes is constructed according to the sample audio frame and the phonemes corresponding to the sample audio frame.

[0054] After the audio to be evaluated is input into the trained speech recognition model, for each frame of the audio to be evaluated, the same analysis as during training is performed to obtain the speech feature parameters to be processed, and the speech feature parameters to be processed are matched one by one with the speech templates in the speech parameter library to obtain the matching probability of the speech feature parameters to be processed and each speech feature parameter in the speech parameter library. Furthermore, the phoneme posterior probability vector of each frame of the audio to be evaluated is obtained according to the mapping table of speech feature parameters and phonemes.

[0055] Compared with the method of training speech recognition models using massive characters / words and character / word audios, this method of training speech recognition models using a small and limited number of phonemes and phoneme audios can reduce model training tasks and quickly obtain a trained speech recognition model.

[0056] The video generation model is obtained by training in the following manner: constructing model training data based on sample audio and sample pronunciation organ action videos corresponding to the sample audio; and obtaining the video generation model by training based on the model training data.

[0057] The present disclosure does not specifically limit the loss function of the video generation model.

[0058] Since the number of characters or words (or phonetic segments) in any language is huge and the number of phonemes is small, and the pronunciation of each character or word (or phonetic segment) is composed of one or more phonemes, therefore, in a preferred embodiment, the sample audio is the audio of all phonemes corresponding to the target language type. The sample pronunciation organ action video can be an animation demonstration video of the pronunciation organ action corresponding to each phoneme made by any animation production rendering software. The sample pronunciation organ action video can also be a pronunciation organ action video corresponding to each phoneme shot by an anatomical imaging instrument such as a camera, a nuclear magnetic resonance apparatus, a CT apparatus, etc. Since the user can not only read the characters or words in various human languages, but also imitate the sounds of animals, musical instruments, etc. Therefore, in order to facilitate ordinary technicians in this field to understand the embodiments of the present disclosure, it is necessary to explain that the above-mentioned sound segment can refer to a sound segment in other sounds that imitate non-human languages ​​(such as a sound segment corresponding to a key or a string of a musical instrument).

[0059] Similarly, in another implementation, the sample audio is the audio of all characters or words (or sound segments) corresponding to the target language type. The sample pronunciation organ action video can be an animation demonstration video of the pronunciation organ action corresponding to each character or word (or sound segment) made by any animation rendering software. The sample pronunciation organ action video can also be a pronunciation organ action video corresponding to each character or word (or sound segment) shot by an anatomical imaging instrument such as a camera, a nuclear magnetic resonance apparatus, or a CT apparatus.

[0060] In an achievable implementation manner, the model training data is constructed based on the sample audio and the sample pronunciation organ action video corresponding to the sample audio, which may specifically include the following steps:

[0061] Convert each frame of audio in the sample audio into a sample phoneme posterior probability vector to obtain a sample phoneme posterior probability vector sequence including at least one sample phoneme posterior probability vector; extract the sample pronunciation organ video features corresponding to each of the sample phoneme posterior probability vectors in the sample phoneme posterior probability vector sequence based on the sample pronunciation organ action video to obtain a sample pronunciation organ video feature sequence; use the sample phoneme posterior probability vector sequence and the sample pronunciation organ video feature sequence as the model training data.

[0062] Among them, each frame of audio in the sample audio corresponds one-to-one to each sample phoneme posterior probability vector in the sample phoneme posterior probability vector sequence, and each sample phoneme posterior probability vector in the sample phoneme posterior probability vector sequence corresponds one-to-one to each sample pronunciation organ video feature in the sample pronunciation organ video feature sequence.

[0063] It is easy to understand that, when one frame of audio corresponds to one phoneme, the pronunciation process of the pronunciation organ corresponding to one phoneme is reflected by one or more frames of video images. Therefore, each of the sample pronunciation organ video features is the pixel feature information of at least one frame of video image in the sample pronunciation organ action video; or, each of the sample pronunciation organ video features is the principal component feature information of at least one frame of video image in the sample pronunciation organ action video.

[0064] It is worth mentioning that the principal component feature information is the principal component coefficient data representing the video image obtained by performing dimensionality reduction processing on the video image through the principal component analysis algorithm.

[0065] A feasible implementation method may also include the following steps before extracting the sample pronunciation organ video features corresponding to each sample phoneme posterior probability vector in the sample phoneme posterior probability vector sequence based on the sample pronunciation organ action video: adjusting the pronunciation organ position in the sample pronunciation organ action video frame by frame so that the same pronunciation organ in each frame of the video image is located at the same image position.

[0066] The adjustment can be performed in the form of pixel tracking or optical flow tracking, or can be performed by extracting and aligning feature points. The processing of each frame of video image includes but is not limited to rotation, translation, enlargement, and reduction. The size of each frame of video image can also be uniformly cropped. The position of the vocal organs in the sample vocal organ action video is adjusted frame by frame so that the same vocal organs in each frame of video image are located at the same image position, which is conducive to reducing the interference on the model training effect and model convergence speed caused by the different positions of the same vocal organs in each frame of video image.

[0067] Since the audio feature vector to be evaluated includes the phoneme posterior probability vector of each frame of audio in the audio to be evaluated, after the audio feature vector to be evaluated is input into the trained video generation model, the pronunciation organ video feature corresponding to the phoneme posterior probability vector of each frame of audio can be obtained. According to the pronunciation organ video feature sequence, the pronunciation organ action video can be generated and output.

[0068] It is worth noting that the sample pronunciation organ action video used for training the video generation model in step S13 is a video corresponding to the sample audio, and the castrated pronunciation organ action video used for training the video feature generation model in step S11 is a video corresponding to the sample text, and the sample audio can be recorded synchronously with the sample pronunciation organ action video. In the case where the sample audio and the sample pronunciation organ action video are synchronously recorded based on the same sample text, the sample pronunciation organ action video in step S11 and step S13 is the same video. In this case, the audio and video alignment, video cropping, video center alignment and other operations for the sample pronunciation organ action video can be performed only once, and when training the two models, the aligned video and audio are used for training.

[0069] S14, generating pronunciation evaluation information based on the pronunciation organ movement video and the pronunciation organ standard movement video corresponding to the example sentence text.

[0070] The pronunciation evaluation information includes at least one of pronunciation score information of the user, pronunciation action suggestion information, and a comparison video between the pronunciation organ action video and the pronunciation organ standard action video.

[0071] The comparison video is generated in the following manner: based on the unit text content of the example sentence text, the video clips representing the same unit text content in the pronunciation organ movement video and the pronunciation organ standard movement video are taken as a group of video clip groups; the video clips belonging to the pronunciation organ movement video and the pronunciation organ standard movement video in each video clip group are aligned; the aligned pronunciation organ movement video and the pronunciation organ standard movement video are spliced ​​to obtain the comparison video.

[0072] In the case where the pronunciation evaluation information includes the pronunciation scoring information and / or the pronunciation action suggestion information, the action difference information is obtained by comparing the pronunciation organ action video with the standard pronunciation organ action video corresponding to the example sentence text; the pronunciation scoring information is generated according to the action difference information, and / or the action difference information is matched with preset pronunciation action suggestion information to obtain target action suggestion information matching the action difference information.

[0073] The motion difference information may refer to the difference information of the motion trajectory of the feature points of the vocal organs.

[0074] The motion trajectory of the characteristic points of the vocal organs is used to reflect the vocal movement process of the vocal organs. The characteristic points of the vocal organs can be the centroid, center point, contour characteristic points, etc. of the vocal organs, or can be other characteristic points outside the vocal organs with the centroid, center point, contour characteristic points, etc. of the vocal organs as reference points. The present disclosure does not impose specific restrictions on the number and type of characteristic points.

[0075] The pronunciation organ action video includes at least one frame of video image. The position coordinates of the characteristic points of the pronunciation organ are determined in each frame of the pronunciation organ action video, and the position coordinates of the characteristic points of the pronunciation organ corresponding to the number of frames of the pronunciation organ action video can be obtained. Based on all the position coordinates of the characteristic points of the pronunciation organ, the motion trajectory of the characteristic points of the pronunciation organ corresponding to the time axis of the pronunciation organ action video can be constructed.

[0076] The preset motion trajectory of the feature points corresponding to the example text is the standard motion trajectory of the feature points of the vocal organs corresponding to the example text. By calculating the similarity between the motion trajectory of the feature points of the vocal organs and the preset motion trajectory of the standard feature points, the similarity information of the two trajectory lines can be obtained.

[0077] In one achievable implementation manner, the preset motion trajectory of the feature points corresponding to the example sentence text can be determined in the following manner:

[0078] From the model training data of the training video generation model, all the phonemes (or other unit-granular information such as characters, words, sentences, etc.) constituting the example sentence text are determined, and the pronunciation organ video feature sequences corresponding to all the phonemes are determined, and the pronunciation organ standard action video of the example sentence text is generated based on the pronunciation organ video feature sequence. The position coordinates of the feature points of the pronunciation organs are determined in each frame of the video image of the pronunciation organ standard action video, and the preset motion trajectory of the feature points corresponding to the example sentence text is obtained.

[0079] In the case of considering accuracy, multiple groups of phoneme sequences constituting example sentence texts can be determined from the model training data of the training video generation model, and multiple feature point preset motion trajectories can be determined based on the multiple groups of phoneme sequences constituting example sentence texts. By performing weighted average processing on the multiple feature point preset motion trajectories, a comprehensive and more accurate feature point preset motion trajectory can be obtained.

[0080] For example, the pronunciation score of the audio to be evaluated is determined according to the size of the similarity value in the similarity information. The pronunciation score of the audio to be evaluated is used as the pronunciation evaluation result. For another example, the pronunciation level of the audio to be evaluated, such as excellent, medium, qualified, unqualified, or missing pronunciation, is determined according to the size of the similarity value in the similarity information. The pronunciation level of the audio to be evaluated, such as excellent, medium, qualified, unqualified, or missing pronunciation, is used as the pronunciation evaluation result.

[0081] By adopting the above-mentioned pronunciation evaluation method, the audio to be evaluated of the user reading the example sentence text can be input into the video generation model, and the user's pronunciation organ movement video can be fitted and restored. The position coordinates of the characteristic points of the pronunciation organ are determined in each frame of the video image of the pronunciation organ movement video, and the motion trajectory of the characteristic points of the pronunciation organ is obtained. The similarity between the motion trajectory of the characteristic points of the pronunciation organ and the preset motion trajectory of the standard characteristic points corresponding to the example sentence text is calculated to obtain the pronunciation movement similarity information of the pronunciation organ. The pronunciation evaluation result can be obtained based on the pronunciation movement similarity information of the pronunciation organ. Since the pronunciation is directly related to the movement of the pronunciation organ, the pronunciation evaluation result obtained in this way is more accurate.

[0082] Generating the pronunciation evaluation result of the audio to be evaluated according to the similarity information may also include the following steps:

[0083] Perform spectral analysis on the audio to be evaluated to extract sound spectrum feature information, calculate the similarity between the extracted sound spectrum feature information and the standard sound spectrum feature information corresponding to the example sentence text to obtain spectrum similarity information, combine the spectrum similarity information with the similarity information determined based on the motion trajectory of the feature points of the vocal organs, and obtain the pronunciation evaluation result.

[0084] In this way, based on the user's pronunciation accuracy calculated based on the information of a single sound spectrum dimension, a more accurate pronunciation evaluation result can be determined by further combining the similarity information determined based on the motion trajectory of the feature points of the pronunciation organs. This method further improves the accuracy of the pronunciation evaluation results.

[0085] Due to individual differences among users, different users have different reading speeds when reading the same example text. That is to say, the duration of the audio to be evaluated is related to the speed of the user's pronunciation, that is, the duration of the audio to be evaluated is variable. In the case where the duration of the audio to be evaluated is different, the number of frames of the audio to be evaluated is different, and then the audio to be evaluated with different durations is input into the video generation model, and the duration of each pronunciation organ action video obtained is also different. When the duration of the pronunciation organ action video is different, the number of video image frames included in the pronunciation organ action video is different. Then, if the number of video image frames in the pronunciation organ action video corresponding to the audio to be evaluated is different from the number of video image frames in the pronunciation organ standard action video corresponding to the example text, the length of the feature point motion trajectory of the pronunciation organ and the preset motion trajectory of the feature point corresponding to the example text will be inconsistent. Then, when the similarity is calculated based on the feature point motion trajectory of the pronunciation organ and the preset motion trajectory of the feature point corresponding to the example text, the similarity information obtained will have a large error. In this regard, the present disclosure provides the following two implementation methods to avoid the problem of large errors in the calculated similarity information.

[0086] In detail, an achievable implementation method is, before calculating the similarity between the feature point motion trajectory of the pronunciation organ and the preset feature point motion trajectory corresponding to the example sentence text to obtain the similarity information, adjusting the number of feature point position coordinates of the feature point motion trajectory of the pronunciation organ according to the number of feature point position coordinates constituting the preset feature point motion trajectory, so that the number of feature point position coordinates of the preset feature point motion trajectory is the same as the number of feature point position coordinates of the feature point motion trajectory of the pronunciation organ.

[0087] For example, assume that the number of feature point position coordinates of the preset motion trajectory of the feature point is 5, which are coordinates A, B, C, D, and E. The number of feature point position coordinates of the feature point motion trajectory is 4, which are coordinates a, b, c, and e. At this time, the number of feature point position coordinates of the feature point motion trajectory can be adjusted. For example, the feature point f (0, 0) is inserted into the feature point motion trajectory of the current pronunciation organ to obtain a feature point motion trajectory composed of coordinates a, b, c, f, and e. Among them, the insertion position of the feature point f (0, 0) can be determined according to the position of the missing phoneme in the audio to be evaluated. It is easy to understand that when the ASR model knows each phoneme in the audio to be evaluated and each phoneme in the known example text, the missing phoneme in the audio to be evaluated can be determined (similarly, the redundant phonemes in the audio to be evaluated can be known, so as to achieve the adjustment of the number of feature point position coordinates of the feature point motion trajectory while reducing the number of feature point position coordinates of the feature point motion trajectory).

[0088] Another feasible implementation manner is, before determining the position coordinates of the feature points of the pronunciation organs in each frame of the video image of the pronunciation organs action video, adjusting the frame number of the video image in the pronunciation organs action video according to the frame number of the video image in the standard action video of the pronunciation organs corresponding to the example sentence text, so that the frame number of the video image in the pronunciation organs action video is the same as the frame number of the video image in the standard action video of the pronunciation organs.

[0089] It is easy to understand that when the number of frames of the video image in the standard movement video of the pronunciation organs is the same as the number of frames of the video image in the movement video of the pronunciation organs of the audio to be evaluated, based on the premise that one frame of video image corresponds to one feature point, it can be known that the number of feature point position coordinates in the preset motion trajectory of the feature points corresponding to the example sentence text is the same as the number of feature point position coordinates in the motion trajectory of the feature points of the audio to be evaluated.

[0090] For example, assume that the number of video image frames in the standard pronunciation organ movement video is 5 frames, namely 1, 2, 3, 4, and 5 frames. And the number of video image frames in the pronunciation organ movement video of the audio to be evaluated is 3 frames, namely 1, 4, and 5 frames. In this case, the video image frame sequence 1, 4, and 5 can be interpolated. For example, insert image frames 1 and 4 into the current video image frame sequence 1, 4, and 5 to obtain a video image frame sequence of 1, 1, 4, 4, and 5. For another example, insert an empty image frame 0 into the current video image frame sequence 1, 4, and 5 to obtain a video image frame sequence of 1, 0, 0, 4, and 5.

[0091] In one achievable implementation mode, in order to further locate which phoneme or which word in the audio to be evaluated is inaccurately pronounced (or missed) based on the pronunciation evaluation result of the audio to be evaluated, the above step S13, in each frame of the video image of the pronunciation organ action video, determines the position coordinates of the feature points of the pronunciation organ, and obtains the motion trajectory of the feature points of the pronunciation organ, can also include the following steps:

[0092] The audio to be evaluated is divided according to a preset pronunciation evaluation granularity to obtain a plurality of sub-audios to be evaluated; in each frame of the pronunciation organ movement video, the position coordinates of the feature points of the pronunciation organ corresponding to each of the sub-audios to be evaluated are determined to obtain a motion trajectory segment of the pronunciation organ feature points corresponding to each of the sub-audios to be evaluated.

[0093] The preset pronunciation evaluation granularity is a pronunciation evaluation unit set according to user needs. The pronunciation evaluation granularity can be a phoneme, a character, a word, a sentence, a paragraph, a chapter, etc., and the present disclosure does not specifically limit this. When the audio to be evaluated is divided according to the preset pronunciation evaluation granularity, the audio to be evaluated can be divided according to the duration corresponding to the preset pronunciation evaluation granularity, thereby obtaining a plurality of sub-audios to be evaluated.

[0094] When determining each sub-audio to be evaluated, a segment of the motion trajectory of the characteristic points of the vocal organs corresponding to each sub-audio to be evaluated can be obtained based on the video of the action of the vocal organs. Specifically, in each frame of the video image of the action of the vocal organs, the position coordinates of the characteristic points of the vocal organs corresponding to each sub-audio to be evaluated can be determined to obtain a segment of the motion trajectory of the characteristic points of the vocal organs corresponding to each sub-audio to be evaluated. Alternatively, after obtaining the complete motion trajectory of the characteristic points of the vocal organs of the entire audio to be evaluated, the motion trajectory of the characteristic points of the vocal organs of the entire audio to be evaluated can be divided according to the way of dividing each sub-audio to be evaluated, thereby obtaining a segment of the motion trajectory of the characteristic points of the vocal organs corresponding to each sub-audio to be evaluated.

[0095] Adaptively, after obtaining the motion trajectory segment of the feature points of the pronunciation organs of each sub-audio to be evaluated, for each sub-audio to be evaluated, the similarity between the motion trajectory segment of the feature points of the pronunciation organs corresponding to the sub-audio to be evaluated and the corresponding preset motion trajectory segment of the feature points is calculated to obtain a first similarity value corresponding to the sub-audio to be evaluated, and the similarity information includes the first similarity value of each sub-audio to be evaluated.

[0096] The feature point preset motion trajectory segment is a trajectory segment in the complete feature point preset motion trajectory. The method of obtaining the feature point preset motion trajectory segment is similar to the method of obtaining the feature point motion trajectory segment of the vocal organ of each sub-audio to be evaluated by dividing the feature point motion trajectory of the vocal organ of the entire audio to be evaluated, and will not be repeated here.

[0097] See also Figure 2 The flowchart of the method for locating which phoneme or which word in the audio to be evaluated is inaccurately pronounced includes the following steps:

[0098] S21, obtaining an audio to be evaluated, where the audio to be evaluated is an audio of a user reading a text of an example sentence;

[0099] S22, inputting the audio to be evaluated into a video generation model, and obtaining a pronunciation organ movement video corresponding to the audio to be evaluated output by the video generation model;

[0100] S23, dividing the audio to be evaluated according to a preset pronunciation evaluation granularity to obtain a plurality of sub-audios to be evaluated;

[0101] S24, determining the position coordinates of the feature points of the vocal organs corresponding to each of the sub-audios to be evaluated in each frame of the vocal organ motion video, and obtaining a motion trajectory segment of the vocal organ feature points corresponding to each of the sub-audios to be evaluated;

[0102] S25, for each of the sub-audio to be evaluated, calculating the similarity between the motion trajectory segment of the feature point of the pronunciation organ corresponding to the sub-audio to be evaluated and the corresponding preset motion trajectory segment of the feature point, to obtain a first similarity value corresponding to the sub-audio to be evaluated;

[0103] S26, determining a target first similarity value that is less than a preset threshold, and determining a target sub-audio to be evaluated corresponding to the target first similarity value;

[0104] The preset threshold value may be 90%, 98%, or other preset values. When the first similarity value is less than the preset threshold value, it is determined that the pronunciation of the target sub-audio to be evaluated corresponding to the first similarity is inaccurate. The magnitude of the first similarity value is used to characterize the similarity between the target consonant to be evaluated and the standard pronunciation corresponding to the target sub-audio to be evaluated.

[0105] S27, determining a target example sentence text segment according to the target sub-audio to be evaluated, wherein the target example sentence text segment is a segment in the example sentence text;

[0106] When the pronunciation of the target sub-audio to be evaluated is determined to be inaccurate, the target example sentence text segment corresponding to the target sub-audio to be evaluated may be determined. The target example sentence text segment may include one or more phonemes / characters / words / sentences, etc.

[0107] S28, displaying the target example sentence text segment and the target first similarity value in association with each other to obtain the pronunciation evaluation result, so as to remind the user of the target example sentence text segment that is mispronounced.

[0108] use Figure 2 The method shown can locate which phoneme or word in the audio to be evaluated is not accurately pronounced, and let the user know. This makes it easier for the user to perform targeted pronunciation exercises on the incorrectly pronounced part. For example, the standard action video of the pronunciation organ corresponding to the incorrectly pronounced part and the preset motion trajectory fragment of the standard pronunciation organ feature point are displayed to the user. At the same time, the user's pronunciation organ action video corresponding to the incorrectly pronounced part and the user's inaccurate pronunciation organ feature point motion trajectory fragment can also be displayed to the user, so that the user knows which pronunciation is inaccurate and where the difference with the standard pronunciation is.

[0109] Since the sound is produced by the coordinated action of multiple vocal organs, the vocal organ action video in the embodiment of the present disclosure includes the action of at least one of the upper lip, lower lip, upper teeth, lower teeth, gums, hard palate, soft palate, uvula, tongue tip, tongue surface, tongue root, nasal cavity, oral cavity, pharynx, epiglottis, esophagus, trachea, vocal cords, and larynx, and the feature point motion trajectory (or feature point motion trajectory fragment) of the vocal organs includes the feature point motion trajectory (or feature point motion trajectory fragment) of each organ.

[0110] That is to say, by adopting the method in the above-mentioned embodiment of the present disclosure, the feature point motion trajectory (or feature point motion trajectory segment) of any kind of vocal organ can be obtained.

[0111] For the feature point motion trajectory (or feature point motion trajectory segment) of each vocal organ, the similarity between the feature point motion trajectory (or feature point motion trajectory segment) of the organ and the feature point preset motion trajectory (or feature point preset motion trajectory segment) corresponding to the organ in the example sentence text can be calculated to obtain a second similarity value, which represents the degree of similarity between the feature point motion trajectory (or feature point motion trajectory segment) of a vocal organ and the standard feature point preset motion trajectory (or feature point preset motion trajectory segment) of the vocal organ.

[0112] Furthermore, a target second similarity value less than a threshold value can be determined, and a target pronunciation organ can be determined according to the target second similarity value, so as to determine which specific pronunciation organ or organs among multiple pronunciation organs have an incorrect pronunciation action that leads to an incorrect pronunciation of the example text (or example text segment).

[0113] In this way, it is possible to locate which phoneme or word in the audio to be evaluated is inaccurately pronounced, and further locate which one or several pronunciation organs cause the inaccurate pronunciation. By showing the user the standard pronunciation organ action video corresponding to the one or several pronunciation organs and the preset motion trajectory of the standard pronunciation organ feature points, it is helpful for the user to carry out targeted correction learning of the pronunciation organ action.

[0114] Among them, the video of the movement of the pronunciation organs is a magnetic resonance imaging MRI video. Correspondingly, the sample video of the movement of the pronunciation organs used to train the video generation model is also a magnetic resonance imaging MRI video. The sample video of the movement of the pronunciation organs includes the movement of at least one of the upper lip, lower lip, upper teeth, lower teeth, gums, hard palate, soft palate, uvula, tongue tip, tongue surface, tongue root, nasal cavity, oral cavity, pharynx, epiglottis, esophagus, trachea, vocal cords, and larynx.

[0115] In addition, since the vocal organs also include vocal power organs such as lungs, diaphragm, and trachea, the vocal organ movement video and the sample vocal organ movement video can also include the movement of at least one of the vocal organs among the lungs, diaphragm, and trachea.

[0116] After obtaining the mispronounced phonemes or words, the action difference information between the mispronounced phonemes or words and the correct action video can be matched with preset pronunciation action suggestion information to obtain target action suggestion information matching the action difference information.

[0117] For example, after obtaining a mispronounced word, the action difference information shows that the upper palate position in the user's pronunciation organ action video is lower than the upper palate position in the standard pronunciation organ action video, then the corresponding target action suggestion information "raise the upper palate" can be matched; the action difference information shows that the tongue position in the user's pronunciation organ action video is backward compared to the tongue position in the standard pronunciation organ action video, then the corresponding target action suggestion information "extend the tongue forward" can be matched.

[0118] S15. Displaying the pronunciation evaluation information to the user.

[0119] The displayed pronunciation evaluation information may be at least one of the user's pronunciation scoring information, pronunciation action suggestion information, and a comparison video between the pronunciation organ action video and the pronunciation organ standard action video. The three may also be displayed in combination of two or at the same time.

[0120] Taking into account that MRI images are not clear enough and non-professionals are not familiar with the shapes of organs, it is difficult for users to extract information from MRI videos. Therefore, when the original pronunciation organ movement video and the original pronunciation organ standard movement video are MRI videos, the animation generation model can be used to render the pronunciation organ movement video or the pronunciation organ standard movement video frame by frame to obtain the pronunciation organ animation video, and the pronunciation organ animation video can be displayed as the pronunciation organ movement video or the pronunciation organ standard movement video.

[0121] Among them, the training samples of the animation generation model include multiple MRI sample images and animated organ maps corresponding to each MRI sample image. The training samples of the animation generation model are obtained in the following manner: determining the position of the organ in each MRI sample image; generating an animated organ corresponding to the position of the organ in each MRI sample image to obtain an animated organ map.

[0122] MRI videos are composed of multiple video frames. When generating animations, you can choose to input all the video frames into the animation generation model. After obtaining the animation frames output by the animation generation model, you can reorganize the animation frames according to the arrangement order of the video frames to obtain the animation video corresponding to the video frames.

[0123] In a possible implementation, video frames can be selected at intervals of preset frames and input into the animation generation model. In this way, after obtaining the animation frames generated by the animation generation model, interpolation can be performed between the animation frames to generate a smooth animation video. In this way, the workload of the animation generation model can be reduced, the consumption of computing resources can be reduced, and the efficiency of animation generation can be improved.

[0124] The animation generation model can be any machine learning model that can learn samples, such as a generative adversarial network model, a recurrent neural network model, a convolutional network model, etc., and the present disclosure does not limit this. The training samples of the model include multiple MRI sample images and animated organ images corresponding to each MRI sample image. By learning the training samples, the animation generation model can generate corresponding animated images based on the input MRI images, thereby achieving the effect of converting MRI video frames into animation frames.

[0125] The animation generation model can output animation frames corresponding to the video frames in sequence according to the order of video frame input, wherein the positions of the pronunciation organs in the animation frames are filled with animation pronunciation organs for easy viewing and understanding by users.

[0126] In a possible implementation, different colors can be filled for each animated pronunciation organ according to the difference of the pronunciation organs, and the organ name can be marked on the animated pronunciation organ. For example, the upper palate position can be filled with light yellow and marked with the character "upper palate", the tongue position can be filled with bright red and marked with the character "tongue", and the teeth position can be filled with white and marked with the character "teeth". In this way, the position and connection relationship of each organ can be more intuitively reflected, which is more conducive to user understanding.

[0127] It is worth noting that the above-mentioned color filling method and name marking method are only used as an example. The present disclosure does not limit the organ color filling method and name marking method. For example, the name marking can also be marked in a foreign language, or the phonetic symbols, pinyin, etc. of the pronunciation can be added.

[0128] By reorganizing the animation frames in the order of the video frames, a straight animation video can be obtained. The playback speed of the animation frames can be consistent with the video frames, or it can be adjusted according to application requirements. For example, when the animation video is used in an educational scenario, the playback speed of the animation frames can be reduced in order to more clearly show the movement and force of the pronunciation organs. When the playback speed of the animation frames is reduced, in order to make the animation video smoother, frames can be added between frames to increase the number of frames in the animation video.

[0129] In a possible implementation, the animation generation model is a generative adversarial network model, the animation generation model includes a generator for generating animation images based on MRI images, and the animation generation model is trained in the following manner:

[0130] Repeat the steps of generating a training animation image based on the MRI sample image by the generator, generating a loss value based on the animation pronunciation organ map corresponding to the MRI sample image and a preset loss function, adjusting the parameters in the generator based on the loss value, and evaluating the training animation image based on the animation pronunciation organ map by the discriminator of the generative adversarial network model, until the evaluation result meets the preset evaluation result condition.

[0131] The generator is used to generate images based on input data, and the discriminator is used to evaluate whether the image output by the generator has consistent features with the images in the specified set, that is, it can determine whether the image is in the specified set. The evaluation result of the discriminator may be correct or wrong. When the features of the image output by the generator are obviously different from those of the images in the specified set, the evaluation result of the discriminator is usually correct, that is, the discriminator can correctly determine whether the image is in the specified set. When the feature difference between the image generated by the generator and the images in the specified set is not obvious, it is difficult for the discriminator to always correctly determine whether the image is in the specified set. In this way, the training stop condition can be set by setting the correct ratio threshold of the discriminant evaluation result, so that the image generated by the generator is more consistent with the features of the training target in the training set.

[0132] Before training the generator, the discriminator can also be pre-trained. For example, random features are input to the generator to obtain an image, and the discriminator compares the features of the image with the animated organs in the training sample. Figure 1 The generator is evaluated and the parameters in the discriminator are adjusted based on whether the evaluation result is correct, until the discriminator can correctly judge whether the image generated by the generator is consistent with the animated pronunciation organ diagram in the training sample. After the discriminator is trained, the generator can be trained with the discriminator. It is worth noting that the training of the generator and the discriminator can also be carried out synchronously, so that they can constrain each other, so that the image generated by the generator is more consistent with the characteristics of the animated pronunciation organ diagram, and the discriminator can evaluate the image more correctly.

[0133] In a possible implementation, the training samples are obtained by determining the position of the pronunciation organ in each MRI sample image, and generating an animated pronunciation organ corresponding to the position of the pronunciation organ in each MRI sample image to obtain an animated pronunciation organ map.

[0134] The position of each organ can be distinguished by the outline of the color block area in the MRI sample image, and the position of the pronunciation organ can also be identified by a recognition model, or the organ template map can be overlapped with the MRI sample image, and the area in the MRI sample image can be merged based on the organ position of the organ template map, and the color block in the area where the pronunciation organ is located is used as the position of the pronunciation organ.

[0135] In a possible implementation, for each MRI sample image, the organ contour of the MRI sample image is extracted, and the organ contour of each pronunciation organ is filled with the organ image corresponding to the pronunciation organ.

[0136] The organ image can be a cartoon image or a realistic image. In a possible implementation, an organ map can be called from a preset flash animation library, and the organ map corresponding to the pronunciation organ can be filled in the organ outline of each pronunciation organ. It is worth noting that there may be multiple organ maps for the same pronunciation organ in the flash animation library, and an organ map can be automatically selected for filling, or the type of map can be modified according to the user's specification for filling.

[0137] In a possible implementation, for the MRI sample image corresponding to the first frame of the MRI sample video, an organ map is called from a preset flash animation library, and the organ map corresponding to the pronunciation organ is filled in the organ contour of each pronunciation organ; for the MRI sample images corresponding to other video frames, the organ map corresponding to each pronunciation organ in the MRI sample image corresponding to the first frame is called from the flash animation library to fill in the organ contour corresponding to each pronunciation organ.

[0138] That is to say, after filling the first frame with texture, other frames can be filled based on the texture type of the first frame, so that the texture style for the same vocal organs in all animation frames is the same, making the final animation video more natural.

[0139] For example, there are 3 types of textures for the tongue and 4 types of textures for the teeth in the flash animation library. When filling the MRI sample image corresponding to the first frame, the tongue 1 texture is selected for the tongue and the teeth 3 texture is selected for the teeth to fill the contours of the tongue and the teeth respectively. When filling other subsequent frames, the tongue 1 texture can be automatically selected to fill the contour of the tongue, and the teeth 3 texture can be selected to fill the contour of the teeth.

[0140] Considering that there may be deviations in the extraction of organ contours, in a possible implementation, after the organ contours are extracted, the organ contours may be corrected. The organ contours may be corrected frame by frame, or after the organ contours of the first frame are corrected, the organ contours may be tracked by identifying feature points, thereby achieving correction of the organ contours of other frames.

[0141] In a possible implementation, for the MRI sample image corresponding to the first frame of the MRI sample video, the organ contour in the MRI sample image is adjusted based on the MRI sample image so that the pronunciation organ contour corresponds to the feature points in the MRI sample image; for the MRI sample images corresponding to other video frames, feature point tracking is performed on the feature points in the MRI sample image and the feature points in the previous video frame of the MRI sample image, and based on the feature point tracking result, the organ contour in the MRI sample image is automatically adjusted.

[0142] It is worth noting that steps S11 to S15 in this embodiment can all be executed on the user terminal. Optionally, in order to reduce the computing pressure of the terminal, steps S13 and S14 can also be executed on the server. After collecting the user's audio to be evaluated, the user terminal can send the audio to the server. After processing the audio, the server returns the pronunciation evaluation information to the user terminal.

[0143] Through the above technical solution, at least the following technical effects can be achieved:

[0144] By obtaining the audio to be evaluated that the user reads aloud based on the example sentence text, and generating pronunciation evaluation information by generating a pronunciation organ movement video based on the audio to be evaluated and the standard pronunciation organ movement video corresponding to the example sentence text, the user's pronunciation can be evaluated more accurately, thereby more intuitively reflecting whether the user's pronunciation is accurate.

[0145] Figure 3 is a block diagram of a pronunciation evaluation device according to an exemplary disclosed embodiment. Figure 3 As shown, the pronunciation evaluation device 300 includes:

[0146] An example sentence display module 310 is used to display example sentence texts to users;

[0147] An audio collection module 320 is used to collect the audio to be evaluated read aloud by the user based on the example sentence text;

[0148] The video generation module 330 is used to generate a pronunciation organ movement video reflecting the movement of the pronunciation organs when the user reads the example sentence text;

[0149] A pronunciation evaluation module 340, for generating pronunciation evaluation information based on the pronunciation organ action video and the pronunciation organ standard action video corresponding to the example sentence text;

[0150] The evaluation display module 350 is used to display the pronunciation evaluation information to the user.

[0151] In a possible implementation, the pronunciation evaluation information includes at least one of pronunciation scoring information for the user, pronunciation action suggestion information, and a comparison video between the pronunciation organ action video and the pronunciation organ standard action video.

[0152] In a possible implementation, the example sentence display module 310 is used to generate example sentence audio based on the example sentence text; synthesize the example sentence audio and the standard movement video of the pronunciation organs into an example sentence demonstration video; and display the example sentence text and the example sentence demonstration video to the user.

[0153] In a possible implementation, the pronunciation evaluation module 340 is used to obtain action difference information by comparing the pronunciation organ action video with the standard pronunciation organ action video corresponding to the example sentence text; generate pronunciation scoring information based on the action difference information, and / or match the action difference information with preset pronunciation action suggestion information to obtain target action suggestion information that matches the action difference information.

[0154] In a possible implementation, the pronunciation evaluation module is used to, based on the unit text content of the example sentence text, take the video clips representing the same unit text content in the pronunciation organ movement video and the pronunciation organ standard movement video as a group of video clip groups; align the video clips belonging to the pronunciation organ movement video and the pronunciation organ standard movement video in each video clip group; and splice the aligned pronunciation organ movement video and the pronunciation organ standard movement video to obtain the comparison video.

[0155] In a possible implementation, the video generation module 330 is used to convert the audio to be evaluated into an audio feature vector to be processed; input the audio feature vector to be processed into a video generation model to obtain a pronunciation organ movement video corresponding to the audio to be evaluated output by the video generation model; wherein the video generation model is trained in the following manner: constructing model training data based on sample audio and sample pronunciation organ movement videos corresponding to the sample audio; and obtaining the video generation model through training based on the model training data.

[0156] In a possible implementation, the video generation module 330 is also used to divide the example text into unit text sequences; input the unit text sequence into a video feature generation model to obtain a video feature sequence; generate a standard movement video of the pronunciation organs based on the video feature sequence; wherein the video feature generation model is trained in the following manner: divide the sample text into sample unit text sequences; construct model training data based on the sample unit text sequences and the sample video feature sequences of the sample pronunciation organ movement videos corresponding to the sample unit text sequences; and obtain the video feature generation model through training based on the model training data.

[0157] In a possible implementation, the pronunciation organ action video and the pronunciation organ standard action video are pronunciation organ animation videos generated based on magnetic resonance imaging (MRI) videos, and the device also includes a video rendering module, which is used to render the pronunciation organ action video or the pronunciation organ standard action video frame by frame through an animation generation model to obtain a pronunciation organ animation video; wherein, the training samples of the animation generation model include multiple MRI sample images and animated pronunciation organ maps corresponding to each MRI sample image, and the training samples of the animation generation model are obtained in the following manner: determining the position of the pronunciation organ in each MRI sample image; at the position of the pronunciation organ in each MRI sample image, generating an animation organ corresponding to the position of the pronunciation organ to obtain an animated pronunciation organ map.

[0158] The specific steps performed by the above modules have been described in detail in the method part of the embodiment, and will not be repeated here.

[0159] Through the above technical solution, at least the following technical effects can be achieved:

[0160] By obtaining the audio to be evaluated that the user reads aloud based on the example sentence text, and generating pronunciation evaluation information by generating a pronunciation organ movement video based on the audio to be evaluated and the standard pronunciation organ movement video corresponding to the example sentence text, the user's pronunciation can be evaluated more accurately, thereby more intuitively reflecting whether the user's pronunciation is accurate.

[0161] Reference below Figure 4 , which shows a schematic diagram of the structure of an electronic device (such as a user device or a server) 400 suitable for implementing the embodiments of the present disclosure. The terminal device in the embodiments of the present disclosure may include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 4The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0162] like Figure 4 As shown, the electronic device 400 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage device 408 into a random access memory (RAM) 403. In the RAM 403, various programs and data required for the operation of the electronic device 400 are also stored. The processing device 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0163] Typically, the following devices may be connected to the I / O interface 405: an input device 406 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 407 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 408 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 409. The communication device 409 may allow the electronic device 400 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 4 The electronic device 400 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0164] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 409, or installed from the storage device 408, or installed from the ROM 402. When the computer program is executed by the processing device 401, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0165] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0166] In some embodiments, the user terminal and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText TransferProtocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0167] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0168] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains at least two Internet Protocol addresses; sends a node evaluation request including the at least two Internet Protocol addresses to a node evaluation device, wherein the node evaluation device selects an Internet Protocol address from the at least two Internet Protocol addresses and returns it; receives the Internet Protocol address returned by the node evaluation device; wherein the obtained Internet Protocol address indicates an edge node in a content distribution network.

[0169] Alternatively, the computer-readable medium carries one or more programs, and when the one or more programs are executed by the electronic device, the electronic device: receives a node evaluation request including at least two Internet Protocol addresses; selects an Internet Protocol address from the at least two Internet Protocol addresses; and returns the selected Internet Protocol address; wherein the received Internet Protocol address indicates an edge node in a content distribution network.

[0170] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0171] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0172] The modules involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a module does not limit the module itself in some cases. For example, the first acquisition module may also be described as a "module for acquiring at least two Internet Protocol addresses".

[0173] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0174] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0175] According to one or more embodiments of the present disclosure, Example 1 provides a pronunciation evaluation method, which includes: displaying an example sentence text to a user; collecting audio to be evaluated that is read aloud by a user based on the example sentence text; generating a pronunciation organ movement video based on the audio to be evaluated; generating a pronunciation organ movement video that reflects the movement of the pronunciation organs when the user reads the example sentence text; and displaying the pronunciation evaluation information to the user.

[0176] According to one or more embodiments of the present disclosure, Example 2 provides the method of Example 1, wherein the pronunciation evaluation information includes at least one of pronunciation scoring information for the user, pronunciation action suggestion information, and a comparison video of the pronunciation organ action video and the standard pronunciation organ action video.

[0177] According to one or more embodiments of the present disclosure, Example 3 provides the method of Example 1, wherein displaying the example sentence text to the user comprises: generating an example sentence audio based on the example sentence text; synthesizing the example sentence audio with the standard movement video of the pronunciation organs into an example sentence demonstration video; and displaying the example sentence text and the example sentence demonstration video to the user.

[0178] According to one or more embodiments of the present disclosure, Example 4 provides the method of Example 2, wherein, when the pronunciation evaluation information includes the pronunciation scoring information and / or the pronunciation action suggestion information, the pronunciation evaluation information is generated based on the pronunciation organ action video and the standard pronunciation organ action video corresponding to the example sentence text, including: obtaining action difference information by comparing the pronunciation organ action video with the standard pronunciation organ action video corresponding to the example sentence text; generating pronunciation scoring information according to the action difference information, and / or matching the action difference information with preset pronunciation action suggestion information to obtain target action suggestion information matching the action difference information.

[0179] According to one or more embodiments of the present disclosure, Example 5 provides the method of Example 2, and the comparison video is generated in the following manner: based on the unit text content of the example sentence text, the video clips representing the same unit text content in the pronunciation organ movement video and the pronunciation organ standard movement video are taken as a group of video clip groups; the video clips belonging to the pronunciation organ movement video and the pronunciation organ standard movement video in each video clip group are aligned; the aligned pronunciation organ movement video and the pronunciation organ standard movement video are spliced ​​to obtain the comparison video.

[0180] According to one or more embodiments of the present disclosure, Example 6 provides the method of Example 1, which generates a pronunciation organ action video that reflects the action of the pronunciation organs when the user reads the example text, including: converting the audio to be evaluated into an audio feature vector to be processed; inputting the audio feature vector to be processed into a video generation model to obtain a pronunciation organ action video corresponding to the audio to be evaluated output by the video generation model; wherein the video generation model is trained in the following manner: constructing model training data based on sample audio and sample pronunciation organ action videos corresponding to the sample audio; and training the video generation model based on the model training data.

[0181] According to one or more embodiments of the present disclosure, Example 7 provides the method of Example 1, wherein the standard movement video of the pronunciation organs is generated in the following manner: segmenting the example sentence text into unit text sequences; inputting the unit text sequence into a video feature generation model to obtain a video feature sequence; generating a standard movement video of the pronunciation organs based on the video feature sequence; wherein the video feature generation model is trained in the following manner: segmenting the sample text into sample unit text sequences; constructing model training data according to the sample unit text sequences and the sample video feature sequences of the sample pronunciation organ movement videos corresponding to the sample unit text sequences; and obtaining the video feature generation model through training according to the model training data.

[0182] According to one or more embodiments of the present disclosure, Example 8 provides the method of Examples 1-7, wherein the pronunciation organ action video and the pronunciation organ standard action video are pronunciation organ animation videos generated based on nuclear magnetic resonance MRI video, and the method further includes: rendering the pronunciation organ action video or the pronunciation organ standard action video frame by frame through an animation generation model to obtain a pronunciation organ animation video; wherein the training samples of the animation generation model include multiple MRI sample images and animated pronunciation organ maps corresponding to each MRI sample image, and the training samples of the animation generation model are obtained in the following manner: determining the position of the pronunciation organ in each MRI sample image; at the position of the pronunciation organ in each MRI sample image, generating animated pronunciation organs corresponding to the position of the pronunciation organ to obtain an animated pronunciation organ map.

[0183] According to one or more embodiments of the present disclosure, Example 9 provides a pronunciation evaluation device, which includes: an example sentence display module for displaying an example sentence text to a user; an audio acquisition module for acquiring audio to be evaluated that is read aloud by a user based on the example sentence text; a video generation module for generating a pronunciation organ action video that reflects the action of the pronunciation organs when the user reads the example sentence text; a pronunciation evaluation module for generating pronunciation evaluation information based on the pronunciation organ action video and the standard pronunciation organ action video corresponding to the example sentence text; and an evaluation display module for displaying the pronunciation evaluation information to the user.

[0184] According to one or more embodiments of the present disclosure, Example 10 provides the device of Example 9, wherein the pronunciation evaluation information includes at least one of pronunciation scoring information for the user, pronunciation action suggestion information, and a comparison video of the pronunciation organ action video and the standard pronunciation organ action video.

[0185] According to one or more embodiments of the present disclosure, Example 11 provides the device of Example 9, wherein the example sentence display module is used to generate example sentence audio based on the example sentence text; synthesize the example sentence audio and the standard movement video of the pronunciation organs into an example sentence demonstration video; and display the example sentence text and the example sentence demonstration video to the user.

[0186] According to one or more embodiments of the present disclosure, Example 12 provides the device of Example 10, wherein the pronunciation evaluation module is used to obtain action difference information by comparing the pronunciation organ action video with the standard pronunciation organ action video corresponding to the example sentence text; generate pronunciation scoring information according to the action difference information, and / or, match the action difference information with preset pronunciation action suggestion information to obtain target action suggestion information matching the action difference information.

[0187] According to one or more embodiments of the present disclosure, Example 13 provides the device of Example 10, wherein the pronunciation evaluation module is further used to, based on the unit text content of the example sentence text, take the video clips representing the same unit text content in the pronunciation organ action video and the pronunciation organ standard action video as a group of video clip groups; align the video clips belonging to the pronunciation organ action video and the pronunciation organ standard action video in each video clip group; and splice the aligned pronunciation organ action video and the pronunciation organ standard action video to obtain the comparison video.

[0188] According to one or more embodiments of the present disclosure, Example 14 provides the apparatus of Example 9, wherein the video generation module is used to convert the audio to be evaluated into an audio feature vector to be processed; the audio feature vector to be processed is input into a video generation model to obtain a pronunciation organ movement video corresponding to the audio to be evaluated output by the video generation model; wherein the video generation model is trained in the following manner: model training data is constructed based on sample audio and sample pronunciation organ movement videos corresponding to the sample audio; and the video generation model is trained based on the model training data.

[0189] According to one or more embodiments of the present disclosure, Example 15 provides the apparatus of Example 9, wherein the video generation module is further used to segment the example sentence text into unit text sequences; input the unit text sequences into a video feature generation model to obtain a video feature sequence; and generate a standard movement video of pronunciation organs based on the video feature sequence; wherein the video feature generation model is trained in the following manner: segmenting the sample text into sample unit text sequences; constructing model training data according to the sample unit text sequences and the sample video feature sequences of the sample pronunciation organ movement videos corresponding to the sample unit text sequences; and training the video feature generation model according to the model training data.

[0190] According to one or more embodiments of the present disclosure, Example 16 provides the apparatus of Examples 9-15, wherein the pronunciation organ action video and the pronunciation organ standard action video are pronunciation organ animation videos generated based on nuclear magnetic resonance MRI video, and the apparatus also includes a video rendering module, which is used to render the pronunciation organ action video or the pronunciation organ standard action video frame by frame through an animation generation model to obtain a pronunciation organ animation video; wherein the training samples of the animation generation model include multiple MRI sample images and animated pronunciation organ maps corresponding to each MRI sample image, and the training samples of the animation generation model are obtained in the following manner: determining the position of the pronunciation organ in each MRI sample image; at the position of the pronunciation organ in each MRI sample image, generating animated pronunciation organs corresponding to the position of the pronunciation organ to obtain an animated pronunciation organ map.

[0191] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0192] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0193] Although the subject matter has been described in language specific to structural features and / or method logic actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims. Regarding the device in the above embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.

Claims

1. A pronunciation evaluation method, characterized in that: The method comprises: Display example text to the user; Collecting the audio to be evaluated read aloud by the user based on the example sentence text; generating a pronunciation organ movement video reflecting the movement of the pronunciation organs when the user reads the example sentence text; Generate pronunciation evaluation information based on the pronunciation organ action video and the pronunciation organ standard action video corresponding to the example sentence text; Displaying the pronunciation evaluation information to the user, the pronunciation evaluation information including pronunciation scoring information and / or pronunciation action suggestion information; The method of generating pronunciation evaluation information based on the pronunciation organ action video and the pronunciation organ standard action video corresponding to the example sentence text by others includes: By comparing the pronunciation organ action video with the pronunciation organ standard action video corresponding to the example sentence text, action difference information is obtained, wherein the action difference information is difference information of motion trajectories of feature points of the pronunciation organ; Generate pronunciation scoring information according to the action difference information, and / or, match the action difference information with preset pronunciation action suggestion information to obtain target action suggestion information matching the action difference information; Dividing the audio to be evaluated according to a preset pronunciation evaluation granularity to obtain a plurality of sub-audios to be evaluated; In each frame of the video of the movement of the vocal organs, the position coordinates of the feature points of the vocal organs corresponding to each of the sub-audios to be evaluated are determined, and a motion trajectory segment of the feature points of the vocal organs corresponding to each of the sub-audios to be evaluated is obtained; For each of the sub-audio to be evaluated, similarity calculation is performed between the motion trajectory segment of the feature point of the pronunciation organ corresponding to the sub-audio to be evaluated and the corresponding preset motion trajectory segment of the feature point, so as to obtain a first similarity value corresponding to the sub-audio to be evaluated; Determine a target first similarity value that is less than a preset threshold, and determine a target sub-audio to be evaluated corresponding to the target first similarity value; For each vocal organ feature point motion trajectory segment, similarity calculation is performed between the feature point motion trajectory segment and a feature point preset motion trajectory segment corresponding to the vocal organ in the example sentence text to obtain a second similarity value; A target second similarity value that is less than a threshold is determined, and a target pronunciation organ is determined according to the target second similarity value.

2. The method according to claim 1, characterized in that: The presenting the example sentence text to the user comprises: Generate example sentence audio based on the example sentence text; Combining the example sentence audio with the standard movement video of the pronunciation organs into an example sentence demonstration video; The example sentence text and the example sentence demonstration video are displayed to the user.

3. The method according to claim 1, characterized in that The step of generating a pronunciation organ motion video reflecting the motion of the pronunciation organ when the user reads the example sentence text includes: converting the audio to be evaluated into an audio feature vector to be processed; Inputting the audio feature vector to be processed into a video generation model to obtain a pronunciation organ movement video corresponding to the audio to be evaluated output by the video generation model; The video generation model is trained in the following way: Constructing model training data based on sample audio and sample articulatory organ movement videos corresponding to the sample audio; The video generation model is obtained by training according to the model training data.

4. The method according to claim 1, characterized in that: The standard motion video of the vocal organs is generated in the following way: Segmenting the example sentence text into unit text sequences; Inputting the unit text sequence into a video feature generation model to obtain a video feature sequence; Generate a standard motion video of the vocal organs based on the video feature sequence; The video feature generation model is trained in the following way: Segment the sample text into sample unit text sequences; Constructing model training data according to a sample unit text sequence and a sample video feature sequence of a sample pronunciation organ action video corresponding to the sample unit text sequence; The video feature generation model is obtained by training according to the model training data.

5. The method according to any one of claims 1 to 4, characterized in that: The pronunciation organ action video and the pronunciation organ standard action video are pronunciation organ animation videos generated based on a nuclear magnetic resonance MRI video, and the method further includes: The video of the movement of the vocal organs or the video of the standard movement of the vocal organs are rendered frame by frame through an animation generation model to obtain an animation video of the vocal organs; The training samples of the animation generation model include a plurality of MRI sample images and animation articulatory organ diagrams corresponding to the MRI sample images. The training samples of the animation generation model are obtained in the following manner: determining the location of the vocal organ in each MRI sample image; At the position of the sound organ in each MRI sample image, an animated sound organ corresponding to the position of the sound organ is generated to obtain an animated sound organ map.

6. A pronunciation evaluation device, characterized in that: The device comprises: An example sentence display module is used to display example sentence texts to users; An audio collection module, used to collect the audio to be evaluated read aloud by the user based on the example sentence text; A video generation module, used to generate a pronunciation organ movement video reflecting the movement of the pronunciation organs when the user reads the example sentence text; A pronunciation evaluation module, used for generating pronunciation evaluation information based on the pronunciation organ action video and the pronunciation organ standard action video corresponding to the example sentence text; An evaluation display module, used to display the pronunciation evaluation information to the user, wherein the pronunciation evaluation information includes pronunciation scoring information and / or pronunciation action suggestion information; The pronunciation evaluation module is further used for: By comparing the pronunciation organ action video with the pronunciation organ standard action video corresponding to the example sentence text, action difference information is obtained, wherein the action difference information is difference information of motion trajectories of feature points of the pronunciation organ; Generate pronunciation scoring information according to the action difference information, and / or, match the action difference information with preset pronunciation action suggestion information to obtain target action suggestion information matching the action difference information; Dividing the audio to be evaluated according to a preset pronunciation evaluation granularity to obtain a plurality of sub-audios to be evaluated; In each frame of the video of the movement of the vocal organs, the position coordinates of the feature points of the vocal organs corresponding to each of the sub-audios to be evaluated are determined, and a motion trajectory segment of the feature points of the vocal organs corresponding to each of the sub-audios to be evaluated is obtained; For each of the sub-audio to be evaluated, similarity calculation is performed between the motion trajectory segment of the feature point of the pronunciation organ corresponding to the sub-audio to be evaluated and the corresponding preset motion trajectory segment of the feature point, so as to obtain a first similarity value corresponding to the sub-audio to be evaluated; Determine a target first similarity value that is less than a preset threshold, and determine a target sub-audio to be evaluated corresponding to the target first similarity value; For each vocal organ feature point motion trajectory segment, similarity calculation is performed between the feature point motion trajectory segment and a feature point preset motion trajectory segment corresponding to the vocal organ in the example sentence text to obtain a second similarity value; A target second similarity value that is less than a threshold is determined, and a target pronunciation organ is determined according to the target second similarity value.

7. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processing device, the steps of the method described in any one of claims 1 to 5 are implemented.

8. An electronic device, characterized in that: include: a storage device having a computer program stored thereon; A processing device, configured to execute the computer program in the storage device to implement the steps of the method according to any one of claims 1 to 5.