English pronunciation teaching method and system based on big model

Abstract

The present invention provides an English pronunciation teaching method and system based on a large model, comprising: collecting pronunciation teaching training data, parsing the pronunciation teaching training data, obtaining pronunciation audio and pronunciation video corresponding to the pronunciation audio, and capturing a three-dimensional facial image of a teaching speaker in the pronunciation video; using the three-dimensional facial image as the input of a first distillation model based on a generative adversarial network, performing image knowledge distillation, outputting facial features, and inputting the pronunciation audio into the first distillation model for audio knowledge distillation, outputting sound features, aligning the time axis of the sound features and the facial features, and obtaining an alignment result; obtaining a joint loss function value according to the alignment result, and updating the parameters of the first distillation model according to the joint loss function value to obtain a second distillation model, so that a corresponding audio and video teaching animation is generated according to the second distillation model; the present invention can improve the user's experience in learning English pronunciation.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an English pronunciation teaching method and system based on a large model. Background Art

[0002] At present, existing English pronunciation teaching can only provide users with standard pronunciation, or provide word scene examples, but cannot provide video images of word scene examples. Users need to search by themselves, or manually organize word scene examples. However, organizing word pronunciation requires capturing massive amounts of audio and video, which consumes a lot of storage space resources and computing resources to extract the required scene example images, making traditional English pronunciation teaching inefficient and the user experience poor. Summary of the invention

[0003] The purpose of the present invention is to address the deficiencies of the above-mentioned existing related technologies and to propose an English pronunciation teaching method and system based on a large model, which can improve the efficiency of English pronunciation teaching and reduce resource consumption, thereby improving user experience.

[0004] In a first aspect, the present invention provides an English pronunciation teaching method based on a large model, comprising:

[0005] Collecting pronunciation teaching training data, parsing the pronunciation teaching training data, obtaining pronunciation audio and a pronunciation video corresponding to the pronunciation audio, and capturing a three-dimensional facial image of a teaching speaker in the pronunciation video;

[0006] Using the three-dimensional facial image as an input of a first distillation model based on a generative adversarial network, performing image knowledge distillation, and outputting facial features, and inputting the pronunciation audio into the first distillation model to perform audio knowledge distillation, outputting sound features, and aligning the time axis of the sound features with the facial features to obtain an alignment result;

[0007] According to the alignment result, a joint loss function value is obtained, and according to the joint loss function value, the parameters of the first distillation model are updated to obtain a second distillation model, so that a corresponding audio and video teaching animation is generated according to the second distillation model.

[0008] The present invention adopts a distillation model based on a generative adversarial network to extract features from the pronunciation video and pronunciation audio in the pronunciation teaching training data respectively, and updates the parameters of the distillation model through a joint loss function value, thereby constructing a distillation model for processing multimodal data to reduce the use of storage space resources and computing resources. There is no need to manually organize word scene examples, thereby improving the efficiency of English pronunciation teaching and further improving the user experience.

[0009] Further, the facial features include: a first facial feature and a second facial feature; the first distillation model includes: a plurality of first image teacher models based on a generative network and a first image student model based on a generative network; the three-dimensional facial image is used as an input of the first distillation model based on a generative adversarial network to perform image knowledge distillation and output facial features, including:

[0010] The three-dimensional facial image is used as the input of the multiple first image teacher models respectively, and image knowledge distillation is performed after expansion in depth, width and channel respectively, and multiple first facial features with different spatial features are output accordingly. At the same time, the three-dimensional facial image is used as the input of the first image student model to perform image knowledge compression and output the second facial feature.

[0011] The present invention uses an image teacher model to expand the depth, width and channels of the three-dimensional facial image to capture the image information of the three-dimensional facial image in different spatial dimensions, which helps to improve the training accuracy of the first distillation model and thus improve the user experience.

[0012] Furthermore, the first distillation model further includes: a first audio teacher model based on a generative network; the pronunciation audio is input into the first distillation model for audio knowledge distillation, the sound features are output, and the time axis of the sound features and the facial features are aligned to obtain an alignment result, including:

[0013] Based on the image information entropy of different facial parts, corresponding weights are given to the multiple first facial features respectively, and multiple facial weight results are obtained accordingly, and the pronunciation audio is used as the input of the first audio teacher model to perform audio knowledge distillation, and the corresponding sound features are output;

[0014] The sound features are introduced into the multiple facial weight results for time axis alignment to synchronize the sound information in the time dimension and the image information in the space dimension, and obtain an alignment result of the facial weight results; wherein the facial weight results and the sound features are in a one-to-one relationship, and an alignment result is generated.

[0015] The present invention uses sound features to align the time axis of the facial weight results to synchronize the sound information in the time dimension and the image information in the space dimension. Compared with the existing related technology of automatically filling in missing continuous frames, the present invention adaptively adjusts the facial features of the facial weight results according to the sound features of the pronunciation audio, which can ensure that the facial weight results are not distorted, thereby improving the user experience of using intelligent voice teaching based on a multimodal large model.

[0016] Furthermore, the first distillation model further includes: a first identification model based on a discriminant network; and obtaining a joint loss function value according to the alignment result includes:

[0017] The three-dimensional facial image and the plurality of alignment results are used as inputs of the first identification model, and the plurality of identification results are outputted correspondingly, and the plurality of alignment results are combined with the plurality of identification results to obtain a plurality of generative adversarial loss values; the alignment result and the identification result are in a one-to-one relationship, and a generative adversarial loss value is generated;

[0018] The network structure similarities and network output similarities of the multiple first image teacher models and the first image student models are respectively obtained, and a knowledge distillation loss value is obtained based on the network structure similarity and the network output similarity, and the multiple generative adversarial loss values ​​and the knowledge distillation loss value are combined to obtain a joint loss function value.

[0019] Furthermore, the three-dimensional facial image and the plurality of alignment results are used as inputs of the first identification model, and the corresponding plurality of identification results outputted include:

[0020] Each time, one alignment result is taken out from the multiple alignment results without replacement and the three-dimensional facial image as a set of input data, and the input data is input into the first identification model of the shared decoder and the branch parallel encoder, and one identification result is output from the corresponding branch parallel encoder each time; the number of the branch parallel encoders is the same as that of the first image teacher model and they correspond one to one.

[0021] The present invention adopts a shared decoder to improve the decoding efficiency of three-dimensional facial images and corresponding alignment results, and adopts a branched parallel encoder to process the extended operation of the first image teacher model in a one-to-one relationship, and can perform one-to-one information feedback on the first image teacher model of image information of different spatial dimensions, which helps to improve the training accuracy of the first distillation model, thereby improving the user experience.

[0022] Further, the second distillation model includes: a second image teacher model, a second image student model, a second audio teacher model and a second identification model; the first distillation model is updated according to the joint loss function value to obtain a second distillation model, including:

[0023] According to the joint loss function value, the first image teacher model, the first image student model, the first audio teacher model and the first identification model are simultaneously updated online parameters to obtain the second image teacher model, the second image student model, the second audio teacher model and the second identification model accordingly.

[0024] Further, generating a corresponding audio and video teaching animation according to the second distillation model includes:

[0025] Acquire a real-time target word, and collect the target pronunciation audio and target pronunciation video of the speaker of the real-time target word, use the target pronunciation audio as the input of the second audio teacher model, output the target sound feature, and use the target pronunciation video as the input of the second image student model, output the target image feature, and combine the target sound feature and the target image feature to construct a three-dimensional demonstration audio and video teaching animation.

[0026] Further, the combining of the target sound feature and the target image feature to construct a three-dimensional audio and video teaching animation includes:

[0027] Collecting the user's character features, the character features including: tongue features, facial features and acoustic features;

[0028] The target sound feature is embedded into the acoustic feature to obtain an audio demonstration feature with the user as the demonstration object, and the target image feature is respectively embedded into the tongue image feature and the facial image feature to obtain a facial image demonstration feature with the user as the demonstration object. Based on the audio demonstration feature and the facial image demonstration feature, an audio and video teaching animation with the user as the demonstration object is generated.

[0029] The present invention uses the user as the demonstration object to generate audio demonstration features and facial image demonstration features, which can facilitate users to find their own pronunciation weaknesses through their own standard demonstration audio and video, and can adapt to the English pronunciation characteristics of different users, thereby improving user experience.

[0030] Further, after generating the audio and video teaching animation with the user as the demonstration object, it also includes:

[0031] Collecting the user's real-time pronunciation audio and video, and analyzing the real-time pronunciation audio and video to obtain a real-time three-dimensional facial image corresponding to the real-time pronunciation audio and the real-time pronunciation video;

[0032] Acquiring an acoustic similarity between the real-time pronunciation audio and the audio demonstration feature, and acquiring a facial action similarity between the real-time three-dimensional facial image and the facial image demonstration feature;

[0033] According to the acoustic similarity and the facial action similarity, the audio segment and the facial action video segment to be taught are respectively extracted from the audio demonstration feature and the audio demonstration feature, so that the user is prompted to learn according to the audio segment and the facial action video segment.

[0034] In a second aspect, the present invention provides an English pronunciation teaching system based on a large model, comprising: a preprocessing module, a feature extraction module and a teaching module; wherein,

[0035] The preprocessing module is used to collect pronunciation teaching training data, analyze the pronunciation teaching training data, obtain pronunciation audio and pronunciation video corresponding to the pronunciation audio, and capture the three-dimensional facial image of the teaching speaker in the pronunciation video;

[0036] The feature extraction module is used to use the three-dimensional facial image as an input of a first distillation model based on a generative adversarial network to perform image knowledge distillation and output facial features, and input the pronunciation audio into the first distillation model to perform audio knowledge distillation and output sound features, and align the time axis of the sound features with the facial features to obtain an alignment result;

[0037] The teaching module is used to obtain a joint loss function value according to the alignment result, and to update the parameters of the first distillation model according to the joint loss function value to obtain a second distillation model, so as to generate a corresponding audio and video teaching animation according to the second distillation model. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 is a flow chart of an English pronunciation teaching method based on a large model provided in this embodiment;

[0039] Figure 2 is a schematic diagram of the structure of a distillation model I based on a generative adversarial network provided in this embodiment;

[0040] Figure 3 is a schematic diagram of the structure of the distillation model II based on the generative adversarial network provided in this embodiment;

[0041] Figure 4 is a schematic diagram of the structure of the distillation model III based on the generative adversarial network provided in this embodiment;

[0042] Figure 5 is a schematic diagram of the structure of the distillation model IV based on the generative adversarial network provided in this embodiment;

[0043] Figure 6 is a schematic diagram of the structure of the distillation model IV based on the generative adversarial network provided in this embodiment;

[0044] Figure 7 It is a schematic diagram of a process of generating audio and video teaching animations based on a distillation model of a generative adversarial network provided in this embodiment;

[0045] Figure 8 It is a structural diagram of an English pronunciation teaching system based on a large model provided in this embodiment. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] It is worth noting that the existing English pronunciation teaching can only provide users with standard pronunciation and word scenario examples, but cannot provide teaching animations suitable for different English pronunciation levels, which makes the user experience low. If the existing English pronunciation teaching provides a large number of teaching animations, it also faces the problem of insufficient storage space and computing space. Based on this, the present invention provides an English pronunciation teaching method and system based on a large model, which uses a distillation model based on a generative adversarial network to realize a multimodal large model to process speech and images at the same time, provide users with adaptive pronunciation teaching animations, and can compress the network model of English pronunciation teaching, thereby reducing resource consumption and achieving efficiency and cost reduction.

[0048] In order to better illustrate the technical solution of the present invention, it will be described in detail from the following embodiments.

[0049] Example 1

[0050] See also Figure 1 , is a flow chart of an English pronunciation teaching method based on a large model provided in this embodiment, including: steps S11 to S13, specifically:

[0051] Step S11, collecting pronunciation teaching training data, parsing the pronunciation teaching training data, obtaining pronunciation audio and a pronunciation video corresponding to the pronunciation audio, and capturing a three-dimensional facial image of a teaching speaker in the pronunciation video.

[0052] In some embodiments, the pronunciation teaching training data can be captured from authorized movie materials, teaching materials, music and video materials, or hot events.

[0053] In some embodiments, the step of acquiring a three-dimensional facial image may be based on existing three-dimensional facial image modeling technology.

[0054] In some embodiments, capturing a three-dimensional facial image of a teaching speaker in a pronunciation video includes: selecting a target frame in the pronunciation video according to a preset frame step, then capturing a two-dimensional facial image of the teaching speaker from the target frame, and then using three-dimensional flexible object modeling technology to perform three-dimensional reconstruction of the two-dimensional facial image to obtain a three-dimensional point distribution model corresponding to the two-dimensional facial image; the three-dimensional point distribution model includes the three-dimensional facial image of the teaching speaker.

[0055] In some embodiments, a two-dimensional facial image is reconstructed in three dimensions to obtain a three-dimensional point distribution model corresponding to the two-dimensional facial image, including: using a non-rigid motion structure algorithm to perform a three-dimensional transformation on the two-dimensional facial image to obtain a reference three-dimensional point distribution model corresponding to the two-dimensional facial image; wherein the reference three-dimensional point distribution model includes: average position information of multiple facial key points and multiple deformation feature vectors; then using a three-dimensional Prucker analysis method, the reference three-dimensional point distribution model is aligned to obtain the aligned three-dimensional point distribution model and transformation parameters of each deformation feature vector; and using a principal component analysis method, the multiple deformation feature vectors are sorted and analyzed to determine at least one target deformation feature vector of the multiple deformation feature vectors; based on the aligned three-dimensional point distribution model and the at least one target deformation feature vector, the three-dimensional point distribution model corresponding to the two-dimensional facial image is determined.

[0056] Step S12: use the three-dimensional facial image as the input of the first distillation model based on the generative adversarial network, perform image knowledge distillation, output facial features, and input the pronunciation audio into the first distillation model to perform audio knowledge distillation, output sound features, align the time axis of the sound features with the facial features, and obtain an alignment result.

[0057] It is worth noting that the first distillation model is based on the initial distillation model of the generative adversarial network, and the second distillation model is the distillation model obtained after training the initial distillation model.

[0058] In some embodiments, the facial feature includes: a first facial feature and a second facial feature.

[0059] In some embodiments, the first facial features and the second facial features both include: tongue features, facial features, and acoustic features of the teaching speaker in the pronunciation video. The difference is that the first facial features are facial features output by image knowledge distillation of the three-dimensional facial image through the initial first image teacher model, and the second facial features are facial features output by image knowledge distillation of the three-dimensional facial image through the initial first image student model.

[0060] It is worth noting that the first image teacher model is an initial image teacher model, which is used to process the knowledge distillation of the image to guide the training of the first image student model, and the first image student model is an initial image student model, which is used to compress the image knowledge with a smaller model, so that the model framework of the obtained first image student model is smaller than that of the first image teacher model. In addition, the second image teacher model and the second image student model mentioned later are the image teacher model and the image student model of the initial image teacher model and the initial image student model after training, respectively.

[0061] In some embodiments, the first distillation model includes: a plurality of first image teacher models based on a generative network and a first image student model based on a generative network.

[0062] In some embodiments, a three-dimensional facial image is used as the input of a first distillation model based on a generative adversarial network to perform image knowledge distillation and output facial features, including: using the three-dimensional facial image as the input of multiple first image teacher models, performing image knowledge distillation after expanding them in depth, width and channel, and correspondingly outputting multiple first facial features with different spatial features, and at the same time using the three-dimensional facial image as the input of the first image student model to perform image knowledge compression and output second facial features.

[0063] In some embodiments, the three-dimensional facial image is used as the input of the first first image teacher model for depth expansion, and the first facial features after depth expansion are output; at the same time, the three-dimensional facial image is used as the input of the second first image teacher model for width expansion, and the first facial features after width expansion are output; and the three-dimensional facial image is used as the input of the third first image teacher model for channel expansion, and the first facial features after channel expansion are output.

[0064] It is worth noting that multiple first image teacher models perform expansions in different dimensions and are independent networks in parallel during network computation.

[0065] In some embodiments, channel expansion is achieved by multiplying a channel expansion coefficient with each channel in a convolutional layer of the first image teacher model.

[0066] In some embodiments, depth expansion and width expansion are achieved by inserting a residual neural network (e.g., ResNet) during upsampling or downsampling.

[0067] It is worth mentioning that by implementing first image teacher models with different structures through width expansion, depth expansion and channel expansion, image information in different spatial dimensions can be obtained, which helps to complement the image information spatially according to the established joint loss function, thereby improving the accuracy of video animations for English phonetic teaching, and further improving the accuracy of English phonetic teaching based on multimodal large models and improving user experience.

[0068] In some embodiments, the first distillation model further includes: a first audio teacher model based on a generative network.

[0069] In some embodiments, the pronunciation audio is input into the first distillation model for audio knowledge distillation, the sound features are output, and the time axis of the sound features and the facial features are aligned to obtain an alignment result, including: based on the image information entropy of different facial parts, corresponding weights are given to multiple first facial features respectively, and multiple facial weight results are obtained accordingly, and the pronunciation audio is used as the input of the first audio teacher model for audio knowledge distillation, and the corresponding sound features are output; the sound features are introduced for multiple facial weight results to align the time axis to synchronize the sound information in the time dimension and the image information in the space dimension, and obtain the alignment result of the facial weight results; wherein the facial weight result and the sound feature are in a one-to-one relationship, and an alignment result is generated.

[0070] It is worth noting that, in fact, the outputs of multiple first image teacher models are weighted according to the corresponding image information entropy to obtain multiple facial weight results, and then the sound features are used to synchronize the time axis for each facial weight result.

[0071] In some embodiments, one or more consecutive frames of image data are taken from the pronunciation video. The accuracy of English teaching can be guaranteed by taking each frame of image data, but this is at the expense of computing resources. Therefore, in a preferred embodiment, image data is taken at intervals of multiple frames, and then a three-dimensional facial image is extracted from the image data. At this time, another problem will arise: the pronunciation audio is continuous data. If multiple frames of image data are taken, the pronunciation audio and the facial weight result length will be inconsistent. Therefore, it is necessary to align the facial weight result with the time axis according to the length of the sound feature to synchronize the sound information in the time dimension and the image information in the space dimension. Compared with automatically filling in missing continuous frames, this embodiment adaptively adjusts the facial features of the facial weight result according to the sound features of the pronunciation audio, which can ensure that the facial weight result is not distorted, thereby improving the user's experience of using intelligent voice teaching based on a multimodal large model.

[0072] Step S13: obtaining a joint loss function value according to the alignment result, and updating the parameters of the first distillation model according to the joint loss function value to obtain a second distillation model, so as to generate a corresponding audio and video teaching animation according to the second distillation model.

[0073] In some embodiments, the first distillation model further includes: a first identification model based on a discriminant network.

[0074] It is worth noting that the first identification model is an initial identification model, and the second identification model mentioned later is a identification model obtained after the initial identification model completes training.

[0075] In some embodiments, a joint loss function value is obtained based on the alignment result, including: taking a three-dimensional facial image and multiple alignment results as inputs of a first identification model, corresponding to the output of multiple identification results, combining multiple alignment results with multiple identification results to obtain multiple generative adversarial loss values; the alignment result and the identification result are in a one-to-one relationship, and a generative adversarial loss value is generated; the network structure similarity and network output similarity of multiple first image teacher models and first image student models are obtained respectively, and a knowledge distillation loss value is obtained based on the network structure similarity and the network output similarity, and multiple generative adversarial loss values ​​and knowledge distillation loss values ​​are combined to obtain a joint loss function value.

[0076] In some embodiments, obtaining the network structure similarity between the first image teacher model and the first image student model includes: comparing the similarity between the first facial feature output by the first image teacher model and the second facial feature output by the first image student model.

[0077] In some embodiments, the network structure similarity can be expressed as:

[0078] ;

[0079] in, and are respectively a first facial feature output by the first image teacher model and a second facial feature output by the first image student model; They are respectively the operations of width expansion, depth expansion and channel expansion; and is a preset constant; Is the first image teacher model The mean of the first facial feature output, is the mean of the second facial feature output by the student model of the first image; is the covariance of the first facial feature and the second facial feature; Is the first image teacher model The variance of the first facial feature output; is the variance of the second facial feature output by the student model for the first image.

[0080] In some embodiments, obtaining the network output similarity between the first image teacher model and the first image student model includes: comparing the similarity of the features of all layers of the first image teacher model on the channel with the features of all layers of the first image student model on the channel one by one, that is, calculating the network output similarity between the first image teacher model and the first image student model according to the channel.

[0081] In some embodiments, a three-dimensional facial image and a plurality of alignment results are used as inputs of a first identification model, and the corresponding plurality of identification results are outputted, including: taking out one alignment result and the three-dimensional facial image from the plurality of alignment results without replacement each time as a set of input data, inputting the input data into a shared decoder and a first identification model of a branched parallel encoder, and outputting one identification result from the corresponding branched parallel encoder each time; the number of branched parallel encoders is the same as that of the first image teacher model and corresponds one to one.

[0082] It is worth noting that the first identification model is the initial identification model, and its structure includes a cascaded decoder and an encoder, and the encoder is multi-branch parallel; wherein, the decoder is shared by multiple first image teacher models, and one encoder corresponds to one first image teacher model.

[0083] This embodiment uses a shared decoder to improve the decoding efficiency of three-dimensional facial images and corresponding alignment results, and uses a branched parallel encoder to process the extended operation of the first image teacher model in a one-to-one relationship, and can perform one-to-one information feedback on the first image teacher model of image information of different spatial dimensions, which helps to improve the training accuracy of the first distillation model, thereby improving the user experience.

[0084] In some embodiments, the second distillation model includes: a second image teacher model, a second image student model, a second audio teacher model, and a second discrimination model.

[0085] It is worth noting that the second image teacher model is the image teacher model of the initial image teacher model after training, the second image student model is the image student model of the initial image student model after training, the second audio teacher model is the audio teacher model of the initial audio teacher model after training, and the second identification model is the identification model of the initial identification model after training.

[0086] In some embodiments, the parameters of the first distillation model are updated according to the joint loss function value to obtain a second distillation model, including: according to the joint loss function value, the first image teacher model, the first image student model, the first audio teacher model and the first identification model are simultaneously updated online parameters to obtain the second image teacher model, the second image student model, the second audio teacher model and the second identification model accordingly.

[0087] In some embodiments, see Figure 2 , is a schematic diagram of the structure of the distillation model I based on the generative adversarial network provided in this embodiment. Figure 2In the present invention, a first distillation model based on a generative adversarial network includes: a first image teacher network, a first image student network and a first audio teacher model based on a generative network, and a first identification model based on a discriminative network; wherein the first image teacher network performs depth expansion, width expansion and channel expansion on the three-dimensional facial image, that is, the first image teacher network includes three parallel sub-image teacher networks, which respectively collect different information of the input in the spatial dimension, and according to the pre-established knowledge distillation loss function, the output of the first image student model is combined to complement the spatial information and the temporal information, and the knowledge distillation loss function is used to measure the complementary result.

[0088] exist Figure 2 It should also be noted that each input of the decoder of the first identification model based on the discriminant network is a different alignment result and a three-dimensional facial image as a set of input data for processing, and then enters the corresponding multi-branch parallel decoder, outputs the corresponding discrimination result, and obtains the generative adversarial loss value based on the discrimination result and the alignment result based on the pre-established generative adversarial loss function, and the weighted sum of multiple generative adversarial loss values ​​and knowledge distillation loss values ​​is the joint loss function value.

[0089] In some embodiments, see Figure 3 , is a schematic diagram of the structure of the distillation model II based on the generative adversarial network provided in this embodiment. Figure 2 The difference is, Figure 3 The first distillation model in the method does not expand the depth, and in the first identification model, there are only two parallel encoders. In some embodiments, the first image teacher model may not expand the width, but only expand the depth and channel, and the corresponding first identification network also has only two parallel decoders for identifying the depth and channel.

[0090] In some embodiments, the first image teacher model may not expand the channel, but only expand the depth and channel, and the corresponding first identification network also has only a two-branch parallel decoder for identifying the depth and width.

[0091] In some embodiments, see Figure 4 , is a schematic diagram of the structure of the distillation model III based on the generative adversarial network provided in this embodiment. The first image teacher model can only perform channel expansion, and the corresponding first identification network also has only one decoder.

[0092] In some embodiments, see Figure 5 , is a schematic diagram of the structure of the distillation model IV based on the generative adversarial network provided in this embodiment. The first image teacher model may not be expanded in any way, and the corresponding first identification network also has only one decoder.

[0093] In some embodiments, see Figure 6 , is a schematic diagram of the structure of the distillation model IV based on the generative adversarial network provided in this embodiment. The first image teacher model can simultaneously perform width expansion and depth expansion, and the corresponding first identification network has only one decoder.

[0094] In some embodiments, the first image teacher model can perform width expansion and channel expansion simultaneously, and there is only one decoder corresponding to the first identification network.

[0095] In some embodiments, the first image teacher model can perform depth expansion and channel expansion simultaneously, and there is only one decoder corresponding to the first identification network.

[0096] It is worth noting that in some embodiments, Figures 2 to 6 The first image teacher model, the first image student model, the first audio teacher model and the first identification model all adopt a cascaded decoder-encoder structure, and the decoder-encoder adopted by the first identification model is a shared decoder-branch parallel encoder structure.

[0097] It is worth mentioning that Figures 3 to 6 is Figure 2 On the basis of the first image teacher model, the expansion is reduced, and the number of multi-branch parallel decoders of the first identification network is correspondingly reduced, which will lose image information in different spatial dimensions to varying degrees, resulting in the accuracy of the English pronunciation teaching animation being not as good as Figure 2 The method is high, but this operation can reduce the amount of image calculation. It can be used as a solution for online training when the real-time communication signal is poor, or as a solution deployed on micro-intelligent devices, such as smart bracelets, smart watches, smart pens, smart glasses or other wearable smart devices; Figure 2 During training, the first image teacher model has a larger model range and requires better communication quality to ensure the efficiency of online training. It can also be used as a response solution deployed on large smart devices, such as computers or notebooks.

[0098] In some embodiments, Figures 2 to 6 It can also be deployed in cluster devices, such as servers or self-organizing networks.

[0099] In some embodiments, according to the second distillation model, a corresponding audio and video teaching animation is generated, including: obtaining a real-time target word, and collecting the target pronunciation audio and target pronunciation video of the real-time target word speaker, using the target pronunciation audio as the input of the second audio teacher model, outputting the target sound features, and using the target pronunciation video as the input of the second image student model, outputting the target image features, and combining the target sound features and the target image features to construct a three-dimensional demonstration audio and video teaching animation.

[0100] In some embodiments, target sound features and target image features are combined to construct a three-dimensional demonstration audio-visual teaching animation, including: collecting the user's character features, the character features include: tongue features, facial features and acoustic features; embedding the target sound features into the acoustic features to obtain audio demonstration features with the user as the demonstration object, and embedding the target image features into the tongue features and facial features respectively to obtain facial image demonstration features with the user as the demonstration object, and generating an audio-visual teaching animation with the user as the demonstration object based on the audio demonstration features and the facial image demonstration features.

[0101] It is worth noting that after the user agrees to authorize the collection of character features and agrees to use his or her own character features as the demonstration object, the user's character features are collected to generate an audio and video teaching animation with the user as the demonstration object based on the user's character features. By comparing the audio and video teaching animation with the user's real-time pronunciation audio and video, the user can find his or her own pronunciation weaknesses and improve them according to his or her own audio and video teaching animation, thereby improving the user experience.

[0102] In some embodiments, after generating an audio and video teaching animation with the user as the demonstration object, it also includes: collecting the user's real-time pronunciation audio and video, and parsing the real-time pronunciation audio and video to obtain a real-time three-dimensional facial image corresponding to the real-time pronunciation audio and the real-time pronunciation video; obtaining the acoustic similarity of the real-time pronunciation audio and the audio demonstration feature, and obtaining the facial movement similarity of the real-time three-dimensional facial image and the facial image demonstration feature; according to the acoustic similarity and the facial movement similarity, respectively, the audio segment and facial movement video segment to be taught are intercepted from the audio demonstration feature and the audio demonstration feature, so that the user is prompted to learn according to the audio segment and the facial movement video segment.

[0103] In some embodiments, if the acoustic similarity is less than the acoustic threshold or the facial action similarity is less than the facial action threshold, the audio segment and the facial action video segment to be taught are respectively extracted from the audio demonstration feature and the audio demonstration feature.

[0104] See also Figure 7 , is a flow chart of generating audio and video teaching animations based on the distillation model of the generative adversarial network provided in this embodiment. Figure 7It should be noted that after obtaining the real-time three-dimensional facial image, the real-time three-dimensional facial image can also be used as the input of the second image student model, and the second image student model of the small model outputs the real-time facial features, and the real-time pronunciation audio is used as the input of the second audio teacher model to output the real-time audio features. Then, the acoustic similarity of the real-time audio features and the audio demonstration features is obtained, and the facial action similarity of the implementation facial features and the facial image demonstration features is obtained. According to the acoustic similarity and the facial action similarity, the audio segment and the facial action video segment to be taught are intercepted to generate an audio and video teaching animation, and the user is prompted to learn according to the audio and video teaching animation.

[0105] It is worth noting that in Figure 7 In the embodiment, the audio demonstration features and the facial image demonstration features can be obtained in real time to reduce the consumption of storage resources, or they can be obtained from the data stored in the database to reduce the consumption of computing resources. In a preferred embodiment, the method of obtaining the audio demonstration features and the facial image demonstration features in real time is adopted. This is because when the second image student model is obtained in real time, it has high processing efficiency because it can provide users with richer English pronunciation teaching without occupying additional storage resources. It can be used immediately after learning, and can provide a higher sense of experience.

[0106] In some embodiments, based on a virtual portrait selected by the user as a demonstration object and based on the character features of the virtual portrait, target sound features and target image features are embedded into the character features of the virtual portrait, and an audio-video teaching animation with the virtual portrait as the demonstration object is generated. The acoustic similarity between the audio demonstration features in the audio-video teaching animation and the user's real-time pronunciation audio is compared, and the facial movement similarity between the facial image demonstration features in the audio-video teaching animation and the real-time three-dimensional facial image is compared at the same time. The audio segments and facial movement video segments to be taught are respectively extracted from the audio demonstration features and the audio demonstration features, so that the user is prompted to learn based on the audio segments and the facial movement video segments.

[0107] It is worth noting that the above-mentioned different embodiments can be combined with each other to generate a new English pronunciation teaching method, but it is still within the scope of the technical solution to be protected by this application, and the English pronunciation teaching method provided in this embodiment can also be applied to the pronunciation teaching methods of other languages ​​or dialects.

[0108] This embodiment adopts a distillation model based on a generative adversarial network to extract features from the pronunciation video and pronunciation audio in the pronunciation teaching training data respectively, and updates the parameters of the distillation model through the joint loss function value, thereby constructing a distillation model for processing multimodal data to reduce the use of storage space resources and computing resources. There is no need to manually organize word scene examples, thereby improving the efficiency of English pronunciation teaching and further improving the user experience.

[0109] Example 2

[0110] See also Figure 8 , is a structural diagram of an English pronunciation teaching system based on a large model provided in this embodiment, including: a preprocessing module 21, a feature extraction module 22 and a teaching module 23.

[0111] In some embodiments, the preprocessing module 21 preprocesses the collected pronunciation teaching training data, outputs pronunciation audio and three-dimensional facial images, and transmits the pronunciation audio and three-dimensional facial images to the feature extraction module 22; after the feature extraction module 22 receives the pronunciation audio and three-dimensional facial images, it outputs the alignment results and facial features, and transmits the alignment results and facial features to the teaching module 23; after the teaching module 23 receives the alignment results and facial features, it outputs audio and video teaching animations.

[0112] The preprocessing module 21 is used to collect pronunciation teaching training data, analyze the pronunciation teaching training data, obtain pronunciation audio and pronunciation video corresponding to the pronunciation audio, and capture the three-dimensional facial image of the teaching speaker in the pronunciation video.

[0113] The feature extraction module 22 is used to use the three-dimensional facial image as the input of the first distillation model based on the generative adversarial network, perform image knowledge distillation, output facial features, and input the pronunciation audio into the first distillation model to perform audio knowledge distillation, output sound features, align the time axis of the sound features with the facial features, and obtain an alignment result.

[0114] It is worth noting that the first distillation model is based on the initial distillation model of the generative adversarial network, and the second distillation model is the distillation model obtained after training the initial distillation model.

[0115] In some embodiments, the facial feature includes: a first facial feature and a second facial feature.

[0116] In some embodiments, the first facial features and the second facial features both include: tongue features, facial features, and acoustic features of the teaching speaker in the pronunciation video. The difference is that the first facial features are facial features output by image knowledge distillation of the three-dimensional facial image through the initial first image teacher model, and the second facial features are facial features output by image knowledge distillation of the three-dimensional facial image through the initial first image student model.

[0117] It is worth noting that the first image teacher model is an initial image teacher model, which is used to process the knowledge distillation of the image to guide the training of the first image student model, and the first image student model is an initial image student model, which is used to compress the image knowledge with a smaller model, so that the model framework of the obtained first image student model is smaller than that of the first image teacher model. In addition, the second image teacher model and the second image student model mentioned later are the image teacher model and the image student model of the initial image teacher model and the initial image student model after training, respectively.

[0118] In some embodiments, the first distillation model includes: a plurality of first image teacher models based on a generative network and a first image student model based on a generative network.

[0119] In some embodiments, a three-dimensional facial image is used as the input of a first distillation model based on a generative adversarial network to perform image knowledge distillation and output facial features, including: using the three-dimensional facial image as the input of multiple first image teacher models, performing image knowledge distillation after expanding them in depth, width and channel, and correspondingly outputting multiple first facial features with different spatial features, and at the same time using the three-dimensional facial image as the input of the first image student model to perform image knowledge compression and output second facial features.

[0120] In some embodiments, the first distillation model further includes: a first audio teacher model based on a generative network.

[0121] In some embodiments, the pronunciation audio is input into the first distillation model for audio knowledge distillation, the sound features are output, and the time axis of the sound features and the facial features are aligned to obtain an alignment result, including: based on the image information entropy of different facial parts, corresponding weights are given to multiple first facial features respectively, and multiple facial weight results are obtained accordingly, and the pronunciation audio is used as the input of the first audio teacher model for audio knowledge distillation, and the corresponding sound features are output; the sound features are introduced for multiple facial weight results to align the time axis to synchronize the sound information in the time dimension and the image information in the space dimension, and obtain the alignment result of the facial weight results; wherein the facial weight result and the sound feature are in a one-to-one relationship, and an alignment result is generated.

[0122] The teaching module 23 is used to obtain the joint loss function value according to the alignment result, and update the parameters of the first distillation model according to the joint loss function value to obtain the second distillation model, so as to generate the corresponding audio and video teaching animation according to the second distillation model.

[0123] In some embodiments, the first distillation model further includes: a first identification model based on a discriminant network.

[0124] In some embodiments, a joint loss function value is obtained based on the alignment result, including: taking a three-dimensional facial image and multiple alignment results as inputs of a first identification model, corresponding to the output of multiple identification results, combining multiple alignment results with multiple identification results to obtain multiple generative adversarial loss values; the alignment result and the identification result are in a one-to-one relationship, and a generative adversarial loss value is generated; the network structure similarity and network output similarity of multiple first image teacher models and first image student models are obtained respectively, and a knowledge distillation loss value is obtained based on the network structure similarity and the network output similarity, and multiple generative adversarial loss values ​​and knowledge distillation loss values ​​are combined to obtain a joint loss function value.

[0125] In some embodiments, a three-dimensional facial image and a plurality of alignment results are used as inputs of a first identification model, and the corresponding plurality of identification results are outputted, including: taking out one alignment result and the three-dimensional facial image from the plurality of alignment results without replacement each time as a set of input data, inputting the input data into a shared decoder and a first identification model of a branched parallel encoder, and outputting one identification result from the corresponding branched parallel encoder each time; the number of branched parallel encoders is the same as that of the first image teacher model and corresponds one to one.

[0126] In some embodiments, the second distillation model includes: a second image teacher model, a second image student model, a second audio teacher model, and a second discrimination model.

[0127] In some embodiments, the parameters of the first distillation model are updated according to the joint loss function value to obtain a second distillation model, including: according to the joint loss function value, the first image teacher model, the first image student model, the first audio teacher model and the first identification model are simultaneously updated online parameters to obtain the second image teacher model, the second image student model, the second audio teacher model and the second identification model accordingly.

[0128] In some embodiments, according to the second distillation model, a corresponding audio and video teaching animation is generated, including: obtaining a real-time target word, and collecting the target pronunciation audio and target pronunciation video of the real-time target word speaker, using the target pronunciation audio as the input of the second audio teacher model, outputting the target sound features, and using the target pronunciation video as the input of the second image student model, outputting the target image features, and combining the target sound features and the target image features to construct a three-dimensional demonstration audio and video teaching animation.

[0129] In some embodiments, target sound features and target image features are combined to construct a three-dimensional demonstration audio-visual teaching animation, including: collecting the user's character features, the character features include: tongue features, facial features and acoustic features; embedding the target sound features into the acoustic features to obtain audio demonstration features with the user as the demonstration object, and embedding the target image features into the tongue features and facial features respectively to obtain facial image demonstration features with the user as the demonstration object, and generating an audio-visual teaching animation with the user as the demonstration object based on the audio demonstration features and the facial image demonstration features.

[0130] In some embodiments, after generating an audio and video teaching animation with the user as the demonstration object, it also includes: collecting the user's real-time pronunciation audio and video, and parsing the real-time pronunciation audio and video to obtain a real-time three-dimensional facial image corresponding to the real-time pronunciation audio and the real-time pronunciation video; obtaining the acoustic similarity of the real-time pronunciation audio and the audio demonstration feature, and obtaining the facial movement similarity of the real-time three-dimensional facial image and the facial image demonstration feature; according to the acoustic similarity and the facial movement similarity, respectively, the audio segment and facial movement video segment to be taught are intercepted from the audio demonstration feature and the audio demonstration feature, so that the user is prompted to learn according to the audio segment and the facial movement video segment.

[0131] This embodiment adopts a distillation model based on a generative adversarial network to extract features from the pronunciation video and pronunciation audio in the pronunciation teaching training data respectively, and updates the parameters of the distillation model through the joint loss function value, thereby constructing a distillation model for processing multimodal data to reduce the use of storage space resources and computing resources. There is no need to manually organize word scene examples, thereby improving the efficiency of English pronunciation teaching and further improving the user experience.

[0132] Those skilled in the art will appreciate that the embodiments of the present application may also provide computer program products. Therefore, the present application may adopt the form of complete hardware embodiments, complete software embodiments, or embodiments in combination with software and hardware. Moreover, the present application may adopt the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program codes.

[0133] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0134] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0135] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0136] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for teaching English pronunciation based on a large model, characterized in that: include: Collecting pronunciation teaching training data, parsing the pronunciation teaching training data, obtaining pronunciation audio and a pronunciation video corresponding to the pronunciation audio, and capturing a three-dimensional facial image of a teaching speaker in the pronunciation video; Using the three-dimensional facial image as an input of a first distillation model based on a generative adversarial network, performing image knowledge distillation, and outputting facial features, and inputting the pronunciation audio into the first distillation model to perform audio knowledge distillation, outputting sound features, and aligning the time axis of the sound features with the facial features to obtain an alignment result; According to the alignment result, a joint loss function value is obtained, and according to the joint loss function value, parameters of the first distillation model are updated to obtain a second distillation model, so that a corresponding audio and video teaching animation is generated according to the second distillation model; The facial features include: a first facial feature and a second facial feature; the first distillation model includes: a plurality of first image teacher models based on a generative network and a first image student model based on a generative network; the three-dimensional facial image is used as an input of the first distillation model based on a generative adversarial network to perform image knowledge distillation and output facial features, including: The three-dimensional facial image is used as the input of the multiple first image teacher models respectively, and image knowledge distillation is performed after expansion in depth, width and channel respectively, and the first facial features with multiple different spatial features are correspondingly outputted; at the same time, the three-dimensional facial image is used as the input of the first image student model to perform image knowledge compression, and the second facial features are outputted; The first distillation model further includes: a first audio teacher model based on a generative network; the step of inputting the pronunciation audio into the first distillation model for audio knowledge distillation, outputting sound features, aligning the time axis of the sound features with the facial features, and obtaining an alignment result includes: Based on the image information entropy of different facial parts, corresponding weights are given to the multiple first facial features respectively, and multiple facial weight results are obtained accordingly, and the pronunciation audio is used as the input of the first audio teacher model to perform audio knowledge distillation, and the corresponding sound features are output; Introducing the sound feature into the multiple facial weight results to perform time axis alignment to synchronize the sound information in the time dimension and the image information in the space dimension, and obtaining an alignment result of the facial weight results; wherein the facial weight results and the sound feature are in a one-to-one relationship, and an alignment result is generated; The first distillation model further includes: a first identification model based on a discriminant network; and obtaining a joint loss function value according to the alignment result includes: The three-dimensional facial image and the plurality of alignment results are used as inputs of the first identification model, and the plurality of identification results are outputted correspondingly, and the plurality of alignment results are combined with the plurality of identification results to obtain a plurality of generative adversarial loss values; the alignment result and the identification result are in a one-to-one relationship, and a generative adversarial loss value is generated; Respectively obtain network structure similarities and network output similarities of the multiple first image teacher models and the first image student model, and obtain a knowledge distillation loss value according to the network structure similarity and the network output similarity, and combine the multiple generative adversarial loss values ​​and the knowledge distillation loss value to obtain a joint loss function value; The three-dimensional facial image and the plurality of alignment results are used as inputs of the first identification model, and the corresponding plurality of identification results outputted include: Each time, one alignment result is taken out from the multiple alignment results without replacement and the three-dimensional facial image as a set of input data, and the input data is input into the first identification model of the shared encoder and parallel branch decoder, and one identification result is output from the corresponding parallel branch decoder each time; the number of the parallel branch decoders is the same as that of the first image teacher model and they correspond one to one.

2. The English pronunciation teaching method according to claim 1, characterized in that: The second distillation model includes: a second image teacher model, a second image student model, a second audio teacher model and a second identification model; the first distillation model is updated according to the joint loss function value to obtain a second distillation model, including: According to the joint loss function value, the first image teacher model, the first image student model, the first audio teacher model and the first identification model are simultaneously updated online parameters to obtain the second image teacher model, the second image student model, the second audio teacher model and the second identification model accordingly.

3. The English pronunciation teaching method as claimed in claim 2, characterized in that: Generating a corresponding audio and video teaching animation according to the second distillation model includes: Acquire a real-time target word, and collect the target pronunciation audio and target pronunciation video of the speaker of the real-time target word, use the target pronunciation audio as the input of the second audio teacher model, output the target sound feature, and use the target pronunciation video as the input of the second image student model, output the target image feature, and combine the target sound feature and the target image feature to construct a three-dimensional demonstration audio and video teaching animation.

4. The English pronunciation teaching method as claimed in claim 3, characterized in that: The step of combining the target sound feature and the target image feature to construct a three-dimensional audio and video teaching animation comprises: Collecting the user's character features, the character features including: tongue features, facial features and acoustic features; The target sound feature is embedded into the acoustic feature to obtain an audio demonstration feature with the user as the demonstration object, and the target image feature is respectively embedded into the tongue image feature and the facial image feature to obtain a facial image demonstration feature with the user as the demonstration object. Based on the audio demonstration feature and the facial image demonstration feature, an audio and video teaching animation with the user as the demonstration object is generated.

5. The English pronunciation teaching method as claimed in claim 4, characterized in that: After generating the audio and video teaching animation with the user as the demonstration object, the method further includes: Collecting the user's real-time pronunciation audio and video, and analyzing the real-time pronunciation audio and video to obtain a real-time three-dimensional facial image corresponding to the real-time pronunciation audio and the real-time pronunciation video; Acquiring an acoustic similarity between the real-time pronunciation audio and the audio demonstration feature, and acquiring a facial action similarity between the real-time three-dimensional facial image and the facial image demonstration feature; According to the acoustic similarity and the facial action similarity, the audio segment and the facial action video segment to be taught are respectively extracted from the audio demonstration feature and the audio demonstration feature, so that the user is prompted to learn according to the audio segment and the facial action video segment.

6. An English pronunciation teaching system based on a large model, the English pronunciation teaching system is used to implement the English pronunciation teaching method as claimed in claim 1, characterized in that: include: Preprocessing module, feature extraction module and teaching module; among them, The preprocessing module is used to collect pronunciation teaching training data, analyze the pronunciation teaching training data, obtain pronunciation audio and pronunciation video corresponding to the pronunciation audio, and capture the three-dimensional facial image of the teaching speaker in the pronunciation video; The feature extraction module is used to use the three-dimensional facial image as an input of a first distillation model based on a generative adversarial network to perform image knowledge distillation and output facial features, and input the pronunciation audio into the first distillation model to perform audio knowledge distillation and output sound features, and align the time axis of the sound features with the facial features to obtain an alignment result; The teaching module is used to obtain a joint loss function value according to the alignment result, and to update the parameters of the first distillation model according to the joint loss function value to obtain a second distillation model, so as to generate a corresponding audio and video teaching animation according to the second distillation model.

Images