Virtual teacher real-time generation method and system in educational universe

Abstract

The invention relates to a virtual teacher real-time generation method and system in an educational universe, and belongs to the field of artificial intelligence and intelligent education. According to the invention, high-quality head rendering is realized through the pre-trained virtual teacher head portrait real-time generation model, the head generation quality is improved through the face restoration module, and accurate posture tracking is ensured through the head posture stabilization module. After a head portrait is generated, high-definition recovery is performed through a face recovery model, then a trunk part is rendered by a trunk generation neural radiation field network (NeRF) with adaptive posture coding, and a complete virtual teacher image is synthesized through a real-time semantic segmentation network. The virtual teacher is driven by audio or text signals, the problem that lip movement is not natural is solved through multiple lip smoothing modules, and the sense of reality of the video is improved. The technology not only maintains the integrity of the generated virtual teacher, but also realizes the multi-pose teacher movement behavior, and provides a highly vivid virtual teacher image solution for the field of education element universe.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and relates to a method and system for real-time generation of virtual teachers in an educational metaverse. Background Art

[0002] Digital human generation is a rapidly developing research field in recent years. With the development of emerging technologies such as the metaverse and generative artificial intelligence (AIGC), more and more researchers are committed to creating highly realistic virtual human images. The application of virtual human generation technology has brought innovative opportunities to the field of smart education. In the educational metaverse, highly realistic virtual teachers can effectively assist students in learning, help students better understand knowledge, improve learning efficiency, and continue to ensure their appeal to students. In terms of digital virtual human generation technology, the generative adversarial network (GAN) has achieved hyper-realistic synthesis of digital content since its launch. Similarly, the voice-driven digital face reconstruction method based on the neural rendering field (NeRF) has been gradually applied to virtual face generation.

[0003] Existing technologies still have problems with synchronization and unstable head posture, especially unnatural jitter and unnatural lip movements, which reduce the overall realism of the video. To solve the above problems, super-resolution face restoration models, head posture stabilization methods, and speech-driven digital face reconstruction methods based on neural rendering fields (NeRF) are applied to virtual face generation to improve the efficiency and quality of virtual teacher generation. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a method and system for real-time generation of virtual teachers in an educational metaverse.

[0005] In order to achieve the above object, the present invention provides the following technical solutions:

[0006] The present invention proposes a method for real-time generation of a virtual teacher in an educational metaverse, the method comprising the following steps:

[0007] S1, obtain the training video data set and use the adversarial network to optimize it;

[0008] S2, extracting the geometric features of the face through the face restoration module, the head posture stabilization module and the three-plane hash representation module, and using the implicit neural field to render the features to generate a head portrait;

[0009] S3, adopting a coarse-to-fine training strategy to train the above process to obtain a pre-trained head portrait generation model;

[0010] S4, using a pre-trained head portrait generation model to generate a head portrait, and processing text features and audio features through a multiple lip smoothing module to drive the virtual teacher;

[0011] S5. The generated head portrait is then processed with high fidelity by the face restoration module, and then the real-time semantic segmentation network superimposes the high-fidelity head portrait with the torso image generated by the torso neural radiation field to obtain a complete virtual teacher image.

[0012] Further, in step S2, the following steps are included:

[0013] S21, firstly restore a high-quality face image HQ from a low-quality face image LQ through a face restoration module;

[0014] S22, using the head posture stabilization module to extract key point features of high-quality face image HQ;

[0015] S23, performing three-plane decomposition on the extracted facial key point features through a three-plane hash representation module, and re-encoding them to obtain geometric features;

[0016] S24. Combining audio features, blinking features, geometric features of the face and the corresponding viewing direction, the implicit neural field is used to render the features to generate a head portrait.

[0017] Further, in step S21, the face restoration module includes an LQ encoder, a Transformer, a pre-trained VQGAN discrete codebook space Codebook and an HQ decoder. In the LQ encoder, based on the pre-trained VQGAN discrete codebook space, the face restoration task is converted into a Code sequence prediction task, and the Transformer is used to model the global face to compensate for the loss of local information, and then the high-quality face image HQ is decoded by the HQ decoder.

[0018] Further, in step S22, the head posture stabilization module adopts a face alignment algorithm based on a regression tree, and regresses the face shape from the current shape to the true shape step by step by establishing a cascade of residual regression trees, wherein the relationship between each tree is serial, and the latter tree is established on the basis of the previous tree. The cascade of regressors is expressed as:

[0019]

[0020] in represents the shape of the t-th level regressor, which is a vector composed of coordinates; t represents the number of cascades, representing the trees of the residual regression tree and are independent of each other; I is the image, r t represents the regressor of the current level;

[0021] The updating strategy adopts the gradient boosting decision tree GBDT, that is, each level of regressor learns the residual between the current shape and the true shape. Each leaf node of each GBDT stores a residual regressor. Through the residual saved on the leaf node, when the input falls on a node, the residual is added to the input, and finally all the residuals are superimposed together to complete the face alignment.

[0022] Further, in step S23, the three-plane hash representation module decomposes the 3D space into three orthogonal planes through NeRF-based three-plane decomposition:

[0023] Given a coordinate x=(x,y,z)∈R XYZ , three 2D multi-resolution hash encoders are used to encode the three projection coordinates respectively:

[0024]

[0025] is the plane-level geometric feature of the projection coordinate (a, b), with L levels and F feature dimensions for each entry; then the encoding results of the three planes are concatenated to obtain the final geometric feature f g ∈R 3×LF :

[0026]

[0027] In the formula, H XY (x,y),H YZ (y,z),H XZ (x,z) represent the multi-resolution hash encoders in three orthogonal planes.

[0028] Further, in step S24, the coordinate x is encoded into f by using the face recovery model processing and three-plane hash coding H based on the lightweight dynamic neural radiation field. g ;

[0029] The encoded features are rendered using a highly robust portrait rendering network, and its implicit function is:

[0030] F G :(x,d,v,e,f g )→(c,θ)

[0031] In the formula, the input of the network includes coordinates x, sight direction d, audio features v, blink features e and geometric features f g , (c,θ) represents the predicted color value c and volume density θ;

[0032] Using the above implicit model, let the camera ray be r(t), the camera center be o, and the viewing direction be d. The actual image color C rendered by the volume is:

[0033]

[0034] T(t) is the distance from t along the direction of light propagation. n to f The cumulative transmittance of the input camera is used to obtain the viewing angle image under the input camera posture condition.

[0035] Furthermore, in step S3, the training adopts a two-step strategy from coarse to fine. First, the MSE loss adopted by ordinary NeRF is used to evaluate the difference between the predicted image I and the actual image color C(r):

[0036]

[0037] In the formula, C(i), Represent the actual image color and the predicted image color respectively, represents the square of the Euclidean norm;

[0038] Then, a random patch P is extracted from the image, and the λ-weighted LPIPS is combined with MSE to improve the detail representation:

[0039]

[0040] Where, L LPIPS (·) represents the structural similarity loss judgment, λ represents the weight, P is the random patch extracted from the image, is the predicted value for the patch.

[0041] Further, in step S4, the virtual teacher is driven by an input audio or text signal, audio features are extracted from the text signal or the input audio, and lip movement smoothing is performed through a multiple lip smoothing module, wherein:

[0042] The multiple lip smoothing module includes an improved exponential moving weighted average (EWMA) algorithm by linear interpolation:

[0043] S t =αx t +βx t-1 +(1-α-β)x t-2

[0044] In the formula, S t is the value processed by the smoothing module at the current moment, x t is the true value at the current moment, α and β are smoothing coefficients;

[0045] The multiple lip smoothing module first divides the audio features used to infer and render the lips into several groups. Continuous multi-frame audio features are regarded as a group. The first value of each group is used as the error-free true value during smoothing. Subsequent data is processed in sequence based on this standard. Each group of data is updated based on the previous EWMA value in the group to achieve multiple smoothing until all audio features are processed.

[0046] Further, in step S5, a head portrait of the virtual teacher is generated, and then the generated face portrait is restored in high definition through the face restoration model to generate a highly realistic head portrait;

[0047] The torso part is rendered by the trunk neural radiation field NeRF with adaptive posture coding to obtain the torso image; then the generated virtual teacher portrait is superimposed with the torso image through the real-time semantic segmentation network to synthesize a complete virtual teacher image;

[0048] The real-time semantic segmentation network includes two paths: a context path for obtaining rich high-level semantic information and a spatial path for capturing fine visual detail information. The human parts are marked through the segmentation network, and then the marked positions and resolutions are recorded, the parts used for training are cropped, and the positions are saved in JSON files. After inference is completed, the JSON is read and ffmpeg is used for overlay synthesis to generate a virtual teacher with a head and torso.

[0049] The present invention also proposes a system for executing the aforementioned method for real-time generation of a virtual teacher in an educational metaverse, the system comprising an input module, a feature extraction and processing module, a pre-trained head portrait generation model, and a torso rendering and virtual teacher generation module, wherein:

[0050] The input module inputs text information and audio information, and the feature extraction module extracts text features and audio features, and the features are processed by the multiple lip smoothing module to drive the virtual teacher;

[0051] The pre-trained head portrait generation model is used to generate head portraits based on the input video set. The head portrait generation model includes a face restoration module, a head posture stabilization module, a three-plane hash representation module and an implicit neural field. The face restoration module optimizes the input, and the head posture stabilization module obtains the identity features and the portrait motion posture. Then, the dynamic head reconstruction is realized through the three-plane hash representation module based on the implicit neural field NeRF.

[0052] The torso rendering and virtual teacher generation module is rendered by the torso NeRF with adaptive pose encoding to predict the torso image; through the real-time semantic segmentation network, the generated virtual teacher head portrait is superimposed with the original input torso to synthesize a complete virtual teacher image. Among them, the head portrait generated by the head portrait generation model needs to be processed again by the face restoration module for high-fidelity processing.

[0053] The beneficial effects of the present invention are:

[0054] The present invention firstly introduces face restoration, head posture stabilization and real-time semantic segmentation models based on neural radiance field (NeRF) to generate a virtual teacher image with emotional expression ability and realistic appearance, including head generation and torso generation. The head generation neural model promotes dynamic head reconstruction by a highly robust portrait generation network, and achieves high-quality rendering, real-time reasoning and fast convergence with a lightweight model. The face restoration module mainly restores the low-quality face (LQ) image with information loss due to degradation into a high-quality face (HQ) portrait, thereby improving the output quality. The head posture stabilization module (HBS) obtains more accurate face key points by optimizing the face alignment algorithm, and estimates all head posture and tracking information.

[0055] The virtual teacher of the present invention is driven by inputting signals such as audio or text, extracting audio features from text signals or input audio, and solving the problem of jitter and unnatural lip movement of the virtual person through multiple lip smoothing modules (DLS). The multiple lip smoothing modules realize smoothed lip movement and eliminate the error generated by the smoothing module. The unnatural jitter and unnatural problems of lip movement are solved, thereby improving the overall realism of the video.

[0056] After generating the head portrait of the virtual teacher, the generated face portrait is restored to high definition by the face restoration model to generate a highly realistic head portrait. The torso part is rendered by the torso NeRF with adaptive pose encoding to predict the torso image. On this basis, the generated virtual teacher portrait is superimposed with the original input torso through the real-time semantic segmentation network to synthesize the complete virtual teacher image. The complete virtual teacher rendering maintains the integrity of the generated subject and can realize the teacher's motion behavior in multiple postures.

[0057] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0059] Figure 1 It is a schematic diagram of the overall architecture of the method for real-time generation of virtual teachers in the educational metaverse of the present invention;

[0060] Figure 2 It is a structural schematic diagram of the face restoration module of the present invention;

[0061] Figure 3 Schematic diagram of the processing process of the real-time semantic segmentation network of the present invention. DETAILED DESCRIPTION

[0062] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0063] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0064] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0065] See also Figure 1 to Figure 3 , which is a method and system for real-time generation of virtual teachers in the educational metaverse.

[0066] Example

[0067] This embodiment provides an overall implementation process of a method for real-time generation of virtual teachers in an educational metaverse. Figure 1 As shown, it includes the following steps:

[0068] Step 1: Obtain a training video dataset and generate an adversarial network to optimize it and improve its anti-interference performance;

[0069] Step 2: Extract audio features and blink features from the optimized video dataset based on its audio input and blink frequency;

[0070] Step 3: Extract the geometric features of the face from the optimized video dataset through the face restoration module, head posture stabilization module and three-plane hash representation module; and combine the audio features, blink features, geometric features of the face and the corresponding viewing direction, use implicit neural fields to render the features and generate a head portrait;

[0071] Step 4: adopt a coarse-to-fine training strategy to train the above process, and obtain a pre-trained head portrait generation model including the above process;

[0072] Step 5: Generate a head portrait using a pre-trained head portrait generation model, and process text features and audio features through a multiple lip smoothing module to drive the virtual teacher;

[0073] Step 6: The generated head portrait is then processed with high realism by the face restoration module, and then the real-time semantic segmentation network superimposes the high-realistic head portrait with the torso image generated by the torso neural radiation field to obtain a complete virtual teacher image.

[0074] In step 1 of this embodiment, the adversarial network includes a pre-trained face GAN as a face prior and degradation removal module. It removes complex degradation through intermediate restoration loss and retains identity loss to maintain the face identity to repair the video dataset, so that its anti-interference performance is improved.

[0075] In step 2 of this embodiment, the following steps are specifically included:

[0076] S21, firstly restore a high-quality face image HQ from a low-quality face image LQ through a face restoration module;

[0077] The low-quality face (LQ) image with information loss caused by degradation is restored to a high-quality face (HQ) portrait, such as Figure 2As shown. In order to improve the output quality, reduce the uncertainty of LQ-HQ mapping, and supplement the auxiliary information of high-quality details. The face restoration module includes LQ encoder, Transformer, pre-trained VQGAN discrete codebook space Codebook and HQ decoder. In the LQ encoder, based on the pre-trained VQGAN discrete codebook space, the face restoration task is converted into a Code sequence prediction task, and the Transformer is used to model the global face to compensate for the loss of local information, and then decoded by the HQ decoder to obtain a high-quality face image HQ.

[0078] S22, using the head posture stabilization module to extract key point features of high-quality face image HQ;

[0079] The head pose stabilization module obtains more accurate facial key points by optimizing the face alignment algorithm and estimates all head pose and tracking information.

[0080] In order to obtain more accurate head key points, the head posture stabilization module (HBS) adopts a face alignment algorithm based on regression tree, which makes the face shape regress from the current shape to the real shape step by step by building a cascaded residual regression tree (GBDT). The relationship between each tree is serial, and the latter tree is built on the basis of the previous tree. The specific cascade of regressors is:

[0081]

[0082] in represents the shape of the t-th level regressor, which is a vector composed of coordinates; t represents the number of cascades, representing the trees of GBDT and are independent of each other; I is the image, r t Represents the regressor of the current level.

[0083] The update strategy uses the GBDT gradient boosting decision tree, that is, each level of regressor learns the residual between the current shape and the groundtruth shape. Each leaf node of each GBDT stores a residual regressor. Through the residual stored on the leaf node, when the input falls on a node, the residual is added to the input to achieve the purpose of regression. Finally, all the residuals are superimposed together to complete the purpose of face alignment.

[0084] S23, performing three-plane decomposition on the extracted facial key point features through a three-plane hash representation module, and re-encoding them to obtain a set feature;

[0085] The three-plane hash representation module decomposes the 3D space into three orthogonal planes through NeRF-based three-plane decomposition. Given a coordinate x = (x, y, z) ∈ R XYZ, three 2D multi-resolution hash encoders can be used to encode the three projection coordinates separately:

[0086]

[0087] is the plane-level geometric feature of the projection coordinate (a, b), with the number of levels L and the feature dimension F of each entry. Then the encoding results of the three planes are concatenated to obtain the final geometric feature f g ∈R 3×LF :

[0088]

[0089] In the formula, H XY (x,y),H YZ (y,z),H XZ (x,z) represent the multi-resolution hash encoders in three orthogonal planes.

[0090] S24, combining audio features, blinking features, geometric features of the face and corresponding viewing directions, using implicit neural fields to render the features and generate a head portrait;

[0091] Based on lightweight dynamic neural radiation field, the face recovery model and three-plane hash coding H are used to encode the coordinate x into f g .

[0092] Therefore, the implicit function of the highly robust portrait rendering network is:

[0093] F G :(x,d,v,e,f g )→(c,θ)

[0094] In the formula, the input of the network includes coordinates x, sight direction d, audio features v, blink features e and geometric features f g , (c,θ) represents the color value and volume density.

[0095] In this embodiment, a content encoder hubert based on the Transformer architecture extracts audio features A from audio input. A deep learning face recognition model OpenFace is used to extract blink features V from blink frequency.

[0096] Assume the camera ray is r(t), the camera center is o, and the viewing direction is d. The actual image color C rendered by the volume is:

[0097]

[0098] T(t) is the distance from t along the direction of light propagation. n tof The cumulative transmittance of the input camera is obtained.

[0099] In step three of this embodiment, the training adopts a two-step strategy from coarse to fine. First, the MSE loss adopted by ordinary NeRF is used to evaluate the difference between the predicted image I and the actual image color C(r).

[0100]

[0101] In the formula, C(i), Represent the actual image color and the predicted image color respectively, represents the square of the Euclidean norm;

[0102] Due to the limitations of MSE in capturing details, a refinement stage is also required to incorporate the image similarity evaluation metric (LPIPS) loss to enhance details, similar to ER-NeRF. Random patches P are extracted from the image, and LPIPS (λ-weighted) is combined with MSE to improve the detail representation:

[0103]

[0104] Where, L LPIPS (·) represents the structural similarity loss judgment, λ represents the weight, P is a random plate extracted from the image, is the predicted value for the patch.

[0105] Through training, a pre-trained highly robust head portrait generation model is obtained. By combining the VQGAN+Transformer face restoration module with NeRF in the portrait rendering part, the face restoration module first optimizes the input to obtain more accurate identity features and portrait motion postures, and then a more accurate dynamic head reconstruction is achieved through an efficient three-plane hash representation based on NeRF.

[0106] In step 4 of this embodiment, the head generation neural model is promoted by a highly robust portrait generation network to promote dynamic head reconstruction, and a lightweight model is used to achieve high-quality rendering, real-time reasoning and fast convergence. The virtual teacher is driven by input signals such as audio or text. Audio features are extracted from text signals or input audio, and the jitter and unnatural phenomenon of virtual human lip movement is solved through a multiple lip smoothing module (DLS).

[0107] The multi-lip smoothing module includes an improved exponential moving weighted average (EWMA) algorithm through linear interpolation:

[0108] S t =αx t +βx t-1+(1-α-β)x t-2

[0109] S t is the value processed by the smoothing module at the current moment, x t is the true value at the current moment, and α and β are smoothing coefficients. First, the audio features used to infer the rendering of the lips are divided into several groups. Continuous multi-frame audio features are taken as a group. The first value of each group is used as the error-free true value during smoothing, and subsequent data is processed in sequence based on this standard. The data in each group can be directly updated based on the previous EWMA value in the group to achieve multiple smoothing until all audio features are processed. In this way, we achieve smoothed lip movement and eliminate the error caused by the smoothing module.

[0110] In step 5 of this embodiment, a head portrait of the virtual teacher is generated, and then the generated face portrait is restored with high definition through the face restoration model to generate a highly realistic head portrait. The torso part is rendered by the torso NeRF with adaptive posture coding to predict the torso image. On this basis, the generated virtual teacher portrait is superimposed with the original input torso through the real-time semantic segmentation network to synthesize a complete virtual teacher image.

[0111] Real-time semantic segmentation network (BiSeNet), such as Figure 3 As shown in the figure, it consists of a context path for obtaining rich high-level semantic information and a spatial path for capturing fine visual detail information. The human parts are marked through the segmentation network, and then the marked positions and resolutions are recorded, the parts used for training are cut, and the positions are saved in a JSON file. After the inference is completed, the JSON is read and superimposed and synthesized using ffmpeg to generate a virtual human with limbs.

[0112] The present invention improves the robustness of the system by introducing a highly robust neural radiation field generation algorithm. The multiple lip smoothing modules and head posture stabilization modules optimize the naturalness of the generated virtual human. The complete virtual teacher rendering maintains the integrity of the generated subject and can realize the teacher's movement behavior in multiple postures. At the same time, the face restoration module is introduced to further optimize the generation effect and realize more realistic virtual teacher generation. In addition, high-quality and high-real-time interactive design is realized in the designed interactive system.

[0113] This embodiment also provides a system for executing the aforementioned method for real-time generation of a virtual teacher in the educational metaverse, which includes an input module, a feature extraction and processing module, a pre-trained head portrait generation model, and a torso rendering and virtual teacher generation module, wherein:

[0114] The input module inputs text information and audio information, and the feature extraction module extracts text features and audio features, and the features are processed by the multiple lip smoothing module to drive the virtual teacher;

[0115] The pre-trained head portrait generation model is used to generate head portraits based on the input video set. The head portrait generation model includes a face restoration module, a head posture stabilization module, a three-plane hash representation module, and an implicit neural field. The face restoration module optimizes the input, and the head posture stabilization module obtains the identity features and the portrait motion posture. Subsequently, a more accurate dynamic head reconstruction is achieved through the three-plane hash representation module based on the implicit neural field NeRF.

[0116] The torso rendering and virtual teacher generation module is rendered by the torso NeRF with adaptive pose coding to predict the torso image. On this basis, the generated virtual teacher head portrait is superimposed with the original input torso through the real-time semantic segmentation network to synthesize a complete virtual teacher image. Among them, the head portrait generated by the head portrait generation model needs to be processed again by the face restoration module for high-fidelity processing.

[0117] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A method for real-time generation of virtual teachers in an educational metaverse, characterized in that: The method comprises the following steps: S1, obtain the training video data set and use the generative adversarial network to optimize it; S2. Build a pre-trained network for real-time generation of virtual teacher head portraits. Through the face restoration module, head posture stabilization module and three-plane hash representation module, extract the geometric features of the face, and use implicit neural fields to render the features to generate head portraits. S3, adopting a coarse-to-fine training strategy to train the above process to obtain a pre-trained head portrait generation model; S4, using a pre-trained head portrait generation model to generate a head portrait, and processing text features and audio features through a multiple lip smoothing module to drive the virtual teacher; S5. The generated head portrait is then processed with high fidelity by the face restoration module, and then the real-time semantic segmentation network superimposes the high-fidelity head portrait with the torso image generated by the torso neural radiation field to obtain a complete virtual teacher image.

2. The method for real-time generation of virtual teachers in an educational metaverse according to claim 1, characterized in that: In step S2, the following steps are included: S21, firstly restore a high-quality face image HQ from a low-quality face image LQ through a face restoration module; S22, using the head posture stabilization module to extract key point features of high-quality face image HQ; S23, performing three-plane decomposition on the extracted facial key point features through a three-plane hash representation module, and re-encoding them to obtain geometric features; S24. Combining audio features, blinking features, geometric features of the face and the corresponding viewing direction, the implicit neural field is used to render the features to generate a head portrait.

3. The method for real-time generation of virtual teachers in an educational metaverse according to claim 2, characterized in that: In step S21, the face restoration module includes an LQ encoder, a Transformer, a pre-trained VQGAN discrete codebook space Codebook and an HQ decoder. In the LQ encoder, based on the pre-trained VQGAN discrete codebook space, the face restoration task is converted into a Code sequence prediction task, and the Transformer is used to model the global face to compensate for the loss of local information, and then the high-quality face image HQ is decoded by the HQ decoder.

4. The method for real-time generation of virtual teachers in an educational metaverse according to claim 3, characterized in that: In step S22, the head posture stabilization module adopts a face alignment algorithm based on a regression tree, and regresses the face shape from the current shape to the true shape step by step by establishing a cascade of residual regression trees, wherein the relationship between each tree is serial, and the latter tree is established on the basis of the previous tree. The cascade of regressors is expressed as: in represents the shape of the t-th level regressor, which is a vector composed of coordinates; t represents the number of cascades, representing the trees of the residual regression tree and are independent of each other; I is the image, r t represents the regressor of the current level; The updating strategy adopts the gradient boosting decision tree GBDT, that is, each level of regressor learns the residual between the current shape and the true shape. Each leaf node of each GBDT stores a residual regressor. Through the residual saved on the leaf node, when the input falls on a node, the residual is added to the input, and finally all the residuals are superimposed together to complete the face alignment.

5. The method for real-time generation of virtual teachers in an educational metaverse according to claim 4, characterized in that: In step S23, the three-plane hash representation module decomposes the 3D space into three orthogonal planes through NeRF-based three-plane decomposition: Given a coordinate x=(x,y,z)∈R XYZ , three 2D multi-resolution hash encoders are used to encode the three projection coordinates respectively: is the plane-level geometric feature of the projection coordinate (a, b), with L levels and F feature dimensions for each entry; then the encoding results of the three planes are concatenated to obtain the final geometric feature f g ∈R 3×LF : In the formula, H XY (x,y),H YZ (y,z),H XZ (x,z) represent the multi-resolution hash encoders in three orthogonal planes.

6. The method for real-time generation of virtual teachers in an educational metaverse according to claim 5, characterized in that: In step S24, the coordinate x is encoded into f by using the face recovery model processing and three-plane hash coding H based on the lightweight dynamic neural radiation field. g ; The encoded features are rendered using a highly robust portrait rendering network, and its implicit function is: F G :(x,d,v,e,f g )→(c,θ) In the formula, the input of the network includes coordinates x, sight direction d, audio features v, blink features e and geometric features f g , (c,θ) represents the predicted color value c and volume density θ; Using the above implicit model, let the camera ray be r(t), the camera center be o, and the viewing direction be d. The actual image color C rendered by the volume is: T(t) is the distance from t along the direction of light propagation. n to f The cumulative transmittance of the input camera is used to obtain the viewing angle image under the input camera posture condition.

7. The method for real-time generation of virtual teachers in an educational metaverse according to claim 2, characterized in that: In step S3, the training adopts a two-step strategy from coarse to fine. First, the MSE loss adopted by the ordinary neural radiation field NeRF is used to evaluate the difference between the predicted image I and the actual image color C(r): In the formula, C(i), Represent the actual image color and the predicted image color respectively, represents the square of the Euclidean norm; Then, a random patch P is extracted from the image, and the λ-weighted LPIPS is combined with MSE to improve the detail representation: Where, L LPIPS (·) represents the structural similarity loss judgment, λ represents the weight, P is the random patch extracted from the image, is the predicted value for the patch.

8. The method for real-time generation of virtual teachers in an educational metaverse according to claim 7, characterized in that: In step S4, the virtual teacher is driven by an input audio or text signal, audio features are extracted from the text signal or input audio, and lip movement is smoothed by a multiple lip smoothing module, wherein: The multiple lip smoothing module includes an improved exponential moving weighted average (EWMA) algorithm by linear interpolation: S t =αx t +βx t-1 +(1-a-b)x t-2 In the formula, S t is the value processed by the smoothing module at the current moment, x t is the true value at the current moment, α and β are smoothing coefficients; The multiple lip smoothing module first divides the audio features used to infer and render the lips into several groups. Continuous multi-frame audio features are regarded as a group. The first value of each group is used as the error-free true value during smoothing. Subsequent data is processed in sequence based on this standard. Each group of data is updated based on the previous EWMA value in the group to achieve multiple smoothing until all audio features are processed.

9. The method for real-time generation of virtual teachers in an educational metaverse according to claim 8, characterized in that: In step S5, a head portrait of the virtual teacher is generated, and then the generated face portrait is restored in high definition through a face restoration model to generate a highly realistic head portrait; The torso part is rendered by the trunk neural radiation field NeRF with adaptive posture coding to obtain the torso image; then the generated virtual teacher portrait is superimposed with the torso image through the real-time semantic segmentation network to synthesize a complete virtual teacher image; The real-time semantic segmentation network includes two paths: a context path for obtaining rich high-level semantic information and a spatial path for capturing fine visual detail information. The human parts are marked through the segmentation network, and then the marked positions and resolutions are recorded, the parts used for training are cropped, and the positions are saved in JSON files. After inference is completed, the JSON is read and ffmpeg is used for overlay synthesis to generate a virtual teacher with a head and torso.

10. A system for executing the method for real-time generation of virtual teachers in the educational metaverse described in any one of 1 to 9 above, characterized in that: The system includes an input module, a feature extraction and processing module, a pre-trained head portrait generation model, and a torso rendering and virtual teacher generation module, wherein: The input module inputs text information and audio information, and the feature extraction module extracts text features and audio features, and the features are processed by the multiple lip smoothing module to drive the virtual teacher; The pre-trained head portrait generation model is used to generate head portraits based on the input video set. The head portrait generation model includes a face restoration module, a head posture stabilization module, a three-plane hash representation module and an implicit neural field. The face restoration module optimizes the input, and the head posture stabilization module obtains the identity features and the portrait motion posture. Then, the dynamic head reconstruction is realized through the three-plane hash representation module based on the implicit neural field NeRF. The torso rendering and virtual teacher generation module is rendered by the torso NeRF with adaptive pose encoding to predict the torso image; through the real-time semantic segmentation network, the generated virtual teacher head portrait is superimposed with the original input torso to synthesize a complete virtual teacher image. Among them, the head portrait generated by the head portrait generation model needs to be processed again by the face restoration module for high-fidelity processing.

Images