Immersive virtual teaching system based on multimodal perception and dynamic evaluation

Abstract

The present invention relates to the field of virtual reality education technology, and in particular to an immersive virtual teaching system based on multimodal perception and dynamic evaluation, comprising: a data acquisition module, a data processing module, a rendering module, and an evaluation module; the data acquisition module is used to collect a user's appearance data and voice data; the data processing module is used to process the appearance data and voice data collected by the data acquisition module in real time and fuse the processed appearance data and voice data with a SMPL-X model to form a dynamic digital human; the rendering module is used to display a virtual teaching scene and render the dynamic digital human into the virtual teaching scene; and the evaluation module is used to evaluate the user's learning concentration and experimental operation accuracy in the virtual teaching scene. The present invention provides an immersive virtual teaching system that supports markerless holographic capture, multimodal data fusion mapping, multi-person low-latency collaboration, and dynamic learning evaluation.

Description

Technical Field

[0001] The present invention relates to the field of virtual reality education technology, and in particular to an immersive virtual teaching system based on multimodal perception and dynamic evaluation. Background Art

[0002] Currently, digital education technology has formed a diversified development pattern, and its main applications include synchronous online teaching platforms, asynchronous learning management systems, virtual reality education systems, AI adaptive systems and metaverse education platforms.

[0003] Synchronous online teaching platforms enable real-time interaction through audio and video streaming, and enhance collaborative efficiency through virtual whiteboards and group discussions. However, traditional architectures suffer from a single data dimension, network fluctuations leading to audio and video asynchrony, and two-dimensional interaction that can easily lead to cognitive fatigue and compromise learning immersion.

[0004] Asynchronous learning management systems are based on modular course design, encapsulating content using the SCORM standard and recording learning trajectories using the xAPI protocol to generate multi-dimensional learning reports. However, pre-set paths lack dynamic adaptability, isolated learning scenarios can easily lead to cognitive alienation, and delayed feedback from experimental operations can lead to persistent errors.

[0005] Virtual reality education systems use head-mounted devices to create a three-dimensional environment, combined with six-degree-of-freedom tracking for spatial interaction, making them suitable for simulating dangerous experiments. However, the complexity of graphics rendering limits high-fidelity synchronization on mobile devices, and continuous use can easily cause visual fatigue. Augmented reality systems reduce device dependence, but lack the accuracy of surface recognition and interaction.

[0006] AI adaptive systems use knowledge graphs and machine learning to build personalized pathways, employing Bayesian models to evaluate effectiveness and recommend resources. However, these algorithms rely on structured data, are weak at analyzing unstructured behavior, and their opaque decision-making processes hinder the development of critical thinking.

[0007] The Metaverse education platform integrates 3D modeling and digital avatar technology, and enhances social presence through spatial audio and blockchain authentication. However, it has the following problems: traditional systems rely on optical markers or wearable sensors, which leads to movement delays and restricts users' natural movement; existing expression mapping methods rely solely on camera image processing, which cannot capture micro-expressions and muscle movements, and the expression recognition accuracy is average; when the number of users is too large, data synchronization delays increase significantly, resulting in asynchronous virtual avatar movements; existing systems only record operation results and lack multi-dimensional quantitative analysis of the learning process. Summary of the Invention

[0008] The purpose of the present invention is to provide an immersive virtual teaching system based on multimodal perception and dynamic evaluation, which supports marker-free holographic capture, multimodal data fusion mapping, multi-person low-latency collaboration and dynamic learning evaluation, so as to solve the technical problems of traditional distance learning systems in the above-mentioned background technology, such as limited natural interaction, distorted emotional expression, multi-user usage delay and single evaluation dimension.

[0009] To achieve the above object, the technical solution of the present invention is as follows:

[0010] An immersive virtual teaching system based on multimodal perception and dynamic evaluation, comprising: a data acquisition module, a data processing module, a rendering module and an evaluation module;

[0011] The data collection module is used to collect the user's appearance data and voice data;

[0012] The data processing module is used to process the appearance data and voice data collected by the data collection module in real time and fuse the processed appearance data and voice data with the SMPL-X model to form a dynamic digital human;

[0013] The rendering module is used to display the virtual teaching scene and render the dynamic digital human into the virtual teaching scene;

[0014] The evaluation module is used to evaluate the user's learning concentration and experimental operation accuracy in the virtual teaching scene.

[0015] Furthermore, the data acquisition module includes: a microphone and several infrared depth-of-field cameras. The several infrared depth-of-field cameras form a circular array, which is used to synchronously collect the user's appearance data and transmit it to the data processing module. The microphone collects the user's voice data in real time and feeds it back to the data processing module.

[0016] Furthermore, the plurality of infrared depth-of-field cameras collect user point cloud data and RGB image data and transmit them to the data processing module.

[0017] Furthermore, the specific steps of the data processing module processing the appearance data and voice data collected by the data collection module in real time and fusing the processed appearance data and voice data with the SMPL-X model are as follows:

[0018] Eliminate depth sensor noise in RGB image data through temporal filtering to preserve key human contour features;

[0019] Locate the user's joint feature points and facial feature points from point cloud data;

[0020] Dynamically align point cloud data with the SMPL-X model;

[0021] Process joint feature points to obtain joint rotation data; map joint rotation data to corresponding bone nodes in the SMPL-X model;

[0022] Process facial feature points to obtain facial feature point displacement; convert facial feature point displacement into expression base in SMPL-X model, and dynamically adjust blink frequency and mouth corner curvature parameters according to the expression base;

[0023] Process the collected voice data and bind the sound source, associate the audio stream of the sound source with the head bone position of the SMPL-X model, and dynamically adjust the volume and high-frequency attenuation based on the distance between the user and the microphone.

[0024] Furthermore, the data processing module separates the hand areas from the point cloud data and extracts several joint points, calculates the hand motion trajectory based on the extracted joint points, and matches the gestures corresponding to the preset control instructions in the SMPL-X model static gesture library and dynamic gesture library based on the hand motion trajectory.

[0025] Furthermore, the specific steps of the rendering module rendering the dynamic digital human into the virtual teaching scene are as follows:

[0026] Establish a 3D model of the virtual teaching scene;

[0027] The status of the SMPL-X model is updated in real time, and the updated SMPL-X model is rendered into a dynamic digital human in the 3D model of the virtual teaching scene.

[0028] Furthermore, the specific steps of the evaluation module evaluating the user's learning concentration in the virtual teaching scene are as follows:

[0029] The OpenEyes eye tracking algorithm is used to calibrate the gaze point coordinates and map them to the virtual teaching scene. The visual attention is obtained by calculating the user's gaze duration and gaze area. The calculation formula for visual attention is:

[0030] ;

[0031] in, Indicates visual attention; G t Indicates that the key teaching area is at the moment t spatial extent; R t Indicates the eye gaze area captured by the RGB camera; IoU Indicates the overlap between the teaching area and the fixation area; Δ t fixation represents the duration of a single fixation; T 1 represents the total duration of the visual attention assessment period;

[0032] The OpenPose body movement recognition algorithm is used to extract several key body angles of the user and calculate the body movement attention. The calculation formula for body movement attention is:

[0033] ;

[0034] in, Indicates attention to body movements; MotionScore ( t ) indicates the time t Body movement score; T 2 represents the total duration of the body movement attention assessment period;

[0035] The expression recognition algorithm is used to identify the user's expression and calculate the expression attention. The calculation formula for expression attention is:

[0036] ;

[0037] in, Indicates facial expression attention; EngagementScore ( t ) represents the facial expression concentration score at time t; T 3 represents the total duration of the facial expression attention assessment period;

[0038] The user's learning concentration is calculated by visual attention, body movement attention, and facial expression attention. The calculation formula is:

[0039] A total =0.5×A eye +0.3×A body +0.2×A face ;

[0040] Among them A total Indicates learning concentration.

[0041] Furthermore, the evaluation module compares the calculated user learning concentration with a set threshold and automatically issues a reminder to users whose learning concentration is lower than the set threshold.

[0042] Furthermore, the evaluation module evaluates the accuracy of the user's experimental operation in the virtual teaching scene, including the evaluation of the process and results. The specific steps are as follows:

[0043] Build a standard operation library, including joint motion trajectories, standard experimental data, and allowable error ranges for standard operations;

[0044] The joint motion trajectory is extracted through the action recognition algorithm, and the accuracy is quantified according to the difference between the extracted joint motion trajectory and the joint motion trajectory of the standard operation. The calculation formula is:

[0045] ;

[0046] in, Indicates the accuracy of the experimental operation process; S t Indicates that the user is at time t The sequence of operation actions; S ref Indicates standard operation; DTW Represents a sequence of actions; MaxDist Indicates the maximum allowable difference value; T 4 represents the total duration of the user experimental operation accuracy evaluation period;

[0047] Then, by comparing the experimental data recorded in the virtual teaching scene with the standard experimental data in the standard operation library, the quantitative difference results are obtained. The calculation formula is:

[0048] ;

[0049] in, Indicates the accuracy of the experimental results; R i Indicates the i The result of the operation; R refi Indicates the i The correct result of the operation; Diff represents the difference calculation function; N Indicates the number of operations; MaxDiff Indicates the maximum allowable difference value;

[0050] according to Evaluate the process, if If the allowable error range recorded in the standard operation library is exceeded, the experimental operation will be interrupted and a warning will be issued;

[0051] like Does not exceed the allowable error range recorded in the standard operation library. Evaluate the results and generate a result evaluation report.

[0052] Compared with the prior art, the present invention has the following beneficial effects:

[0053] The immersive virtual teaching system provided by the present invention collects the user's appearance data and voice data through a data acquisition module, processes the data collected by the data acquisition module in real time through a data processing module integrated with the SMPL-X model, and fuses the processed data with the SMPL-X model to form a dynamic digital human. The virtual teaching scene is displayed through a rendering module and the dynamic digital human is rendered into the virtual teaching scene. The evaluation module evaluates the user's learning concentration and experimental operation accuracy in the virtual teaching scene and generates an evaluation report. The entire system can achieve natural interaction and realistic emotional expression. There will be no delay when multiple users use it. Through multi-dimensional evaluation, the reliability of the evaluation results of user learning concentration and experimental operation accuracy is guaranteed. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is a system framework diagram of the present invention. DETAILED DESCRIPTION

[0055] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0056] See also Figure 1 As shown, an immersive virtual teaching system based on multimodal perception and dynamic evaluation includes: a data acquisition module, a data processing module, a rendering module and an evaluation module;

[0057] The data collection module is used to collect the user's appearance data and voice data;

[0058] The data processing module integrates the SMPL-X model to process the appearance data and voice data collected by the data acquisition module in real time and fuse the processed appearance data and voice data with the SMPL-X model to form a dynamic digital human;

[0059] The rendering module is used to display the virtual teaching scene and render the dynamic digital human into the virtual teaching scene;

[0060] The evaluation module is used to evaluate the user's learning concentration and experimental operation accuracy in the virtual teaching scene.

[0061] In this embodiment, the data acquisition module includes: a microphone and four D435 infrared depth-of-field cameras. The four D435 infrared depth-of-field cameras form a circular array, covering a 240° viewing angle. The field of view of a single D435 infrared depth-of-field camera is 85°×58°, and hardware-level frame synchronization is achieved through a synchronization line with an error of less than 5ms. The four D435 infrared depth-of-field cameras synchronously collect the user's appearance data and transmit it to the data processing module. The microphone collects the user's voice data in real time and feeds it back to the data processing module.

[0062] The D435 infrared depth-of-field camera is calibrated using the checkerboard CharUco composite calibration method, utilizing the camera's RGB-IR dual inputs. The depth sensor calibration of the D435 infrared depth-of-field camera is based on dynamic depth error compensation using a 10×7 checkerboard grid. Multi-machine alignment achieves sub-pixel spatial registration using OpenCV's aruco module, with an RMS reprojection error of less than 0.3 pixels.

[0063] In this embodiment, the specific steps for the data acquisition module to collect user appearance data are as follows: the user stands in the center of a circular array composed of four D435 infrared depth-of-field cameras and rotates one circle at a constant speed of 20 seconds per circle. The user's point cloud data and RGB image data are collected and transmitted to the data processing module.

[0064] In this embodiment, the specific steps of the data processing module processing the appearance data and voice data collected by the data collection module in real time and fusing the processed appearance data and voice data with the SMPL-X model are as follows:

[0065] Eliminate depth sensor noise in RGB image data through temporal filtering to preserve key human contour features;

[0066] Locate the user's joint feature points and facial feature points from point cloud data; joint feature points include: shoulder joints, elbow joints, hip joints, etc., and facial feature points include: eyes, eyebrows, mouth, etc.;

[0067] Dynamically align point cloud data with the SMPL-X model; the SMPL-X model is a universal digital human model that can adjust body shape, posture, and expression; the details of the scanned model are adapted to the SMPL-X model through the non-rigid registration algorithm ICP (Iterative Closest Point).

[0068] When users participate in virtual teaching, joint feature points are processed to obtain joint rotation data; the captured joint rotation data is mapped to the corresponding bone nodes in the SMPL-X model; the SMPL-X model consists of a set of joints and bones, and the rotation data of each joint will be applied to the corresponding part of the model.

[0069] Process facial landmarks to obtain their displacements. Convert the displacements into expression bases from the 10 basic expression bases in the SMPL-X model (see Table 1) that match the current facial landmark displacements. Dynamically adjust the blink frequency and mouth corner curvature parameters based on the expression bases.

[0070] Table 1 - List of the 10 basic expression bases and their corresponding muscle movements in the SMPL-X model

[0071]

[0072] Process the collected voice data and bind the sound source, associate the audio stream of the sound source with the head bone position of the SMPL-X model, and dynamically adjust the volume and high-frequency attenuation based on the distance between the user and the microphone.

[0073] The data processing module processes hand data to support gesture control. It separates the hand regions from the point cloud data and extracts 21 joint points (see Table 2). It then calculates the hand motion trajectory, including position, velocity, and acceleration. It then matches the hand motion trajectory to the preset gestures in the SMPL-X model static gesture library, such as making a fist for confirmation and spreading five fingers for cancellation. It also matches the hand motion trajectory to the gestures corresponding to the preset control commands in the SMPL-X model dynamic gesture library, such as drawing a circle for rotation and pinching for zoom.

[0074] Table 2 - List of joint points for separating the hands area and extracting 21 joint points from point cloud data

[0075]

[0076] In this embodiment, the specific steps of the rendering module rendering the dynamic digital human into the virtual teaching scene are as follows:

[0077] Build a 3D model of the virtual teaching scene, including classrooms, laboratories, or conference rooms. The interiors of classrooms, laboratories, and conference rooms all include walls, tables and chairs, equipment, and ambient lighting. The laboratory also contains experimental equipment and materials. Each of these is then bound to specific trigger conditions.

[0078] The user's motion data and expression data are synchronized to the SMPL-X model in real time; the status of the SMPL-X model is updated in real time, and the updated SMPL-X model is rendered into a dynamic digital human in the 3D model of the virtual teaching scene.

[0079] In the virtual teaching scene, the teacher uses a wide-angle lens to cover the entire audience and automatically highlights the current speaking student through the focus following system; students can switch between third-person and first-person perspectives.

[0080] In the virtual teaching scene, the teacher selects or changes the virtual teaching scene, loads the 3D model and texture of the virtual teaching scene, and the system loads the pre-set rendering effect according to the virtual teaching scene. The virtual teaching scene is rendered on the display device and displayed to all users. Different users have different scene contents due to their different locations.

[0081] After students enter the virtual teaching scene, the system generates an initial position for each student based on the seating layout of the virtual teaching scene and renders the student's dynamic digital human to the corresponding seat, ensuring that each student can see the teacher and teaching content from their perspective;

[0082] The system receives the teacher and student's movement and expression data in real time, maps this data to the SMPL-X model, updates the dynamic digital human's posture and expression, and renders the updated dynamic digital human into the virtual teaching scene; the system then sends the rendering results to each user's display device;

[0083] When teachers move around, operate experimental equipment, or change courseware in the virtual teaching scene, the system renders these actions in real time; when teachers operate equipment, operation trajectory lines are generated, and the operation steps are rendered with particle effects based on the preset object properties and trigger conditions;

[0084] When students move around in the virtual teaching scene, operate experimental equipment, or raise their hands to ask questions, the system renders these movements in real time. When a hand-raising action is detected, the system renders a question mark icon above the student. When conducting virtual experiments, the operation steps are displayed with a semi-transparent guide outline, and the system issues a warning when an incorrect operation is performed.

[0085] When the virtual teaching scene is a multi-person collaborative experiment or multi-person discussion, operating the devices simultaneously will trigger the negotiation interface, which will issue instructions based on the division of labor among the group members and complete the collaboration linearly or in parallel.

[0086] In this embodiment, the specific steps of the evaluation module evaluating the user's learning concentration in the virtual teaching scene are as follows:

[0087] The OpenEyes eye tracking algorithm is used to calibrate the gaze point coordinates and map them to the virtual teaching scene. The visual attention is obtained by calculating the user's gaze duration and gaze area. The calculation formula for visual attention is:

[0088] ;

[0089] in, Indicates visual attention; G t Indicates that key teaching areas (such as virtual blackboards and lab tables) are open at the moment t spatial extent; R t Represents the eye gaze area captured by the RGB camera (calculated by the eye tracking algorithm); IoU Indicates the overlap between the teaching area and the fixation area; Δ t fixation represents the duration of a single fixation (threshold > 200ms); T 1 represents the total duration of the visual attention assessment period;

[0090] The OpenPose body movement recognition algorithm extracts the user's 18 key body angles (see Table 3) and calculates body movement attention. It analyzes posture stability and detects abnormal movements. The calculation formula for body movement attention is:

[0091] ;

[0092] in, Indicates attention to body movements; MotionScore ( t ) indicates the time t Scoring of body movements (e.g., frequent head turning and body shaking will reduce the score); Scoring rules: facing the teaching area: 0 points; slight deviation: 0.5 points; complete deviation: 1 point; T 2 represents the total duration of the body movement attention assessment period;

[0093] Table 3 - List of key body parts of the user's 18 key body part angles extracted by the OpenPose body movement recognition algorithm

[0094]

[0095] The expression recognition algorithm (FER2013 model) is used to identify the user's seven basic expressions (see Table 4), calculate the expression attention, calculate the concentration index, and detect fatigue signals. The calculation formula for expression attention is:

[0096] ;

[0097] in, Indicates facial expression attention; EngagementScore ( t ) represents the facial concentration score at time t; focused expression (such as staring, thinking): 1 point; neutral expression: 0.5 point; distracted expression (such as yawning, frowning): 0 point; T 3 represents the total duration of the facial expression attention assessment period;

[0098] Table 4 - List of 7 basic expressions of users recognized by expression recognition algorithm

[0099]

[0100] The user's learning concentration is calculated by visual attention, body movement attention, and facial expression attention. The calculation formula is:

[0101] A total =0.5×A eye +0.3×A body +0.2×A face ;

[0102] Among them A total Indicates learning concentration.

[0103] Weight description: Visual attention (50%): directly reflects the user's attention to the teaching content; Body movement attention (30%): reflects the user's physical participation; Facial expression attention (20%): reflects the user's emotional involvement;

[0104] The evaluation module compares the calculated user's learning concentration with the set threshold and automatically issues reminders to users whose learning concentration is lower than the set threshold;

[0105] The evaluation module assesses the accuracy of the user's experimental operation in the virtual teaching scene, including the evaluation of the process and results. The specific steps are as follows:

[0106] Build a standard operation library, including joint motion trajectories, standard experimental data, and allowable error ranges for standard operations;

[0107] Then, the key joint motion trajectories are extracted through the action recognition algorithm (OpenPose). The accuracy is quantified based on the difference between the extracted joint motion trajectory and the joint motion trajectory of the standard operation. The calculation formula is:

[0108] ;

[0109] in, Indicates the accuracy of the experimental operation process; S t Indicates that the user is at time t The sequence of operation actions; S ref Indicates standard operation; action sequence DTW : Dynamic Time Warping algorithm, used to calculate the difference between two sequences; MaxDist Indicates the maximum allowable difference value (set according to the complexity of the experimental operation); T 4 represents the total duration of the user experimental operation accuracy evaluation period;

[0110] Then, by comparing the experimental data recorded in the virtual teaching scene with the standard experimental data in the standard operation library, the quantitative difference results are obtained. The calculation formula is:

[0111] ;

[0112] in, Indicates the accuracy of the experimental results; R i Indicates the i The result of the operation; R refi Indicates the i The correct result of the operation; Diff Indicates the difference calculation function, which is selected according to the experiment type;N Indicates the number of operations; MaxDiff Indicates the maximum allowable difference value (set according to the experiment type);

[0113] according to Evaluate the process, if If the allowable error range recorded in the standard operation library is exceeded, the experimental operation will be interrupted and a warning will be issued;

[0114] like Does not exceed the allowable error range recorded in the standard operation library. Evaluate the results and generate a result evaluation report.

[0115] Through multi-dimensional data fusion and intelligent diagnosis, the present invention achieves accurate evaluation of virtual learning scenarios and provides real-time feedback and improvement suggestions.

[0116] The above description is only a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Any simple modification, equivalent replacement and improvement made by any technician familiar with the profession to the above embodiment without departing from the scope of the technical solution of the present invention and based on the technical essence of the present invention shall still fall within the scope of protection of the technical solution of the present invention.

Claims

1. An immersive virtual teaching system based on multimodal perception and dynamic evaluation, characterized by: include: Data acquisition module, data processing module, rendering module and evaluation module; The data collection module is used to collect the user's appearance data and voice data; The data processing module is used to process the appearance data and voice data collected by the data collection module in real time and fuse the processed appearance data and voice data with the SMPL-X model to form a dynamic digital human; The rendering module is used to display the virtual teaching scene and render the dynamic digital human into the virtual teaching scene; The evaluation module is used to evaluate the user's learning concentration and experimental operation accuracy in the virtual teaching scene; The specific steps of the evaluation module evaluating the user's learning concentration in the virtual teaching scene are as follows: The OpenEyes eye tracking algorithm is used to calibrate the gaze point coordinates and map them to the virtual teaching scene. The visual attention is obtained by calculating the user's gaze duration and gaze area. The calculation formula for visual attention is: ； in, Indicates visual attention; G t Indicates that the key teaching area is at the moment t spatial extent; R t Indicates the eye gaze area captured by the RGB camera; IoU Indicates the overlap between the teaching area and the fixation area; Δ t fixation represents the duration of a single fixation; T 1 represents the total duration of the visual attention assessment period; The OpenPose body movement recognition algorithm is used to extract several key body angles of the user and calculate the body movement attention. The calculation formula for body movement attention is: ； in, Indicates attention to body movements; MotionScore ( t ) indicates the time t Body movement score; T 2 represents the total duration of the body movement attention assessment period; The expression recognition algorithm is used to identify the user's expression and calculate the expression attention. The calculation formula for expression attention is: ； in, Indicates facial expression attention; Engagement Score ( t ) represents the facial expression concentration score at time t; T 3 represents the total duration of the facial expression attention assessment period; The user's learning concentration is calculated by visual attention, body movement attention, and facial expression attention. The calculation formula is: A total =0.5×A eye +0.3×A body +0.2×A face ; Among them A total Indicates learning concentration.

2. The immersive virtual teaching system based on multimodal perception and dynamic evaluation according to claim 1 is characterized by: The data acquisition module includes: a microphone and several infrared depth-of-field cameras. The several infrared depth-of-field cameras form a ring array, which is used to synchronously collect the user's appearance data and transmit it to the data processing module. The microphone collects the user's voice data in real time and feeds it back to the data processing module.

3. The immersive virtual teaching system based on multimodal perception and dynamic evaluation according to claim 2, characterized in that: The plurality of infrared depth-of-field cameras collect user point cloud data and RGB image data and transmit them to the data processing module.

4. The immersive virtual teaching system based on multimodal perception and dynamic evaluation according to claim 3 is characterized by: The specific steps of the data processing module processing the appearance data and voice data collected by the data collection module in real time and fusing the processed appearance data and voice data with the SMPL-X model are as follows: Eliminate depth sensor noise in RGB image data through temporal filtering to preserve key human contour features; Locate the user's joint feature points and facial feature points from point cloud data; Dynamically align point cloud data with the SMPL-X model; Process joint feature points to obtain joint rotation data; Map joint rotation data to the corresponding bone nodes in the SMPL-X model; Process facial feature points to obtain facial feature point displacement; convert facial feature point displacement into expression base in SMPL-X model, and dynamically adjust blink frequency and mouth corner curvature parameters according to the expression base; Process the collected voice data and bind the sound source, associate the audio stream of the sound source with the head bone position of the SMPL-X model, and dynamically adjust the volume and high-frequency attenuation based on the distance between the user and the microphone.

5. The immersive virtual teaching system based on multimodal perception and dynamic evaluation according to claim 4 is characterized by: The data processing module also separates the hand areas from the point cloud data and extracts several joint points, calculates the hand motion trajectory based on the extracted joint points, and matches the gestures corresponding to the preset control instructions in the SMPL-X model static gesture library and dynamic gesture library based on the hand motion trajectory.

6. The immersive virtual teaching system based on multimodal perception and dynamic evaluation according to claim 1, characterized in that: The specific steps of the rendering module rendering the dynamic digital human into the virtual teaching scene are as follows: Establish a 3D model of the virtual teaching scene; The status of the SMPL-X model is updated in real time, and the updated SMPL-X model is rendered into a dynamic digital human in the 3D model of the virtual teaching scene.

7. The immersive virtual teaching system based on multimodal perception and dynamic evaluation according to claim 1 is characterized by: The evaluation module compares the calculated user learning concentration with a set threshold and automatically issues a reminder to users whose learning concentration is lower than the set threshold.

8. The immersive virtual teaching system based on multimodal perception and dynamic evaluation according to claim 1 is characterized by: The evaluation module evaluates the accuracy of the user's experimental operation in the virtual teaching scene, including the evaluation of the process and results. The specific steps are as follows: Build a standard operation library, including joint motion trajectories, standard experimental data, and allowable error ranges for standard operations; The joint motion trajectory is extracted through the action recognition algorithm, and the accuracy is quantified according to the difference between the extracted joint motion trajectory and the joint motion trajectory of the standard operation. The calculation formula is: ； in, Indicates the accuracy of the experimental operation process; S t Indicates that students are at t The sequence of operation actions; S ref Indicates standard operation; DTW Represents a sequence of actions; MaxDist Indicates the maximum allowable difference value; T 4 represents the total duration of the student experimental operation accuracy evaluation period; Then, by comparing the experimental data recorded in the virtual teaching scene with the standard experimental data in the standard operation library, the quantitative difference results are obtained. The calculation formula is: ； in, Indicates the accuracy of the experimental results; R i Indicates the i The result of the operation; R refi Indicates the i The correct result of the operation; Diff represents the difference calculation function; N Indicates the number of operations; MaxDiff Indicates the maximum allowable difference value; according to Evaluate the process, if If the allowable error range recorded in the standard operation library is exceeded, the experimental operation will be interrupted and a warning will be issued; like Does not exceed the allowable error range recorded in the standard operation library. Evaluate the results and generate a result evaluation report.

Images