Fusion rendering method of digital human and virtual teaching scene

Abstract

The invention relates to the technical field of virtual reality, in particular to a digital human and virtual teaching scene fusion rendering method, which comprises the following steps of: acquiring sight line staying time and sight line frequency of a virtual teacher image, extracting a current explanation object in a teacher instruction, and measuring the spatial distance between the camera and each teaching courseware model in the virtual teaching scene, and calculating the sight stay time, the explanation object matching degree and the spatial distance to obtain the teaching focus original score of each teaching courseware model. Through dynamic fusion of the visual focus of the virtual teacher and spatial positioning of the teaching model, teaching key points in a scene can be quantified in real time, so that a virtual environment can intelligently deploy rendering resources, the visual fidelity of a current explanation object is automatically improved, secondary elements are weakened, and the teaching efficiency is improved. The self-adaptive visual hierarchical structure effectively guides the attention of students, avoids cognitive overload, and ensures that core teaching content is always presented in the clearest mode.

Description

Technical Field

[0001] This invention relates to the field of virtual reality technology, and in particular to a method for fusion rendering of digital humans and virtual teaching scenarios. Background Technology

[0002] The field of virtual reality technology integrates knowledge from multiple disciplines such as computer graphics, real-time rendering, sensor technology, human-computer interaction, motion tracking and positioning, 3D sound effects, and haptic feedback to build a digital world that allows users to feel as if they are actually there.

[0003] Current virtual reality (VR) technologies often present users with perceptually rich but passively educational environments. In these computer-generated worlds, the allocation of details and visual focus is usually dominated by fixed rules, such as the distance between objects and the user's viewpoint. This lack of contextual awareness means that the virtual environment struggles to distinguish a core teaching component from a background element. For example, in a complex virtual mechanical repair training, a brightly colored but non-critical lever might unintentionally attract more attention than a specific and inconspicuous valve that the instructor is explaining. This leads to distraction and reduced efficiency in knowledge transfer. Furthermore, user interaction and performance evaluation mechanisms are often crude and lagging. User actions are recorded, but the quality of those actions, such as smoothness, accuracy, or speed, is rarely analyzed in real time. Feedback is usually simplified to a simple success or failure summary after the task is completed. This binary result fails to inform users of the specific nature of their errors, leaving them only able to guess the reason for a failed attempt, hindering the gradual improvement of operational skills. Therefore, improvements are needed. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies by proposing a method for fusion rendering of digital humans and virtual teaching scenarios.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: a method for fusion rendering of digital humans and virtual teaching scenarios, comprising the following steps:

[0006] The system obtains the gaze dwell time and gaze frequency of the virtual teacher avatar, extracts the current object of explanation from the teacher's instructions, and measures the spatial distance between the camera and each teaching courseware model in the virtual teaching scene. It calculates the gaze dwell time, the matching degree of the object of explanation, and the spatial distance to obtain the original teaching focus score of each teaching courseware model. It normalizes all the original teaching focus scores to obtain the teaching importance score of the scene model.

[0007] The teaching importance score of the scene model is compared with the preset switching threshold to determine the target level of detail identifier of the teaching courseware model. The vertex position and normal data corresponding to the target level of detail identifier are retrieved and associated with the environment occlusion map of the current student digital avatar view to establish model geometric transition state data.

[0008] Based on the geometric transition state data of the model, the real-time transformation matrix of the teaching courseware model manipulated by the current student digital avatar through skeletal binding is extracted, and the operation deviation quantification value is obtained by comparing it with the standard operation path. The operation deviation quantification value is used to limit the range and obtain the interaction fit score.

[0009] The interaction fit score is used as the interpolation weight. Linear interpolation is performed between the student's free control state and the guided correct state to obtain the guiding interpolation transformation matrix of the manipulated teaching courseware model. The guiding interpolation transformation matrix is ​​then applied to update the model state and generate a fused guiding rendering frame.

[0010] Preferably, the steps for obtaining the teaching importance score of the scenario model are as follows:

[0011] Based on the gaze trajectory sequence and time markers of the virtual teacher avatar, continuous gaze segments are aggregated according to the teaching courseware model index, and the gaze duration and gaze count are accumulated. The teacher's voice commands are analyzed to locate the current object of explanation and calculate the object of explanation matching degree. The spatial distance is obtained by measuring the difference between the camera coordinates and the spatial coordinates of each teaching courseware model. The gaze dwell time, gaze frequency, object of explanation matching degree and spatial distance are generated.

[0012] The original score of the teaching focus is calculated based on the gaze duration, gaze frequency, matching degree of the object of explanation, and spatial distance.

[0013] Based on the original scores of the teaching focus, all the original scores of the teaching focus are normalized to form a teaching importance score for the scenario model.

[0014] Preferably, the step of obtaining the target detail level identifier is as follows:

[0015] Based on the teaching importance score of the scenario model and the preset switching threshold, the teaching courseware model index is compared item by item and the model index and corresponding time identifier exceeding the threshold are recorded. The continuous trigger records are merged and the non-triggered records are removed to obtain the switching trigger determination result.

[0016] Based on the switching trigger determination result, the triggered teaching courseware model is located and the target detail level identifier of the teaching courseware model is read. Vertex position and normal data are retrieved from the resources according to the target detail level identifier of the teaching courseware model and the vertex order is checked to generate the target detail level identifier of the teaching courseware model.

[0017] Preferably, the steps for obtaining the model geometric transition state data are as follows:

[0018] Based on the vertex positions and normal data corresponding to the target detail level identifier, an environment occlusion map is selected from the student's digital avatar perspective, and a texture coordinate mapping is established. The environment occlusion map is bound to the vertex positions and normal data, and the target detail level identifier of the teaching courseware model is recorded to form the model's geometric transition state data.

[0019] Preferably, the step of obtaining the quantized value of the operational deviation is as follows:

[0020] Based on the geometric transition state data of the model, the posture control commands of the teaching courseware model manipulated by the student digital avatar through skeletal binding are analyzed, the rotation matrix and translation matrix of each frame are extracted to form a real-time transformation matrix sequence, the key frame transformation matrices in the standard operation path are aligned in time order, and a pairing set of real-time transformation matrices and key frame transformation matrices is generated.

[0021] Based on the pairing set of the real-time transformation matrix and the keyframe transformation matrix, the rotation and translation components are decomposed for each pair of matrices. The rotation angle deviation is calculated and the rotation difference is obtained in quaternion form. The translation deviation distance is calculated and expressed in three-dimensional Euclidean distance. The rotation deviation and translation deviation of all time frames are calculated respectively to obtain the total rotation deviation and the total translation deviation, which are then summarized into the operation deviation quantization value.

[0022] Preferably, the step of obtaining the interaction compatibility score is as follows:

[0023] The interaction fit score is calculated based on the total rotational deviation and total translational deviation in the quantified operation deviation value.

[0024] Preferably, the step of obtaining the guided interpolation transformation matrix is ​​as follows:

[0025] Based on the interaction fit score, a one-to-one correspondence is established between the student's free control state and the correct guided state according to the skeletal nodes. The rotation, translation, and scaling of the postures at both ends are decomposed and linearly interpolated according to the time markers, and then the matrix of each frame is recombined to obtain the guided interpolation transformation matrix.

[0026] Preferably, the step of obtaining the fusion-guided rendering frame is as follows:

[0027] Based on the guided interpolation transformation matrix, the skeleton index of the manipulated teaching courseware model is traversed, the vertex position and vertex normal are transformed by matrix according to the skeleton weight, and the bounding volume and level update flags are refreshed to generate the updated model state.

[0028] Based on the updated model state, scene lighting probe data is queried for each fragment in world coordinates and trilinear interpolation irradiance coefficients are calculated. The lighting responses of the digital human and the scene are combined and color space conversion is completed to form a fusion-guided rendering frame.

[0029] Compared with the prior art, the advantages and positive effects of the present invention are as follows:

[0030] This invention dynamically integrates the visual focus of the virtual teacher with the spatial positioning of the teaching model, enabling real-time quantification of teaching priorities within the scene. This allows the virtual environment to intelligently allocate rendering resources, automatically enhancing the visual fidelity of the current subject while weakening secondary elements. This adaptive visual hierarchy effectively guides students' attention, avoids cognitive overload, and ensures that core teaching content is always presented in the clearest way. Simultaneously, student actions on the teaching model are tracked, and deviations from the standard path are continuously calculated. High consistency results in minimal intervention, granting users ample control. Conversely, significant deviations trigger stronger but smoother interpolation corrections towards the correct state. This constructs an adaptive learning scaffold, providing immediate and proportional assistance to correct errors without disrupting immersion and cultivating users' procedural skills. Attached Figure Description

[0031] Figure 1 This is a schematic diagram of the steps of the present invention. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0033] Please see Figure 1 This invention provides a technical solution, a method for fusion rendering of digital humans and virtual teaching scenarios, comprising the following steps:

[0034] The system obtains the gaze dwell time and gaze frequency of the virtual teacher avatar, extracts the current object of explanation from the teacher's instructions, and measures the spatial distance between the camera and each teaching courseware model in the virtual teaching scene. It calculates the gaze dwell time, the matching degree of the object of explanation, and the spatial distance to obtain the original score of the teaching focus of each teaching courseware model. It normalizes all the original scores of the teaching focus to obtain the teaching importance score of the scene model.

[0035] The importance score of the scene model teaching is compared with the preset switching threshold to determine the target level of detail of the teaching courseware model. The vertex position and normal data corresponding to the target level of detail are retrieved and associated with the environment occlusion map of the current student digital avatar view to establish the geometric transition state data of the model.

[0036] Based on the model's geometric transition state data, the real-time transformation matrix of the teaching courseware model manipulated by the current student digital avatar through skeletal binding is extracted, and the operation deviation quantification value is obtained by comparing it with the standard operation path. The operation deviation quantification value is used to limit the range and obtain the interaction fit score.

[0037] Using the interaction fit score as the interpolation weight, linear interpolation is performed between the student's free control state and the guided correct state to obtain the guiding interpolation transformation matrix of the manipulated teaching courseware model. The guiding interpolation transformation matrix is ​​then applied to update the model state and generate a fused guiding rendering frame.

[0038] The steps to obtain the importance score of the scenario model instruction are as follows:

[0039] Based on the gaze trajectory sequence and time markers of the virtual teacher avatar, continuous gaze segments are aggregated according to the teaching courseware model index, and the gaze duration and gaze count are accumulated. The teacher's voice commands are analyzed to locate the current object of explanation and calculate the object of explanation matching degree. The spatial distance is obtained by measuring the difference between the camera coordinates and the spatial coordinates of each teaching courseware model. The gaze dwell time, gaze frequency, object of explanation matching degree and spatial distance are generated.

[0040] The raw score for teaching focus is calculated based on gaze duration, gaze frequency, relevance of the subject being explained, and spatial distance. The calculation formula is as follows:

[0041] ;

[0042] in, For the first The original score of the teaching focus of each teaching courseware model. For the first The standardized score of gaze dwell time for each teaching courseware model is calculated as follows: , For the first The standardized score of the gaze frequency of each teaching courseware model is calculated as follows: , For the first The duration of eye contact in a teaching courseware model. This represents the average gaze duration across all teaching courseware models. The standard deviation of the gaze dwell time for all teaching courseware models. For the first The gaze frequency of a teaching courseware model This represents the average gaze frequency across all teaching courseware models. The standard deviation of the gaze frequency for all teaching courseware models. For the first The matching degree of the teaching courseware model to the teaching object. To explain the enhancement coefficient, For the first Spatial distance of each teaching courseware model The maximum value among all teaching courseware model spatial distances. This is the distance attenuation coefficient. These are the weighting coefficients corresponding to gaze duration, gaze frequency, and spatial distance, respectively.

[0043] Based on the original scores of the teaching focus, the original scores of all teaching focus are normalized to form a teaching importance score for the scenario model.

[0044] Specifically, the processing of the virtual teacher's gaze trajectory sequence and time markers begins by extracting valid data points from the unprocessed eye-tracking data stream. This stream contains a series of timestamps, two-dimensional screen coordinates, and potentially mapped three-dimensional spatial coordinates. The teacher's gaze movement trajectory is continuously recorded at a preset sampling frequency, such as 60 times per second. Next, a three-dimensional bounding box is defined as the interaction area for each teaching material model in the scene, such as an engine model, gearbox model, or circuit board model. When a series of consecutive gaze coordinate points, for example, more than 18 data points within 300 milliseconds, fall within the bounding box of the same teaching material model, and the spatial distance between these points is... When the divergence, i.e., the maximum Euclidean distance of the point set, is less than 10% of the diagonal length of the bounding box, this series of data points is aggregated into a continuous gaze segment, and its start and end times, as well as its associated teaching courseware model index, are recorded. This process iterates through all data within a complete teaching cycle, such as 5 minutes, grouping the indices of each teaching courseware model and accumulating the duration of all continuous gaze segments associated with that index to obtain the total gaze dwell time for that model. Simultaneously, the number of independent gaze segments observed in that model is counted to obtain the gaze frequency. At the same time, the teacher's voice command audio stream is processed synchronously, utilizing a speech recognition service based on deep neural networks, such as those used in education and engineering corpora. The finely tuned Wav2Vec2 model converts speech into text strings in real time. Then, a Named Entity Recognition (NER) module is used to extract keywords related to the teaching content from the converted text. This NER module is pre-loaded with a dictionary containing the names, aliases, and related terms of all teaching courseware models, such as "engine," "piston," and "connecting rod." When the teacher says, "Let's observe the movement of this piston," the keyword "piston" is recognized. Next, the matching degree of the teaching object is calculated, that is, the similarity between the recognized keyword and the label of each teaching courseware model is calculated. The similarity here is quantified using word vector cosine similarity, for example... The cosine similarity between the word vector of "piston" and the label vector of "piston model" is 0.98, while the similarity with the label vector of "gearbox model" is 0.15. This highest similarity score is taken as the matching degree of the teaching object of the model. Finally, the spatial distance between the camera coordinates and each teaching courseware model is measured. By obtaining the position coordinates of the virtual camera representing the student's perspective in the world coordinate system, and the coordinates of the geometric center of each teaching courseware model, the three-dimensional Euclidean distance between the two is calculated. The data obtained after these calculations and processing, namely the gaze dwell time, gaze frequency, matching degree of the teaching object and spatial distance corresponding to each teaching courseware model, are stored in a structured manner.

[0045] In the formula for calculating the original score of teaching focus, gaze data, as the core indicator of attention, is weighted and summed to form the base score. Voice commands, as strong indicators of intent, are amplified in the form of multiplicative enhancement terms to increase the importance of the current subject being explained. Spatial distance, as a physical constraint, provides a basic importance weight for nearby objects in the form of exponentially decaying additive terms. This additive and multiplicative structure ensures that the calculated original score of teaching focus can sensitively reflect the teacher's teaching rhythm and focus, and realize intelligent scheduling of scene resources.

[0046] For the first The standardized score of gaze dwell time for each teaching courseware model reflects the persistence of a teacher's attention to a particular model over a period of time. This parameter is obtained by Z-score standardization of the original gaze dwell time, and the calculation formula is as follows: ,in The teacher's data was collected and accumulated from the eye-tracking system in the first... Total fixation time on each model and These are the average and standard deviation of the gaze dwell time for all models in the current scene. This standardization process eliminates the influence of differences in the total duration of different teaching segments, allowing scores to be compared across time periods. For example, within a 60-second observation period, the gaze dwell time of three teaching courseware models (Model 1, Model 2, and Model 3) is statistically analyzed, resulting in a data sequence. for First, calculate the average value. Then calculate the standard deviation. Based on these values, the standardized score of gaze dwell time for Model 2 can be calculated. .

[0047] For the first The standardized score for gaze frequency of a teaching courseware model measures how frequently a teacher focuses on that model. Similar to the standardized score for gaze dwell time, this parameter is also calculated using Z-score standardization. The calculation formula is as follows: ,in It is the teacher's gaze in the first place The number of independent fixations occurring on each model and These are the mean and standard deviation of gaze frequencies across all models. This parameter supplements attention-switching behavior, which cannot be reflected by dwell time. High-frequency saccades also indicate high attention levels. The method for obtaining this data is to record each gaze behavior while calculating gaze dwell time, and count the occurrences by model index. For example, within the same observation period, the gaze frequency data sequence for the three models... for First, calculate the average value. Then calculate the standard deviation. Therefore, the standardized score of the gaze frequency in Model 2 .

[0048] For the first The matching degree of the teaching courseware model is a numerical value between 0 and 1, used to quantify the correlation strength between the teacher's current oral explanation and the specific model. This value is obtained through speech recognition and natural language understanding technologies. First, the teacher's explanation is converted into text using speech-to-text technology. Then, core words are extracted using named entity recognition (NER) or keyword extraction algorithms (such as TextRank). These core words are then compared semantically with a pre-defined tag library for each teaching courseware model. The similarity calculation can use a cosine similarity algorithm based on word vectors (such as Word2Vec or BERT). The calculation formula is as follows: ,in It is a set of keywords extracted from voice commands. It is the first A set of labels for each model. This is a vector representation of the corresponding words. For example, when a teacher says, "Observe the wear condition of this main bearing," the keyword "main bearing" is extracted. The tag library of Model 2 is {"engine," "main bearing," "crankshaft"}. The cosine similarity between "main bearing" and "main bearing" is calculated to be 1.0. However, there are no matching items in the tag libraries of Model 1 and Model 3, so their matching degree is 0. Therefore, we get... , , .

[0049] For the first The spatial distance between the teaching courseware model and the virtual camera (representing the student's perspective) represents the straight-line distance between the virtual camera and the teaching courseware model. This parameter is calculated directly using three-dimensional spatial coordinates to obtain the camera's position in the world coordinate system. and the The position of the geometric center of the model Then spatial distance This value directly reflects scene awareness; the closer the distance, the clearer the observation and the higher the possibility of interaction. For example, if the camera position is... The center positions of the three models are respectively , , The unit is meters, so the spatial distances of the three models can be calculated as follows: , , .

[0050] The maximum value among all spatial distances of the teaching courseware models is the distance set. ,therefore .

[0051] These are weighting coefficients corresponding to gaze duration, gaze frequency, and spatial distance, respectively. The sum of these coefficients is set to 1 to adjust the contribution of different information modalities in the final score calculation. Their setting is based on teaching theory and experience; for example, in teaching emphasizing fine motor skills, prolonged gaze (…) ) compared to frequent scanning ( ) is more important, while spatial distance ( As a basic environmental factor, its weight is relatively low. The weight can be determined using the Analytic Hierarchy Process (AHP). Five senior teaching experts were invited to conduct pairwise comparisons of the three indicators: "sight duration," "sight frequency," and "spatial distance," constructing a judgment matrix. For example, the experts generally considered "duration" slightly more important than "frequency" (assigned a value of 3), "duration" much more important than "distance" (assigned a value of 5), and "frequency" slightly more important than "distance" (assigned a value of 3). By calculating the largest eigenvalue and corresponding eigenvector of the judgment matrix and performing a consistency check, a set of weight assignments was obtained, for example... , , .

[0052] To explain the enhancement coefficient, which determines the extent to which the teacher's verbal explanation increases the score for the teaching focus, its value reflects the degree of reliance on explicit instructions in the teaching strategy. In a teaching model dominated by teacher explanation, It should be set relatively high to ensure that the subject being explained receives the highest level of attention. The specific value can be calibrated experimentally and adjusted through multiple teaching simulations. The value, until the verbally mentioned model's teaching focus raw score ranked first in 95% of cases, was tested and set. This effect can be achieved.

[0053] This is the distance attenuation coefficient, which controls the rate attenuation of the effect of spatial distance on the focal score. The larger the value, the faster the effect of distance decays. Only very close objects will receive a high base score. This value is related to the scale of the virtual scene. For a desktop-sized virtual experimental platform, where the distance between objects is small, a larger value can be set. To highlight the priority of nearby objects, for example However, for a scenario requiring observation of large equipment at a distance, a smaller setting should be used. To ensure that distant objects also receive a certain base score, the scene here is a virtual engine assembly / disassembly platform, which is relatively small in scale, hence the setting. .

[0054] Calculations based on parameters:

[0055] The calculations are performed using the parameter values ​​obtained for Model 2 in the previous example:

[0056] ;

[0057] ;

[0058] ;

[0059] ;

[0060] ;

[0061] , , ;

[0062] ;

[0063] ;

[0064] Substitute the above values ​​into the formula to calculate the original score of the teaching focus in Model 2. :

[0065] ;

[0066] ;

[0067] ;

[0068] The results indicate that the original score of the teaching focus of Model 2 is 3.3122, which is a relatively high score. The score of 3.3122 obtained by Model 2 means that the model is the absolute core of the current teaching activity because it has received long-term and high-frequency attention from the teacher and is being verbally explained.

[0069] Based on the original scores of a set of teaching focus points for each teaching courseware model calculated in the previous step, for example... These raw scores need to be normalized, mapping them to a unified and clearly interpretable range, such as between 0 and 1, to eliminate the impact of score fluctuations in different teaching scenarios. The specific normalization operation uses the min-max normalization method. First, it iterates through the set of raw scores for all teaching focus models in the current scenario and finds the maximum value. and minimum value In the above example, , Subsequently, the raw scores for the teaching focus of each model were determined. The importance score of the teaching of the scenario model is obtained by applying the following normalization formula. The calculation formula is: For Model 1, its importance score is For Model 2, its importance score is For model 3, its importance score is Through this process, the original scores of all models are linearly transformed to the interval [0, 1], where the most important model scores exactly 1 and the least important model scores exactly 0. This score not only preserves the relative size of the original scores, but also provides a standardized metric that can be directly used for subsequent threshold judgment and rendering parameter modulation. Finally, the normalized scores calculated for all models are stored in the form of key-value pairs, namely model index and corresponding scene model teaching importance score.

[0070] The steps for obtaining the target detail level identifier are as follows:

[0071] Based on the teaching importance score of the scenario model and the preset switching threshold, the model index and corresponding time identifier of the teaching courseware model index are compared item by item and recorded. The continuous trigger records are merged and the non-triggered records are removed to obtain the switching trigger judgment result.

[0072] Based on the switching trigger determination result, locate the triggered teaching courseware model and read the target detail level identifier of the teaching courseware model. Retrieve vertex position and normal data from the resources according to the target detail level identifier of the teaching courseware model and check the vertex order to generate the target detail level identifier of the teaching courseware model.

[0073] Specifically, based on the scene model teaching importance score obtained from the aforementioned steps, this score is a set of floating-point numbers in the range [0,1] with the teaching courseware model index as the key and the value as a value. These scores are compared one by one with a dynamically adjusted preset switching threshold. This preset switching threshold is the core parameter of teaching interaction sensitivity. Its setting is not a fixed value, but is calculated by combining a baseline value and a floating value. The baseline value is determined based on the previous user experience test, for example, set to 0.75. This value represents the minimum importance level that needs to trigger a system response. The floating value is calculated based on the standard deviation of the importance scores of all models in a recent period, such as the past 5 seconds. The final switching threshold is formed by adding 0.5 times the standard deviation to the baseline value. This setting allows the threshold to be appropriately increased when the teaching focus changes rapidly and the score difference increases, and appropriately decreased when the teaching content is stable and the scores of each model are generally low, thereby avoiding overly frequent or overly sluggish switching responses. In specific execution, for each frame rendered, all teaching courseware models are traversed. For each model, its current scene model teaching importance score is extracted and compared with the calculated switching threshold of the current frame. If a model's score is greater than or equal to the threshold, a trigger record is generated. This record contains the model's unique index, the current timestamp, and the model's score. All these instantaneous trigger records are stored in a temporary buffer. Then, the records in the buffer are merged. A time window is set, such as 300 milliseconds. The consecutive trigger records for the same model index in the buffer are traversed. If the time interval between two records is less than 300 milliseconds, they are considered as a continuation of the same teaching attention event. The latter record is merged into the former, the end timestamp of the event is updated, and the highest score during the period is taken as the intensity value of the event. Conversely, if the time interval exceeds 300 milliseconds, the previous attention event is considered to have ended, and a new attention event begins. After traversal and merging, all empty records of models that have not triggered any threshold judgment are cleared, and finally a structured switching trigger judgment result is formed.

[0074] Based on the switching trigger determination result generated in the previous step, this result is a list containing the triggered model index and the corresponding trigger time period. Each item in the list is processed. First, the specific teaching courseware model object is located in the scene's resource manager using the model index, for example, the model with the index "Engine_Block_01". Next, the pre-configured metadata of this model is read. The metadata defines identifiers for different levels of detail (LOD) and their mapping relationships to the scene model's teaching importance score range. For example, a score of [0.8, 1.0] maps to "LOD0" (highest detail), a score of [0.5, 0.8) maps to "LOD1", and a score of [0.2, 1.0] maps to "LOD1" (highest detail). 0.5) is mapped to "LOD2". Since the switching trigger determination itself is based on the score exceeding a high threshold, the target level of detail identifier of the triggered model is determined to be the highest level, i.e., "LOD0". Then, using this "LOD0" identifier as an index, the corresponding geometric data is asynchronously loaded from the project's resource package. This data is usually stored in binary format and contains the coordinates of all vertex positions of the model (an array of x, y, z floating-point numbers) and vertex normal vectors (also x, y, z). (An array of floating-point numbers). After the data is loaded, a crucial calibration step is performed: verifying the consistency of the vertex order. To achieve a smooth LOD transition animation, the vertices shared between the low-detail model and the high-detail model must have a consistent index order. This calibration process is accomplished by loading a vertex index mapping table corresponding to the model. This table is automatically generated when the 3D modeling software exports the model. It records that vertex i in "LOD1" corresponds to vertex j in "LOD0". By traversing this mapping table, the loaded "LOD0" vertex data buffer is rearranged so that its vertex order matches the vertex order of the currently displayed low-detail model. For newly added vertices in "LOD0", they are appended to the end of the buffer. Through this series of operations, a final result is generated that contains the target level of detail identifier "LOD0" and the vertex position and normal data that have been calibrated in order.

[0075] The steps for obtaining model geometric transition state data are as follows:

[0076] Based on the vertex positions and normal data corresponding to the target detail level identifier, select the environment occlusion map from the student's digital avatar perspective and establish texture coordinate mapping. Bind the environment occlusion map to the vertex positions and normal data and record the target detail level identifier of the teaching courseware model to form the model's geometric transition state data.

[0077] Specifically, after obtaining the vertex positions and normals corresponding to the target level of detail (LOD0) identifier and with the vertex order already corrected, the next step is to configure appropriate lighting and surface detail information for this geometric data, especially the environment occlusion map. This map is pre-baked and generated separately for different levels of detail of each model to simulate the soft shadows produced by geometric occlusion on the model surface. First, based on the current viewpoint of the student's digital avatar, i.e., the position and orientation of the virtual camera, the closest one is selected from a set of pre-computed lighting probes. However, this does not directly affect the selection of the environment occlusion map. The selection of the environment occlusion map is strictly bound to the model's level of detail; that is, the "LOD0" model corresponds to the "AO_LOD0.dds" map, and the "LOD1" model corresponds to the "AO_LOD1.dds" map. Therefore, here, the environment map to be loaded is directly specified based on the target level of detail identifier "LOD0". Next, after creating the occlusion map resource, a texture coordinate mapping is established. This step utilizes the UV coordinates (also known as texture coordinates) inherent in the model data. Each vertex, in addition to its position and normal data, also contains a two-dimensional UV coordinate, which specifies which point on the two-dimensional texture map the vertex corresponds to. In the rendering pipeline, the loaded environment occlusion map is bound to a specific texture unit, and the color value of this map is sampled in the shader program using the model's UV coordinates. This process essentially "pastes" the two-dimensional shadow information correctly onto the three-dimensional model surface. Then, this selected environment occlusion map, along with the previously prepared target vertex position data, target normal data, the source vertex data and source normal data currently being used by the model, and the recorded current level of detail (LMD) and target LMD, are packaged together to form a complete set of model geometric transition state data.

[0078] The steps for obtaining the quantized value of operational deviation are as follows:

[0079] Based on the model's geometric transition state data, the posture control commands of the teaching courseware model manipulated by the student digital avatar through skeletal binding are analyzed. The rotation matrix and translation matrix of each frame are extracted to form a real-time transformation matrix sequence. The keyframe transformation matrices in the standard operation path are aligned in time order to generate a pairing set of real-time transformation matrices and keyframe transformation matrices.

[0080] Based on the pairing set of real-time transformation matrix and keyframe transformation matrix, the rotation and translation components are decomposed for each pair of matrices. The rotation angle deviation is calculated and the rotation difference is obtained in quaternion form. The translation deviation distance is calculated and expressed in three-dimensional Euclidean distance. The rotation deviation and translation deviation of all time frames are calculated respectively to obtain the total rotation deviation and the total translation deviation, which are then summarized into the operation deviation quantization value.

[0081] Specifically, based on the model geometric transition state data obtained from the aforementioned steps, the posture control commands issued by the student digital avatar are first analyzed in real time. These commands originate from virtual reality controllers, data gloves, or motion capture devices, capturing the user's manipulation actions on the bound teaching material model, such as a virtual wrench, at a frequency of no less than 60 times per second. Each frame's command is parsed into a transformation request for the model's root bone or a specified control bone, containing translation, rotation, and scaling components. These components are combined into a 4x4 homogeneous transformation matrix, which is the real-time transformation matrix for the current frame. These matrices are continuously recorded to form a real-time transformation matrix sequence synchronized with the rendering frame rate. Next, a preset standard operation path is loaded. This path is pre-recorded or manually edited by the instructional designer and contains a series of timestamped key points. Keyframes store a standard 4x4 transformation matrix, representing the correct pose the model should be in at that moment. For comparison, discrete keyframe paths need to be aligned with continuous real-time paths. Using temporal linear interpolation, for each matrix in the real-time transformation matrix sequence, its corresponding timestamp is recorded. Two temporally adjacent keyframes are found in the standard operation path. By performing spherical linear interpolation (for rotation) and linear interpolation (for translation) between the transformation matrices of these two keyframes, the interpolated transformation matrix corresponding to the standard path at the current real-time timestamp is calculated. The real-time transformation matrix of each frame is paired with the interpolated standard transformation matrix of its corresponding timestamp, ultimately generating a set of paired real-time transformation matrices and keyframe transformation matrices that contains the entire operation process.

[0082] Based on the pairing set of real-time transformation matrices and keyframe transformation matrices generated in the previous step, we iterate through each pair of matrices in the set (real-time matrix, standard matrix), decompose each pair, and extract their respective rotation and translation components. The rotation component is usually represented as a 3x3 rotation matrix or a quaternion, while the translation component is a three-dimensional vector. To calculate the rotation deviation, both rotation matrices are converted into quaternion representations, and then the angular distance between them is calculated. Specifically, the quaternion corresponding to the real-time rotation is multiplied by the conjugate of the quaternion corresponding to the standard rotation to obtain a difference quaternion representing the difference between the two. Twice the arccosine of the scalar part of this difference quaternion is the rotation angle deviation of the current frame, in radians. For the translation deviation, we calculate the two translation components... The three-dimensional Euclidean distance between the translation vectors, which is the square root of the sum of the squares of the differences in the corresponding coordinates of the two vectors, is the translation deviation of the current frame, in meters. This process is repeated for all frames in the pairing set to obtain two deviation sequences: one records the rotation angle deviation of each frame, and the other records the translation deviation of each frame. Finally, in order to obtain an overall deviation value that can represent the entire operation process, the two deviation sequences are summed and calculated. The sum of the squares of the rotation angle deviation values ​​of all time frames is calculated, and the square root is taken to obtain the total rotation deviation. Similarly, the sum of the squares of the translation deviation distances of all time frames is calculated, and the square root is taken to obtain the total translation deviation. The two summed values, namely the total rotation deviation and the total translation deviation, together constitute the quantified value of the operation deviation.

[0083] The steps to obtain the interaction compatibility score are as follows:

[0084] The interaction fit score is calculated based on the total rotational deviation and total translational deviation in the quantified operational deviation value. The calculation formula is as follows:

[0085] ;

[0086] in, The interaction compatibility score is used to determine the interaction fit. The rotation penalty coefficient is... This is the translation penalty coefficient. The total rotation deviation is calculated as the square root of the sum of the squares of the rotation differences across all time frames. The total translation deviation is calculated as the square root of the sum of the squares of the translation differences across all time frames. This is the maximum rotational tolerance error value. This represents the maximum tolerance value for translation.

[0087] Specifically, in the formula for calculating the interaction fit score, the greater the deviation, the heavier the penalty and the lower the score. The formula structure includes the total rotation deviation. and total translation deviation First, based on their respective maximum tolerance error values... and Normalization eliminates scale issues caused by varying dimensions and different task difficulties. Then, the normalized deviation is squared, which exponentially penalizes larger deviations, thus increasing sensitivity to serious operational errors. (Rotation penalty coefficient) Translation penalty coefficient This allows for flexible adjustment of the weights of the two types of deviations in the total score based on the different requirements of posture and position accuracy for different teaching tasks. Finally, the calculated total weighted penalty value (a non-positive number) is mapped to the (0, 1] interval through the natural exponential function, resulting in an intuitive and easy-to-use fit score for subsequent interpolation calculations.

[0088] The total rotational deviation is a portion of the quantified operational deviation calculated using the preceding steps. It comprehensively reflects the cumulative difference between the student-operated teaching material model and the standard path in terms of rotational posture throughout the entire operation. This parameter is calculated by squaring the rotational angle deviation (the angle between the student's rotational posture and the standard posture, in radians) for each time frame, summing the squared values ​​for all time frames, and finally taking the square root of the sum. ,in It is the first Frame rotation angle deviation, This refers to the total number of frames. This calculation method essentially uses the L2 norm of the rotation deviation vector over time, effectively measuring the overall magnitude of the rotation error. For example, in an operation sequence containing 100 frames, the calculated sum of squared rotation deviations for each frame is 0.64. So the total rotational deviation radian.

[0089] The total translational deviation, also derived from the quantified operational deviation, quantifies the cumulative deviation of the student-manipulated teaching material model from the standard path in spatial position throughout the entire operation. Its calculation method is similar to the total rotational deviation: the translational deviation distance (the three-dimensional Euclidean distance between the student's operating position and the standard position, in meters) for each time frame is squared, then the squared values ​​of all frames are summed, and finally the square root of the sum is taken. ,in It is the first Frame translation deviation distance, This is the total number of frames. This value is the L2 norm of the translation deviation vector over time, which intuitively reflects the accuracy of position control. For example, in a 100-frame operation sequence, the calculated sum of squares of the translation deviations for each frame is 0.0225. So the total translation deviation rice.

[0090] The maximum rotation tolerance error value is a preset threshold parameter representing the maximum total rotation deviation the system can tolerate in a specific teaching task. Its setting should be based on the analysis of the task's accuracy requirements and the statistical analysis of expert operation data. Specifically, the setting process involves inviting at least 10 skilled operators or experts to repeatedly complete the same teaching task 5 times in the same virtual environment, recording the complete data for each operation, and calculating the corresponding total rotation deviation. Collect all of these The values ​​form a sample set, and the mean of that sample set is calculated. and standard deviation To define a reasonable upper limit for tolerance, the maximum rotational tolerance error can be set as the average value plus three times the standard deviation, i.e. This value covers approximately 99.7% of expert operation performance, representing a lenient but still within the normal range of deviation. For example, statistical analysis shows that the average total rotational deviation of expert operations is 0.3 radians, with a standard deviation of 0.05 radians. radian.

[0091] This is the maximum translation tolerance error value. Its meaning and setting method are similar to the maximum rotation tolerance error value. It is for the total translation deviation. The set tolerance limit is also determined by collecting and analyzing expert operational data, i.e. ,in and These are the mean and standard deviation of the total translational deviation of the expert operation sample set, respectively. Setting this parameter provides the system with an objective basis for evaluating positional deviation, enabling it to adapt to the spatial positioning accuracy requirements of different tasks. For example, for a task requiring precise insertion of parts into slots, high translational accuracy is required, and the expert data... and It will be very small, thus resulting in a smaller one. For example, if the average total translational deviation of the expert operation is 0.05 meters and the standard deviation is 0.01 meters, then... rice.

[0092] The rotation penalty coefficient is a dimensionless adjustment parameter used to control the sensitivity of the interaction fit score to rotational deviation. Its setting is based on the dependence of the teaching task itself on the accuracy of rotational posture. For tasks where rotation is a core element, such as tightening screws or aligning valves, a higher penalty coefficient should be set. The value is set so that even a small rotational deviation can lead to a decrease in the score; conversely, a lower value can be set. The value can be determined as follows: Set a desired penalty effect, for example, when the student's total rotational deviation exactly reaches the maximum tolerance error value. When (e.g., the translation deviation is 0), the expected interaction fit score is... Drop to a specific value, for example According to the formula Substitute and , can be obtained Solving for If you want a harsher punishment, such as lowering the score... ,but For the assembly task in this example, rotation and translation are equally important, therefore, we set... .

[0093] This is the translation penalty coefficient, similar to the rotation penalty coefficient, which controls the sensitivity of the score to translational deviations. Its setting is based on the task's requirements for spatial accuracy, and the setting method is also similar to... Similarly, by setting the total translation deviation to achieve the maximum tolerance error value. The coefficient is derived by inversely from the expected score, i.e. ,in This is the target score. In many operations, rotation and translation are equally important, and in this case, it can be set... However, in certain tasks, such as placing an object on a wide platform, the accuracy of the position is not high, but the orientation (e.g., keeping it horizontal) is crucial. In such cases, a more precise setting should be used. In this example, we set It is consistent with the rotation penalty coefficient.

[0094] Calculations based on parameters:

[0095] Calculations are performed using the parameter values ​​obtained from the previous example:

[0096] radian;

[0097] rice;

[0098] radian;

[0099] rice;

[0100] ;

[0101] ;

[0102] Substitute the above values ​​into the formula to calculate the interaction compatibility score. :

[0103] ;

[0104] ;

[0105] ;

[0106] ;

[0107] The results indicate that the current student's interaction fit score is 0.0000447, which is very close to 0. This indicates a significant deviation between the student's actual operation and the preset standard operation path. Specifically, the calculated total rotation deviation (0.8 radians) and total translation deviation (0.15 meters) far exceed the system's maximum tolerance error values ​​(0.45 radians and 0.08 meters, respectively), resulting in an extremely low fit score. This score will be used as an interpolation weight in subsequent steps. Such a low score means that the system will provide very strong guidance. In the subsequently generated rendering frames, the posture of the manipulated teaching courseware model will be almost entirely determined by the correct state of the guidance, rather than the state that the student currently controls freely.

[0108] The steps for obtaining the guided interpolation transformation matrix are as follows:

[0109] Based on the interaction fit score, a one-to-one correspondence is established between the student's free control state and the correct guided state according to the skeletal nodes. The rotation, translation, and scaling of the postures at both ends are decomposed and linearly interpolated according to the time markers, and then the matrix of each frame is recombined to obtain the guided interpolation transformation matrix.

[0110] Specifically, based on the interaction fit score calculated in the aforementioned steps, for example, a value close to 0 such as 0.0000447, this score is used as an interpolation weight to smoothly transition between the student's current free operating state and the system's preset guided correct state. First, for each skeletal node in the manipulated teaching courseware model, such as the "root bone" and "handle end bone" of a virtual wrench, two corresponding posture sources are established. The posture of the student's free control state comes from the local transformation matrix of each skeletal node in the current frame obtained after real-time parsing of user input device commands, while the guided correct state comes from the standard operation path data. By querying the current time marker, the standard local transformation matrix that each skeletal node should be at at this moment is interpolated from the standard path. Next, for each pair of corresponding skeletal transformation matrices, i.e., the student's matrix and the standard matrix, they are decomposed. Each 4x4 matrix is ​​decomposed into a three-dimensional translation vector, a quaternion representing rotation, and a three-dimensional scaling vector. Then, the interaction fit score is used as an interpolation factor to linearly interpolate the decomposed components. Specifically, the interpolation of the translation vector is interpolated translation = (1 - Interaction Fit Score) * Standard Translation + Interaction Fit Score * Student translation. For the rotation component, spherical linear interpolation (Slerp) is used to obtain the shortest and constant-speed rotation path, i.e., interpolated rotation = Slerp(standard rotation, student rotation, interaction fit score). The interpolation method of the scaling vector is the same as that of the translation vector. When the interaction fit score is extremely low, such as 0.0000447, the interpolation result will be infinitely close to the correct guided state. Conversely, when the score is close to 1, the student's free operation is almost completely preserved. After the interpolation calculation is completed for all components of a certain bone node, the obtained interpolated translation vector, interpolated rotation quaternion, and interpolated scaling vector are recombined into a new 4x4 local transformation matrix. This process is repeated for all nodes in the model bone level, and finally a complete set of guided interpolation transformation matrices for all bones in the current frame is obtained.

[0111] The steps for obtaining the merged guided rendering frame are as follows:

[0112] Based on the guided interpolation transformation matrix, the skeleton index of the manipulated teaching courseware model is traversed, the vertex position and vertex normal are transformed by matrix according to the skeleton weight, and the bounding volume and level update flags are refreshed to generate the updated model state.

[0113] Based on the updated model state, the scene lighting probe data is queried for each fragment in world coordinates, and the trilinear interpolation irradiance coefficient is calculated. The lighting response of the digital human and the scene is combined and the color space conversion is completed to form a fusion guided rendering frame.

[0114] Specifically, based on the guided interpolation transformation matrix generated for each node in the model's skeletal hierarchy in the previous step, a standard graphics skinning process is executed to update the geometry of the entire teaching material model. First, all vertices constituting the model are traversed. For each vertex, its bound bone information is read from the model data. This typically includes up to four bone indices affecting the vertex and their corresponding weight values, the sum of which is 1. Next, for each bone affecting the current vertex, its corresponding guided interpolation transformation matrix and an inverse binding matrix of that bone, pre-calculated and stored when the model is posed, are obtained. These two matrices are multiplied to obtain the final skinning transformation matrix applied to the vertex. Then, the vertex's original model space position (position when posed) is transformed using this skinning transformation matrix to obtain the vertex's new position under the influence of the current bone. The new positions of all bones affecting this vertex are calculated. The vertex is weighted and summed according to its respective weights to obtain its final world space position in the current frame. A similar process is used for the transformation of vertex normals, but the inverse of the skinning transformation matrix is ​​used to avoid erroneous deformation of normals under non-uniform scaling. After all vertex positions and normals have been calculated and updated to the vertex buffer, the model's bounding volume needs to be refreshed, that is, the axis-aligned bounding box (AABB) of the model in world space needs to be recalculated. By traversing all updated vertex positions, the minimum and maximum values ​​on the X, Y, and Z coordinate axes are found to determine the new bounding box range. At the same time, a hierarchical update flag is set. This is a boolean flag used to notify the scene graph management system that the world transformation matrix of the model and all its child nodes has become invalid and needs to be recalculated before the next frame is rendered. Finally, these updated vertex data, normal data, new bounding boxes, and update flags together constitute the updated model state.

[0115] Using the updated model state, the final rendering and shading stage begins. First, the updated vertex data is submitted to the graphics processing unit (GPU). In the vertex shader, a model-view-projection transformation is performed on the vertices. Then, the rasterizer converts the 3D triangles into 2D fragments on the screen. For each fragment, lighting calculations are performed in the fragment shader. Using its position in the world coordinate system, the pre-placed lighting probe mesh in the scene is queried. The lighting probes are pre-calculated and stored when the scene is loaded, containing lighting information of the surrounding environment, usually in the form of cube maps. Based on the fragment's world coordinates, the probe mesh cell in which it belongs is determined, and the lighting probes at the eight corner points surrounding that cell are found. Then, the relative position of the fragment within that cell is calculated, and this relative position is used as a weight to calculate the irradiance data (used to calculate diffuse indirect lighting) collected by the eight probes. Trilinear interpolation is performed to obtain the precise indirect diffuse color of the fragment location. Similarly, trilinear interpolation is also performed on the pre-filtered environment map stored by the probe (used to calculate the indirect lighting for specular reflection) to obtain the indirect specular color of the fragment. These indirect lighting results are combined with the direct lighting results calculated by resolving light sources (such as point lights and parallel lights), as well as the model's own material properties (such as base color, roughness, and metallicity) and the occlusion values ​​sampled from the environment occlusion map, under a physically based rendering (PBR) lighting model to calculate the final lighting color of the fragment. This process is also applied to the student digitized human model in the scene. Finally, the calculated high dynamic range (HDR) color is tone-mapped and converted from linear color space to sRGB color space to form the final blended guided rendering frame that can be displayed.

[0116] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments that can be applied to other fields. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A method for fusion rendering of digital humans and virtual teaching scenarios, characterized in that, Includes the following steps: The system obtains the gaze dwell time and gaze frequency of the virtual teacher avatar, extracts the current object of explanation from the teacher's instructions, and measures the spatial distance between the camera and each teaching courseware model in the virtual teaching scene. It calculates the gaze dwell time, the matching degree of the object of explanation, and the spatial distance to obtain the original teaching focus score of each teaching courseware model. It normalizes all the original teaching focus scores to obtain the teaching importance score of the scene model. The teaching importance score of the scene model is compared with the preset switching threshold to determine the target level of detail identifier of the teaching courseware model. The vertex position and normal data corresponding to the target level of detail identifier are retrieved and associated with the environment occlusion map of the current student digital avatar view to establish model geometric transition state data. Based on the geometric transition state data of the model, the real-time transformation matrix of the teaching courseware model manipulated by the current student digital avatar through skeletal binding is extracted, and the operation deviation quantification value is obtained by comparing it with the standard operation path. The operation deviation quantification value is used to limit the range and obtain the interaction fit score. The interaction fit score is used as the interpolation weight. Linear interpolation is performed between the student's free control state and the guided correct state to obtain the guiding interpolation transformation matrix of the manipulated teaching courseware model. The guiding interpolation transformation matrix is ​​then applied to update the model state and generate a fused guiding rendering frame.

2. The method for fusion rendering of digital humans and virtual teaching scenarios according to claim 1, characterized in that, The steps for obtaining the teaching importance score of the scenario model are as follows: Based on the gaze trajectory sequence and time markers of the virtual teacher avatar, continuous gaze segments are aggregated according to the teaching courseware model index, and the gaze duration and gaze count are accumulated. The teacher's voice commands are analyzed to locate the current object of explanation and calculate the object of explanation matching degree. The spatial distance is obtained by measuring the difference between the camera coordinates and the spatial coordinates of each teaching courseware model. The gaze dwell time, gaze frequency, object of explanation matching degree and spatial distance are generated. The original score of the teaching focus is calculated based on the gaze duration, gaze frequency, matching degree of the object of explanation, and spatial distance. Based on the original scores of the teaching focus, all the original scores of the teaching focus are normalized to form a teaching importance score for the scenario model.

3. The method for fusion rendering of digital humans and virtual teaching scenarios according to claim 1, characterized in that, The steps for obtaining the target detail level identifier are as follows: Based on the teaching importance score of the scenario model and the preset switching threshold, the teaching courseware model index is compared item by item and the model index and corresponding time identifier exceeding the threshold are recorded. The continuous trigger records are merged and the non-triggered records are removed to obtain the switching trigger determination result. Based on the switching trigger determination result, the triggered teaching courseware model is located and the target detail level identifier of the teaching courseware model is read. Vertex position and normal data are retrieved from the resources according to the target detail level identifier of the teaching courseware model and the vertex order is checked to generate the target detail level identifier of the teaching courseware model.

4. The method for fusion rendering of digital humans and virtual teaching scenarios according to claim 1, characterized in that, The steps for obtaining the model's geometric transition state data are as follows: Based on the vertex positions and normal data corresponding to the target detail level identifier, an environment occlusion map is selected from the student's digital avatar perspective, and a texture coordinate mapping is established. The environment occlusion map is bound to the vertex positions and normal data, and the target detail level identifier of the teaching courseware model is recorded to form the model's geometric transition state data.

5. The method for fusion rendering of digital humans and virtual teaching scenarios according to claim 1, characterized in that, The steps for obtaining the quantified value of the operational deviation are as follows: Based on the geometric transition state data of the model, the posture control commands of the teaching courseware model manipulated by the student digital avatar through skeletal binding are analyzed, the rotation matrix and translation matrix of each frame are extracted to form a real-time transformation matrix sequence, the key frame transformation matrices in the standard operation path are aligned in time order, and a pairing set of real-time transformation matrices and key frame transformation matrices is generated. Based on the pairing set of the real-time transformation matrix and the keyframe transformation matrix, the rotation and translation components are decomposed for each pair of matrices. The rotation angle deviation is calculated and the rotation difference is obtained in quaternion form. The translation deviation distance is calculated and expressed in three-dimensional Euclidean distance. The rotation deviation and translation deviation of all time frames are calculated respectively to obtain the total rotation deviation and the total translation deviation, which are then summarized into the operation deviation quantization value.

6. The method for fusion rendering of digital humans and virtual teaching scenarios according to claim 1, characterized in that, The steps for obtaining the interaction compatibility score are as follows: The interaction fit score is calculated based on the total rotational deviation and total translational deviation in the quantified operation deviation value.

7. The method for fusion rendering of digital humans and virtual teaching scenarios according to claim 1, characterized in that, The steps for obtaining the guided interpolation transformation matrix are as follows: Based on the interaction fit score, a one-to-one correspondence is established between the student's free control state and the correct guided state according to the skeletal nodes. The rotation, translation, and scaling of the postures at both ends are decomposed and linearly interpolated according to the time markers, and then the matrix of each frame is recombined to obtain the guided interpolation transformation matrix.

8. The method for fusion rendering of digital humans and virtual teaching scenarios according to claim 1, characterized in that, The steps for obtaining the fused guided rendering frame are as follows: Based on the guided interpolation transformation matrix, the skeleton index of the manipulated teaching courseware model is traversed, the vertex position and vertex normal are transformed by matrix according to the skeleton weight, and the bounding volume and level update flags are refreshed to generate the updated model state. Based on the updated model state, scene lighting probe data is queried for each fragment in world coordinates and trilinear interpolation irradiance coefficients are calculated. The lighting responses of the digital human and the scene are combined and color space conversion is completed to form a fusion-guided rendering frame.

Images