Virtual Reality-Based Digital Processing System for Educational Courses

By establishing a closed-loop processing pipeline that enables semantic-driven automatic generation of 3D scenes, real-time perception of multimodal cognitive load and dynamic scene control, and unified cross-platform encapsulation and parsing, the problems of automated generation of virtual reality education courses, cognitive load perception, and poor cross-platform interoperability have been solved, achieving efficient and low-cost production of virtual reality education content and learning experience.

CN123114253APending Publication Date: 2026-11-17BEIJING ALIXUE EDUCATION TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611250138.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-18
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing virtual reality education courses suffer from low levels of digital production automation, inability to perceive and dynamically intervene in cognitive load in real time, and extremely poor cross-platform interoperability, resulting in long course production cycles, high costs, poor learning outcomes, and the inability to circulate and reuse content across platforms.

Method used

This invention provides a virtual reality-based digital processing system for educational courses, including a closed-loop processing pipeline for semantically driven automatic generation of 3D scenes, real-time perception of multimodal cognitive load and dynamic scene control, and cross-platform unified intermediate layer representation encapsulation and parsing. It realizes end-to-end automated generation of lesson plan texts into 3D virtual reality scenes, real-time perception of cognitive load and adaptive scene control, and unified encapsulation and parsing of cross-platform content.

Benefits of technology

It enables rapid and efficient production of virtual reality courses, real-time control of learners' cognitive load and optimal flow state, and free circulation and reuse of cross-platform content, reducing production thresholds and costs, and improving learning efficiency and the large-scale development of educational content.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123114253A_ABST
    Figure CN123114253A_ABST
Patent Text Reader

Abstract

This invention discloses a virtual reality-based digital processing system for educational courses, belonging to the field of digital cultural and creative activities. The system consists of three functional subsystems: a semantically driven 3D scene automatic generation subsystem, which parses lesson plan texts using a large language model, constructs a teaching knowledge graph, automatically solves the 3D scene spatial layout using a semantically constrained wave function collapse algorithm, and drives the engine to dynamically instantiate and output virtual teaching scenes; a multimodal cognitive load real-time perception and scene dynamic control subsystem, which uses the native sensors of virtual reality devices to collect five-dimensional implicit behavioral features such as pupil diameter change rate and gaze point dispersion, outputs a cognitive load index through a lightweight federated learning classifier, and drives the adaptive control of scene detail level, lighting, and semantic node visibility in real time; and a cross-platform unified intermediate layer representation encapsulation and parsing subsystem, which encapsulates course content into a four-layer structured format file carrying educational semantic weights and cognitive control strategies.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention belongs to the field of digital cultural and creative activities, and involves the cross-application of virtual reality education technology, artificial intelligence technology and three-dimensional graphics rendering technology. Specifically, it relates to a virtual reality-based digital processing system for educational courses, which can be applied to the automated production, cognitive adaptive presentation and cross-platform digital asset packaging and distribution of virtual reality educational courses. Background Technology

[0002] I. Current Status of Digital Production of Virtual Reality Education Courses

[0003] Virtual reality technology generates three-dimensional immersive environments through computers, providing learners with embodied and contextualized interactive learning experiences. It has shown significant application value in fields such as science and engineering experimental teaching, vocational skills training, and cultural heritage education. The digital production of virtual reality education courses usually involves multiple stages, including three-dimensional scene modeling, engine integration, and content packaging and publishing. The production cycle is long, the professional threshold is high, and it is highly dependent on professional artists and engine engineers. In recent years, with the development of artificial intelligence content generation technology, some toolchains can generate three-dimensional assets from images or text. However, the above technologies have not yet formed a systematic end-to-end integration with the semantic structure of education courses, and cannot automatically transform the teaching logic contained in the lesson plan text into a structured three-dimensional interactive scene.

[0004] II. Cognitive Load Theory and Virtual Reality Learning

[0005] Cognitive load theory categorizes the burden on the brain's working memory during learning into three types: intrinsic load, extrinsic load, and proliferative load. Studies have shown that virtual reality learning environments, due to their high information density and rich spatial stimuli, are highly likely to induce cognitive overload and learning fatigue in learners. This not only affects learning outcomes but can also trigger physiological discomfort such as motion sickness. At the level of physiological signal perception, eye tracking has been widely validated as an effective means of assessing cognitive load. Increased pupil diameter and increased fixation point dispersion are reliable physiological indicators of cognitive overload. Currently, mainstream virtual reality headsets have integrated eye tracking modules and six-degrees-of-freedom head tracking modules and opened up data access through the OpenXR standard interface, which provides a sufficient hardware foundation for non-invasive cognitive load perception.

[0006] III. Programmatic Content Generation and Knowledge Graphs

[0007] Programmatic content generation is a technique that uses algorithms to automatically create game levels, maps, or scenes. Wave function collapse algorithm is an efficient spatial layout solution algorithm based on constraint propagation, which can automatically generate spatial layout schemes that conform to rules while satisfying global constraints. Knowledge graph is a graph-structured database that organizes knowledge in the form of entity-relation-entity triples. It has been widely used in the field of education for knowledge point association modeling and adaptive recommendation. Combining the semantic constraints of teaching knowledge graphs with programmatic content generation algorithms is an important technical foundation for realizing the automated production of educational 3D scenes.

[0008] IV. Existing technical solutions and their defects

[0009] The shortcomings of existing technical solutions can be summarized and analyzed from three dimensions.

[0010] The first type of solution is virtual reality course production tools, represented by virtual reality educational content creation platforms such as CoSpaces Edu and Engage XR. These platforms require teachers to manually drag and drop pre-set 3D models and configure text descriptions in the scene within a graphical interface. The configured scene is then published to the platform's private cloud, and learners access the content on a virtual reality headset through the platform's client. This type of solution still relies on manual scene construction, does not support the automatic generation of 3D scenes from lesson plan text, and lacks cognitive state perception and adaptive control capabilities. Furthermore, the content is locked within the platform's private ecosystem, resulting in a long virtual reality course production cycle, high costs, and a much slower pace of content updates compared to the changing teaching needs. Ultimately, this restricts the realization of large-scale, low-cost virtual reality education content production.

[0011] The second type of solution is the adaptive learning system, represented by Knewton and DreamBox. Adaptive learning platforms are designed for two-dimensional web pages or application interfaces. They build user profiles through answer history and behavior logs, predict the probability of mastery based on knowledge tracking models, and dynamically push two-dimensional learning content with matching difficulty. The adaptive object of this type of solution is the question push strategy rather than the visual complexity of three-dimensional scenes. It cannot be used for real-time scene perception and control in virtual reality immersive environments. When learners experience cognitive overload, the system cannot identify and intervene, causing learners to be in a high-load state for a long time, resulting in decreased attention, increased risk of motion sickness, reduced learning efficiency, and the inability to guarantee the quality of the virtual reality learning experience.

[0012] The third type of solution is the cross-platform distribution of 3D content. The distribution method using glTF format in conjunction with 3D viewers solves the problem of cross-platform transmission of basic 3D models. However, this type of solution does not carry educational semantic information, interaction logic, and cognitive control strategies. Existing virtual reality educational content is deeply bound to the private ecosystem of specific platforms and lacks a unified format standard that carries educational semantics. This results in the inability to seamlessly circulate and reuse between different educational institutions and different brands of terminal devices. Educational institutions are forced to repeatedly produce the same courses for different brands of devices. Digital cultural and creative content cannot circulate and be reused across institutions and devices, which seriously hinders the large-scale development of the virtual reality education ecosystem.

[0013] In summary, existing technologies have three core shortcomings: First, they lack an end-to-end processing system capable of automatically transforming the semantics of lesson plan texts into structured 3D interactive scenes; second, they cannot adaptively adjust the visual complexity of virtual reality scenes based on the learner's real-time cognitive state without relying on invasive external physiological sensors; and third, they lack a unified cross-platform format standard that carries educational semantics, preventing the free circulation and reuse of digital cultural and creative educational content. Summary of the Invention

[0014] The technical problem this invention aims to solve is to address the three core shortcomings of existing virtual reality education courses: low automation in digital production, inability to perceive and dynamically intervene in cognitive load in real time, and extremely poor cross-platform interoperability. This invention provides a virtual reality-based digital processing system for education courses, enabling end-to-end automated generation from lesson plan text to deployable 3D virtual reality scenes; non-intrusive real-time perception and adaptive scene control based on native sensors of virtual reality devices; and cross-platform unified encapsulation and parsing capabilities that support one-time processing for deployment on multiple platforms.

[0015] This invention provides a virtual reality-based digital processing system for educational courses, including a semantically driven 3D scene automatic generation subsystem, a multimodal cognitive load real-time perception and scene dynamic control subsystem, and a cross-platform unified intermediate layer representation encapsulation and parsing subsystem. The three subsystems share a unified teaching knowledge graph data layer, forming a closed-loop processing pipeline for content generation, adaptive presentation, and cross-platform distribution.

[0016] (I) Semantic-driven automatic 3D scene generation subsystem

[0017] The semantically driven 3D scene automatic generation subsystem takes lesson plan text as input and goes through five stages in sequence: semantic parsing of a large language model, construction of a teaching knowledge graph, semantic retrieval of 3D assets, wave function collapse spatial layout based on semantic constraints, and dynamic instantiation of the engine, and outputs a structured 3D virtual teaching scene.

[0018] In the semantic parsing stage of the large language model, the original lesson plan text is received and processed by sentence segmentation, noise reduction, and format normalization to obtain a cleaned sentence sequence. This sentence sequence is then fed into a pre-trained large language model based on the Transformer architecture to perform named entity recognition and concept extraction tasks, resulting in a set of knowledge entities. ,in For the first Each knowledge entity node The total number of extracted knowledge entities, each entity Carry semantic tags (The range of values ​​includes categories such as concepts, principles, experimental equipment, and scene environments), weighting of teaching importance. and education stage labels (The range of values ​​includes stages such as introduction, lecture, practice, and summary.) Extract relations from entity pairs in the entity set and output directed relation labels. Relational label types include precondition, contain, coordinate, causal, and demonstrative, constructing a set of relational triples. ,in A predefined set of labels for educational relationship types.

[0019] In the stage of constructing a knowledge graph for instruction, a set of knowledge entities is used. For a set of nodes, and for a set of relation triples For directed edge sets, construct a directed graph of teaching knowledge map. A connectivity test is performed on the graph. If isolated nodes or weakly connected components exist, supplementary relation inference is performed using a large language model to ensure the overall connectivity of the graph. A semantic label mapping function is then established. The semantic label of each entity node is mapped to a predefined set of three-dimensional asset categories. The category identifiers (including static models, animated characters, particle effects, interface panels, environment bases, etc.) are mapped by rules that are hard-coded in the form of rule tables and can be dynamically expanded through machine learning classifiers.

[0020] In the 3D asset semantic retrieval stage, the teaching knowledge graph nodes are traversed, and a search is initiated in the 3D asset library according to the semantic tags to obtain a set of candidate assets. It is then determined whether the asset matching confidence is not lower than a preset threshold. If the threshold is met, the asset is confirmed and a set of scene tiles is generated. If the threshold is not met, the 3D content generation module is triggered to generate supplementary assets before confirmation is performed.

[0021] In the semantically constrained wavefunction collapse spatial layout stage, the target 3D virtual scene space is discretized into a 3D mesh, and the initial wavefunction of each mesh cell is a uniform superposition state of all legal tile types. ,in For the first Types of tiles, Given the total number of tile types, a semantic adjacency constraint matrix is ​​constructed between tiles based on the node relationship types and node teaching importance weights in the teaching knowledge graph. The third dimension corresponds to the six adjacency directions in three-dimensional space. The matrix elements represent the adjacency and legality of the two tile types in the corresponding directions. The adjacency and legality rules are driven by the graph relationship type. The inclusion relationship is mapped to the parent node asset being able to wrap the child node asset. The parallel relationship is mapped to the two tiles being able to be arranged horizontally at the same level. The precondition relationship is mapped to the tile corresponding to the predecessor knowledge point being placed in the near-end priority position of the learner's line of sight. The three sub-steps of minimum Shannon entropy unit selection, wave function collapse, and constraint propagation are executed in a loop. If the wave function of a unit is reduced to an empty set during constraint propagation, a backtracking mechanism is triggered until all grid units have completed collapse. The complete mapping table of each grid coordinate and the determined tile type is output.

[0022] During the engine's dynamic instantiation phase, prefabs are instantiated in the 3D engine scene according to the layout results, interactive logic scripts are configured, and a structured initial 3D virtual teaching scene is output.

[0023] (II) Multimodal cognitive load real-time perception and scene dynamic control subsystem

[0024] The multimodal cognitive load real-time perception and scene dynamic control subsystem takes the multimodal implicit behavioral data stream output by the native sensors of the virtual reality device as input. Through four stages, namely, synchronous acquisition of sensor data, multimodal feature extraction, federated learning cognitive load assessment, and generation and execution of scene control instructions, it realizes closed-loop monitoring of learners' cognitive state and real-time adaptive control of the three-dimensional scene. The entire closed loop runs in a cycle of fifty milliseconds.

[0025] The sensor data synchronization acquisition module acquires eye-tracking data streams (including left and right eye pupil diameters, unit vectors of binocular gaze direction, and eyelid opening and closing degrees), head six-degree-of-freedom tracking data streams (including three-dimensional coordinates of head position and quaternion pose) and controller interaction data streams (including button event timestamps and trigger simulation values) through the OpenXR standard interface.

[0026] The multimodal feature extraction module extracts five-dimensional implicit behavioral features within a time window of three seconds with a sliding step of 0.05 seconds, forming a feature vector. Mean pupil diameter change rate reflects the degree of change in cognitive load; fixation point dispersion reflects the degree of attention concentration; blink frequency, abnormal values ​​indicate abnormal cognitive state; head motion entropy, usually increases when cognitive overload occurs; mean interaction delay, usually increases significantly when cognitive load increases.

[0027] The federated learning cognitive load assessment module performs Z-score standardization on the feature vectors and then feeds them into a lightweight three-layer fully connected neural network classifier deployed locally on the virtual reality device, outputting a cognitive load index. Where zero represents no load and one represents extreme overload, the model parameters are updated by the federated learning coordination server after aggregating gradients from multiple devices. Each device only uploads the locally computed parameter gradients and does not upload the original sensor data to protect learner privacy.

[0028] The scene dynamic adjustment execution module compares the cognitive load index with the overload threshold of 0.75 and the underload threshold of 0.3, and generates adjustment instructions: when the cognitive load index is not lower than the overload threshold, the detail level of non-core objects with a teaching importance weight of less than 0.5 is forcibly reduced by two levels (the number of triangle faces is reduced by no less than 60%), global illumination is turned off or downgraded, and the rendering visibility of the 3D objects corresponding to the top 20% of semantic nodes with the lowest teaching importance weight in the current frame is set to invisible. When the cognitive load index is not higher than the underload threshold, the reverse recovery of the above downgrade operation is performed. When the cognitive load index is between the two thresholds, the current scene configuration remains unchanged. All adjustment operations are implemented through engine coroutines to achieve a gradual transition with a transition time of 500 milliseconds.

[0029] (III) Cross-platform unified middleware representation encapsulation and parsing subsystem

[0030] The cross-platform unified middleware representation, encapsulation, and parsing subsystem is extended based on the glTF 2.0 specification. It carries educational semantics and cognitive regulation-specific data through a custom extended namespace, encapsulating 3D scenes into a unified digital course encapsulation format file, which consists of the following four data layers:

[0031] The first layer, geometry and material, follows the glTF 2.0 standard and performs Draco geometry compression with a compression ratio of no less than 3:1 on the geometry data. It stores the vertex data, coordinate data, physically based material maps, and skeletal animation data of all 3D meshes in the scene.

[0032] The second layer is the scene hierarchy and spatial layout layer, which stores the scene node tree and the spatial transformation matrix of each node, and adds the layout parameters output by the wave function collapse algorithm.

[0033] The third layer, the educational semantic weight layer, writes a unique identifier, semantic label, and teaching importance weight value for each 3D object node in the scene into a custom extended field, and aligns it with the second-layer node array in an index-based manner.

[0034] The fourth layer, the cognitive regulation strategy layer, writes the set of cognitive regulation rules into the global extended field in the form of an array. Each rule includes the triggering condition (the lower and upper bounds of the cognitive load index interval), the set of objects to be affected, the type of operation to be performed, and the operation parameters.

[0035] The package file is parsed on the target virtual reality hardware platform by a lightweight parsing engine plugin with a size of no more than two megabytes. It sequentially completes file header verification, 3D mesh and material reconstruction, scene node tree reconstruction, semantic metadata injection, and cognitive control rule registration. It also automatically selects the default display level of detail level according to the computing power index of the target platform's graphics processing unit, realizing progressive loading based on capabilities.

[0036] The present invention has the following beneficial effects:

[0037] 1. This invention enables end-to-end automated generation from lesson plan text to deployable 3D virtual reality scenes, compressing the content production cycle from weeks to hours, significantly reducing the threshold and cost of virtual reality course production, enabling educators without technical backgrounds to quickly produce high-quality virtual reality teaching content, and effectively promoting the large-scale production of digital cultural and creative education content.

[0038] 2. This invention achieves real-time perception and scene-adaptive control of cognitive load without using any invasive external sensors, enabling learners to maintain an optimal flow state throughout the virtual reality learning process, effectively reducing the risk of motion sickness and improving learning efficiency.

[0039] 3. This invention enables virtual reality education course content to be processed once and deployed across multiple platforms by using a unified intermediate layer representation format, breaking down platform data silos, promoting the free flow and reuse of digital cultural and creative education content among different institutions and devices, and driving the large-scale and standardized development of the virtual reality education ecosystem.

[0040] 4. The federated learning mechanism of this invention enables the cognitive load assessment model to be trained collaboratively across devices without uploading raw physiological data, effectively protecting the privacy and security of learners' personal data while continuously improving the model's generalization ability. Attached Figure Description

[0041] Figure 1 This is the overall closed-loop flowchart of the system of this invention;

[0042] Figure 2 This is a flowchart of the semantic-driven 3D scene automatic generation subsystem of the present invention;

[0043] Figure 3 This is a flowchart of the multimodal cognitive load real-time sensing and control subsystem of the present invention;

[0044] Figure 4This is a flowchart of the cross-platform four-layer encapsulation and parsing subsystem of the present invention. Detailed Implementation

[0045] The following is in conjunction with the appendix Figure 1-4 The specific embodiments of the present invention will be further described below. It should be noted that the description of these embodiments is for the purpose of helping to understand the present invention, but does not constitute a limitation of the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0046] I. System Overall Architecture

[0047] The virtual reality-based digital processing system for educational courses provided by this invention consists of three functional subsystems: a semantically driven 3D scene automatic generation subsystem, a multimodal cognitive load real-time perception and scene dynamic control subsystem, and a cross-platform unified intermediate layer representation encapsulation and parsing subsystem. These three subsystems share a unified teaching knowledge graph data layer, which runs through both content generation and runtime presentation stages, forming a complete closed-loop processing pipeline of content generation, adaptive presentation, and cross-platform distribution. Specifically, the semantically driven 3D scene automatic generation subsystem is responsible for transforming lesson plan text into structured 3D virtual teaching scenes; the multimodal cognitive load real-time perception and scene dynamic control subsystem is responsible for monitoring learners' cognitive states in real time and dynamically adjusting the scene's visual complexity during scene operation; and the cross-platform unified intermediate layer representation encapsulation and parsing subsystem is responsible for encapsulating the generated scene content into a cross-platform format file carrying complete educational semantics and cognitive control strategies, and completing parsing and deployment on the target platform.

[0048] II. Specific Implementation Methods of the Semantic-Driven Automatic Generation Subsystem for 3D Scenes

[0049] The semantically driven 3D scene automatic generation subsystem takes lesson plan text as input and outputs a structured 3D virtual teaching scene through five processing stages.

[0050] (I) Semantic analysis of large language models and construction of teaching knowledge graphs

[0051] The system receives the original lesson plan text (supporting portable document format, word processing document format, plain text or structured data format), performs sentence segmentation, noise reduction and format normalization on the text, and obtains a cleaned sentence sequence.

[0052] The sentence sequence is fed into a pre-trained large language model (using a Transformer-based pre-trained model with no less than seven billion parameters) to perform named entity recognition and concept extraction tasks, resulting in a set of knowledge entities:

[0053] ;

[0054] in, This refers to the set of knowledge entities extracted from the lesson plan text. For the first Each knowledge entity node The total number of extracted knowledge entities, each entity Carry the following attributes: semantic tags (The range of values ​​includes categories such as concepts, principles, experimental equipment, and scene environments), weighting of teaching importance. A higher value indicates greater importance of the knowledge point in the current lesson, and it is used to calculate the priority of subsequent scene spatial layout and cognitive regulation. (Educational stage label) (The range of values ​​includes stages such as introduction, lecture, practice, and summary), which is used to organize the presentation order of knowledge points in a 3D scene according to the teaching process.

[0055] For entity collections Entity pairs in Perform relation extraction and leverage the relation classification capabilities of large language models to output directed relation labels. Relationship label types include preconditions ( It is understanding (prerequisite), including () Conceptually includes Parallel () and (existing in parallel at the same level), causality () yes (Causes of occurrence), demonstration () pass (To demonstrate and illustrate), construct a set of relational triples:

[0056] ;

[0057] in, For a set of relation triples, To be from the entity To the entity Directed relation labels, A predefined set of labels for educational relationship types.

[0058] With entity set For a set of nodes, for a set of triples For directed edge sets, construct a directed graph of teaching knowledge:

[0059] ;

[0060] in, For teaching directed graphs of knowledge graphs, A connectivity test is performed. If isolated nodes or weakly connected components exist, supplementary relation inferences are performed using a large language model to ensure the overall connectivity of the graph, so that all knowledge nodes are in an interconnected graph structure.

[0061] Establish semantic tag mapping function Map the semantic label of each entity node to a 3D asset class identifier. ,in The system predefines a set of 3D asset categories, including static models, animated characters, particle effects, interface panels, and environment bases. The mapping rules are hard-coded and stored in the form of rule tables, and can be dynamically expanded through machine learning classifiers to adapt to the needs of new teaching scenarios.

[0062] (ii) Three-dimensional asset semantic retrieval

[0063] Iterates through each node of the teaching knowledge graph, initiates a semantic search request to the 3D asset library according to the 3D asset category identifier of the node, obtains a set of candidate assets, and determines whether the asset matching confidence is not lower than a preset threshold. If it is satisfied, the candidate asset is confirmed as the scene tile of the corresponding node. If it is not satisfied, the 3D content generation module is triggered to generate supplementary 3D assets according to the semantic tags of the node and incorporates the generated assets into the scene tile set. Then, the confirmation process is performed. After the scene tile sets of all nodes are summarized, the wave function collapse space layout stage is entered.

[0064] (III) Spatial Layout of Wavefunction Collapse Based on Semantic Constraints

[0065] Discretize the target 3D virtual scene space into A 3D mesh of various sizes, each mesh cell The initial wavefunction is a uniform superposition state of all legal tile types:

[0066] ;

[0067] in, For the coordinates The initial wavefunction state of a grid cell represents the set of tile types that the cell can be placed in. For the first Each tile type corresponds to a specific 3D asset prefab. This represents the total number of tile types.

[0068] Based on the node relationship types in the teaching knowledge graph And the importance weight of node teaching Construct semantic adjacency constraint matrices between graph tiles The third dimension of six corresponds to the six adjacent directions in three-dimensional space (positive). ,burden ,just ,burden ,just ,burden The meaning of the matrix elements is: when the tile type In direction The above can be with When an adjacency is valid, the corresponding element takes a value of one; otherwise, it takes a value of zero. The adjacency legality rule is driven by the graph relationship type: containment relationship maps to parent node assets being able to wrap child node assets; parallel relationship maps to two tiles being able to be arranged horizontally at the same level; prerequisite relationship maps to the tile corresponding to the predecessor knowledge point being placed in the learner's line of sight with priority.

[0069] Repeat the following three sub-steps until all mesh cells have collapsed:

[0070] Calculate the Shannon entropy of the current wavefunction for each grid cell:

[0071] ;

[0072] in, For grid cells The wavefunction entropy represents the current degree of uncertainty of the unit. To determine the tile type under the current constraints The probability weights of selection are initially uniformly distributed, and subsequently determined by the teaching importance weights of the corresponding knowledge nodes. The algorithm dynamically adjusts to reflect teaching priorities, selecting the uncollapsed grid cell with the smallest entropy value as the target cell for this iteration. (Unlike the traditional wave function collapse algorithm, which only randomly samples based on static adjacency rules, this invention innovatively injects the teaching importance weight of the teaching knowledge graph nodes into the block selection probability distribution, so that the blocks corresponding to the core knowledge points receive priority in the spatial layout solution process, realizing the deep intervention of teaching semantics in the scene generation process.)

[0073] Target units by probability weight Randomly sample a tile type As a result of determining the placement, the wavefunction of the unit is contracted into a set of single elements. This completes the collapse.

[0074] Starting from the collapsed target cell, constraint propagation is performed to its six adjacent cells, removing all elements that are related to the collapsed tiles from the current wavefunction of the adjacent cells. If a tile type that does not satisfy the adjacency constraint has its wave function reduced to an empty set during propagation, a backtracking mechanism is triggered. The current collapse step is undone, and the uncollapsed tile with the smallest entropy value is reselected to perform collapse. The constraint propagation is recursively extended to all affected adjacent tiles until the constraint propagation is stable, i.e., no new tile type is removed.

[0075] After all grid cells have collapsed, the final scene layout result is output, which is a complete mapping table of each grid coordinate and its corresponding tile type, for use by the engine's dynamic instantiation module.

[0076] (iv) Dynamic instantiation of the engine

[0077] Based on the layout results, prefabs are instantiated in the 3D engine scene. The corresponding interactive logic scripts are configured according to the educational stage tags of each node in the teaching knowledge graph. The structured initial 3D virtual teaching scene is then output and encapsulated into a unified intermediate layer representation format.

[0078] III. Specific Implementation Methods of the Multimodal Cognitive Load Real-time Perception and Scene Dynamic Control Subsystem

[0079] See Figure 3 The multimodal cognitive load real-time perception and scene dynamic control subsystem takes the multimodal implicit behavior data stream output by the native sensors of the virtual reality device as input, and realizes closed-loop monitoring of cognitive state and real-time adaptive control of three-dimensional scene through four processing stages. The entire system runs in a cycle of fifty milliseconds.

[0080] (a) Synchronous acquisition of sensor data

[0081] The following three data streams are synchronously acquired at a fixed sampling period through the OpenXR standard interface: eye tracking data stream, including the diameter of the left and right pupils (in millimeters), the unit vector of the gaze direction of both eyes, and the eyelid opening degree (the value ranges from zero to one, where one indicates that the eyelids are fully open); head six-degree-of-freedom tracking data stream, including the three-dimensional coordinates of the head position and the quaternion representing the head posture; and controller interaction data stream, including the press timestamp and release timestamp of the controller button events, and the trigger simulation value (the value ranges from zero to one).

[0082] (II) Extraction of Multimodal Implicit Behavioral Features

[0083] Within a time window of three seconds with a sliding step of 0.05 seconds, five-dimensional implicit behavioral features are extracted from the three data streams to form a feature vector. .

[0084] Average pupil diameter change rate The calculation formula is:

[0085] ;

[0086] in, This represents the mean absolute rate of change of the average pupil diameter of both eyes within the time window, expressed in millimeters per second. A higher value indicates a more dramatic change in cognitive load. and They are respectively The pupil diameter of the left and right eyes at all times. The time window length is set to three seconds.

[0087] gaze dispersion The calculation formula is:

[0088] ;

[0089] in, This represents the root mean square deviation of the gaze direction vector relative to the mean direction within the time window, expressed in radians. A larger value indicates greater distraction. for The unit vector of the direction of eye fixation at any given moment. It is the mean vector of the gaze direction vector within the time window.

[0090] blink frequency The blink rate is obtained by counting the number of zero-crossing events in the eyelid opening and closing signal within a time window. The unit is blinks per minute. A blink rate of less than ten times per minute or an abnormally high blink rate indicates an abnormal cognitive state.

[0091] Head motion entropy Head angular velocity sequence After performing histogram statistics, calculate the Shannon entropy:

[0092] ;

[0093] in, Head motor entropy reflects the randomness and irregularity of head movements; it typically increases during cognitive overload. For the head angular velocity to fall into the first The probability of each histogram interval. This represents the interval number of the histogram, with a value of sixteen.

[0094] Mean interaction delay Defined as the arithmetic mean of the response delays of all controllers within a time window, in milliseconds. Interaction delays typically increase significantly when cognitive load rises.

[0095] (III) Assessment of Cognitive Load in Federated Learning

[0096] For eigenvectors Each feature dimension in the dataset undergoes Z-score standardization. The mean and standard deviation parameters required for standardization are obtained through global aggregation via federated learning to ensure consistency in feature scale across devices. The standardized feature vectors are then fed into a lightweight three-layer fully connected neural network classifier (approximately 50,000 parameters, inference latency less than 5 milliseconds) deployed locally on the virtual reality device, outputting a cognitive load index. Where zero represents no cognitive load and one represents extreme cognitive overload, the model parameters are updated by the federated learning coordination server after aggregating gradients from multiple devices. Each device only uploads the locally computed parameter gradients and does not upload the original sensor data to protect the learner's privacy.

[0097] (iv) Generation and execution of scene control instructions

[0098] Cognitive Load Index Compared with the preset overload threshold and underload threshold The system compares data, generates scene control commands, and executes them.

[0099] when At that time, three degradation operations are performed: the detail level of non-core objects in the scene with a teaching importance weight of less than 0.5 is forcibly reduced by two levels, reducing the number of triangle faces by no less than 60% to reduce the graphics rendering load; global illumination is turned off or downgraded to reduce visual complexity; and the 3D objects corresponding to the top 20% of semantic nodes with the lowest teaching importance weight in the current frame are queried from the teaching knowledge graph and their rendering visibility is set to invisible to focus learners' attention on core knowledge point objects.

[0100] when At that time, a reverse recovery operation is performed: the downgraded level of detail is restored level by level, real-time lighting is re-enabled, and the rendering visibility of hidden objects is restored to visible, thereby stimulating learners' interest in in-depth exploration through rich scene details.

[0101] when If the current scenario configuration remains unchanged, this cognitive load range is the optimal learning range for the learner to be in a flow state.

[0102] All scene adjustment operations are implemented through engine coroutines to achieve gradual transitions with a transition time of 500 milliseconds to avoid visual abrupt changes that could interrupt learners' attention. After the adjustment is completed, the adjustment log (containing only model gradients and not the original data) is sent back to the federated learning coordination server to continuously optimize the generalization ability of the cognitive load assessment model.

[0103] IV. Specific Implementation Methods of the Cross-Platform Unified Middleware Representation Encapsulation and Parsing Subsystem

[0104] The cross-platform unified middleware representation, encapsulation, and parsing subsystem is extended based on the glTF 2.0 specification. It carries educational semantics and cognitive regulation-specific data through a custom extended namespace, and encapsulates the 3D scene into a unified digital course encapsulation format file, which consists of four data layers in sequence.

[0105] (a) Packaging stage

[0106] The semantic-driven 3D scene automatic generation subsystem collects full data of instantiated 3D scenes, including geometric data, material data and scene node trees of all mesh objects. The subsystem collects cognitive regulation rule sets from the multimodal cognitive load real-time perception and scene dynamic regulation subsystem. The subsystem collects semantic metadata of each node from the teaching knowledge graph data layer.

[0107] Perform Draco geometry compression on the geometry data (target compression ratio not less than 3:1), and serialize the compressed geometry data and material data into binary blocks according to the glTF 2.0 standard to form the first layer of geometry and material data segments.

[0108] The scene node tree and the spatial transformation matrices (translation vector, rotation quaternion, scaling vector) of each node are serialized into a standard node array, and the layout parameters (mapping relationship from grid coordinates to tile type) output by the wave function collapse algorithm are added to form the second layer of scene hierarchy and spatial layout layer data segment.

[0109] Traverse each node in the scene node tree and write the corresponding educational semantic metadata into its custom extended field, including the unique identifier string of the teaching knowledge graph node, the semantic tag string, and the floating-point value of the teaching importance weight. Align and associate it with the node array of the second layer in an indexed manner to ensure that each three-dimensional object node has corresponding semantic metadata, forming the third layer of educational semantic weight layer data segment.

[0110] The set of cognitive regulation rules is serialized into an array and written into a global custom extended field. Each rule includes a trigger condition (two floating-point values: the lower and upper bounds of the cognitive load index interval), a set of objects to be affected (specified by a list of teaching knowledge graph node identifiers), an operation type (one of two types: downgrade or enhancement), and operation parameters (a target integer value for the level of detail and a Boolean value for rendering visibility), forming the fourth cognitive regulation strategy layer data segment.

[0111] The four data layers are merged into a complete package file according to a predefined file header format (including version number, layer offset index table, and digital signature) and output to the distribution server.

[0112] (II) Analysis Phase

[0113] On the target virtual reality hardware platform, the lightweight parsing engine plugin (a dynamic link library with a size of no more than two megabytes) sequentially performs the following parsing operations: reads the file header and verifies the version number and digital signature, confirms the integrity of the package, calls the target platform's native graphics interface to reconstruct the 3D mesh and materials according to the first layer of data, reconstructs the scene node tree and spatial layout according to the second layer of data, injects semantic metadata tags into each node according to the third layer of data, builds a local lightweight teaching knowledge graph copy at runtime, registers cognitive control rules according to the fourth layer of data, and interfaces with the running instance of the local multimodal cognitive load real-time perception and scene dynamic control subsystem to complete the activation of scene adaptive control capabilities. During the parsing process, the plugin automatically selects the default display level for each level of detail based on the target platform's graphics processing unit computing power indicators (including video memory capacity and floating-point operation capability, obtained through platform interface queries), realizing progressive loading according to capabilities, thereby supporting unified deployment of platforms with different computing power levels such as high-performance workstations, professional virtual reality all-in-one machines, and mobile entry-level headsets. Specific Implementation

[0114] The complete workflow of this invention is illustrated using a lesson plan for Lenz's Law in high school physics as an example.

[0115] After receiving the course lesson plan text, the system extracts knowledge entities such as Lenz's law, magnetic flux, induced current, coil, magnet, ammeter, and experimental platform from the large language model. It also extracts relationships between entities such as causal relationships (changes in magnetic flux lead to the generation of induced current), demonstration relationships (the relative motion of the coil and magnet demonstrates Lenz's law), inclusion relationships (the experimental platform contains both the coil and the magnet), and precondition relationships (Faraday's law of electromagnetic induction precedes the explanation of Lenz's law). The system constructs a teaching knowledge graph with approximately fifteen nodes. The teaching importance weights of the core knowledge entities are as follows: Lenz's law node weight is 0.95, magnetic flux node weight is 0.85, induced current node weight is 0.80, experimental equipment node weight is 0.55 to 0.75, and background display board node weight is 0.20 to 0.35.

[0116] The wave function collapse spatial layout algorithm divides the laboratory scene space into an 8x4x8 grid, constructs a semantic adjacency constraint matrix based on graph relation constraints, and automatically solves the spatial layout: the demonstration experimental platform is preferentially placed in the core area near the learner's line of sight, the ammeter and wire model are placed in the mid-ground auxiliary area, and the background display board is placed in the far-ground area. The engine's dynamic instantiation module renders the above layout into a virtual reality 3D scene that learners can operate in real time.

[0117] During scene execution, the multimodal cognitive load real-time perception and scene dynamic control subsystem collects learners' eye movement and head tracking data at 50-millisecond intervals. When learners first encounter the concept of Lenz's law, the system detects that the cognitive load index rises to 0.78, exceeding the overload threshold of 0.75. It immediately executes downgrade control, reducing the detail level of the background display board and auxiliary decorations with a weight below 0.5 by two levels, turning off real-time global illumination, and hiding distant background objects. After about 40 seconds, the learner gradually understands the concept, and the cognitive load index drops to 0.55, returning to the flow zone. The system maintains the current configuration. After entering the practical operation stage, the cognitive load index further drops to 0.25, below the underload threshold of 0.3. The system then executes enhanced control, restoring all details and lighting effects, and increasing scene richness to stimulate learners' in-depth exploration.

[0118] Once the course is generated, the system will output a package file containing complete educational semantic weights and cognitive regulation rule sets to the distribution server. This file can be seamlessly parsed and deployed on multiple target platforms: on mobile headsets, the parsing plugin automatically reduces the initial level of detail to adapt to computing power; on high-performance workstations, it renders the complete scene with full precision; and on web browsers, it provides access capabilities through progressive loading. Throughout the process, there is no need to recreate the course content for different platforms, enabling digital cultural and creative educational content to be processed once and deployed to multiple platforms.

[0119] Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of the present invention. Therefore, any modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall fall within the protection scope defined by the claims of the present invention.

Claims

1. A virtual reality-based digital processing system for educational courses, characterized in that, include: The semantically driven 3D scene automatic generation subsystem is used to receive lesson plan text and sequentially perform semantic parsing of large language model, construction of teaching knowledge graph, semantic retrieval of 3D assets, wave function collapse spatial layout based on semantic constraints and dynamic instantiation of engine, and output a structured 3D virtual teaching scene. The multimodal cognitive load real-time perception and scene dynamic control subsystem is used to collect multimodal implicit behavioral data through the native sensors of virtual reality devices, extract five-dimensional feature vectors, output cognitive load index through federated learning cognitive load assessment, and perform real-time closed-loop control on the visual complexity of the three-dimensional scene based on the comparison result of the cognitive load index and the preset threshold. A cross-platform unified middleware representation encapsulation and parsing subsystem is used to encapsulate 3D virtual teaching scenarios into a four-layer structured encapsulation format file carrying educational semantic weights and cognitive regulation strategies, and to parse and deploy them on multiple target platforms through a lightweight parsing engine plugin. The three subsystems mentioned above share a unified teaching knowledge graph data layer, forming a closed-loop processing pipeline for content generation, adaptive presentation, and cross-platform distribution.

2. The system according to claim 1, characterized in that, The specific method for constructing the teaching knowledge graph is as follows: The cleaned sentence sequence is fed into a pre-trained large language model to perform named entity recognition and concept extraction, resulting in a knowledge entity set. Each knowledge entity carries a semantic label, a teaching importance weight, and an educational stage label. The teaching importance weight ranges from zero to one. Relationship extraction is performed on entity pairs in the entity set, outputting directed relation labels. The types of directed relation labels include preconditions, inclusion, parallelism, causality, and demonstration. A set of relation triples is constructed. Using the knowledge entity set as the node set and the set of relation triples as the directed edge set, a directed graph of teaching knowledge is constructed. A connectivity test is performed on the graph. If isolated nodes exist, supplementary relation inference is performed through the large language model. A semantic label mapping function is established to map the semantic label of each entity node to a predefined three-dimensional asset class identifier.

3. The system according to claim 2, characterized in that, The specific method for the wavefunction collapse space layout based on semantic constraints is as follows: The target 3D scene space is discretized into a 3D mesh. The initial wave function of each mesh cell is a uniform superposition of all legal tile types. Based on the node relationship types and node teaching importance weights of the teaching knowledge graph, a semantic adjacency constraint matrix between tiles is constructed. The matrix elements represent the adjacency legality of two tile types in the corresponding spatial direction. This includes relationship mapping that parent node assets can wrap child node assets, parallel relationship mapping that two tiles can be arranged horizontally at the same level, and precondition relationship mapping that the tile corresponding to the predecessor knowledge point must be placed in the near-end priority position of the learner's line of sight. The minimum Shannon entropy mesh cell selection, wave function collapse, and constraint propagation are executed iteratively. If the wave function of a mesh cell shrinks to an empty set during constraint propagation, a backtracking mechanism is triggered until all mesh cells have completed collapse. The complete mapping table of each mesh coordinate and the determined tile type is output.

4. The system according to claim 1, characterized in that, The five-dimensional feature vector includes: The average pupil diameter change rate represents the mean absolute change rate of the average pupil diameter of both eyes within the time window; fixation point dispersion represents the root mean square deviation of the fixation direction vector relative to the mean direction within the time window; blink frequency represents the number of eyelid descent zero-crossing events per unit time within the time window; head motion entropy represents the Shannon entropy value calculated after histogram statistics of the head angular velocity sequence, with the histogram interval number being sixteen; and the mean interaction delay represents the arithmetic mean of the response delays of all controllers within the time window. The five-dimensional feature vectors are fed into the classifier after Z-score standardization, and the standardization parameters are obtained through global aggregation via federated learning.

5. The system according to claim 4, characterized in that, The specific method for assessing the cognitive load of federated learning is as follows: The standardized five-dimensional feature vector is fed into a lightweight three-layer fully connected neural network classifier deployed locally on the virtual reality device. The output is a cognitive load index with a value ranging from zero to one, where zero represents no cognitive load and one represents extreme cognitive overload. The model parameters are updated by the federated learning coordination server after aggregating gradients from multiple devices. Each device only uploads the locally calculated parameter gradients and does not upload the original sensor data. The cognitive load index is compared with the preset overload threshold of 0.75 and the underload threshold of 0.

3. When the cognitive load index is not lower than the overload threshold, a degradation instruction is generated. When the cognitive load index is not higher than the underload threshold, an enhancement instruction is generated. When the cognitive load index is between the underload threshold and the overload threshold, the current scene configuration remains unchanged.

6. The system according to claim 5, characterized in that, The scene control operations corresponding to the downgrade command include: forcibly reducing the detail level of non-core objects in the scene with a teaching importance weight of less than 0.5 by two levels, reducing the number of triangle faces by no less than 60%, reducing the global illumination intensity to less than 30% of the original intensity, querying the teaching knowledge graph for the 3D objects corresponding to the top 20% of semantic nodes with the lowest teaching importance weight in the current frame, and setting their rendering visibility to invisible. The scene control operations corresponding to the enhancement command are the reverse recovery of the above downgrade operations. All control operations are implemented through engine coroutines to achieve a gradual transition with a transition time of 500 milliseconds.

7. The system according to claim 1, characterized in that, The four-layer structured encapsulation format file consists of the following four data layers: The first layer, geometry and material layer, stores 3D mesh vertex data, coordinate data, physically based material maps, and skeletal animation data compressed using the glTF 2.0 standard and with a compression ratio of no less than 3:

1. The second layer, scene hierarchy and spatial layout layer, stores the scene node tree, the spatial transformation matrix of each node, and the layout parameters output by the wave function collapse algorithm. The third layer, educational semantic weight layer, adds a unique identifier, semantic label, and educational importance weight to each 3D object node in the scene, aligning it with the second-layer node array using an index. The fourth layer, cognitive regulation strategy layer, stores a set of scene regulation rules triggered by cognitive load index intervals. Each rule includes a set of objects to be affected, the type of operation to be performed, and operation parameters.

8. The system according to claim 7, characterized in that, The lightweight parsing engine plugin has a size of no more than two megabytes. On the target platform, it sequentially performs file header verification, 3D mesh and material reconstruction, scene node tree reconstruction, semantic metadata injection, and cognitive control rule registration. During the parsing process, the plugin automatically selects the default display level for each level of detail based on the graphics processing unit's memory capacity and floating-point operation capability of the target platform, thus achieving progressive loading based on capability.

9. The system according to claim 1, characterized in that, The multimodal implicit behavioral data is collected through the OpenXR standard interface, including eye-tracking data stream, head six-degree-of-freedom tracking data stream, and controller interaction data stream. Feature extraction is performed within a time window of three seconds with a sliding step size of 0.05 seconds. The entire cognitive load perception and scene regulation closed loop runs in a 50-millisecond cycle.

10. The system according to claim 1, characterized in that, In the 3D asset semantic retrieval stage, when the asset matching confidence is lower than a preset threshold of 85%, the 3D content generation module is triggered to generate supplementary 3D assets based on the semantic tags of the corresponding nodes. The supplementary assets and the retrieved assets are uniformly incorporated into the scene tile set, and then the subsequent wave function collapse spatial layout is executed.