Middle and primary school multi-person interactive teaching method and system based on VR technology

Abstract

The invention provides a middle and primary school multi-person interactive teaching method and system based on a VR technology, and relates to the technical field of virtual reality, and the method comprises the steps: generating a model through deep learning based on pre-obtained teaching aid model data, determining subcomponents, combining a teaching aid generation engine, and constructing a virtual teaching aid; a color image and a depth image are obtained through a structured light depth camera, two-dimensional pixel coordinates are obtained, depth information is combined, a three-dimensional skeleton curve of limb movement of a student is generated, an optical motion capture result is obtained, an inertial motion capture result is obtained through inertial drift calibration based on the three-dimensional skeleton curve, and a limb movement three-dimensional track of the student is reconstructed; generating a force feedback signal by solving a kinetic equation through a collision detection algorithm based on the three-dimensional trajectory of the limb movement of the trainee; and rendering the virtualization model into a virtual teaching scene, synchronizing the limb movement three-dimensional track and the force feedback signal in real time, and generating a multi-person collaborative interaction process of the virtual teaching aid.

Description

Technical Field

[0001] This invention relates to the field of virtual reality technology, and in particular to a multi-person interactive teaching method and system for primary and secondary schools based on VR technology. Background Technology

[0002] With the continuous development of science and technology, Virtual Reality (VR) technology, with its immersive, interactive, and imaginative characteristics, has been widely applied and researched in the field of education. Introducing VR technology into primary and secondary school teaching can create vivid and realistic virtual teaching environments, stimulate students' learning interest, and promote their understanding and mastery of knowledge.

[0003] Traditional primary and secondary school teaching methods are teacher-led, with students passively receiving knowledge and lacking initiative and participation. Existing VR teaching systems are mostly designed for individual students, lacking features for multi-person collaboration and interaction. Students' learning activities in virtual environments are typically conducted independently, without the ability to communicate and collaborate with other students in real time.

[0004] In summary, there is an urgent need for a VR-based interactive teaching method for primary and secondary schools, which can fully leverage the advantages of VR technology to provide students with an immersive learning experience, while supporting multi-person collaboration and interaction to promote communication and cooperation among students. This invention can solve the problems in the existing technology. Summary of the Invention

[0005] This invention provides a multi-person interactive teaching method and system for primary and secondary schools based on VR technology, which can solve the problems in the prior art.

[0006] A first aspect of the present invention, A VR-based interactive teaching method for primary and secondary schools is provided, including: Based on the pre-acquired teaching aid model data, a model is generated through deep learning to learn the inherent distribution characteristics. Based on the inherent distribution characteristics, multiple sub-components are determined. Combined with the teaching aid generation engine, the corresponding three-dimensional shapes are established through mesh components. A transformation node is created for each sub-component, and the parent-child relationship between transformation nodes is set to generate a tree-like hierarchical structure and construct a virtual teaching aid. A structured light depth camera is used to acquire color and depth images of the trainee. Based on the color images, a convolutional neural network is used to detect key points of body parts and obtain two-dimensional pixel coordinates. Based on the depth information in the depth images, the key points of body parts are mapped into space to obtain three-dimensional spatial coordinates and generate three-dimensional skeletal curves of the trainee's limb movements to obtain optical motion capture results. Based on the three-dimensional skeletal curves, an inertial measurement unit is used to acquire the acceleration and angular velocity of key points of body parts. After inertial drift calibration, inertial motion capture results are obtained. The optical motion capture results and the inertial motion capture results are fused to reconstruct the three-dimensional trajectory of the trainee's limb movements. Based on the three-dimensional trajectory of the trainees' limb movements, the motion of the trainee's virtual model is mapped. Through a collision detection algorithm, the contact state between the virtual model and the virtual teaching aid is calculated in real time. According to the material properties of the virtual teaching aid and the contact state, the motion response of the virtual teaching aid is calculated by solving the dynamic equation and generating a force feedback signal. The virtual models of multiple trainees are rendered into the same virtual teaching scene, and the three-dimensional trajectory of the limb movements and the force feedback signal of different trainees are synchronized in real time to generate a multi-person collaborative interaction process of the virtual teaching aid.

[0007] In one alternative embodiment, Based on pre-acquired teaching aid model data, a model is generated through deep learning to learn the inherent distribution characteristics, including: Acquire teaching aid model data; construct a geometric feature network based on the PointNet++ structure to represent the teaching aid model data as a point cloud and learn the geometric shape features corresponding to the point cloud; construct a visual feature network based on the ResNet structure to render the teaching aid model data as multi-view images and learn the appearance texture features corresponding to the multi-view images; represent the material attributes, size attributes, and usage attributes in the teaching aid model data as corresponding attribute vectors, and learn functional semantic features through fully connected layers to determine the functional attribute network; The geometric feature network, the visual feature network, and the functional attribute network are dynamically adjusted through an attention mechanism and combined into a cross-modal fusion network. The intrinsic distribution characteristics of the teaching aid model data are learned through a modal alignment loss function and a cross-modal distillation learning strategy. In one alternative embodiment, Based on inherent distribution characteristics, multiple sub-components are identified. Using a teaching aid generation engine, corresponding 3D shapes are created through mesh components. A transformation node is created for each sub-component, and parent-child relationships between these nodes are set to generate a tree-like hierarchical structure. This constructs the virtual teaching aid, including: The intrinsic distribution features are input into the generator of the cross-modal conditional generative adversarial network. The generator determines the generator teaching aid model based on the intrinsic distribution features. The generator teaching aid model is input into the discriminator of the cross-modal conditional generative adversarial network. The discriminator judges the realism of the generator teaching aid model and the consistency between the generator teaching aid model and the intrinsic distribution features. Through adversarial iteration between the generator and the discriminator, the optimal teaching aid model is learned. The optimal teaching aid model is input into a sub-component discovery network, which includes a variational autoencoder module and an active learning module. The variational autoencoder module encodes the optimal teaching aid model by minimizing the reconstruction error and KL divergence to obtain a low-dimensional latent space representation. Based on the low-dimensional latent space representation, similar sub-components are clustered to form a cluster structure. The active learning module adopts a maximum entropy-based sampling strategy and labels the sub-components according to the cluster structure in the low-dimensional latent space representation. Combined with manifold-based regularization constraints, the uniform distribution of sub-component labels is determined. By alternating between unsupervised learning by the variational autoencoder module and supervised learning by the active learning module, the labeling cost is minimized, and the virtual teaching aid sub-components are determined. Based on each virtual teaching aid sub-component, a corresponding three-dimensional shape is established through a mesh component. Based on the three-dimensional shape, a transformation node is created for each virtual teaching aid sub-component. The parent-child relationship between the transformation nodes is set to generate a tree-like hierarchical structure. Combined with the attributes of the transformation nodes, the constraint relationship between the virtual teaching aid sub-components is set. At the same time, the texture and color of the virtual teaching aid sub-components are set through a material component, and the virtual teaching aid sub-components are assembled into a virtual teaching aid.

[0008] In one alternative embodiment, Using a structured light depth camera, color and depth images of the trainee are acquired. Based on the color image, a convolutional neural network is used to detect key points of body parts and obtain two-dimensional pixel coordinates. Based on the depth information in the depth image, the key points of the body parts are mapped into space to obtain three-dimensional spatial coordinates. This generates a three-dimensional skeletal curve of the trainee's limb movement, resulting in optical motion capture results including: A structured light depth camera simultaneously acquires color and depth images of the trainee. The structured light depth camera includes an infrared projector, two infrared cameras, and a color camera. The infrared projector projects structured light according to a preset pattern, generating deformable stripe images on the trainee's surface. The two infrared cameras acquire these deformable stripe images from two different perspectives. Based on the two sets of deformable stripe images, the depth value of each pixel is calculated, and a depth image is synthesized. The color camera acquires the trainee's color image. The color and depth images are then spatiotemporally synchronized and aligned at different perspectives to obtain a synchronized image set. A keypoint detection network is constructed based on the U-Net structure. The keypoint detection network includes an encoder and a decoder. The encoder is based on a ResNet-50 structure and extracts multi-scale image features from color images in a synchronously aligned image set. The decoder fuses the multi-scale image features through skip connections to generate a feature map. The feature map is then mapped to a heatmap corresponding to keypoints of body parts through convolutional layers. Each keypoint of a body part corresponds to a heatmap channel. The probability that a body part is a keypoint is determined based on the pixel value. The two-dimensional pixel coordinates of the keypoints of body parts are extracted through threshold filtering and non-maximum suppression. Based on the depth image in the synchronized image set, the depth value corresponding to the two-dimensional pixel coordinates is determined. Combined with the intrinsic parameters of the structured light depth camera, the three-dimensional coordinates of the key points of the body parts in the corresponding coordinate system of the structured light depth camera are calculated by the coordinate transformation formula. Based on the three-dimensional coordinates, with the key point of the body part corresponding to the head as the root node, the key points of each body part are linked sequentially according to the predetermined human motion chain topology to form a tree-like limb hierarchy. By connecting the key points of adjacent body parts and adjusting the translation and rotation parameters, a three-dimensional skeletal curve of the trainee's limb movement is generated. The three-dimensional coordinates corresponding to the key points of the body parts in adjacent frames are combined into a three-dimensional coordinate sequence and smoothed and filtered to obtain the optical motion capture result.

[0009] In one alternative embodiment, Based on 3D skeletal curves, acceleration and angular velocity of key points in body parts are acquired through an inertial measurement unit. After inertial drift calibration, inertial motion capture results are obtained. By fusing the optical motion capture results and the inertial motion capture results, the 3D trajectory of the trainee's limb movements is reconstructed, including: Based on the extracted key points of the body parts, an inertial measurement unit is attached to determine the three-axis acceleration and angular velocity. Through attitude calculation and integration, the preliminary pose of the body parts is determined. The inertial measurement unit includes an accelerometer, a gyroscope and a magnetometer, which measure the three-axis acceleration and angular velocity corresponding to the key points of the body parts, respectively. Initiating the inertial drift calibration process: Using the initial pose as the pose of the previous moment, and combining the triaxial acceleration and angular velocity, the current pose of each part of the body at the current moment is predicted through the inertial navigation equation, thus obtaining the current prior state estimate and the state estimate covariance matrix. Using the optical motion capture results as observations, the error between the observations and the current prior state estimate is calculated, an observation equation is constructed, error statistics are determined, and the observation noise covariance matrix is ​​obtained. Based on the state estimate covariance matrix and the observation noise covariance matrix, the optimal Kalman gain matrix is ​​calculated. Based on the optimal Kalman gain matrix, the current prior state estimate is corrected to obtain the posterior state estimate, and the state estimate covariance matrix is ​​updated at the same time. Using the posterior state estimate as the initial pose for the next moment, the inertial drift calibration process is repeated to obtain the inertial motion capture result; Based on the 3D skeletal curves corresponding to optical motion capture results and the body part poses corresponding to inertial motion capture results, temporal features are extracted, a multi-scale spatiotemporal feature pyramid is constructed, and the features are encoded and fused through an attention-gated recurrent unit network to obtain fused features. Through adaptive weight adjustment, combined with the mutual compensation between optical motion capture results and inertial motion capture results, the fused features are mapped to the 3D trajectory of the trainee's limb movement through a multilayer perceptron.

[0010] In one alternative embodiment, Based on the three-dimensional trajectory of the student's limb movements, the motion of the student's virtual model is mapped. A collision detection algorithm is used to calculate in real time the contact state between the virtual model and the virtual teaching aid, including: Based on the three-dimensional geometric data corresponding to the virtual teaching aid, the mathematical equation form is determined, a discrete point set in three-dimensional space is generated, the surface feature points corresponding to the virtual teaching aid are extracted, virtual teaching aid point cloud data is generated, the normal vector of the virtual teaching aid point cloud data is estimated, and the surface triangular facet mesh topology of the virtual teaching aid is reconstructed with the point position and normal vector as constraints. After mesh post-processing, the virtual teaching aid triangular facet mesh model is obtained. The bounding box calculation is performed on the triangular facet mesh model of the virtual teaching aid to obtain the corresponding AABB bounding box. Based on the geometric features of the virtual teaching aid, the AABB bounding box is adaptively and recursively partitioned until the size of a single triangular facet within the AABB bounding box is less than the preset unit size threshold. The AABB bounding box is then used as a leaf node to construct the octree space partitioning structure corresponding to the virtual teaching aid. For leaf nodes, calculate the minimum distance from the corresponding AABB bounding box to the surface of the student virtual model, and use it as the distance field value of the leaf node; traverse the octree from bottom to top, and for each non-leaf node, take the minimum value of the distance field value corresponding to the lower-level child node of the non-leaf node, and use it as the distance field value of the non-leaf node; traverse the octree from top to bottom, and for each node, update the distance field value of the node with the distance field value corresponding to the parent node of the node, and construct the octree hierarchical distance field; During the movement, based on the octree hierarchical distance field, starting from the root node of the octree, the system recursively traverses and determines whether each node has collided. Nodes with distance field values ​​greater than a preset collision threshold are pruned, and leaf nodes with distance field values ​​less than or equal to the preset collision threshold are marked as potential collision areas. For the set of triangular facets in the potential collision region, a preset bounding volume detection algorithm is applied to the triangular facets corresponding to the student's virtualization model to perform pairwise collisions and determine the intersecting triangular facet pairs. Based on intersecting triangular facet pairs, continuous collision detection is performed. The earliest collision time and corresponding collision information of the intersecting triangular facet pairs in the continuous motion space are calculated. Based on the collision position coordinates and normal vector in the collision information, the collision penetration depth is calculated. Neighboring collision points are clustered and fused to generate a collision region data structure. Collision data is statistically analyzed to determine the contact state of the collision.

[0011] In one alternative embodiment, Based on the material properties and contact state of the virtual teaching aid, the motion response of the virtual teaching aid is calculated by solving the dynamic equations, and force feedback signals are generated, including: Based on the material properties and collision data of the virtual teaching aids, combined with the geometry of the virtual teaching aids and the stress distribution of the contact area, different contact types are determined. Based on the contact types, a physical model is determined and the contact force is calculated. The motion equations of the virtual teaching aid are constructed based on the Lagrange dynamics equations. By combining one or more constraints such as non-penetrating constraints, hinge constraints, and sliding constraints, the Lagrange multipliers are determined, and a system of differential algebraic equations is generated. Based on the constraints, a numerical solution algorithm is determined, and the motion response of the virtual teaching aid is obtained by solving the equations. Based on the motion response, the acceleration vectors of the corresponding surface mesh vertices in the virtual teaching aid are extracted. The product of the acceleration vector and the vertex mass is taken as the vertex force. Through the connection relationship between mesh vertices, a sparse mapping matrix from vertex force to contact force is constructed. The inverse of the sparse mapping matrix is ​​solved using a preset singular value decomposition algorithm to determine the distribution of contact force to vertex force and convert the contact force into a force feedback signal.

[0012] A second aspect of the present invention, A VR-based interactive teaching system for primary and secondary schools is provided, including: The first unit is used to generate a model based on pre-acquired teaching aid model data through deep learning, learn the inherent distribution characteristics, determine multiple sub-components based on the inherent distribution characteristics, combine the teaching aid generation engine, establish the corresponding three-dimensional shape through mesh components, create a transformation node for each sub-component, set the parent-child relationship between transformation nodes, generate a tree-like hierarchical structure, and construct virtual teaching aids. The second unit is used to acquire color and depth images of the trainee through a structured light depth camera. Based on the color image, it detects key points of body parts through a convolutional neural network to obtain two-dimensional pixel coordinates. Based on the depth information in the depth image, it maps the key points of body parts into space to obtain three-dimensional spatial coordinates and generates a three-dimensional skeletal curve of the trainee's limb movement to obtain optical motion capture results. Based on the three-dimensional skeletal curve, it acquires the acceleration and angular velocity of key points of body parts through an inertial measurement unit. After inertial drift calibration, it obtains inertial motion capture results. It fuses the optical motion capture results and the inertial motion capture results to reconstruct the three-dimensional trajectory of the trainee's limb movement. The third unit is used to map the motion of the virtual model of the student based on the three-dimensional trajectory of the student's limb movement. Through a collision detection algorithm, the contact state between the virtual model and the virtual teaching aid is calculated in real time. Based on the material characteristics of the virtual teaching aid and the contact state, the motion response of the virtual teaching aid is calculated by solving the dynamic equation and generating a force feedback signal. The virtual models of multiple students are rendered into the same virtual teaching scene, and the three-dimensional trajectory of the limb movement and the force feedback signal of different students are synchronized in real time to generate a multi-person collaborative interaction process of the virtual teaching aid.

[0013] A third aspect of the present invention, An electronic device is provided, comprising: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0014] Fourth aspect of the embodiments of the present invention, A computer-readable storage medium is provided, having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0015] In this embodiment of the invention, a deep learning-based generative model can accurately learn the inherent distribution characteristics of teaching aids, automatically identify and generate complex structures, and improve the accuracy and efficiency of the generative model. By establishing three-dimensional shapes using mesh components, the geometric details of each sub-component can be precisely described, making the teaching aid model more realistic. A tree-like hierarchical structure is formed, enabling structured management and operation of sub-components, facilitating the editing and adjustment of complex teaching aids. High-precision limb motion capture is achieved by using a structured light depth camera and a convolutional neural network to detect key points, combined with acceleration and angular velocity data from an inertial measurement unit. Based on color and depth images, two-dimensional pixel coordinates are mapped to three-dimensional space to generate a three-dimensional skeleton. The iliac curve ensures spatial accuracy and three-dimensionality in motion capture; data calibration using the inertial measurement unit eliminates accumulated drift errors, improving the accuracy and robustness of inertial motion capture and ensuring accuracy over long periods; the trainee's limb movements are accurately mapped onto the virtual model, and forward and inverse kinematics algorithms ensure that the joint movements of the virtual model are consistent with the actual movements; the contact state between the virtual model and the virtual teaching aid is detected in real time, and contact point information is obtained to support subsequent physical simulation and force feedback calculation, enhancing the realism of the interaction; based on the material properties and contact state of the virtual teaching aid, the motion response of the teaching aid is calculated through dynamic equations to achieve realistic deformation, displacement, and reaction force effects. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating a VR-based interactive teaching method for primary and secondary schools according to an embodiment of the present invention. Figure 2 This is a schematic diagram of the structure of a VR-based interactive teaching system for primary and secondary schools according to an embodiment of the present invention. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0019] Figure 1 This is a flowchart illustrating a VR-based interactive teaching method for primary and secondary schools according to an embodiment of the present invention. Figure 1 As shown, the method includes: S101. Based on the pre-acquired teaching aid model data, a model is generated through deep learning to learn the inherent distribution characteristics. Based on the inherent distribution characteristics, multiple sub-components are determined. Combined with the teaching aid generation engine, the corresponding three-dimensional shape is established through the mesh component. A transformation node is created for each sub-component, the parent-child relationship between transformation nodes is set, a tree-like hierarchical structure is generated, and a virtual teaching aid is constructed. The sub-components specifically refer to the components or modules of a virtual teaching aid. Sub-components can be different structural parts of the teaching aid, each with independent geometry and functional characteristics, and can be defined and operated individually.

[0020] The mesh component specifically refers to a set of triangular facets used to define the surface of a 3D model. Mesh components form the basis of the 3D shape of virtual teaching aids, allowing for the description of the aid's appearance and surface details.

[0021] The transformation node specifically refers to a node used to describe the position, rotation, and scaling of an object in three-dimensional space. Transformation nodes organize sub-components through a hierarchical structure, forming a parent-child relationship tree, which facilitates the management and operation of the complex structure of the virtual teaching aid.

[0022] In this embodiment, the deep learning generative model can accurately learn the inherent distribution characteristics of the teaching aids, automatically identify and generate complex structures, and improve the accuracy and efficiency of the generated model. By establishing three-dimensional shapes through mesh components, the geometric details of each sub-component can be accurately described, making the teaching aid model more realistic. The tree-like hierarchical structure enables structured management and operation of sub-components, facilitating the editing and adjustment of complex teaching aids. The automated process based on deep learning and generative models reduces the workload of manual intervention and manual modeling, improving production efficiency. The intelligent sub-component division and transformation node management make the generation and operation of virtual teaching aids more intelligent and convenient.

[0023] In one optional embodiment, based on pre-acquired teaching aid model data, a model is generated through deep learning to learn the inherent distribution characteristics, including: Acquire teaching aid model data; construct a geometric feature network based on the PointNet++ structure to represent the teaching aid model data as a point cloud and learn the geometric shape features corresponding to the point cloud; construct a visual feature network based on the ResNet structure to render the teaching aid model data as multi-view images and learn the appearance texture features corresponding to the multi-view images; represent the material attributes, size attributes, and usage attributes in the teaching aid model data as corresponding attribute vectors, and learn functional semantic features through fully connected layers to determine the functional attribute network; The geometric feature network, the visual feature network, and the functional attribute network are dynamically adjusted through an attention mechanism and combined into a cross-modal fusion network. The intrinsic distribution characteristics of the teaching aid model data are learned through a modal alignment loss function and a cross-modal distillation learning strategy. A large amount of teaching aid model data needs to be collected, including the three-dimensional geometric model of the teaching aid, multi-view rendered images, material properties, size properties and usage properties, etc., and then preprocessed and labeled to form a structured dataset.

[0024] PointNet++ was chosen as the backbone network for geometric feature extraction. The 3D geometric data of each teaching aid model was represented as a point cloud, i.e., a set of 3D coordinate points. PointNet++ was used to learn the local and global features of the point cloud layer by layer, and geometric shape features were extracted through operations such as dense connections and multi-scale grouping. The network input is the point cloud data, and the output is a feature vector representing the geometric features.

[0025] ResNet was chosen as the backbone network for visual feature extraction. Each manipulative model was rendered as a 2D image from multiple viewpoints. By changing the camera position and angle, the appearance information of the manipulative from different viewpoints was obtained. The pre-trained ResNet was used to learn visual features such as texture, color, and material of the images layer by layer. Through mechanisms such as residual connections and cross-layer information flow, the appearance and texture features of the multi-view images were extracted. The network input is multi-view image data, and the output is a feature vector representing the visual features.

[0026] Material, size, and usage attributes in the teaching aid model data are each represented as a one-dimensional vector, which serves as the input to the functional attribute network. A multi-layer fully connected neural network learns the non-linear mapping relationship between the attribute vectors to extract the functional semantic features of the teaching aid. The network's input is the attribute vector, and its output is a feature vector representing the functional semantics.

[0027] The output feature vectors of the geometric feature network, visual feature network, and functional attribute network are dynamically weighted and fused using an attention mechanism. This attention mechanism adaptively adjusts the weights of features from each modality based on their importance, achieving effective integration of cross-modal information. The fused feature vector serves as the output of the cross-modal fusion network, representing the inherent distribution characteristics of the teaching aid model data.

[0028] To ensure the consistency of features across different modalities in the semantic space, a modality alignment loss function is needed. This function measures the similarity between features from different modalities, encouraging semantically similar teaching aids to be closer together in the feature space, and semantically different ones to be further apart. Preferably, the modality alignment loss function is a triplet loss, which promotes semantic alignment of cross-modal features by minimizing the loss function. To further improve the learning performance of the cross-modal fusion network, a cross-modal distillation learning strategy can be employed. Specifically, a geometric feature network, a visual feature network, and a functional attribute network are pre-trained on single-modal data to obtain their respective teacher models. The knowledge from these teacher models guides the learning process of the cross-modal fusion network. By minimizing the distillation loss between the student model and the teacher model, knowledge from single modalities is transferred to the cross-modal fusion network, improving the network's generalization ability and robustness.

[0029] In this embodiment, feature learning is performed by extracting geometric features, visual features, and functional attributes comprehensively. Geometric, visual, and functional attribute features are dynamically weighted and fused through an attention mechanism to achieve a comprehensive feature representation of the teaching aid model data. The fused feature vector can represent the inherent distribution characteristics of the teaching aid model, enabling the model to exhibit consistent semantic features in different modalities. Through high-precision feature extraction and fusion, a more realistic virtual teaching aid model can be generated, improving the simulation and modeling effect in the virtual environment.

[0030] In one optional embodiment, based on inherent distribution characteristics, multiple sub-components are determined. Combined with a teaching aid generation engine, corresponding 3D shapes are established using mesh components. A transformation node is created for each sub-component, and parent-child relationships between transformation nodes are set to generate a tree-like hierarchical structure. The virtual teaching aid is constructed by including: The intrinsic distribution features are input into the generator of the cross-modal conditional generative adversarial network. The generator determines the generator teaching aid model based on the intrinsic distribution features. The generator teaching aid model is input into the discriminator of the cross-modal conditional generative adversarial network. The discriminator judges the realism of the generator teaching aid model and the consistency between the generator teaching aid model and the intrinsic distribution features. Through adversarial iteration between the generator and the discriminator, the optimal teaching aid model is learned. The optimal teaching aid model is input into a sub-component discovery network, which includes a variational autoencoder module and an active learning module. The variational autoencoder module encodes the optimal teaching aid model by minimizing the reconstruction error and KL divergence to obtain a low-dimensional latent space representation. Based on the low-dimensional latent space representation, similar sub-components are clustered to form a cluster structure. The active learning module adopts a maximum entropy-based sampling strategy and labels the sub-components according to the cluster structure in the low-dimensional latent space representation. Combined with manifold-based regularization constraints, the uniform distribution of sub-component labels is determined. By alternating between unsupervised learning by the variational autoencoder module and supervised learning by the active learning module, the labeling cost is minimized, and the virtual teaching aid sub-components are determined. Based on each virtual teaching aid sub-component, a corresponding three-dimensional shape is established through a mesh component. Based on the three-dimensional shape, a transformation node is created for each virtual teaching aid sub-component. The parent-child relationship between the transformation nodes is set to generate a tree-like hierarchical structure. Combined with the attributes of the transformation nodes, the constraint relationship between the virtual teaching aid sub-components is set. At the same time, the texture and color of the virtual teaching aid sub-components are set through a material component, and the virtual teaching aid sub-components are assembled into a virtual teaching aid.

[0031] The learned intrinsic distribution features are input into the generator as conditions. Guided by these features, the generator produces matching educational toy models. These models are then fed into a discriminator for realism and consistency assessment. The discriminator continuously improves its ability to distinguish between real and generated models through comparative learning. Simultaneously, the generator adjusts its generation strategy based on feedback from the discriminator, aiming to produce more realistic and matching educational toy models. Through repeated interaction between the generator and the discriminator, a generator capable of producing high-quality educational toy models based on intrinsic distribution features is ultimately obtained.

[0032] The optimal teaching aid model generated by a cross-modal conditional generative adversarial network is input into the sub-part discovery network. A variational autoencoder (VAE) module encodes the teaching aid model, mapping it to a low-dimensional latent space by minimizing the difference between the input and reconstructed teaching aid models, and minimizing the KL divergence between the latent space distribution and the prior distribution. In this low-dimensional latent space, semantically similar sub-parts automatically cluster together. Next, an active learning module selects the most informative sub-parts for annotation based on a maximum entropy sampling strategy. By introducing a manifold-based regularization term, the annotation samples are ensured to be evenly distributed in the low-dimensional latent space, covering different sub-part types. By alternately training the VAE and active learning modules, unsupervised discovery and supervised annotation of sub-parts are achieved, ultimately resulting in a sub-part discovery network that accurately identifies and classifies teaching aid sub-parts.

[0033] For each sub-part identified by the sub-part discovery network, its corresponding 3D shape is constructed using mesh components. These mesh components are based on triangular or quadrilateral faces and describe the geometry of the sub-part using attributes such as vertex positions, normal vectors, and texture coordinates. By adjusting the mesh resolution and level of detail, 3D shapes of varying precision and complexity can be generated. For sub-parts with symmetry or repetition, a procedural modeling approach can be used, parametrically controlling the mesh generation process to improve modeling efficiency.

[0034] A corresponding transformation node is created for the 3D shape of each sub-component. The transformation node records the sub-component's position, rotation, scaling, and other transformation information in virtual space. By setting parent-child relationships between transformation nodes, a tree-like hierarchical structure is established for the entire virtual teaching aid. The transformation of a parent node affects the transformation of all its child nodes, ensuring that different sub-components maintain their relative positions and constraints. The root node of the tree structure represents the entire virtual teaching aid, and the leaf nodes represent the most basic sub-components. Through recursive traversal and transformation propagation within the tree structure, coordinated movement and interaction of the various parts of the virtual teaching aid are achieved.

[0035] Based on the transformation nodes and combined with prior knowledge of the teaching aid's function and structure, constraint relationships are set between sub-components. Common constraint relationships include hinge constraints, sliding constraints, and fixed constraints. Hinge constraints restrict the relative rotation between two sub-components, simulating the behavior of a hinge connection. Sliding constraints restrict the relative translation between two sub-components, simulating the motion of a slide rail mechanism. Fixed constraints fix multiple sub-components together, forming a whole. By reasonably setting constraint relationships, the movement and interaction of the virtual teaching aid are ensured to conform to physical laws and functional requirements.

[0036] Each sub-component is given texture and color attributes using material components. Textures are applied via UV mapping, projecting 2D images onto a 3D mesh surface to simulate the surface details and texture of the sub-components. Tile, blend, and transition operations on the textures enrich the visual effects of the sub-components. Color attributes control the overall tone and brightness variations of the sub-components and can be customized according to the design style and color scheme of the teaching aid. A reasonable combination of material attributes enhances the realism and aesthetics of the virtual teaching aids.

[0037] Sub-components with pre-defined material properties are assembled in a hierarchical tree structure to form a complete virtual teaching aid. The assembly process requires consideration of factors such as dimensional matching, connection methods, and motion constraints between sub-components. By adjusting the properties of transformation nodes, the relative positions and postures of the sub-components are controlled to meet the structural and functional requirements of the teaching aid. For detachable and combinable teaching aids, suitable connection mechanisms and interaction methods also need to be designed to facilitate user assembly and operation. Ultimately, the virtual teaching aid obtained through virtual assembly should have a reasonable structure, realistic appearance, and good interactivity.

[0038] In this embodiment, the cross-modal conditional generative adversarial network generates a high-quality teaching aid model that highly matches the inherent distribution characteristics through repeated game interaction, improving the model's realism and consistency. The sub-component discovery network utilizes variational autoencoders and active learning modules to achieve unsupervised discovery and supervised annotation of teaching aid sub-components, ensuring the accuracy and comprehensiveness of sub-component identification. The mesh component constructs a 3D shape, and by adjusting the resolution and level of detail, as well as adopting a procedural modeling approach, it improves the efficiency and accuracy of sub-component modeling. Transformation nodes and tree-like hierarchical structures record and propagate the transformation information of sub-components, enabling coordinated movement and interaction of various parts of the teaching aid, and improving the dynamic management capability of the virtual teaching aid. The assembly process of the virtual teaching aid considers factors such as size matching, connection methods, and motion constraints to ensure that the final assembled virtual teaching aid has a reasonable structure, realistic appearance, and good interactivity, making it convenient for users to assemble and operate.

[0039] S102. Using a structured light depth camera, acquire color and depth images of the trainee. Based on the color image, use a convolutional neural network to detect key points of body parts and obtain two-dimensional pixel coordinates. Based on the depth information in the depth image, map the key points of the body parts into space to obtain three-dimensional spatial coordinates and generate a three-dimensional skeletal curve of the trainee's limb movement to obtain optical motion capture results. Based on the three-dimensional skeletal curve, use an inertial measurement unit to acquire the acceleration and angular velocity of the key points of the body parts. After inertial drift calibration, obtain inertial motion capture results. Fuse the optical motion capture results and the inertial motion capture results to reconstruct the three-dimensional trajectory of the trainee's limb movement.

[0040] In this embodiment, a structured light depth camera and a convolutional neural network are used to detect key points. Combined with acceleration and angular velocity data from an inertial measurement unit (IMU), high-precision limb motion capture is achieved. Based on color and depth images, two-dimensional pixel coordinates are mapped to three-dimensional space to generate three-dimensional skeletal curves, ensuring spatial accuracy and stereoscopic effect of motion capture. Through data calibration of the IMU, accumulated drift errors are eliminated, improving the accuracy and robustness of inertial motion capture and ensuring accuracy of long-term motion capture. The results of optical and inertial motion capture are fused, combining the advantages of both to achieve more accurate and stable limb motion reconstruction, improving the overall performance of motion capture. Through the fused motion capture results, the three-dimensional trajectory of the trainee's limb movement is reconstructed, realizing continuous and dynamic motion tracking, suitable for real-time interactive scenarios.

[0041] In one optional embodiment, a structured light depth camera is used to acquire color and depth images of the trainee. Based on the color image, a convolutional neural network is used to detect key points of body parts and obtain two-dimensional pixel coordinates. Based on the depth information in the depth image, the key points of the body parts are mapped into space to obtain three-dimensional spatial coordinates. A three-dimensional skeletal curve of the trainee's limb movement is generated, and the optical motion capture results include: A structured light depth camera simultaneously acquires color and depth images of the trainee. The structured light depth camera includes an infrared projector, two infrared cameras, and a color camera. The infrared projector projects structured light according to a preset pattern, generating deformable stripe images on the trainee's surface. The two infrared cameras acquire these deformable stripe images from two different perspectives. Based on the two sets of deformable stripe images, the depth value of each pixel is calculated, and a depth image is synthesized. The color camera acquires the trainee's color image. The color and depth images are then spatiotemporally synchronized and aligned at different perspectives to obtain a synchronized image set. A keypoint detection network is constructed based on the U-Net structure. The keypoint detection network includes an encoder and a decoder. The encoder is based on a ResNet-50 structure and extracts multi-scale image features from color images in a synchronously aligned image set. The decoder fuses the multi-scale image features through skip connections to generate a feature map. The feature map is then mapped to a heatmap corresponding to keypoints of body parts through convolutional layers. Each keypoint of a body part corresponds to a heatmap channel. The probability that a body part is a keypoint is determined based on the pixel value. The two-dimensional pixel coordinates of the keypoints of body parts are extracted through threshold filtering and non-maximum suppression. Based on the depth image in the synchronized image set, the depth value corresponding to the two-dimensional pixel coordinates is determined. Combined with the intrinsic parameters of the structured light depth camera, the three-dimensional coordinates of the key points of the body parts in the corresponding coordinate system of the structured light depth camera are calculated by the coordinate transformation formula. Based on the three-dimensional coordinates, with the key point of the body part corresponding to the head as the root node, the key points of each body part are linked sequentially according to the predetermined human motion chain topology to form a tree-like limb hierarchy. By connecting the key points of adjacent body parts and adjusting the translation and rotation parameters, a three-dimensional skeletal curve of the trainee's limb movement is generated. The three-dimensional coordinates corresponding to the key points of the body parts in adjacent frames are combined into a three-dimensional coordinate sequence and smoothed and filtered to obtain the optical motion capture result.

[0042] The structured light depth camera consists of an infrared projector, two infrared cameras, and a color camera. The infrared projector projects structured light onto the trainee according to a preset pattern, forming distorted striped images on the trainee's body surface. The two infrared cameras capture these distorted striped images from different perspectives, obtaining two sets of image data. Simultaneously, the color camera captures a color image of the trainee, obtaining information about the trainee's appearance.

[0043] Two sets of deformed fringe images captured by two infrared cameras are used as input. The depth value of each pixel is calculated using the principle of triangulation. By matching the fringe deformation at the same object surface location in the two sets of images, disparity information is obtained. Combining the intrinsic and extrinsic parameters of the infrared cameras, the disparity is converted into a depth value using the triangulation equation. Interpolation and filtering are performed on the depth values ​​of all pixels to generate a complete depth image. Each pixel value in the depth image represents the depth distance of that point from the structured light depth camera.

[0044] Due to differences in position and viewing angle between the color camera and the infrared camera, spatiotemporal synchronization and viewing angle alignment of the color and depth images are necessary. Hardware triggering or software timestamps are used to ensure synchronized acquisition of the color and depth images in time. Utilizing extrinsic parameters (relative position and angle) between the cameras and intrinsic parameters of the images (focal length, principal point coordinates, etc.), coordinate transformation and image correction algorithms are employed to align the color and depth images to a unified viewing angle and coordinate system. The aligned color and depth images form a synchronized image set.

[0045] A keypoint detection network is constructed based on the U-Net architecture, consisting of an encoder and a decoder. The encoder uses a ResNet-50 structure, extracting multi-scale image features from color images in a synchronized image set through convolution and downsampling operations. The decoder fuses the multi-scale features extracted by the encoder through upsampling and skip connections to generate a high-resolution feature map. Convolutional layers map the feature map into heatmaps corresponding to keypoints of body parts, with each keypoint corresponding to a heatmap channel. The pixel values ​​in the heatmap represent the probability that the location is a keypoint of a body part.

[0046] Color images from the synchronized image set are input into a trained keypoint detection network to obtain heatmaps corresponding to body part keypoints. Threshold filtering is applied to the heatmaps to remove pixels with probability values ​​below a threshold, retaining high-probability candidate keypoints. A non-maximum suppression algorithm is used to select local maxima among the candidate keypoints, yielding the final 2D pixel coordinates of the body part keypoints. The depth image of the corresponding location in the synchronized image set is used to find the depth value corresponding to the 2D pixel coordinates. Combining the intrinsic parameters of the structured light depth camera, the pixel coordinates and depth values ​​are converted into 3D coordinates of the body part keypoints in the camera coordinate system using a coordinate transformation formula.

[0047] Based on the 3D coordinates of key points in body parts, a hierarchical structure of the trainee's limbs is generated according to a predefined human motion chain topology. The key point corresponding to the head is used as the root node, and a tree-like hierarchical structure of limbs is formed by connecting adjacent key points. Each key point represents a joint or endpoint of the human body, and the line connecting adjacent key points represents a segment of the limb. By adjusting the translation and rotation parameters between key points, the limbs conform to the anatomical constraints of human movement. Finally, a 3D skeletal curve representing the trainee's limb movement is generated, with one skeletal curve corresponding to each time frame.

[0048] The 3D coordinates of key points on body parts are tracked between adjacent frames to form a continuous 3D coordinate sequence. Due to potential noise and jitter during key point detection and coordinate calculation, the 3D coordinate sequence needs to be smoothed. High-frequency noise is removed using Kalman filtering to obtain a smooth 3D coordinate sequence, representing the continuous temporal trajectory of the trainee's limbs. This smoothed 3D coordinate sequence is then used as the output of the optical motion capture.

[0049] In this embodiment, structured light projection and two infrared cameras can accurately acquire depth information of the object's surface, enabling the generation of high-resolution depth images. Hardware triggering or software timestamps are used to ensure synchronous acquisition of color and depth images, improving data consistency and accuracy. Based on a keypoint detection network, multi-scale feature extraction and fusion can accurately predict the location of key points on body parts, improving the accuracy of keypoint detection. Combining the depth images and intrinsic parameters of the structured light depth cameras, two-dimensional pixel coordinates and depth values ​​are converted into three-dimensional coordinates, enabling three-dimensional localization of key points on body parts. A hierarchical structure of the student's limbs is generated based on the three-dimensional coordinates, facilitating the description and analysis of limb movements.

[0050] In one optional embodiment, based on a three-dimensional skeletal curve, the acceleration and angular velocity of key points of body parts are acquired through an inertial measurement unit. After inertial drift calibration, inertial motion capture results are obtained. The optical motion capture results and the inertial motion capture results are fused to reconstruct the three-dimensional trajectory of the trainee's limb movements, including: Based on the extracted key points of the body parts, an inertial measurement unit is attached to determine the three-axis acceleration and angular velocity. Through attitude calculation and integration, the preliminary pose of the body parts is determined. The inertial measurement unit includes an accelerometer, a gyroscope and a magnetometer, which measure the three-axis acceleration and angular velocity corresponding to the key points of the body parts, respectively. Initiating the inertial drift calibration process: Using the initial pose as the pose of the previous moment, and combining the triaxial acceleration and angular velocity, the current pose of each part of the body at the current moment is predicted through the inertial navigation equation, thus obtaining the current prior state estimate and the state estimate covariance matrix. Using the optical motion capture results as observations, the error between the observations and the current prior state estimate is calculated, an observation equation is constructed, error statistics are determined, and the observation noise covariance matrix is ​​obtained. Based on the state estimate covariance matrix and the observation noise covariance matrix, the optimal Kalman gain matrix is ​​calculated. Based on the optimal Kalman gain matrix, the current prior state estimate is corrected to obtain the posterior state estimate, and the state estimate covariance matrix is ​​updated at the same time. Using the posterior state estimate as the initial pose for the next moment, the inertial drift calibration process is repeated to obtain the inertial motion capture result; Based on the 3D skeletal curves corresponding to optical motion capture results and the body part poses corresponding to inertial motion capture results, temporal features are extracted, a multi-scale spatiotemporal feature pyramid is constructed, and the features are encoded and fused through an attention-gated recurrent unit network to obtain fused features. Through adaptive weight adjustment, combined with the mutual compensation between optical motion capture results and inertial motion capture results, the fused features are mapped to the 3D trajectory of the trainee's limb movement through a multilayer perceptron.

[0051] Inertial measurement units (IMUs), including a three-axis accelerometer, a three-axis gyroscope, and a three-axis magnetometer, are attached to key points on the body. The accelerometer measures the linear acceleration of the body part, the gyroscope measures the angular velocity, and the magnetometer measures the orientation of the body part relative to the Earth's magnetic field. Using an attitude calculation algorithm, preferably the quaternion method or the Euler angle method, the raw data from the IMUs is converted into attitude angles (pitch, roll, and yaw) of the body part. Combining the acceleration data and attitude angles, an integral calculation is performed to obtain the preliminary position and attitude of the body part, i.e., the preliminary pose. The preliminary pose provides initial values ​​for subsequent inertial drift calibration.

[0052] Using the pose from the previous moment as the initial pose for the current moment, and combining it with the triaxial acceleration and angular velocity measured by the inertial measurement unit at the current moment, state prediction is performed using the inertial navigation equations. The inertial navigation equations, based on Newton's laws of motion and Euler's rotation theorem, describe the dynamic relationship between position, velocity, and attitude. Velocity is obtained by integrating acceleration, position by integrating velocity, and attitude angle by integrating angular velocity. The state prediction result is a priori pose estimate of each part of the body at the current moment. Based on the noise and uncertainty in the state prediction process, the covariance matrix of the state estimate is calculated, representing the uncertainty of the prior estimate.

[0053] The 3D coordinates of key points on body parts obtained through optical motion capture are used as observations and compared with the prior pose estimate at the current moment to calculate the error between the two. An observation equation is constructed to establish a mapping relationship between the observations and state variables. The observation equation considers noise and errors in the optical motion capture process. Based on the statistical characteristics of the error between the observations and the prior estimate, a covariance matrix of the observation noise is constructed to represent the uncertainty of the observations.

[0054] Based on the covariance matrix of the prior state estimate and the covariance matrix of the observation noise, the optimal Kalman gain matrix is ​​calculated through the update step of the Kalman filter. The Kalman gain matrix determines the degree to which the observations correct the state estimate, representing a trade-off between the uncertainty of the state estimate and the uncertainty of the observations. The optimal Kalman gain matrix is ​​multiplied by the errors of the observations and the prior estimate to correct the prior state estimate, yielding the posterior state estimate, i.e., the optimal pose estimate of the body part at the current moment. Simultaneously, the covariance matrix of the state estimate is updated based on the Kalman gain matrix, reducing the uncertainty of the state estimate.

[0055] The posterior state estimate at the current moment is used as the initial pose at the next moment, and the optimization process is repeated iteratively. By continuously introducing new observations to correct the state estimate and propagating the uncertainty of the state estimate, continuous optimization and drift calibration of the inertial motion capture results are achieved. After multiple iterations, the cumulative drift of inertial motion capture is effectively suppressed, and a stable and accurate sequence of body part poses, i.e., the inertial motion capture result, is output.

[0056] Temporal features were extracted from the 3D skeletal curves obtained by optical motion capture and the body part pose sequences obtained by inertial motion capture. These temporal features include the changes in physical quantities such as joint angles, angular velocities, and accelerations over time, as well as the spatiotemporal correlations between different body parts. Through Fourier transform, temporal features were extracted at different time scales, constructing a multi-scale spatiotemporal feature pyramid. This feature pyramid contains spatiotemporal features at different levels of abstraction and temporal granularity, comprehensively depicting the dynamic changes in limb movement.

[0057] Multi-scale spatiotemporal feature pyramids from optical and inertial motion capture are input into an attention-gated recurrent unit (RNU) network for feature encoding and fusion. The attention mechanism dynamically adjusts the importance weights among different features, highlighting those most relevant to the current motion state. The RNU network, by introducing forget and update gates, controls the transmission and updating of features over time, capturing temporal dependencies. Through multiple stacked layers of attention-gated RNUs, the network progressively extracts and fuses complementary features from optical and inertial motion capture to obtain a high-level semantic feature representation, i.e., fused features.

[0058] Based on the reliability and accuracy of optical and inertial motion capture, their weights in the fusion features are adaptively adjusted. When optical motion capture is affected by occlusion, edge distortion, etc., its weight is reduced and the weight of inertial motion capture is increased; when inertial motion capture experiences drift or noise interference, its weight is reduced and the weight of optical motion capture is increased. Through weight adjustment, mutual compensation and optimization between optical and inertial motion capture results are achieved. Finally, the fused features are input into a multilayer perceptron, and through nonlinear transformation and parameter learning, the fused features are mapped to the three-dimensional trajectory of the trainee's limb movements, i.e., the final motion capture result.

[0059] In this embodiment, attitude angles of body parts are obtained using attitude calculation algorithms based on data from accelerometers, gyroscopes, and magnetometers, providing accurate initial poses. Combining the current acceleration and angular velocity, state prediction is performed using inertial navigation equations to obtain prior pose estimates for each body part. Three-dimensional coordinates from optical motion capture are introduced as observations, errors are calculated, observation equations are constructed, and error statistics are performed to calibrate the observations with prior estimates. The optimal Kalman gain matrix is ​​calculated using a Kalman filter to correct prior state estimates and update the covariance matrix of state estimates, reducing uncertainty in state estimation. Temporal features from optical and inertial motion capture are extracted to construct a multi-scale spatiotemporal feature pyramid, comprehensively characterizing the dynamic changes in limb movement. Attention-gated recurrent unit networks are used to encode and fuse multi-scale spatiotemporal features, highlighting features most relevant to the current motion state and capturing temporal dependencies.

[0060] S103. Based on the three-dimensional trajectory of the student's limb movement, the motion of the student's virtual model is mapped. Through a collision detection algorithm, the contact state between the virtual model and the virtual teaching aid is calculated in real time. According to the material characteristics of the virtual teaching aid and the contact state, the motion response of the virtual teaching aid is calculated by solving the dynamic equation and generating a force feedback signal. The virtual models of multiple students are rendered into the same virtual teaching scene, and the three-dimensional trajectory of the limb movement and the force feedback signal of different students are synchronized in real time to generate a multi-person collaborative interaction process of the virtual teaching aid.

[0061] The trainee's limb movement 3D trajectory is mapped onto a virtual model. Using a forward kinematics algorithm, the 3D spatial coordinates of each joint in the virtual model are calculated based on the trainee's joint angles and bone length. Then, using an inverse kinematics algorithm, the trainee's extremities, such as hands and feet, are mapped to corresponding parts of the virtual model, enabling posture control of the virtual model.

[0062] Virtual teaching aids related to the teaching content are placed in the virtual teaching scenario. The contact state between the virtual model and the virtual teaching aids is calculated in real time through a collision detection algorithm. The collision detection results include information such as the position of the contact point, the normal vector, and the penetration depth, providing input conditions for subsequent physical simulation and force feedback calculation.

[0063] Based on the material properties (such as stiffness, coefficient of friction, and elasticity) and contact state of the virtual teaching aid, the motion response of the virtual teaching aid is calculated by solving the dynamic equations. The results of the physical simulation include state variables such as deformation, displacement, and velocity of the virtual teaching aid, as well as the reaction forces acting on the virtual model. Through numerical integration and constraint solving, realistic motion effects of the virtual teaching aid are achieved.

[0064] Based on the physical simulation results of the virtual teaching aids, force feedback signals are generated, allowing learners to experience tactile feedback when interacting with the virtual teaching aids. The force feedback signals include information such as the magnitude, direction, and point of application of the force, and can be transmitted to learners through force feedback devices (such as force feedback gloves, force feedback joysticks, etc.).

[0065] This system renders virtual models of multiple learners into the same virtual teaching scene, enabling multi-user collaborative interaction. Through network communication and data synchronization technologies, it transmits the 3D trajectories of different learners' limb movements and force feedback signals in real time, ensuring that the position, posture, and actions of each learner's virtual model remain consistent within the virtual scene. Distributed rendering and load balancing technologies optimize rendering performance and response speed for multi-user collaborative interaction, providing a smooth and real-time interactive experience. During multi-user collaborative interaction, different learners can interact with shared virtual teaching aids through their respective virtual models. For example, multiple learners can collaborate to complete an experiment, assemble a virtual device together, or pass objects within the virtual scene. Collision detection and physics simulation handle the interaction between multiple virtual models and virtual teaching aids, calculating the motion response and force feedback effects of the virtual teaching aids under multi-user interaction. Synchronization locking and concurrency control mechanisms resolve conflicts and competition issues during multi-user interaction, ensuring the consistency of the virtual teaching aids' state and the fairness of the interaction.

[0066] In this embodiment, the trainee's limb movements are accurately mapped onto the virtual model. Forward and inverse kinematics algorithms ensure that the joint movements of the virtual model are consistent with the actual movements. The contact state between the virtual model and the virtual teaching aid is detected in real time, acquiring contact point information to support subsequent physical simulation and force feedback calculations, enhancing the realism of the interaction. Based on the material properties and contact state of the virtual teaching aid, the motion response of the teaching aid is calculated using dynamic equations, achieving realistic deformation, displacement, and reaction force effects. Force feedback signals are generated and transmitted to the trainee through a force feedback device, allowing them to experience tactile feedback when interacting with the virtual teaching aid, providing a realistic, natural, and comfortable tactile experience. Network communication and data synchronization technologies ensure that the virtual model states of each trainee are consistent, providing a smooth and real-time interactive experience. The system handles the interaction between multiple virtual models and virtual teaching aids, calculating the motion response and force feedback effects of the teaching aid under the influence of multiple users. Synchronization locking and concurrent control mechanisms ensure the fairness and consistency of the interaction.

[0067] In one optional embodiment, the motion of the virtual model of the student is mapped based on the three-dimensional trajectory of the student's limb movements. A collision detection algorithm is then used to calculate in real time the contact state between the virtual model and the virtual teaching aid, including: Based on the three-dimensional geometric data corresponding to the virtual teaching aid, the mathematical equation form is determined, a discrete point set in three-dimensional space is generated, the surface feature points corresponding to the virtual teaching aid are extracted, virtual teaching aid point cloud data is generated, the normal vector of the virtual teaching aid point cloud data is estimated, and the surface triangular facet mesh topology of the virtual teaching aid is reconstructed with the point position and normal vector as constraints. After mesh post-processing, the virtual teaching aid triangular facet mesh model is obtained. The bounding box calculation is performed on the triangular facet mesh model of the virtual teaching aid to obtain the corresponding AABB bounding box. Based on the geometric features of the virtual teaching aid, the AABB bounding box is adaptively and recursively partitioned until the size of a single triangular facet within the AABB bounding box is less than the preset unit size threshold. The AABB bounding box is then used as a leaf node to construct the octree space partitioning structure corresponding to the virtual teaching aid. For leaf nodes, calculate the minimum distance from the corresponding AABB bounding box to the surface of the student virtual model, and use it as the distance field value of the leaf node; traverse the octree from bottom to top, and for each non-leaf node, take the minimum value of the distance field value corresponding to the lower-level child node of the non-leaf node, and use it as the distance field value of the non-leaf node; traverse the octree from top to bottom, and for each node, update the distance field value of the node with the distance field value corresponding to the parent node of the node, and construct the octree hierarchical distance field; During the movement, based on the octree hierarchical distance field, starting from the root node of the octree, the system recursively traverses and determines whether each node has collided. Nodes with distance field values ​​greater than a preset collision threshold are pruned, and leaf nodes with distance field values ​​less than or equal to the preset collision threshold are marked as potential collision areas. For the set of triangular facets in the potential collision region, a preset bounding volume detection algorithm is applied to the triangular facets corresponding to the student's virtualization model to perform pairwise collisions and determine the intersecting triangular facet pairs. Based on intersecting triangular facet pairs, continuous collision detection is performed. The earliest collision time and corresponding collision information of the intersecting triangular facet pairs in the continuous motion space are calculated. Based on the collision position coordinates and normal vector in the collision information, the collision penetration depth is calculated. Neighboring collision points are clustered and fused to generate a collision region data structure. Collision data is statistically analyzed to determine the contact state of the collision.

[0068] Based on the 3D geometric data of the virtual teaching aid, a discrete point set of the virtual teaching aid in 3D space is generated. Feature points are extracted from the surface of the virtual teaching aid using a sampling algorithm to generate point cloud data. Normal vector estimation is performed on the point cloud data to calculate the normal vector of each point, representing the orientation of the local surface. Using the position and normal vector of the points as constraints, the surface triangular mesh topology of the virtual teaching aid is reconstructed using triangulation algorithms, such as Delaunay triangulation and Poisson surface reconstruction. Post-processing is performed on the reconstructed mesh, such as hole filling, mesh simplification, and smoothing, to obtain a high-quality triangular mesh model of the virtual teaching aid.

[0069] Bounding box calculations are performed on the triangular mesh model of the virtual teaching aid to obtain the smallest axis-aligned bounding box (AABB bounding box) that can completely contain the virtual teaching aid. Based on the geometric characteristics of the virtual teaching aid, such as size, curvature, and detail, the AABB bounding box is adaptively and recursively partitioned. The AABB bounding box is evenly divided into eight sub-bounding boxes along the three coordinate axes, and the triangular faces within each sub-bounding box are statistically analyzed. If the size of the triangular faces within a sub-bounding box is greater than a preset unit size threshold, the recursive partitioning of that sub-bounding box continues; otherwise, the sub-bounding box is designated as a leaf node, and the partitioning stops. Through recursive partitioning, an octree spatial partitioning structure corresponding to the virtual teaching aid is constructed, where each leaf node corresponds to an AABB bounding box containing local triangular faces.

[0070] For each leaf node of the octree, calculate the minimum distance from its corresponding AABB bounding box to the surface of the student virtual model, and use this as the distance field value for that leaf node. Distance calculation can employ algorithms that minimize distances from a point to a triangular facet, such as the distance from a point to a plane or the distance from a point to an edge. Traverse the octree from bottom to top. For each non-leaf node, take the minimum distance field value corresponding to its lower-level child nodes, and use this as the distance field value for that non-leaf node. Through bottom-up propagation, obtain the initial distance field values ​​for all nodes in the octree. Then, traverse the octree from top to bottom. For each node, update its distance field value using the distance field value corresponding to its parent node. Through top-down updates, obtain the hierarchical distance field of the octree. The distance field value of each node represents a conservative estimate of the minimum distance from its corresponding AABB bounding box to the surface of the student virtual model.

[0071] During the movement of the virtual teaching aids and the student virtual model, collision detection is performed based on an octree hierarchical distance field. Starting from the root node of the octree, a recursive traversal is performed to determine whether each node is in conflict. For each node, its distance field value is compared with a preset collision threshold. If the distance field value is greater than the collision threshold, it indicates that the AABB bounding box corresponding to the node will not collide with the student virtual model, and the node can be pruned without further traversing its child nodes; otherwise, if the node is a non-leaf node, its child nodes are recursively traversed; if the node is a leaf node, it is marked as a potential collision region, awaiting further fine-grained collision detection. By recursively traversing and pruning, leaf nodes that may collide are quickly filtered out, reducing the computational load of subsequent fine-grained collision detection.

[0072] For the set of triangular faces in the potential collision region, pairwise collision detection is performed on the triangular faces corresponding to the student's virtual model. A pre-defined bounding volume detection algorithm can be used to quickly determine whether two triangular faces intersect. For intersecting triangular face pairs, their indices are recorded as input for subsequent continuous collision detection. Through collision detection at the triangular facet level, the range of potential collision regions is further narrowed, providing more accurate collision candidates for continuous collision detection.

[0073] Continuous collision detection is performed on intersecting pairs of triangular facets, calculating their earliest collision time and corresponding collision information in continuous motion space. Continuous collision detection obtains the collision time and collision point coordinates by solving the intersection equations between the triangular facets. Based on the order of collision times, the pair of triangular facets that collided earliest is determined. For the pair of triangular facets that collided earliest, the normal vector at the collision point is calculated, representing the contact direction at the time of the collision. Based on the collision point coordinates and the normal vector, the collision penetration depth is calculated, representing the degree of intersection between the two objects. Neighboring collision points are clustered and merged into a single collision region. Collision data such as the number of collision points and the average penetration depth within the collision region are statistically analyzed to generate a data structure for the collision region. Based on the collision data, the contact state of the collision is determined.

[0074] In this embodiment, a high-quality virtual teaching aid triangular mesh model is reconstructed through point cloud data sampling and triangulation algorithms to ensure model accuracy and detail. An octree structure is used to adaptively partition the space, improving collision detection efficiency and reducing unnecessary computation. An octree hierarchical distance field is constructed to provide minimum distance estimates from each node to the surface of the student's virtual model, improving collision detection accuracy. Recursive traversal and pruning mechanisms quickly filter out potential collision regions, reducing the computational load of fine-grained collision detection. Fine-grained collision detection at the triangular mesh level is performed on potential collision regions to further narrow the collision detection range and ensure accuracy. The earliest collision time and collision information are calculated, and a collision region data structure is generated through clustering and fusion to accurately determine the contact state, achieving a realistic and continuous collision effect.

[0075] In one optional embodiment, based on the material properties and contact state of the virtual teaching aid, the motion response of the virtual teaching aid is calculated by solving the dynamic equations, and a force feedback signal is generated, including: Based on the material properties and collision data of the virtual teaching aids, combined with the geometry of the virtual teaching aids and the stress distribution of the contact area, different contact types are determined. Based on the contact types, a physical model is determined and the contact force is calculated. The motion equations of the virtual teaching aid are constructed based on the Lagrange dynamics equations. By combining one or more constraints such as non-penetrating constraints, hinge constraints, and sliding constraints, the Lagrange multipliers are determined, and a system of differential algebraic equations is generated. Based on the constraints, a numerical solution algorithm is determined, and the motion response of the virtual teaching aid is obtained by solving the equations. Based on the motion response, the acceleration vectors of the corresponding surface mesh vertices in the virtual teaching aid are extracted. The product of the acceleration vector and the vertex mass is taken as the vertex force. Through the connection relationship between mesh vertices, a sparse mapping matrix from vertex force to contact force is constructed. The inverse of the sparse mapping matrix is ​​solved using a preset singular value decomposition algorithm to determine the distribution of contact force to vertex force and convert the contact force into a force feedback signal.

[0076] Based on the material properties of the virtual teaching aid, such as stiffness, damping, and coefficient of friction, and collision data, such as collision point location, normal vector, and penetration depth, combined with the geometry of the virtual teaching aid and the stress distribution in the contact area, different contact types are determined. For example, contact types include point contact, line contact, and surface contact. For point contact, Hertz contact theory can be used to calculate the contact force, considering the local curvature and material properties of the object; for line and surface contact, contact force models based on pressure distribution, such as uniform pressure distribution, can be used. The corresponding physical model is selected according to the contact type to calculate the contact force. The calculation of the contact force needs to consider both normal and tangential forces. The normal force is related to the penetration depth and material stiffness, while the tangential force is related to the relative sliding velocity and coefficient of friction. Through numerical integration and iterative solving, the numerical solution of the contact force at each time step is obtained.

[0077] Based on the Lagrange dynamics equations, the motion equations of a virtual teaching aid are constructed. The Lagrange dynamics equations consider the system's kinetic energy, potential energy, and external forces. Through the variational principle of Lagrange functions and generalized coordinates, a system of differential equations describing the system's motion is obtained. Appropriate generalized coordinates, such as the position and orientation of a rigid body or the modal coordinates of a flexible body, are selected according to the virtual teaching aid's geometry and mass distribution. Based on the interaction between the virtual teaching aid and its environment, corresponding constraints are introduced, such as non-penetrating constraints, hinge constraints, and sliding constraints. Each constraint corresponds to a Lagrange multiplier, representing the constraint force generated to satisfy the constraint. These constraints are introduced into the Lagrange dynamics equations to generate a system of differential-algebraic equations. Based on the type and number of constraints, appropriate numerical solution algorithms, such as the Lagrange multiplier method, penalty function method, and sequential quadratic programming method, are selected to solve the system of differential-algebraic equations, obtaining the virtual teaching aid's motion response, including position, velocity, and acceleration, at each time step.

[0078] Based on the motion response of the virtual teaching aid, the acceleration vectors of the corresponding vertices of the surface mesh in the virtual teaching aid are extracted. The acceleration vector represents the effect of the net force on the vertex at the current time step. Multiplying the acceleration vector of the vertex by its corresponding mass yields the net force on the vertex, i.e., the vertex force. According to the topology of the virtual teaching aid's surface mesh, a connection matrix between vertices is constructed to represent the force transmission and influence between vertices. Using the connection matrix, a sparse mapping matrix from vertex forces to contact forces is constructed. This matrix describes the distribution and transmission of contact forces on the virtual teaching aid and is a large sparse matrix. Using a pre-defined singular value decomposition algorithm, the inverse matrix of the sparse mapping matrix is ​​solved to obtain the distribution relationship between contact forces and vertex forces. Substituting the calculated contact forces into the inverse matrix yields the numerical value of the force on each vertex. Based on the characteristics of human tactile perception, the vertex forces are filtered, amplified, and mapped to obtain a force feedback signal that matches human perception. The force feedback signal can be applied to the student's hands or other parts through force feedback devices, such as force feedback gloves or force feedback joysticks, to produce a tactile feedback effect.

[0079] In this embodiment, different contact types are determined based on collision data and material properties to provide a more accurate contact description; different physical models are used to calculate contact forces to adapt to different contact types and improve calculation accuracy; normal and tangential forces are calculated, taking into account penetration depth, material stiffness, slip velocity, and friction coefficient to ensure the comprehensiveness and accuracy of the forces; numerical integration and iterative solving are used to obtain the numerical solution of the contact forces at each time step to ensure the accuracy and temporal continuity of the results; the force feedback signal is obtained through the singular value decomposition algorithm to ensure that the accuracy of the force feedback matches human perception; the force feedback signal is transmitted to the student through the device to provide a realistic and natural tactile experience and enhance the realism of virtual interaction.

[0080] Figure 2 This is a schematic diagram of the structure of a VR-based interactive teaching system for primary and secondary schools according to an embodiment of the present invention. Figure 2 As shown, the system includes: The first unit is used to generate a model based on pre-acquired teaching aid model data through deep learning, learn the inherent distribution characteristics, determine multiple sub-components based on the inherent distribution characteristics, combine the teaching aid generation engine, establish the corresponding three-dimensional shape through mesh components, create a transformation node for each sub-component, set the parent-child relationship between transformation nodes, generate a tree-like hierarchical structure, and construct virtual teaching aids. The second unit is used to acquire color and depth images of the trainee through a structured light depth camera. Based on the color image, it detects key points of body parts through a convolutional neural network to obtain two-dimensional pixel coordinates. Based on the depth information in the depth image, it maps the key points of body parts into space to obtain three-dimensional spatial coordinates and generates a three-dimensional skeletal curve of the trainee's limb movement to obtain optical motion capture results. Based on the three-dimensional skeletal curve, it acquires the acceleration and angular velocity of key points of body parts through an inertial measurement unit. After inertial drift calibration, it obtains inertial motion capture results. It fuses the optical motion capture results and the inertial motion capture results to reconstruct the three-dimensional trajectory of the trainee's limb movement. The third unit is used to map the motion of the virtual model of the student based on the three-dimensional trajectory of the student's limb movement. Through a collision detection algorithm, the contact state between the virtual model and the virtual teaching aid is calculated in real time. Based on the material characteristics of the virtual teaching aid and the contact state, the motion response of the virtual teaching aid is calculated by solving the dynamic equation and generating a force feedback signal. The virtual models of multiple students are rendered into the same virtual teaching scene, and the three-dimensional trajectory of the limb movement and the force feedback signal of different students are synchronized in real time to generate a multi-person collaborative interaction process of the virtual teaching aid.

[0081] A third aspect of the present invention, An electronic device is provided, comprising: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0082] Fourth aspect of the embodiments of the present invention, A computer-readable storage medium is provided, having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0083] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0084] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-person interactive teaching method for primary and secondary schools based on VR technology, characterized by: include: Based on the pre-acquired teaching aid model data, a model is generated through deep learning to learn the intrinsic distribution characteristics. Based on the intrinsic distribution characteristics, multiple sub-components are determined. In combination with the teaching aid generation engine, the corresponding three-dimensional shapes are established through mesh components. A transformation node is created for each sub-component, and the parent-child relationship between the transformation nodes is set to generate a tree-like hierarchical structure to construct a virtual teaching aid. Using a structured light depth camera, a color image and a depth image of the trainee are obtained. Based on the color image, a convolutional neural network is used to detect key points of body parts to obtain two-dimensional pixel coordinates. Based on the depth information in the depth image, the key points of the body parts are mapped into space to obtain three-dimensional spatial coordinates, and a three-dimensional skeleton curve of the trainee's limb movement is generated to obtain an optical motion capture result. Based on the three-dimensional skeleton curve, the acceleration and angular velocity of the key points of the body parts are obtained through an inertial measurement unit. After inertial drift calibration, an inertial motion capture result is obtained. The optical motion capture result and the inertial motion capture result are fused to reconstruct the three-dimensional trajectory of the trainee's limb movement. Based on the three-dimensional trajectory of the students' limb movements, the movement of the students' virtual models is mapped. Through the collision detection algorithm, the contact state between the virtual model and the virtual teaching aids is calculated in real time. According to the material properties and contact state of the virtual teaching aids, the motion response of the virtual teaching aids is calculated by solving the dynamic equations, and a force feedback signal is generated. The virtual models of multiple students are rendered into the same virtual teaching scene, and the three-dimensional limb movement trajectories and force feedback signals of different students are synchronized in real time to generate a multi-person collaborative interaction process of the virtual teaching aids.

2. The method according to claim 1, characterized in that Based on the pre-acquired teaching aid model data, the model is generated through deep learning, and the intrinsic distribution features learned include: Acquire teaching aid model data; construct a geometric feature network based on the PointNet++ structure, represent the teaching aid model data as a point cloud, and learn the geometric shape features corresponding to the point cloud; construct a visual feature network based on the ResNet structure, render the teaching aid model data as a multi-view image, and learn the appearance texture features corresponding to the multi-view image; represent the material attributes, size attributes, and use attributes in the teaching aid model data as corresponding attribute vectors, respectively, and learn the functional semantic features through the fully connected layer to determine the functional attribute network; The geometric feature network, the visual feature network and the functional attribute network are dynamically adjusted through the attention mechanism to form a cross-modal fusion network. The intrinsic distribution characteristics of the teaching aid model data are learned through the modal alignment loss function and the cross-modal distillation learning strategy.

3. The method according to claim 2, characterized in that Based on the inherent distribution characteristics, multiple subcomponents are determined, combined with the teaching aid generation engine, the corresponding three-dimensional shapes are established through the grid components, a transformation node is created for each subcomponent, the parent-child relationship between the transformation nodes is set, and a tree hierarchy is generated. The construction of the virtual teaching aid includes: The intrinsic distribution feature is input into the generator of the cross-modal conditional generative adversarial network, the generator determines the generator teaching aid model based on the intrinsic distribution feature, the generator teaching aid model is input into the discriminator of the cross-modal conditional generative adversarial network, the discriminator determines the authenticity of the generator teaching aid model and the consistency of the generator teaching aid model with the intrinsic distribution feature, and the optimal teaching aid model is learned through adversarial iteration of the generator and the discriminator; The optimal teaching aid model is input into a sub-component discovery network, and the sub-component discovery network includes a variational autoencoder module and an active learning module, wherein the variational autoencoder module encodes the optimal teaching aid model by minimizing reconstruction error and KL divergence to obtain a low-dimensional latent space representation, and based on the low-dimensional latent space representation, similar sub-components are clustered to form a cluster structure; the active learning module adopts a sampling strategy based on maximum entropy, and performs sub-component labeling according to the cluster structure in the low-dimensional latent space representation, and combines the regularization term constraint based on the manifold to determine that the sub-component labeling is uniformly distributed; by alternately performing unsupervised learning of the variational autoencoder module and supervised learning of the active learning module, the labeling cost is minimized and the virtual teaching aid sub-components are determined; Based on each virtual teaching aid sub-component, a corresponding three-dimensional shape is established through a grid component. Based on the three-dimensional shape, a transformation node is created for each virtual teaching aid sub-component, and a parent-child relationship between the transformation nodes is set to generate a tree-like hierarchical structure. In combination with the attributes of the transformation nodes, a constraint relationship between the virtual teaching aid sub-components is set. At the same time, the texture and color of the virtual teaching aid sub-components are set through the material component, and the virtual teaching aid sub-components are assembled into a virtual teaching aid.

4. The method according to claim 1, characterized in that: Through the structured light depth camera, the color image and depth image of the trainee are obtained. Based on the color image, the key points of the body parts are detected through the convolutional neural network to obtain the two-dimensional pixel coordinates. Based on the depth information in the depth image, the key points of the body parts are mapped to the space to obtain the three-dimensional space coordinates, and the three-dimensional bone curve of the trainee's limb movement is generated. The optical motion capture results include: The color image and depth image of the trainee are simultaneously acquired through a structured light depth camera, wherein the structured light depth camera includes an infrared projector, two infrared cameras and a color camera, wherein the infrared projector projects structured light according to a preset pattern to generate a deformed stripe image on the surface of the trainee, and the two infrared cameras respectively acquire the deformed stripe images from two perspectives, and based on the two groups of the deformed stripe images, the depth value of each pixel is calculated to synthesize the depth image, and the color camera acquires the color image of the trainee; the color image and the depth image are synchronized in time and space and aligned in perspective to obtain a synchronized aligned image set; A key point detection network is constructed based on a U-Net structure, wherein the key point detection network includes an encoder and a decoder, wherein the encoder is based on a ResNet-50 structure, extracts multi-scale image features from color images in a synchronously aligned image set, and the decoder fuses the multi-scale image features through a jump connection to generate a feature map, and maps the feature map to a heat map corresponding to a key point of a body part through a convolutional layer, wherein each key point of a body part corresponds to a heat map channel, and determines the probability that a body part is a key point of a body part based on a pixel value, and extracts the two-dimensional pixel coordinates of the key point of the body part through threshold filtering and non-maximum suppression; Based on the depth image in the synchronously aligned image set, the depth value corresponding to the two-dimensional pixel coordinate is determined, and the three-dimensional coordinates of the key points of the body parts in the corresponding coordinate system of the structured light depth camera are calculated by the coordinate transformation formula in combination with the intrinsic parameters of the structured light depth camera; Based on the three-dimensional coordinates, with the key points of the body parts corresponding to the head as the root node, the key points of each body part are linked in sequence according to the predetermined human motion chain topology to form a tree-like limb hierarchy. By connecting the key points of adjacent body parts and adjusting the translation parameters and rotation parameters, the three-dimensional bone curve of the student's limb movement is generated, the three-dimensional coordinates corresponding to the key points of the body parts of adjacent frames are combined into a three-dimensional coordinate sequence, and smoothing filtering is performed to obtain the optical motion capture result.

5. The method according to claim 4, characterized in that Based on the three-dimensional bone curve, the acceleration and angular velocity of the key points of the body parts are obtained through the inertial measurement unit, and the inertial motion capture result is obtained after inertial drift calibration. The optical motion capture result and the inertial motion capture result are integrated to reconstruct the three-dimensional trajectory of the trainee's limb movement, including: According to the extracted key points of the body parts, an inertial measurement unit is bound to determine the three-axis acceleration and angular velocity, and the preliminary posture of the body parts is determined through attitude solution and integral operation, wherein the inertial measurement unit includes an accelerometer, a gyroscope and a magnetometer, which respectively measure the three-axis acceleration and angular velocity corresponding to the key points of the body parts; To begin the inertial drift calibration process: Taking the preliminary posture as the posture at the previous moment, combining the three-axis acceleration and angular velocity, and using the inertial navigation equation, predict the current posture of each body part corresponding to the current moment, and obtain the current prior state estimation and state estimation covariance matrix; Taking the optical motion capture result as the observation value, calculating the error between the observation value and the current prior state estimation, constructing the observation equation, determining the error statistics, obtaining the observation noise covariance matrix, and calculating the optimal Kalman gain matrix based on the state estimation covariance matrix and the observation noise covariance matrix; Based on the optimal Kalman gain matrix, the current prior state estimate is corrected to obtain the posterior state estimate, and the state estimate covariance matrix is ​​updated at the same time; Using the posterior state estimation as the preliminary posture at the next moment, cyclically executing the inertial drift calibration process to obtain an inertial motion capture result; Based on the three-dimensional skeleton curves corresponding to the optical motion capture results and the body part postures corresponding to the inertial motion capture results, the temporal features are extracted and a multi-scale spatiotemporal feature pyramid is constructed. The fused features are encoded through an attention-gated recurrent unit network to obtain the fused features. Through adaptive weight adjustment and the mutual compensation between the optical motion capture results and the inertial motion capture results, the fused features are mapped into the three-dimensional trajectory of the trainee's limb movement through a multi-layer perceptron.

6. The method according to claim 1, characterized in that Based on the three-dimensional trajectory of the student's limb movement, the movement of the student's virtual model is mapped, and the contact state between the virtual model and the virtual teaching aid is calculated in real time through a collision detection algorithm, including: Based on the three-dimensional geometric data corresponding to the virtual teaching aid, a mathematical equation form is determined to generate a discrete point set in the three-dimensional space, surface feature points corresponding to the virtual teaching aid are extracted, and point cloud data of the virtual teaching aid is generated. Normal vectors are estimated for the point cloud data of the virtual teaching aid, and the surface triangular patch mesh topology structure of the virtual teaching aid is reconstructed with point positions and normal vectors as constraints, and a triangular patch mesh model of the virtual teaching aid is obtained after mesh post-processing; Calculate the bounding box of the virtual teaching aid triangle mesh model to obtain the corresponding AABB bounding box, and adaptively recursively divide the AABB bounding box based on the geometric features of the virtual teaching aid until the size of a single triangle in the AABB bounding box is smaller than a preset unit size threshold. Use the AABB bounding box as a leaf node to construct an octree space partition structure corresponding to the virtual teaching aid. For leaf nodes, the minimum distance from the corresponding AABB bounding box to the surface of the student's virtualized model is calculated as the distance field value of the leaf node; the octree is traversed from bottom to top, and for each non-leaf node, the minimum value of the distance field value corresponding to the lower-level child nodes of the non-leaf node is taken as the distance field value of the non-leaf node; the octree is traversed from top to bottom, and for each node, the distance field value corresponding to the parent node of the node is used to update the distance field value of the node, and the octree hierarchical distance field is constructed; During the movement, based on the octree hierarchical distance field, starting from the octree root node, recursively traverse to determine whether each node collides, prune the nodes whose corresponding distance field values ​​are greater than a preset collision threshold, and mark the leaf nodes whose values ​​are less than or equal to the preset collision threshold as potential collision areas; For the set of triangular facets in the potential collision area, the triangular facets corresponding to the student virtual model are subjected to pair-by-pair collision by applying a preset bounding volume detection algorithm to determine intersecting triangular facet pairs; Based on intersecting triangle face pairs, continuous collision detection is performed to calculate the earliest collision time and corresponding collision information of the intersecting triangle face pairs in the continuous motion space. Based on the collision position coordinates and normal vector in the collision information, the collision penetration depth is calculated, and adjacent collision points are clustered and fused to generate a collision area data structure. The collision data is counted to determine the contact state of the collision.

7. The method according to claim 6, characterized in that According to the material properties and contact state of the virtual teaching aid, the motion response of the virtual teaching aid is calculated by solving the dynamic equation, and the force feedback signal generated includes: According to the material properties and collision data of the virtual teaching aids, combined with the geometric shape of the virtual teaching aids and the stress distribution of the contact area, different contact types are determined, and based on the contact type, the physical model is determined to calculate the contact force; Constructing the motion equation of the virtual teaching aid based on the Lagrangian dynamic equation, combining one or more constraints of non-penetration constraint, hinge constraint and sliding constraint, determining the Lagrangian multiplier term, generating a differential algebraic equation group, determining the numerical solution algorithm based on the constraint condition, and solving to obtain the motion response of the virtual teaching aid; Based on the motion response, the acceleration vector of the surface mesh vertex at the corresponding position in the virtual teaching aid is extracted, and the product of the acceleration vector and the vertex mass is used as the vertex force. Through the connection relationship between the mesh vertices, a sparse mapping matrix from the vertex force to the contact force is constructed, and the inverse of the sparse mapping matrix is ​​solved using a preset singular value decomposition algorithm to determine the distribution of the contact force to the vertex force, and the contact force is converted into a force feedback signal.

8. A multi-person interactive teaching system for primary and secondary schools based on VR technology, used to implement the method described in any one of claims 1 to 7, characterized in that: include: The first unit is used to generate a model through deep learning based on pre-acquired teaching aid model data, learn intrinsic distribution characteristics, determine multiple sub-components based on the intrinsic distribution characteristics, combine with the teaching aid generation engine, establish corresponding three-dimensional shapes through mesh components, create a transformation node for each sub-component, set the parent-child relationship between the transformation nodes, generate a tree hierarchy, and construct a virtual teaching aid; The second unit is used to obtain a color image and a depth image of the trainee through a structured light depth camera, detect key points of body parts through a convolutional neural network based on the color image, obtain two-dimensional pixel coordinates, map the key points of the body parts into space based on the depth information in the depth image, obtain three-dimensional spatial coordinates, generate a three-dimensional skeleton curve of the trainee's limb movement, obtain an optical motion capture result, obtain the acceleration and angular velocity of the key points of the body parts through an inertial measurement unit based on the three-dimensional skeleton curve, obtain an inertial motion capture result after inertial drift calibration, fuse the optical motion capture result and the inertial motion capture result, and reconstruct the three-dimensional trajectory of the trainee's limb movement; The third unit is used to map the movement of the student's virtual model based on the three-dimensional trajectory of the student's limb movement, calculate the contact state between the virtual model and the virtual teaching aid in real time through a collision detection algorithm, calculate the motion response of the virtual teaching aid according to the material properties and contact state of the virtual teaching aid by solving the dynamic equation, and generate a force feedback signal; render the virtual models of multiple students into the same virtual teaching scene, synchronize the three-dimensional limb movement trajectories and force feedback signals of different students in real time, and generate a multi-person collaborative interaction process of the virtual teaching aid.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method described in any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images