Somatosensory interaction supported educational scenario simulation system and method

Abstract

This application provides a motion-sensing interaction-supported educational scenario simulation system and method. The system collects motion sensor data from learners; constructs a motion skeleton model of the learners; generates a predicted motion trajectory of a virtual character at the end-effector in the educational scenario simulation; spatially fits the predicted motion trajectory to obtain interaction conflict risk; when the interaction conflict risk is less than a preset risk threshold, a dynamic conflict compensation vector is generated based on the directional difference between the virtual character's motion direction and the surface normal of the virtual object; the motion trajectory of the virtual character is constrained and corrected according to the dynamic conflict compensation vector to obtain conflict-corrected virtual motion features; and then the virtual character is driven to perform interactive actions conforming to the scenario rules through these virtual motion features. Using the solution of this application, motion mapping conflicts in dynamic scenarios can be avoided.

Description

Technical Field

[0001] This application relates to the field of educational scenario simulation technology, and more specifically, to an educational scenario simulation system and method supported by motion-sensing interaction. Background Technology

[0002] Educational scenario simulation is a teaching application area that enables learners to experience, practice, and learn in virtual or semi-virtual educational scenarios through immersive technology and interactive instructional design. This application area breaks through the limitations of time and space and the single theoretical transmission in traditional teaching. By recreating real educational scenarios (such as classroom teaching simulation, vocational skills practice scenarios, subject experiment scenarios, etc.), learners can enter the learning situation as participants rather than bystanders, and deepen their understanding of knowledge in the process of completing tasks and solving problems.

[0003] Motion-sensing interaction-supported educational scenario simulation is an innovative approach that uses motion capture technology to acquire learners' body movement data in real time and combines it with the logic of virtual educational scenarios for immersive learning interaction. This method breaks through the limitations of traditional educational scenario simulations that rely on keyboard and mouse or touchscreen interaction. With the help of motion-sensing cameras, motion sensors, and other devices, it can capture dynamic information such as learners' gestures, postures, and body movements, and convert them into operation commands in the virtual scene. However, traditional motion-sensing interaction-supported educational scenario simulations often use fixed motion mapping algorithms and single-dimensional conflict judgment rules, which are difficult to dynamically adapt to changes in scene elements within the educational context. Complex dynamic factors such as the diversity of learners' movements and fluctuations in the accuracy of equipment data acquisition can lead to conflicts and misalignments in the mapping between learners' real movements and the behavior of virtual characters and the logic of scene interaction. This seriously affects the continuity of interaction and the sense of immersion in learning. For example, in an educational context simulating mechanical repair, if the learner's hand action data for tightening screws is not correlated with the state of the virtual character's arm skeleton, direct mapping may result in collisions where the virtual character's hand penetrates the screw model. Therefore, how to avoid action mapping conflicts in dynamic scenes of motion-sensing interaction in order to achieve accurate synchronization between learners' movements and virtual scene interaction has become a challenge for the industry. Summary of the Invention

[0004] This application provides an educational scenario simulation system and method that supports motion-sensing interaction, which can avoid motion mapping conflicts in dynamic scenes.

[0005] In a first aspect, this application provides a method for simulating educational scenarios supported by motion-sensing interaction, comprising the following steps: During the simulated interactive educational scenario, learners' motion sensor data are collected synchronously. A motion skeleton model of the learner is constructed based on the motion sensor data, wherein the motion skeleton model includes the angle parameters of each joint of the learner during the educational scenario simulation. By generating the motion prediction trajectory of the virtual character at the end execution point in the educational scenario simulation using the angle parameters of each joint of the learner, the motion prediction trajectory is spatially fitted with the boundary constraints of the virtual object in the educational scenario simulation to obtain the interaction conflict risk between joint movements and virtual objects during somatosensory interaction. When the risk of interaction conflict is less than a preset risk threshold, a dynamic conflict compensation vector is generated in the haptic interaction process based on the directional difference between the direction of the virtual character's action and the surface normal of the virtual object. The motion trajectory of the virtual character is constrained and corrected based on the dynamic conflict compensation vector to obtain the conflict-corrected virtual motion features. Then, the virtual character is driven to perform interactive actions that conform to the context rules through the virtual motion features.

[0006] In some embodiments, constructing a learner's motion skeleton model based on the motion sensor data specifically includes: The motion sensor data is mapped onto the learner's skeletal topology to establish the initial positions of each joint in three-dimensional space; The rotational parameters of each joint are determined by the initial position of each joint in three-dimensional space and the constraint of the length of the skeleton link. Based on the rotation parameters and the parent-child joint relationships of all joints, a motion skeleton model of the learner is constructed.

[0007] In some embodiments, generating the motion prediction trajectory of the virtual character at the end execution point in the educational scenario simulation using the angle parameters of each joint of the learner specifically includes: The angle parameters of each joint of the learner are passed along the skeleton topology chain to obtain the rotation matrix sequence of each joint in three-dimensional space. Based on the rotation matrix sequence and combined with forward kinematics, the position change sequence of each end execution point of the virtual character in three-dimensional space is calculated; The predicted trajectory of a virtual character at the end execution point in an educational scenario simulation is generated using the position change sequence.

[0008] In some embodiments, spatial fitting is performed between the motion prediction trajectory and the boundary constraints of the virtual object during educational scenario simulation to obtain the interaction conflict risk between joint movements and the virtual object during motion-sensing interaction, specifically including: Construct a boundary space model of the virtual objects based on the boundary constraints of the virtual objects during the educational scenario simulation; Determine the spatial overlap between the predicted motion trajectory and the boundary space model at different trajectory points; The risk of interaction conflict between joint movements and virtual objects during motion-sensing interaction is determined based on all spatial overlaps.

[0009] In some embodiments, generating a dynamic conflict compensation vector during the haptic interaction process based on the directional difference between the virtual character's action direction and the surface normal of the virtual object specifically includes: Obtain the surface normal of the virtual object; Determine the directional difference vector between the virtual character's movement direction and the surface normal; By integrating all directional difference vectors according to the skeleton topology chain, the dynamic conflict compensation vector in the haptic interaction process is obtained.

[0010] In some embodiments, constraining and correcting the motion trajectory of the virtual character based on the dynamic conflict compensation vector to obtain conflict-corrected virtual motion features specifically includes: The motion prediction trajectory is dynamically compensated using the dynamic conflict compensation vector to obtain the constraint correction offset of each joint. By reverse mapping all constraint correction offsets, conflict correction parameters for each joint in the motion trajectory of the virtual character are obtained. Generate conflict-corrected virtual motion features based on all conflict correction parameters.

[0011] In some embodiments, driving a virtual character to perform interactive actions conforming to contextual rules through the virtual action features specifically includes: The virtual character's corrected motion trajectory is generated using the aforementioned virtual motion features; Based on the modified motion trajectory, the virtual character is driven to perform interactive actions that conform to the context rules.

[0012] Secondly, this application provides a motion-sensing interactive-supported educational scenario simulation system, comprising: The data acquisition module is used to synchronously collect learners' motion sensor data during simulated interactive educational scenarios. The processing module is used to construct a motion skeleton model of the learner based on the motion sensor data, wherein the motion skeleton model includes the angle parameters of each joint of the learner during the educational scenario simulation. The processing module is also used to generate the motion prediction trajectory of the virtual character at the end execution point in the educational scenario simulation by using the angle parameters of each joint of the learner, and to spatially fit the motion prediction trajectory with the boundary constraints of the virtual object in the educational scenario simulation to obtain the interaction conflict risk between joint movements and virtual objects during somatosensory interaction. The processing module is also used to generate a dynamic conflict compensation vector in the haptic interaction process based on the directional difference between the virtual character's action direction and the surface normal of the virtual object when the interaction conflict risk is less than a preset risk threshold. The execution module is used to constrain and correct the motion trajectory of the virtual character based on the dynamic conflict compensation vector, obtain the conflict-corrected virtual motion features, and then drive the virtual character to perform interactive actions that conform to the context rules through the virtual motion features.

[0013] Thirdly, this application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described method for simulating educational scenarios supported by haptic interaction.

[0014] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method for simulating educational scenarios supported by haptic interaction.

[0015] The technical solutions provided by the embodiments disclosed in this application have the following beneficial effects: The motion-sensing interaction-supported educational scenario simulation system and method provided in this application involves synchronously collecting learner motion sensor data during the educational scenario simulation interaction process; constructing a learner motion skeleton model based on the motion sensor data, wherein the motion skeleton model includes angle parameters of each joint of the learner during the educational scenario simulation; generating a motion prediction trajectory of the virtual character at the end execution point in the educational scenario simulation using the angle parameters of each joint of the learner; spatially fitting the motion prediction trajectory with the boundary constraints of the virtual object during the educational scenario simulation to obtain the interaction conflict risk between joint movements and the virtual object during motion-sensing interaction; when the interaction conflict risk is less than a preset risk threshold, generating a dynamic conflict compensation vector during the motion-sensing interaction process based on the directional difference between the direction of the virtual character's movement and the surface normal of the virtual object; constraining and correcting the motion trajectory of the virtual character according to the dynamic conflict compensation vector to obtain conflict-corrected virtual action features, and then driving the virtual character to perform interactive actions that conform to the scenario rules through the virtual action features.

[0016] Therefore, this application first constructs a learner's motion skeleton model based on motion sensor data, extracts the angle parameters of each joint to generate the predicted motion trajectory of the virtual character's end effector, and spatially fits this trajectory with the boundary constraints of the virtual object to determine the risk of interaction conflict. This interaction conflict risk reflects the potential conflict between the learner's joint movements and the virtual object on the spatial trajectory, making the assessment of motion mapping conflict more consistent with dynamic scene characteristics. Especially in scenarios where the virtual object's position changes in real time and the learner's motion amplitude suddenly adjusts, it significantly improves the timeliness and targeting of conflict identification. Compared to the traditional method of directly mapping actions to the virtual character without prior prediction, this process strengthens the coupling relationship between motion data and virtual scene constraints, thereby... This approach more effectively captures potential conflicts between actions and virtual objects in dynamic scenes, providing a scientific basis for predicting and avoiding action mapping conflicts. Then, after determining that the risk of interaction conflict is less than a preset threshold, a dynamic conflict compensation vector is generated based on the directional difference between the virtual character's action direction and the surface normal of the virtual object. This vector is then used to constrain and correct the virtual character's action trajectory. The dynamic conflict compensation vector can provide real-time feedback on the adaptation deviation between actions and virtual objects, optimizing the depiction of the learner's real actions by the virtual character's actions without affecting the smoothness of the haptic interaction, thereby reducing action mapping conflicts in dynamic scenes. In summary, this solution can avoid action mapping conflicts in haptic interaction in dynamic scenes, achieving precise synchronization between the learner's actions and the virtual scene interaction. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating a motion-sensing interactive support method for simulating educational scenarios according to some embodiments of this application; Figure 2 This is a flowchart illustrating the process of determining a motion prediction trajectory according to some embodiments of this application; Figure 3 This is a flowchart illustrating the execution of interactive actions according to some embodiments of this application; Figure 4 This is a schematic diagram of the structure of an educational scenario simulation system supported by motion-sensing interaction, as shown in some embodiments of this application; Figure 5 This is an internal structural diagram of a computer device that implements a method for simulating educational scenarios with haptic interaction support, according to some embodiments of this application. Detailed Implementation

[0018] To better understand the technical solutions in this embodiment, the technical solutions in this embodiment will be described in detail below with reference to the accompanying drawings and specific implementation methods.

[0019] refer to Figure 1The figure is a flowchart illustrating a motion-sensing interaction-supported educational scenario simulation method according to some embodiments of this application. This motion-sensing interaction-supported educational scenario simulation method mainly includes the following steps: In step 101, during the simulated interactive process in the educational context, the learner's motion sensor data is collected synchronously.

[0020] In practical implementation, a multimodal motion sensing acquisition device array can be deployed, including optical motion capture sensors and inertial measurement units. Throughout the entire cycle of simulated interaction in the educational context, the learner's limb movement data is synchronously collected at a preset sampling frequency. The optical motion capture sensors capture the three-dimensional coordinates of marker points on the learner's body surface through an infrared camera. The inertial measurement units are worn on the learner's key joints (e.g., shoulder, elbow, wrist, hip, knee, ankle, etc.) to collect the joint motion angular velocity and posture angle. The sensor data can be transmitted to the edge computing node in real time through a network protocol. The data synchronization module within the node performs frame alignment processing on the multi-source sensor data based on timestamps, and finally outputs time-consistent motion sensor data.

[0021] It should be noted that the motion sensor data mentioned in this application refers to a multi-dimensional quantitative data set that can accurately characterize the learner's limb movement state, collected by professional sensing equipment. It is mainly used for subsequent construction of motion skeleton models and restoration of virtual character movements. The optical motion capture sensor refers to a sensing device that realizes limb movement tracking based on optical imaging principles. Common types include binocular infrared motion capture cameras and depth imaging cameras. The inertial measurement unit refers to a miniature sensing module that integrates accelerometers, gyroscopes, and magnetometers. It is usually designed as a wearable device (e.g., wristband or patch type) and worn on the learner's key joints. Its working principle is to measure the linear acceleration of joint movement through accelerometers and to measure the rotational angular velocity of the joint around each axis through gyroscopes.

[0022] In step 102, a motion skeleton model of the learner is constructed based on the motion sensor data, wherein the motion skeleton model includes the angle parameters of each joint of the learner during the educational scenario simulation.

[0023] In some embodiments, constructing a learner's motion skeleton model based on the motion sensor data can be achieved through the following steps: The motion sensor data is mapped onto the learner's skeletal topology to establish the initial positions of each joint in three-dimensional space; The rotational parameters of each joint are determined by the initial position of each joint in three-dimensional space and the constraint of the length of the skeleton link. Based on the rotation parameters and the parent-child joint relationships of all joints, a motion skeleton model of the learner is constructed.

[0024] In specific implementation, mapping the motion sensor data to the learner's skeletal topology and establishing the initial positions of each joint in three-dimensional space can be achieved in the following way: First, the three-dimensional coordinate information is mapped to the joint nodes in the predefined learner skeletal topology. This includes: matching the output of optical markers or inertial measurement units to the corresponding joint nodes based on the spatial position and functional correspondence of each joint in the skeletal topology, and, if necessary, calculating the positions of joint nodes for which data was not directly collected. The calculation method can use a linear interpolation algorithm based on parent-child joint chains, that is, using the known three-dimensional coordinates of the parent and child joints, and based on the link length... The initial position of the unmeasured joints is calculated based on constraints. Then, according to the link length constraints of the skeleton topology, the mapped 3D coordinates of the joints are corrected. Specifically, this includes calculating the difference between the actual spatial distance between each pair of connected joints and the preset link length, and adjusting the position of each joint according to the least squares error criterion so that the spatial distance between adjacent joints matches the preset skeleton link length as closely as possible. This correction process can be implemented through iterative optimization algorithms, such as gradient descent, by continuously updating the joint coordinates until all joint spacing errors converge to below a preset threshold. Finally, the corrected 3D coordinates of each joint are used as the initial position of each joint in the learner's motion skeleton model in 3D space.

[0025] It should be noted that the initial position mentioned in this application refers to the starting coordinate position of each joint node in three-dimensional space when establishing the learner's motion skeleton model, which is used as a reference benchmark for subsequent motion tracking, motion prediction and virtual character driving.

[0026] In specific implementation, determining the rotation parameters of each joint based on its initial position in three-dimensional space and the constraints of the skeleton link lengths can be achieved in the following way: First, based on the initial position of each joint in three-dimensional space, calculate the local coordinate system of each joint. Specifically, this includes performing a vector difference operation between the position vector of each joint and the three-dimensional coordinate vector of its parent joint to obtain a local direction vector, which serves as a preliminary reference for the joint rotation axis direction. Subsequently, according to the link length constraints in the skeleton topology, calculate the rotation matrix of the joint relative to the parent joint. Specifically, this includes using the known parent joint position, child joint position, and preset link length, through triangulation... The rotation angles of the joints are solved using a rotation matrix decomposition method; the rotation matrix can be represented in Euler angles or quaternion form; then, using a forward kinematics algorithm in the prior art, the rotation matrix of each joint is passed along the skeleton chain to calculate the spatial position of each sub-joint after rotation and compare it with the preset link length; when it is found that the rotation result causes the sub-joint position to be inconsistent with the link length constraint, an iterative optimization method (such as least squares error optimization or gradient descent algorithm) is used to adjust the joint rotation angle until the rotation matrix of all joints satisfies the link length constraint; finally, the obtained rotation matrix is ​​used as the rotation parameter of each joint in the learner motion skeleton model.

[0027] It should be noted that the rotation parameters mentioned in this application refer to the parameter values ​​used to describe the rotational state of each joint in the learner's motion skeleton model relative to its parent joint in three-dimensional space. They are used to characterize the direction and posture changes of the joints in space and to guide the spatial rotation of the joints in the virtual character skeleton, thereby ensuring the continuity of the skeleton structure and the accuracy of the joint movements.

[0028] In specific implementation, the learner's motion skeleton model, based on the rotation parameters and the parent-child joint relationships of all joints, can be constructed in the following way: First, using a predefined root joint as the starting node, a global reference coordinate system is established in three-dimensional space according to its rotation parameters, with the initial position of the root joint as the origin of the skeleton; then, according to the hierarchical order of the skeleton topology, coordinate transformation operations are performed on each child joint, specifically including: performing matrix multiplication between the global rotation matrix of the parent joint and the rotation matrix of the child joint to obtain the global rotation matrix of the child joint; next, using the global rotation matrix and the spatial position of the parent joint, combined with the link length constraint, the three-dimensional coordinates of the child joint are calculated, specifically: determining the direction vector of the link in the parent joint coordinate system through the rotation matrix, multiplying it by the link length to obtain the displacement vector of the child joint relative to the parent joint, and then superimposing this displacement vector with the position of the parent joint to obtain the global position of the child joint; In one step, during the overall skeleton generation process, to avoid skeleton distortion caused by accumulated errors, a global consistency correction method is used to correct the skeleton structure. This method may include: establishing an objective function with the lengths of all skeleton links as constraints, and minimizing the squared error between the actual position and the target position of each joint; this optimization process can be implemented using an iterative solution algorithm, such as a constraint optimization method based on gradient descent or Lagrange multipliers; finally, the corrected 3D coordinates, rotation matrices, and skeleton topology of each joint are integrated to form a motion skeleton model that can drive the virtual character. For example, the 3D position, rotation state, and parent-child relationship of each joint are encapsulated as a joint node object; subsequently, all nodes are organized according to the skeleton hierarchy to form a tree structure; finally, this skeleton structure is mapped to the virtual character model, enabling each joint node to drive the corresponding virtual joint, thereby completing the construction of the overall motion skeleton model.

[0029] It should be noted that this application calculates the rotation angle around each rotation axis using matrix decomposition methods (such as Euler angle decomposition or quaternion to Euler angle conversion) through the rotation matrix of each joint in the motion skeleton model; then, the rotation angles of each joint are stored in the order of skeleton topology to form a set of joint angle parameters in the learner motion skeleton model, that is: the motion skeleton model includes the angle parameters of each joint of the learner during the educational scenario simulation.

[0030] It should be noted that the motion skeleton model described in this application refers to a structured model of a driveable virtual character skeleton established based on the rotation parameters of each joint of the learner, the three-dimensional spatial position, and the skeleton topology. It is used to represent the relative position, rotation state, and parent-child connection relationship of each joint of the virtual character in three-dimensional space, thereby ensuring the coherence of the overall skeleton structure and the accuracy of the movements.

[0031] In step 103, the motion prediction trajectory of the virtual character at the end execution point in the educational scenario simulation is generated by the angle parameters of each joint of the learner. The motion prediction trajectory is then spatially fitted with the boundary constraints of the virtual object in the educational scenario simulation to obtain the interaction conflict risk between joint movements and the virtual object during somatosensory interaction.

[0032] In some embodiments, reference Figure 2 As shown in the figure, this is a flowchart illustrating the process of determining a motion prediction trajectory in some embodiments of this application. The generation of the motion prediction trajectory of a virtual character at the end-effector point in an educational scenario simulation, based on the angle parameters of each joint of the learner, can be achieved through the following steps: First, in step 1031, the angle parameters of each joint of the learner are passed along the skeleton topology chain to obtain the rotation matrix sequence of each joint in three-dimensional space. Then, in step 1032, the position change sequence of each end execution point of the virtual character in three-dimensional space is calculated based on the rotation matrix sequence and positive kinematics. Finally, in step 1033, the motion prediction trajectory of the virtual character at the end execution point in the educational scenario simulation is generated through the position change sequence.

[0033] In practice, the angle parameters of each joint of the learner are passed along the skeleton topology chain to obtain the rotation matrix sequence of each joint in three-dimensional space. This can be achieved as follows: First, a global reference coordinate system is established in three-dimensional space, using a predefined root joint as the starting node. Then, according to the hierarchical order of the skeleton topology chain, the angle parameters of each joint are processed level by level from the root joint to the end joint, converting the angle parameters of each joint into the corresponding rotation matrix. The rotation matrix can be obtained using Euler angle transformation or quaternion transformation. Next, the global rotation matrix of the parent joint is multiplied by the local rotation matrix of the child joint to calculate the global rotation matrix of the child joint. This process is passed level by level along the skeleton chain until the global rotation matrix calculation of all joints is completed. Furthermore, during the transmission process, the accumulated rotation error can be corrected using constraint optimization methods to ensure the consistency of the overall skeleton rotation. Finally, the processed global rotation matrices of each joint are recorded according to the skeleton topology order to form the rotation matrix sequence of each joint in three-dimensional space in the learner's motion skeleton model.

[0034] It should be noted that the rotation matrix sequence mentioned in this application refers to a set of matrices that represent the global rotation state of each joint in three-dimensional space according to the topological order of the learner's motion skeleton model. It is used to describe the rotation information of each joint in the skeleton relative to the global reference coordinate system. The rotation matrix sequence can reflect the spatial posture of the motion skeleton at each trajectory point or time step and is used to drive the joint rotation and motion reproduction of the virtual character.

[0035] In specific implementation, the position change sequence of each end execution point of the virtual character in three-dimensional space can be calculated based on the rotation matrix sequence combined with forward kinematics, as follows: First, for each end execution point in the virtual character skeleton, obtain its parent-child joint order along the skeleton chain and the initial position vector of each joint relative to the parent joint; then, starting from the root joint, pass the rotation matrix sequence level by level along the skeleton chain, specifically including: for the rotation matrix sequence of each time step, select the rotation matrix of the corresponding joint in sequence according to the skeleton topology order, starting from the root joint; apply the rotation matrix of the selected joint to the relative position vector of its child joint, transforming the position of the child joint in the local coordinate system to the global coordinate system of the parent joint; continue to superimpose the position of the parent joint on the global position of the child joint to obtain the three-dimensional position of the child joint in the global coordinate system; repeat this process until the end execution point, ensuring that the global position of each end execution point simultaneously considers the rotation state of all parent joints along the skeleton chain; for each rotation matrix in the time sequence, perform the above steps to obtain the global position of the end execution point of the corresponding time step, forming the position change sequence of each end execution point of the virtual character in three-dimensional space.

[0036] It should be noted that the position change sequence mentioned in this application refers to a set of continuous position vectors of each end execution point in three-dimensional space that change with time or trajectory points, calculated according to the topology of the virtual character skeleton and the rotation matrix sequence. This set is used to characterize the spatial motion trajectory and displacement changes of each end execution point of the virtual character during the action process, thereby driving the virtual character to perform precise spatial actions and interactions.

[0037] In specific implementation, the motion prediction trajectory of the virtual character at the end execution point in the educational scenario simulation can be generated by the position change sequence in the following way: First, the three-dimensional positions of each end execution point in the time series are arranged sequentially to form a continuous spatial coordinate point sequence; then, the motion trajectory is generated according to the coordinate point sequence, and interpolation methods (such as linear interpolation or spline curve interpolation) can be used to smoothly connect adjacent position points to obtain the continuous motion path of the end execution point in three-dimensional space; finally, the smoothed spatial path is used as the motion prediction trajectory of the virtual character at the end execution point in the educational scenario simulation.

[0038] It should be noted that the motion prediction trajectory described in this application refers to a continuous spatial path generated by time series arrangement and interpolation smoothing based on the position change sequence of each end execution point of the virtual character in three-dimensional space. It is used to characterize the future motion trend and possible trajectory of each end execution point of the virtual character in the educational scenario simulation.

[0039] In some embodiments, spatially fitting the motion prediction trajectory with the boundary constraints of the virtual object during educational scenario simulation to obtain the risk of interaction conflict between joint movements and virtual objects during motion-sensing interaction can be achieved through the following steps: Construct a boundary space model of the virtual objects based on the boundary constraints of the virtual objects during the educational scenario simulation; Determine the spatial overlap between the predicted motion trajectory and the boundary space model at different trajectory points; The risk of interaction conflict between joint movements and virtual objects during motion-sensing interaction is determined based on all spatial overlaps.

[0040] It should be noted that the boundary constraints of the virtual objects described in this application refer to the allowed range of motion of the virtual objects in three-dimensional space and the spatial restrictions on other objects or scene elements in the educational scenario simulation, including but not limited to the surface boundaries, collision volumes, motion radii, pose constraints, and impenetrable areas in the scene; the boundary constraints can be obtained through the geometric model of the virtual object (such as vertex coordinates, mesh, bounding box, or voxel representation), and spatially calibrated in combination with the object's position, orientation, and scaling in the virtual scene; the acquisition method may include: exporting boundary data from virtual object modeling software, obtaining collision volume parameters from the physics engine, or manually or automatically on-site. The scene sets non-intersecting areas and movement restrictions; in other embodiments, boundary constraints can also be dynamically generated based on the object's movement history trajectory or scene rules, which is not limited in this application; therefore, as a preferred embodiment, the boundary space model of the virtual object can be constructed based on the boundary constraints of the virtual object during the educational scenario simulation in the following way: First, based on the geometric model and boundary constraints of the virtual object, a boundary representation of the object in three-dimensional space is generated, including but not limited to bounding boxes, collision volumes or voxelized meshes; then, the position, orientation and scaling of the object in the virtual scene are applied to the boundary representation to obtain the boundary space model in the global coordinate system.

[0041] It should be noted that the boundary space model described in this application refers to a spatial representation model constructed in the global coordinate system of a virtual scene based on the boundary constraints and geometric model of the virtual object. It is used to characterize the allowed range of motion of the virtual object in three-dimensional space and the spatial interaction restrictions with other objects or scene elements, including but not limited to bounding boxes, collision volumes or voxelized meshes, in order to guide the virtual character to interact reasonably with the virtual object in the educational scenario simulation, and to ensure that the action is limited by the boundary of the object and the scene rules.

[0042] In specific implementation, determining the spatial overlap between the predicted motion trajectory and the boundary space model at different trajectory points can be achieved in the following way: First, the predicted motion trajectory of the virtual character's end execution point is discretized into multiple trajectory sampling points, and the three-dimensional position of each trajectory point in the global coordinate system is obtained; then, for each trajectory point, the boundary space model of the corresponding virtual object in the same global coordinate system is obtained, including bounding boxes, collision volumes, or voxelized mesh representations; next, the spatial overlap between the trajectory point and the boundary space model is calculated, specifically including: if a mesh representation is used, the trajectory point can be projected onto a mesh cell and it can be determined whether the point falls inside the mesh or near the boundary; if a voxelized representation is used, the overlap is determined by detecting whether the voxel cell corresponding to the trajectory point is occupied; if a bounding box representation is used, the intersection is determined by comparing the range relationship between the trajectory point coordinates and the bounding box in each axis; then, the determination result or distance metric value of each trajectory point is recorded as the spatial overlap of that point, thereby obtaining the spatial overlap between the predicted motion trajectory and the boundary space model at different trajectory points.

[0043] It should be noted that the spatial overlap mentioned in this application refers to a quantitative parameter used to characterize the degree of spatial proximity or interaction between the virtual character's end execution point and the virtual object boundary space model in the motion prediction trajectory. It is used to reflect whether the trajectory point intersects, overlaps, or approaches the object boundary, as well as the intensity and degree of overlap or proximity.

[0044] In specific implementation, determining the interaction conflict risk between joint movements and virtual objects during motion-sensing interaction based on all spatial overlap values ​​can be achieved in the following way: First, collect the spatial overlap values ​​corresponding to each trajectory point in the predicted trajectory of the virtual character's motion in chronological order to form a spatial overlap value sequence; then, normalize the spatial overlap value sequence, specifically including: subtracting the minimum value from each spatial overlap value in the sequence and dividing by the difference between the maximum and minimum values ​​to obtain a normalized overlap value sequence, with the range limited to 0 to 1; further, use the normalized spatial overlap value sequence as the interaction conflict risk of the corresponding trajectory point, thereby obtaining the interaction conflict risk between joint movements and virtual objects during motion-sensing interaction.

[0045] It should be noted that the interaction conflict risk mentioned in this application refers to the parameter value used to characterize the degree of risk that the joint movements of the virtual character may collide, overlap or get too close to the boundary of the virtual object during the motion interaction process, in order to quantify the potential conflict risk between each trajectory point in the motion trajectory and the boundary of the object.

[0046] In step 104, when the risk of interaction conflict is less than a preset risk threshold, a dynamic conflict compensation vector is generated in the haptic interaction process based on the directional difference between the direction of the virtual character's action and the surface normal of the virtual object.

[0047] It should be noted that the preset risk threshold mentioned in this application refers to a critical value used to determine the risk of interaction conflict between the joint movements of a virtual character and a virtual object during motion-sensing interaction. This threshold is used to distinguish between normal movements and movements that may collide or do not conform to spatial constraints. The risk threshold can be set based on historical movement data statistics, the geometric dimensions of the virtual object, the range of joint movement, and user safety or interaction experience requirements to limit the acceptable range of spatial overlap or directional deviation. Furthermore, the risk threshold can be a fixed constant, such as a specific numerical range (e.g., a certain proportion between 0 and 1) determined experimentally in a normalized spatial overlap sequence, or it can vary depending on different scenarios, different virtual objects, or different... The joint dynamic adjustment is adapted to an adaptive threshold. In a preferred embodiment, the risk threshold setting can take into account both the naturalness of the action and the safety of the interaction. By adjusting the threshold, the triggering frequency and amplitude of dynamic conflict compensation can be controlled, thereby ensuring the smoothness of the action and the spatial constraints of the virtual object during the haptic interaction. This application does not limit this. In addition, in some preferred embodiments, when the interaction conflict risk is greater than or equal to the preset risk threshold, the collision area between the virtual character's end execution point and the virtual object (such as the collision point coordinates and collision area) can be marked by a collision detection algorithm, and the trajectory calculation of the area can be frozen to prevent the virtual character's action from penetrating the geometric boundary of the virtual object (such as the wall of the virtual container or the operation restriction zone of the experimental instrument). This is not limited here.

[0048] In some embodiments, generating a dynamic conflict compensation vector during the haptic interaction process based on the directional difference between the virtual character's movement direction and the surface normal of the virtual object can be achieved through the following steps: Obtain the surface normal of the virtual object; Determine the directional difference vector between the virtual character's movement direction and the surface normal; By integrating all directional difference vectors according to the skeleton topology chain, the dynamic conflict compensation vector in the haptic interaction process is obtained.

[0049] It should be noted that the surface normal of the virtual object mentioned in this application refers to the normal vector information of each surface point of the virtual object in three-dimensional space, which is used to describe the directional characteristics of the object surface at that point. The normal can be used to calculate the directional difference between the virtual character's action and the object surface. The surface normal can be obtained through the geometric model of the virtual object, including but not limited to the face normal or vertex normal of the triangular mesh model, the surface normal of the voxelized mesh, and the analytical normal of the surface parametric model. The acquisition method may include: for the triangular mesh model, calculating the face normal by traversing the vertex coordinates of each face and using the cross product; for the vertex normal, it can be generated by the weighted average method of the normals of adjacent face patches; for the voxelized model, the normal can be calculated based on the neighborhood occupancy of each surface voxel; in the surface parametric model, the normal can be obtained by taking the cross product of the tangent vector of the surface; in other embodiments, the normal information can also be obtained directly through the collision detection interface of the physics engine or the normal cache in the rendering pipeline, which is not limited in this application.

[0050] In specific implementation, the direction difference vector between the virtual character's action direction and the surface normal can be determined in the following way: First, obtain the action direction vector of the virtual character's action point in three-dimensional space. This vector can be determined by the difference vector between the current position of the joint and the position of the previous time step, or extracted from the direction component of the corresponding rotation axis in the joint rotation matrix. Then, obtain the surface normal vector of the virtual object at the corresponding position of the action point. Next, calculate the direction difference vector through vector operations, specifically including: normalizing the action direction vector and the surface normal vector, calculating the vector difference between the two, or determining the vertical offset direction through cross product, thereby obtaining the direction difference vector between the virtual character's action direction and the surface normal. Preferably, the above steps can be repeated to perform the same calculation on all time steps and corresponding surface points in the virtual character's action trajectory to obtain multiple direction difference vector sequences.

[0051] It should be noted that the directional difference vector mentioned in this application refers to a vector value used to characterize the degree of spatial deviation between the direction of motion of each action point of the virtual character and the normal of the virtual object surface during the motion interaction process, in order to reflect the direction and magnitude of the deviation of the action direction relative to the object surface.

[0052] It should be noted that the topological structure of the virtual character skeleton described in this application refers to a hierarchical structural model used to describe the joints of the virtual character and their connection relationships, including joint nodes, link relationships, and parent-child association information; this topological structure can reflect the hierarchical order of the skeletal chain, the degrees of freedom of movement of each joint, and the spatial dependencies between joints; therefore, in specific implementation, integrating all directional difference vectors according to the skeleton topological chain to obtain the dynamic conflict compensation vector in the motion-sensing interaction process can be achieved in the following way: First, based on the topological structure of the virtual character skeleton, determine the parent-child relationship and hierarchical order of each joint in the skeleton chain, wherein the skeleton topological structure defines the connection relationship and relative spatial position of each joint in the overall skeleton, so that the transmission path of joint movements can be clearly defined and Dependency relationships are established; subsequently, each joint is processed level by level along the skeleton chain starting from the root joint. The directional difference vector corresponding to each joint is regarded as a local conflict compensation amount. Combined with the integrated directional difference information of the parent joint, the local compensation is spatially mapped and accumulated through hierarchical transmission. Preferably, during the integration process, the local directional difference vector can be weighted or projected according to the hierarchical depth and range of the joint in the skeleton chain, so that the directional compensation of the higher-level joints can reasonably affect the lower-level joints. Finally, the directional difference vectors of each joint obtained by integrating level by level along the skeleton chain are vectorized as a whole. For example, the directional difference vectors of all joints can be arranged according to the skeleton hierarchy to form a vector set, and then the vector generated by the weighted average method can be used as the dynamic conflict compensation vector in the somatosensory interaction process.

[0053] It should be noted that the dynamic conflict compensation vector mentioned in this application refers to a vector value used to characterize the overall deviation of the direction of movement of each joint of a virtual character from the skeleton topology chain and the surface normal of the virtual object during the motion interaction process. It is used to reflect the direction and magnitude of the movement correction that each joint needs to make in space. This vector is generated by weighting or projecting the direction difference vectors of each joint integrated step by step along the skeleton chain, and is used to guide the conflict compensation and correction of the virtual character's movements.

[0054] In step 105, the motion trajectory of the virtual character is constrained and corrected according to the dynamic conflict compensation vector to obtain the conflict-corrected virtual motion features, and then the virtual character is driven to perform interactive actions that conform to the context rules through the virtual motion features.

[0055] In some embodiments, constraining and correcting the motion trajectory of the virtual character based on the dynamic conflict compensation vector to obtain the conflict-corrected virtual motion features can be achieved through the following steps: The motion prediction trajectory is dynamically compensated using the dynamic conflict compensation vector to obtain the constraint correction offset of each joint. By reverse mapping all constraint correction offsets, conflict correction parameters for each joint in the motion trajectory of the virtual character are obtained. Generate conflict-corrected virtual motion features based on all conflict correction parameters.

[0056] In specific implementation, the constraint correction offset of each joint is obtained by dynamically compensating the motion prediction trajectory using the dynamic conflict compensation vector, which can be achieved in the following way: First, based on the topology of the virtual character skeleton, obtain the set of movable directions of each joint in the local coordinate system. This set can be defined by the rotational degrees of freedom or translational constraints of the joint. Then, map the dynamic conflict compensation vector to this local coordinate system, and extract the components of the compensation vector in the movable directions of the joint through vector projection, as candidate correction offsets for the joint. Further, to ensure the continuity between the correction result and the motion prediction trajectory, the candidate correction offsets are adjusted by magnitude. The constraint, for example, limits the joint to a preset threshold that does not exceed its historical range of motion. In a preferred embodiment, a weighting factor can also be used to adjust the compensation component. The weighting factor can be determined by the joint's hierarchical depth in the skeleton chain or its distance from the end effector, so that joints closer to the end effector have a higher weight in conflict correction. Finally, the compensation component after amplitude constraint and weight adjustment is used as the constraint correction offset of the joint. It should be noted that the set of movable directions of the joint in the local coordinate system mentioned in this application refers to the set of directions in which each joint is allowed to rotate or translate in its own local coordinate system, used to limit the degree of freedom of joint movement. The set of movable directions can be determined according to the joint type and skeleton topology. For example, for ball joints, the set of movable directions includes rotational directions about the three local coordinate axes X, Y, and Z, or for hinge joints, the set of movable directions is a single rotational direction of a fixed axis.

[0057] It should be noted that the constraint correction offset mentioned in this application refers to the vector value used to characterize the spatial position or rotation adjustment required for each joint of a virtual character during the motion interaction process in order to eliminate the conflict between the motion prediction trajectory and the environment or virtual object. It is used to reflect the correction magnitude and direction of the joint along its movable direction. This offset is generated by mapping the dynamic conflict compensation vector to the joint local coordinate system, projecting it in the movable direction, and adjusting the amplitude and weight, thereby guiding the conflict correction of each joint on the motion trajectory.

[0058] In specific implementation, the conflict correction parameters for each joint in the virtual character's motion trajectory are obtained by back-mapping all constraint correction offsets. This can be achieved as follows: First, based on the topology of the virtual character's skeleton, determine the parent-child relationship of each joint and the transformation relationship between the local and global coordinate systems. Then, for each joint, map its constraint correction offset in the local coordinate system back to the global coordinate system through coordinate transformation. This specifically includes multiplying the local offset vector by the joint rotation matrix or using a homogeneous transformation matrix for coordinate mapping to obtain the global offset vector. Further, apply the global offsets of each joint to the original motion prediction trajectory. This specifically includes: For joints with translational degrees of freedom, the global offset vector is directly added to the original joint position vector to obtain the corrected position vector. For joints with rotational degrees of freedom, the global offset vector is projected through the joint rotation axis to obtain the rotation angle correction, which is then superimposed on the original joint rotation matrix or quaternion to obtain the corrected joint rotation state. For joints with both translational and rotational degrees of freedom, the position and rotation corrections are combined and superimposed using a homogeneous transformation matrix to simultaneously correct both position and attitude. Finally, the correction result of each joint in the motion trajectory after applying the global offset is used as the conflict correction parameter for that joint.

[0059] It should be noted that the conflict correction parameters mentioned in this application refer to the parameter values ​​used to characterize the global position and posture correction of each joint of the virtual character in the motion trajectory in order to eliminate the conflict between the motion prediction trajectory and the environment or virtual object, so as to reflect the translation and rotation adjustment range and direction of the joint in three-dimensional space.

[0060] In specific implementation, generating conflict-corrected virtual motion features based on all conflict correction parameters can be achieved in the following way: First, according to the topology of the virtual character skeleton, the conflict correction parameters corresponding to each joint in the motion trajectory are combined with the original motion prediction trajectory point by point to obtain the corrected joint position and rotation state of each trajectory point; then, for each trajectory point, the corrected position vector of each joint is directly superimposed with the original position, and the rotation correction amount is applied to the original rotation state through rotation matrix or quaternion superposition; next, according to the parent-child relationship of each joint in the skeleton topology, the corrected position and rotation state are passed down from the root joint in sequence to ensure that the child joint posture correctly reflects the correction effect of the parent joint; then, the global position vector and rotation state of each joint are encapsulated as joint node objects, and the skeleton posture data obtained after forming a tree structure according to the skeleton hierarchy is used as the conflict-corrected virtual motion features of the corresponding trajectory points.

[0061] It should be noted that the conflict-corrected virtual motion features mentioned in this application refer to the skeleton posture information formed by correcting the position and translation of the original motion prediction trajectory at each trajectory point based on the conflict correction parameters of each joint of the virtual character. This information is used to characterize the corrected position vector and rotation state of each joint of the virtual character in three-dimensional space. This feature is generated by passing the corrected position and rotation state along the parent-child relationship according to the skeleton topology, encapsulating them into joint nodes, and organizing them into a tree structure.

[0062] In some embodiments, driving a virtual character to perform interactive actions that conform to contextual rules through the virtual action features can be achieved by the following steps: The virtual character's corrected motion trajectory is generated using the aforementioned virtual motion features; Based on the modified motion trajectory, the virtual character is driven to perform interactive actions that conform to the context rules.

[0063] In specific implementation, the generation of the corrected motion trajectory of the virtual character through the virtual motion features can be achieved in the following way: First, for the conflict-corrected virtual motion features of each trajectory point, according to the topological structure of the virtual character skeleton, the global position vector and rotation state of each joint are sequentially passed from the root joint to each sub-joint to form complete skeleton posture data during haptic interaction. The global position and rotation state of each joint take into account the influence of the parent joint correction on its spatial position and posture, thereby ensuring the continuity and consistency of the overall skeleton structure. Then, the skeleton posture data corresponding to each trajectory point are arranged in the time order of the trajectory points to construct a continuous time series as the corrected motion trajectory of the virtual character.

[0064] It should be noted that the corrected motion trajectory of the virtual character mentioned in this application refers to the skeleton posture sequence formed by sequentially transmitting the global position vector and rotation state of each joint of the virtual character along the skeleton topology from the root joint based on the virtual motion features after conflict correction, and arranging them in chronological order to generate a continuous motion sequence.

[0065] For specific implementation, refer to Figure 3As shown in the figure, this is a flowchart illustrating the execution of interactive actions according to some embodiments of this application. Driving a virtual character to execute interactive actions conforming to contextual rules based on the corrected motion trajectory can be achieved in the following way: First, the corrected motion trajectory of the virtual character is parsed into a continuous sequence of skeleton poses in chronological order, with each skeleton pose including the global position vector and rotation state of each joint; then, in the virtual environment, the parsed skeleton poses are sequentially mapped to the corresponding joints of the virtual character model, causing the virtual character to update its pose and position step-by-step in the 3D scene; further, combined with rule constraints in educational context simulation, such as task objectives, The interaction conditions and environmental constraints of the object are judged and adjusted in real time to make the virtual character's actions meet the requirements of the scene rules based on the corrected action trajectory. Preferably, existing virtual character control technology and physical simulation engine can be used to achieve this, such as the Inverse Kinematics (IK) system. IK constrains the position of the execution point at the end of the skeleton, and at the same time combines physical collision detection and joint constraints to achieve coordination between action continuity and rule constraints. This enables the virtual character to continuously execute interactive actions that conform to the situation rules according to the corrected action trajectory. Other methods can also be used in other embodiments, which are not limited here.

[0066] Furthermore, in another aspect of this application, in some embodiments, this application provides an educational scenario simulation system supported by motion-sensing interaction, referencing... Figure 4 The figure is a schematic diagram of the structure of a motion-sensing interactive educational scenario simulation system 200 according to some embodiments of this application. The motion-sensing interactive educational scenario simulation system 200 includes: a data acquisition module 201, a processing module 202, and an execution module 203, which are described below: The acquisition module 201 in this application is mainly used to synchronously acquire the learner's motion sensor data during the simulated interaction in the educational scenario. Processing module 202, in this application, is mainly used to construct a learner's motion skeleton model based on the motion sensor data, wherein the motion skeleton model includes the angle parameters of each joint of the learner during the educational scenario simulation; In addition, the processing module 202 in this application is also used to generate the motion prediction trajectory of the virtual character at the end execution point in the educational scenario simulation by using the angle parameters of each joint of the learner, and to spatially fit the motion prediction trajectory with the boundary constraint conditions of the virtual object in the educational scenario simulation to obtain the interaction conflict risk between joint movements and virtual objects during somatosensory interaction. In addition, the processing module 202 in this application is also used to generate a dynamic conflict compensation vector in the haptic interaction process based on the directional difference between the direction of the virtual character's action and the surface normal of the virtual object when the interaction conflict risk is less than a preset risk threshold. The execution module 203 in this application is mainly used to constrain and correct the motion trajectory of the virtual character based on the dynamic conflict compensation vector, obtain the conflict-corrected virtual motion features, and then drive the virtual character to perform interactive actions that conform to the context rules through the virtual motion features.

[0067] In addition, this application also provides a computer device, the computer device including a memory and a processor, the memory storing code, the processor being configured to acquire the code and execute the above-described method for simulating educational scenarios supported by motion-sensing interaction.

[0068] In some embodiments, reference Figure 5 This figure is an internal structural diagram of a computer device implementing a motion-sensing interaction-supported educational scenario simulation method according to some embodiments of this application. The motion-sensing interaction-supported educational scenario simulation method in the above embodiments can be implemented through... Figure 5 The computer device shown is used to implement this, and the computer device 300 includes at least one processor 301, a communication bus 302, a memory 303, and at least one communication interface 304.

[0069] The processor 301 may be a general-purpose central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more devices used to control the execution of the motion-sensing interactive educational scenario simulation method in this application.

[0070] The communication bus 302 is used to transmit information between the aforementioned components.

[0071] Memory 303 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CDROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital versatile optical discs, Blu-ray discs, etc.), magnetic disks or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. Memory 303 may exist independently and be connected to processor 301 via communication bus 302. Memory 303 may also be integrated with processor 301.

[0072] The memory 303 stores program code for executing the scheme of this application, and its execution is controlled by the processor 301. The processor 301 executes the program code stored in the memory 303. The program code may include one or more software modules. In the above embodiments, the educational scenario simulation method supported by motion-sensing interaction can be implemented by the processor 301 and one or more software modules in the program code in the memory 303.

[0073] Communication interface 304 uses any transceiver-like device to communicate with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area networks (WLAN), etc.

[0074] In a specific implementation, as one example, a computer device may include multiple processors, each of which may be a single-core processor or a multi-core processor. Here, a processor may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0075] The aforementioned computer device can be a general-purpose computer device or a special-purpose computer device. In specific implementations, the computer device may be a desktop computer, a portable computer, a network server, a handheld digital assistant (PDA), a mobile phone, a tablet computer, a wireless terminal device, a communication device, or an embedded device. This application does not limit the type of computer device.

[0076] In addition, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for simulating educational scenarios supported by motion-sensing interaction.

[0077] In summary, the motion-sensing interaction-supported educational scenario simulation system and method disclosed in this application synchronously collects learner motion sensor data during the educational scenario simulation interaction process; constructs a learner motion skeleton model based on the motion sensor data, wherein the motion skeleton model includes angle parameters of each joint of the learner during the educational scenario simulation; generates a motion prediction trajectory of the virtual character at the end execution point in the educational scenario simulation using the angle parameters of each joint of the learner; spatially fits the motion prediction trajectory with the boundary constraints of the virtual object during the educational scenario simulation to obtain the interaction conflict risk between joint movements and the virtual object during motion-sensing interaction; when the interaction conflict risk is less than a preset risk threshold, a dynamic conflict compensation vector is generated during the motion-sensing interaction process based on the directional difference between the direction of the virtual character's movement and the surface normal of the virtual object; constrains and corrects the motion trajectory of the virtual character according to the dynamic conflict compensation vector to obtain conflict-corrected virtual action features, and then drives the virtual character to perform interactive actions that conform to the scenario rules through the virtual action features; this can avoid motion mapping conflicts in dynamic scenarios of motion-sensing interaction, so as to achieve accurate synchronization between learner's actions and virtual scene interaction.

[0078] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0079] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for simulating educational scenarios supported by motion-sensing interaction, characterized in that, Includes the following steps: During the simulated interactive educational scenario, learners' motion sensor data are collected synchronously. A motion skeleton model of the learner is constructed based on the motion sensor data, wherein the motion skeleton model includes the angle parameters of each joint of the learner during the educational scenario simulation. By generating the motion prediction trajectory of the virtual character at the end execution point in the educational scenario simulation using the angle parameters of each joint of the learner, the motion prediction trajectory is spatially fitted with the boundary constraints of the virtual object in the educational scenario simulation to obtain the interaction conflict risk between joint movements and virtual objects during somatosensory interaction. When the risk of interaction conflict is less than a preset risk threshold, a dynamic conflict compensation vector is generated in the haptic interaction process based on the directional difference between the direction of the virtual character's action and the surface normal of the virtual object. The motion trajectory of the virtual character is constrained and corrected based on the dynamic conflict compensation vector to obtain the conflict-corrected virtual motion features. Then, the virtual character is driven to perform interactive actions that conform to the context rules through the virtual motion features.

2. The method as described in claim 1, characterized in that, Constructing a learner's motion skeleton model based on the motion sensor data specifically includes: The motion sensor data is mapped onto the learner's skeletal topology to establish the initial positions of each joint in three-dimensional space; The rotational parameters of each joint are determined by the initial position of each joint in three-dimensional space and the constraint of the length of the skeleton link. Based on the rotation parameters and the parent-child joint relationships of all joints, a motion skeleton model of the learner is constructed.

3. The method as described in claim 1, characterized in that, Specifically, generating the predicted motion trajectory of the virtual character at the end-effector in the educational scenario simulation based on the angle parameters of each joint of the learner includes: The angle parameters of each joint of the learner are passed along the skeleton topology chain to obtain the rotation matrix sequence of each joint in three-dimensional space. Based on the rotation matrix sequence and combined with forward kinematics, the position change sequence of each end execution point of the virtual character in three-dimensional space is calculated; The predicted trajectory of a virtual character at the end execution point in an educational scenario simulation is generated using the position change sequence.

4. The method as described in claim 1, characterized in that, By spatially fitting the predicted motion trajectory with the boundary constraints of the virtual object during educational scenario simulation, the specific risks of interaction conflict between joint movements and virtual objects during motion-sensing interaction are obtained, including: Construct a boundary space model of the virtual objects based on the boundary constraints of the virtual objects during the educational scenario simulation; Determine the spatial overlap between the predicted motion trajectory and the boundary space model at different trajectory points; The risk of interaction conflict between joint movements and virtual objects during motion-sensing interaction is determined based on all spatial overlaps.

5. The method as described in claim 1, characterized in that, The dynamic conflict compensation vector generated based on the directional difference between the virtual character's movement direction and the surface normal of the virtual object specifically includes: Obtain the surface normal of the virtual object; Determine the directional difference vector between the virtual character's movement direction and the surface normal; By integrating all directional difference vectors according to the skeleton topology chain, the dynamic conflict compensation vector in the haptic interaction process is obtained.

6. The method as described in claim 1, characterized in that, The virtual character's motion trajectory is constrained and corrected based on the dynamic conflict compensation vector to obtain the conflict-corrected virtual motion features, specifically including: The motion prediction trajectory is dynamically compensated using the dynamic conflict compensation vector to obtain the constraint correction offset of each joint. By reverse mapping all constraint correction offsets, conflict correction parameters for each joint in the motion trajectory of the virtual character are obtained. Generate conflict-corrected virtual motion features based on all conflict correction parameters.

7. The method as described in claim 1, characterized in that, The specific methods for driving virtual characters to perform interactive actions that conform to contextual rules through the aforementioned virtual action features include: The virtual character's corrected motion trajectory is generated using the aforementioned virtual motion features; Based on the modified motion trajectory, the virtual character is driven to perform interactive actions that conform to the context rules.

8. A motion-sensing interactive educational scenario simulation system, characterized in that, include: The data acquisition module is used to synchronously collect learners' motion sensor data during simulated interactive educational scenarios. The processing module is used to construct a motion skeleton model of the learner based on the motion sensor data, wherein the motion skeleton model includes the angle parameters of each joint of the learner during the educational scenario simulation. The processing module is also used to generate the motion prediction trajectory of the virtual character at the end execution point in the educational scenario simulation by using the angle parameters of each joint of the learner, and to spatially fit the motion prediction trajectory with the boundary constraints of the virtual object in the educational scenario simulation to obtain the interaction conflict risk between joint movements and virtual objects during somatosensory interaction. The processing module is also used to generate a dynamic conflict compensation vector in the haptic interaction process based on the directional difference between the virtual character's action direction and the surface normal of the virtual object when the interaction conflict risk is less than a preset risk threshold. The execution module is used to constrain and correct the motion trajectory of the virtual character based on the dynamic conflict compensation vector, obtain the conflict-corrected virtual motion features, and then drive the virtual character to perform interactive actions that conform to the context rules through the virtual motion features.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the educational scenario simulation method supported by motion-sensing interaction as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the educational scenario simulation method supported by motion-sensing interaction as described in any one of claims 1 to 7.

Images