An interactive system and method based on virtual reality

Abstract

The application provides an interactive system and method based on virtual reality, comprising: a virtual scene construction module: dynamically generating a virtual scene according to user behavior data and teaching needs, including scene geometry modeling, texture mapping, light rendering and interactive logic driven by a physical engine; a multi-modal data acquisition module: including a motion capture unit, a micro-expression recognition unit and a professional personality analysis unit, for real-time acquisition of user action images, facial key point data and personality characteristic parameters; an intelligent analysis module, based on a motion classification model of a convolutional neural network, a micro-expression emotion analysis model of a support vector machine and a professional personality classification model of a decision tree. The application has the beneficial effect that: through dynamic scene adaptation and multi-modal data analysis, the teaching effect of ethics experiments is significantly improved, the system adjusts the light, object position and conflict intensity of the virtual scene in real time based on user attention scores and behavior preferences, ensuring the real-time and accuracy of the generated teaching strategy.

Description

Technical Field

[0001] This invention belongs to the field of education systems, and in particular relates to an interactive system and method based on virtual reality. Background Technology

[0002] Currently, the application of virtual reality (VR) technology in education is mainly concentrated in practical disciplines such as medicine and engineering. However, it still faces significant technical bottlenecks in humanities disciplines such as ethics. First, existing systems rely on static scene construction and cannot dynamically adjust the teaching environment based on real-time user behavior (such as decision-making hesitation and emotional fluctuations), leading to a disconnect between teaching content and user needs. Second, data collection and analysis methods are limited. Traditional methods only obtain information through questionnaires or basic motion capture, making it difficult to deeply integrate multimodal data such as actions, micro-expressions, and personality, resulting in coarse user profiles and a lack of targeted teaching strategies. Third, the interactive feedback mechanism is rigid, resulting in low efficiency of teacher-student collaboration and an inability to generate moral decision-making reports or conflict resolution suggestions in real time. For example, in the classic ethics experiment "Trolley Problem," existing systems can only simulate fixed scenes and cannot dynamically adjust the intensity of conflict (such as adding data prompts or emotional rendering) based on the user's micro-expressions (such as the frequency of frowning) or motion delays during decision-making, thus limiting the teaching effectiveness. Summary of the Invention

[0003] In view of this, the present invention aims to provide an interactive system and method based on virtual reality, so as to at least solve one of the problems in the background art.

[0004] To achieve the above objectives, the technical solution of the present invention is implemented as follows: A virtual reality-based interactive system, comprising: Virtual scene construction module: used to dynamically generate virtual scenes based on user behavior data and teaching needs, including scene geometry modeling, texture mapping, lighting rendering and physical engine-driven interaction logic; Multimodal data acquisition module: including motion capture unit, micro-expression recognition unit and occupational personality analysis unit, used to collect user motion images, facial key point data and personality characteristic parameters in real time; The intelligent analysis module, based on the action classification model of convolutional neural network, the micro-expression emotion analysis model of support vector machine, and the occupational personality classification model of decision tree, analyzes the collected data and generates dynamic teaching strategies. The cloud platform processing module is used for data compression and storage, scene rendering command issuance, and multi-terminal collaborative interaction; The interactive feedback module enables real-time interaction between the user and the virtual scene through VR devices, including gesture control, voice commands, and facial expression feedback.

[0005] Furthermore, the dynamic generation process of the virtual scene construction module includes the following steps: The scene lighting intensity is dynamically adjusted based on the user's attention score, which is calculated by combining the duration of the user's gaze focus and the tension of their micro-expressions. The placement of objects in a virtual scene is optimized based on user action preferences, which are obtained through continuous frame action classification results and scene interaction frequency analysis.

[0006] Furthermore, the motion capture unit of the multimodal data acquisition module performs the following operations: The system uses a binocular camera to capture images of the user's movements and employs an edge detection algorithm to extract key point data. The 3D convolutional neural network is input with consecutive frames of images within a time window and outputs action type classification results.

[0007] Furthermore, the micro-expression emotion analysis process of the intelligent analysis module includes: Facial key points are detected using the MTCNN model, and the displacement vectors of key points in consecutive frames are calculated using optical flow. Input the displacement vector into the support vector machine model and output the sentiment classification label.

[0008] Furthermore, the cloud platform processing module includes: The data fusion unit is used to align action data, micro-expression data, and personality data with timestamps and assign weights to generate user behavior profiles. The real-time optimization unit dynamically adjusts the complexity of teaching content and the latency of interactive feedback in the virtual scene based on user behavior profiles. The optimization logic is as follows: When the user's attention score is below the threshold, reduce the scene complexity and add virtual teacher guidance prompts; When the user's sentiment analysis result is "anxiety", the scene conflict intensity adjustment mechanism is triggered.

[0009] Furthermore, the data fusion unit performs the following steps: Data from different sensors can be mapped to the same timeline by timestamp alignment; Weighting coefficients are assigned based on data type: action data has a weight of 0.5, micro-expression data has a weight of 0.3, and personality data has a weight of 0.2. A weighted fusion formula is used to generate a comprehensive score for user behavior profiles.

[0010] Furthermore, the interactive feedback module supports multi-terminal collaborative interaction, including: Teachers can view student behavior profiles and teaching strategy suggestions in real time. Students can use gestures to manipulate ethical conflict props in a virtual scene, triggering the system to record the operation trajectory and generate a moral decision report.

[0011] Furthermore, this solution discloses a virtual reality-based interaction method, based on a virtual reality-based interaction system, including: S1. Multimodal data synchronous acquisition: Through motion capture equipment, micro-expression sensor and biosignal acquisition device, the user's action sequence, facial expression changes and physiological index data are acquired synchronously. A unified timestamp is applied to the acquired multi-source data, and the time sequence is aligned by using a sliding window algorithm with a preset teaching experiment response period as the window length. S2. User behavior profile construction: The user's decision-making delay is analyzed based on action sequence analysis. The delay is calculated by comparing the user's actual decision-making time with the experimental preset average time and maximum allowed time. At the same time, facial expression features and physiological indicators are combined to calculate the user's emotional tension score. S3. Dynamic Ethical Conflict Parameter Generation: Based on user personality classification results and behavioral profile data, dynamically generate conflict adjustment parameters for virtual scenarios. S4. Real-time scene rendering and differentiated feedback: The scene lighting intensity is dynamically adjusted according to the user's decision-making delay and personality type. The adjustment logic is as follows: the higher the delay, the higher the lighting intensity with the personality weight factor. At the same time, the tactile feedback module generates a vibration pattern that matches the user's decision-making direction, including high-frequency short pulses and low-frequency long vibrations. The high-frequency short pulses correspond to quick decision-making, and the low-frequency long vibrations correspond to hesitant decision-making. S5. Multi-terminal collaboration and teaching analysis: The teacher's end generates an ethical decision-making heatmap based on student behavior profiles, and the heatmap marks the distribution of conflict choices of user groups in virtual scenarios; the student's end outputs a moral decision-making analysis report, which includes a radar chart based on behavioral profiles and improvement suggestions for rational / emotional tendencies.

[0012] Furthermore, in step S3, the user personality classification results include rational and emotional types. For rational users, quantitative decision indicators are superimposed on the scene, including casualty statistics and probability calculations; for emotional users, emotional rendering elements are inserted, including the intensity of the virtual character's crying and close-up shots of facial expressions.

[0013] Compared with existing technologies, the virtual reality-based interactive system and method described in this invention have the following advantages: (1) The interactive system and method based on virtual reality described in this invention significantly improves the teaching effect of ethics experiments through dynamic scene adaptation and multimodal data analysis. The system adjusts the lighting, object position and conflict intensity of the virtual scene in real time based on user attention scores and behavioral preferences to ensure the real-time and accurate generation of teaching strategies. (2) The interactive system and method based on virtual reality described in this invention supports multi-terminal collaboration and personalized teaching guidance. The teacher can quickly analyze the distribution of group decision-making through heat maps, improve data analysis efficiency, reduce the delay in generating moral reports on the student side, and provide targeted improvement suggestions. Attached Figure Description

[0014] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings: Figure 1 This is a schematic diagram of an interactive system and method based on virtual reality according to an embodiment of the present invention. Detailed Implementation

[0015] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0016] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0017] 1. To address the problem that traditional virtual scenes in existing technologies cannot be dynamically adjusted according to real-time user behavior, resulting in poor adaptability of teaching content, the attention score calculation method in this solution is as follows: The user's gaze focus is collected by an eye tracker to determine the duration of the user's gaze focus on the core teaching area in the virtual scene (such as the control lever in the "trolley problem"), and the percentage of the gaze focus within the maximum preset duration is calculated. At the same time, the user's tension coefficient is calculated by combining the frowning frequency detected by the micro-expression recognition module and the heart rate sensor data (such as the offset of heart rate variability relative to the baseline). Finally, the attention score is generated by fusing the gaze focus duration percentage and the tension coefficient with a weight of 6:4. The dynamic lighting adjustment method is as follows: when the attention score is below the threshold, the lighting intensity of non-core areas is reduced to 60% of the baseline value, and the core teaching elements are highlighted (such as the control lever, whose lighting intensity is increased to 1.8 times the baseline value). If the user is detected to be in a state of high anxiety (such as a tension coefficient exceeding 0.8), the background sound intensity of the scene is simultaneously reduced to 50%. The object position optimization method is as follows: based on the frequent areas of the user's historical grabbing actions, the optimal placement position of the props is calculated through the physics engine to make them close to the user's habitual interaction area; 2. To address the problem that traditional motion capture relies on preset rules and cannot recognize complex ethical decision-making behaviors (such as hesitant hand-raising) in the existing technology, the motion image acquisition and processing method of this solution in the motion capture and classification part is as follows: use a binocular camera to acquire user gesture images at a frame rate of 60Hz, extract the contour of hand joints through edge detection algorithm, and calculate the image gradient magnitude and direction to locate motion features. The action classification logic is as follows: five consecutive frames of images (time window 0.1 seconds) are input into a 3D convolutional neural network (3D-CNN). The network extracts spatiotemporal features through three layers of 3×3×3 convolutional kernels. After dimensionality reduction by pooling layers, the fully connected layer outputs the action classification results (e.g., "raising hand" probability 93%, "grabbing" probability 5%). 3. To address the issue that existing technologies suffer from subtle changes in micro-expressions and the difficulty of traditional algorithms in capturing instantaneous emotional fluctuations, the facial key point detection method in this solution for micro-expression emotion analysis is as follows: the MTCNN model is used to detect the coordinates of key points on the user's face, including the eyes, nose tip, and corners of the mouth, with data updated every 10ms. The facial expression change analysis method is as follows: the sum of displacement vectors of key points between consecutive frames is calculated by optical flow. If the displacement exceeds 15 pixels, it is judged as a significant facial expression change (such as surprise or frowning). Combined with the drooping angle parameter of the corner of the mouth, the displacement vector and angle data are input into the support vector machine (SVM) model, classified based on the radial basis function kernel, and output sentiment label (such as "anxiety" with a confidence of 88%). 4. To address the problem of isolated data between teachers and students in existing technologies, which prevents real-time strategy synchronization, the teacher-side decision heatmap generation method in this solution is as follows: Collect the selection data of multiple students in the "trolley problem" experiment (e.g., 60% choose "sacrifice 1 person" and 40% choose "sacrifice 5 people"), generate a heatmap through color gradient mapping, with red indicating high conflict areas, and the conflict score is calculated by weighting the selection ratio, highlighting the group decision-making tendency. The method for generating student-side moral reports is as follows: analyze user decision-making time (e.g., 8.2 seconds), anxiety peak (e.g., tension level 0.75), and personality type (e.g., rational type 85%), generate a radar chart to show moral tendencies, and provide improvement suggestions based on the analysis results (e.g., "Significant tendency towards rational decision-making, it is recommended to add empathy training scenarios"). Based on the virtual reality-based interactive system disclosed in this solution, the specific implementation method is as follows: Step S1, Multimodal Data Acquisition and Synchronization: Using a binocular camera, infrared sensor and biosignal acquisition device, the user's motion images, facial micro-expression data and heart rate variability parameters are acquired in real time. A unified timestamp is applied to the multi-source data, and the timing is aligned through a sliding window mechanism. The window length is the standard response cycle (5 seconds) of the teaching experiment. Step S2, User Behavior Profile Construction: Based on action image analysis, the user's decision hesitation level is calculated using the following formula: ,in, For the user's actual decision-making time, The average time was preset for the experiment. The maximum allowed time; Combining micro-expression recognition results (such as the frequency of drooping corners of the mouth) Using heart rate variability (HRV) parameters, an emotional stress score is calculated. ; Step S3: Dynamic Ethical Conflict Parameter Generation: Based on the user's personality classification results (rational / emotional) and behavioral profile, generate dynamic ethical conflict parameters. Rational users: Overlaying quantitative decision-making indicators (such as probability of injury or death) in virtual scenarios. ); For emotionally driven users: Insert emotionally evocative elements, including virtual crying intensity. ( k (For scene coefficients) Step S4, Real-time Scene Rendering and Interactive Feedback: Input the conflict parameters into the Unity engine's shader script to dynamically adjust scene lighting and sound effects. The lighting intensity formula is: ,in, α Personality weighting factor (rational type) α =0.8, emotional type α =0.5); At the same time, through the VR controller's haptic feedback module, differentiated vibration patterns are generated according to the user's decision direction (such as pushing the control stick left / right): left-direction selection triggers high-frequency short pulses (frequency 120Hz, duration 0.1 seconds); right-direction selection triggers low-frequency long vibrations (frequency 60Hz, duration 0.3 seconds). Step S5, Multi-terminal Collaboration and Teaching Report Generation: The teacher receives student behavior profile data in real time, generates an ethical decision-making heatmap, and marks the distribution of conflict choices; the student outputs a decision analysis report after the experiment, including a moral tendency radar chart and improvement suggestions (such as "significant tendency towards rational decision-making, empathy training needs to be strengthened").

[0018] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0019] In the several embodiments provided in this application, it should be understood that the disclosed methods and systems can be implemented in other ways. For example, the division of units described above is merely a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. The aforementioned units may or may not be physically separated. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of the present invention according to actual needs.

[0020] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be covered within the scope of the claims and specification of the present invention.

[0021] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An interactive system based on virtual reality, characterized in that, include: Virtual scene construction module: used to dynamically generate virtual scenes based on user behavior data and teaching needs, including scene geometry modeling, texture mapping, lighting rendering and physical engine-driven interaction logic; Multimodal data acquisition module: including motion capture unit, micro-expression recognition unit and occupational personality analysis unit, used to collect user motion images, facial key point data and personality characteristic parameters in real time; The intelligent analysis module, based on the action classification model of convolutional neural network, the micro-expression emotion analysis model of support vector machine, and the occupational personality classification model of decision tree, analyzes the collected data and generates dynamic teaching strategies. The cloud platform processing module is used for data compression and storage, scene rendering command issuance, and multi-terminal collaborative interaction; The interactive feedback module enables real-time interaction between the user and the virtual scene through VR devices, including gesture control, voice commands, and facial expression feedback.

2. The virtual reality-based interactive system according to claim 1, characterized in that: The dynamic generation process of the virtual scene construction module includes the following steps: The scene lighting intensity is dynamically adjusted based on the user's attention score, which is calculated by combining the duration of the user's gaze focus and the tension of their micro-expressions. The placement of objects in a virtual scene is optimized based on user action preferences, which are obtained through continuous frame action classification results and scene interaction frequency analysis.

3. The virtual reality-based interactive system according to claim 1, characterized in that: The motion capture unit of the multimodal data acquisition module performs the following operations: The system uses a binocular camera to capture images of the user's movements and employs an edge detection algorithm to extract key point data. The 3D convolutional neural network is input with consecutive frames of images within a time window and outputs action type classification results.

4. The interactive system based on virtual reality according to claim 1, characterized in that: The micro-expression emotion analysis process of the intelligent analysis module includes: Facial key points are detected using the MTCNN model, and the displacement vectors of key points in consecutive frames are calculated using optical flow. Input the displacement vector into the support vector machine model and output the sentiment classification label.

5. The virtual reality-based interactive system according to claim 1, characterized in that: The cloud platform processing module includes: The data fusion unit is used to align action data, micro-expression data, and personality data with timestamps and assign weights to generate user behavior profiles. The real-time optimization unit dynamically adjusts the complexity of teaching content and the latency of interactive feedback in the virtual scene based on user behavior profiles. The optimization logic is as follows: When the user's attention score is below the threshold, reduce the scene complexity and add virtual teacher guidance prompts; When the user's sentiment analysis result is "anxiety", the scene conflict intensity adjustment mechanism is triggered.

6. The virtual reality-based interactive system according to claim 5, characterized in that: The data fusion unit performs the following steps: Data from different sensors can be mapped to the same timeline by timestamp alignment; Weighting coefficients are assigned based on data type: action data has a weight of 0.5, micro-expression data has a weight of 0.3, and personality data has a weight of 0.

2. A weighted fusion formula is used to generate a comprehensive score for user behavior profiles.

7. The virtual reality-based interactive system according to claim 1, characterized in that: The interactive feedback module supports multi-terminal collaborative interaction, including: Teachers can view student behavior profiles and teaching strategy suggestions in real time. Students can use gestures to manipulate ethical conflict props in a virtual scene, triggering the system to record the operation trajectory and generate a moral decision report.

8. A virtual reality-based interaction method, comprising a virtual reality-based interaction system according to any one of claims 1-7, characterized in that, include: S1. Multimodal data synchronous acquisition: Through motion capture equipment, micro-expression sensor and biosignal acquisition device, the user's action sequence, facial expression changes and physiological index data are acquired synchronously. A unified timestamp is applied to the acquired multi-source data, and the time sequence is aligned by using a sliding window algorithm with a preset teaching experiment response period as the window length. S2. User behavior profile construction: The user's decision-making delay is analyzed based on action sequence analysis. The delay is calculated by comparing the user's actual decision-making time with the experimental preset average time and maximum allowed time. At the same time, facial expression features and physiological indicators are combined to calculate the user's emotional tension score. S3. Dynamic Ethical Conflict Parameter Generation: Based on user personality classification results and behavioral profile data, dynamically generate conflict adjustment parameters for virtual scenarios. S4. Real-time scene rendering and differentiated feedback: The scene lighting intensity is dynamically adjusted according to the user's decision-making delay and personality type. The adjustment logic is as follows: the higher the delay, the higher the lighting intensity with the personality weight factor. At the same time, the tactile feedback module generates a vibration pattern that matches the user's decision-making direction, including high-frequency short pulses and low-frequency long vibrations. The high-frequency short pulses correspond to quick decision-making, and the low-frequency long vibrations correspond to hesitant decision-making. S5. Multi-terminal collaboration and teaching analysis: The teacher's end generates an ethical decision-making heatmap based on student behavior profiles, and the heatmap marks the distribution of conflict choices of user groups in virtual scenarios; the student's end outputs a moral decision-making analysis report, which includes a radar chart based on behavioral profiles and improvement suggestions for rational / emotional tendencies.

9. The virtual reality-based interactive system according to claim 8, characterized in that: In step S3, the user personality classification results include rational and emotional types. For rational users, quantitative decision indicators are superimposed on the scene, including casualty statistics and probability calculations. For emotional users, emotional rendering elements are inserted, including the intensity of the virtual character's crying and close-up shots of facial expressions.

Images