Education training system based on VR and device thereof

Abstract

The invention discloses a VR-based educational training system, and relates to the technical field of virtual reality educational training, and the system comprises a client which is used for logging in an account; the central processing module is used for processing data; the visual module is used for identifying the facial expression of the user and is connected with the central processing module; the modeling module is used for building a user virtual model and is linked with the visual module; the invention further discloses an educational training device, specifically, the visual module identifies the facial expression of the user and is linked with the modeling module to update the virtual model in real time, so that the system can accurately monitor the state of the user, and the immersion of educational training is improved; the method has the advantages that the user state monitoring capability is effectively improved, the authenticity and interactivity of the virtual environment are enhanced, and thus the learning effect is improved.

Description

Technical Field

[0001] This invention relates to the field of virtual reality education and training technology, and in particular, to a VR-based education and training system and apparatus. Background Technology

[0002] Existing online education and training systems generally suffer from low user engagement and poor learning outcomes. In remote learning scenarios, students frequently engage in "idle" behavior, meaning they log into the system but leave their devices for extended periods or remain unfocused. The system cannot effectively identify and intervene, leading to wasted training resources and difficulty in achieving learning objectives. Insufficient interaction is prominently manifested in the lack of natural emotional exchange between teachers and students. Interactions in the virtual environment are often superficial, failing to simulate the immediate feedback and emotional resonance of a real classroom. Poor realism stems from a severe disconnect between the virtual avatar and the user's actual state. For example, changes in user facial expressions cannot be mapped to the virtual model in real time, making the virtual environment appear rigid and unrealistic, weakening immersion and learning motivation. Traditional VR education systems typically only have two simple account types: teacher and student. They lack multi-level management mechanisms such as administrator accounts, teaching accounts, and supervisory accounts, failing to support role division and collaboration in complex teaching scenarios. More importantly, these systems fail to integrate facial expression recognition technology for real-time user status monitoring. For example, they cannot assess a user's level of concentration, emotional fluctuations, or cognitive load by analyzing subtle facial expressions such as eye movements and changes in the corners of the mouth. This prevents the system from dynamically adjusting teaching strategies to address user distraction or confusion. Although existing technologies attempt to improve this by constructing virtual scenes or capturing body movements, these solutions often neglect the crucial dimension of facial expressions, resulting in incomplete user status detection. For instance, while some systems can generate virtual student avatars and analyze their behavior, they fail to dynamically link facial expression data with the virtual model, causing the virtual avatar to fail to accurately reflect the user's current psychological state. Other systems, while possessing basic facial capture capabilities, only use them for simple emotion classification and are not deeply integrated with the education and training process, making it difficult to effectively prevent idle behavior or improve interaction quality. Specifically, existing solutions have significant limitations: some VR-based remote education systems focus on virtual scene construction but lack facial expression recognition modules, failing to capture subtle emotional changes in users; while intelligent teaching systems integrate motion capture and facial capture, their evaluation mechanisms operate in isolation, failing to drive real-time updates of the virtual model using facial expression data; multi-person virtual experimental teaching systems focus on synchronizing interactive information but neglect the role of facial expression monitoring in assessing user focus; nursing operation practice platforms record limb movement trajectories but focus on skills assessment, failing to cover multi-level account management and comprehensive analysis of facial expression states in education and training; remote virtual teaching systems match different levels of teacher resources but lack facial expression recognition technology, failing to ensure genuine user participation throughout the process. These deficiencies collectively result in significant shortcomings in preventing idle behavior, enhancing the realism of interaction, and achieving comprehensive user status awareness. Users may remain in a non-learning state for extended periods without the system noticing, leading to a persistent lack of realism in the virtual environment and targeted teaching.

[0003] Therefore, it is necessary to propose a VR-based education and training system and device to solve or at least alleviate the above-mentioned defects. Summary of the Invention

[0004] The main objective of this invention is to provide a VR-based education and training system and device, which has the advantages of effectively improving user status monitoring capabilities, enhancing the realism and interactivity of the virtual environment, and thus improving learning outcomes.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows: This application provides a VR-based education and training system, the technical solution of which is as follows: It includes: a client for logging into an account; a central processing module for processing data; a vision module for recognizing user facial expressions and connected to the central processing module; and a modeling module for building a virtual user model and linked with the vision module. Furthermore, this application also proposes that it includes a temperature acquisition module for monitoring user body temperature and ambient temperature data; the output of the temperature acquisition module is connected to the central processing module for transmitting user body temperature and ambient temperature data to the central processing module. Furthermore, this application also proposes that the account includes an administrative account, a teaching account, and a supervisory account, which work together to enable users to log in on the client. Furthermore, this application also proposes that the management account is a system configuration account that logs in through the client and connects to the central processing module, and is used to send system configuration instructions to the central processing module to set and manage parameters of the client, central processing module, vision module or modeling module. Furthermore, this application proposes that the supervision account is a passive observation account; when logging into the supervision account through the client, the supervision account establishes a data connection with the central processing module to receive and display user facial expression data forwarded by the central processing module and identified by the vision module, as well as user virtual model images generated by the modeling module. Furthermore, this application proposes that the teaching account is a learning account, and when a user builds their own virtual model through the management account, a data connection is established with the central processing module to achieve real-time learning. Furthermore, this application also proposes to integrate the system and its components into an electronic device. As can be seen from the above, the VR-based education and training system provided in this application recognizes the user's facial expressions through a visual module and links with a modeling module to update the virtual model in real time. The system can accurately monitor the user's status, enhance the immersion of education and training, and has the advantages of effectively improving the user status monitoring capability, enhancing the realism and interactivity of the virtual environment, thereby improving the learning effect.

[0006] In addition to the objectives, features and advantages described above, the present invention has other objectives, features and advantages.

[0007] The present invention will now be described in further detail with reference to the figures. Attached Figure Description

[0008] The accompanying drawings, which form part of this application, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings: Figure 1 This is a schematic diagram of the organizational structure of a VR-based education and training system in this invention; Detailed Implementation

[0009] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0010] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0011] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indication will also change accordingly.

[0012] Furthermore, the use of terms such as "first" and "second" in this invention is for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of that feature. Additionally, the technical solutions of the various embodiments can be combined with each other, but only on the basis of being achievable by those skilled in the art. When the combination of technical solutions is contradictory or impossible to implement, such a combination of technical solutions should be considered non-existent and not within the scope of protection claimed by this invention. Detailed Implementation

[0013] The technical solutions of this application will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments. The components of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0014] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0015] Traditional online education and training systems often lack multi-dimensional data fusion capabilities in their user status detection mechanisms, particularly the absence of dynamic linkage between facial expression recognition technology and virtual models. This prevents the system from acquiring real-time information on changes in users' emotional states. Consequently, the timeliness of teaching interaction is limited, the realism of the virtual environment is insufficient, and consequently, user engagement and the accuracy of teaching effectiveness evaluation are affected.

[0016] For example, in a virtual nursing operation training scenario, students wear VR devices to conduct cardiopulmonary resuscitation simulation training, and their operation trajectory is recorded by the system through a motion capture module. However, when students show facial fatigue or become distracted due to prolonged operation, the lack of a linkage mechanism between the facial expression recognition module and the virtual model means that the teaching prompts are not dynamically adjusted by the virtual assistant, resulting in the student's operational errors not being corrected in a timely manner, and reducing the immersive experience of the virtual scene.

[0017] If these issues are not addressed, user idling behavior cannot be effectively monitored and intervened in by the system, resulting in a lack of substantive participation in the training process and a decline in the utilization rate of teaching resources. Furthermore, the system will be unable to generate targeted teaching strategies based on user status, which will weaken the effectiveness of education and training in the long run and reduce users' trust in the system.

[0018] In response, this application proposes a VR-based education and training system, including: a client 1 for logging into an account; a central processing module 2 for processing data; a vision module 3 for recognizing user facial expressions and connecting to the central processing module; and a modeling module 4 for building a virtual model of the user and interacting with the vision module.

[0019] For ease of understanding, the following explains some key terms in this embodiment: Client 1 refers to the terminal device used by the user to access the VR-based education and training system. Client 1 can be a standalone VR headset, or a computer, tablet, or smartphone with integrated VR functionality. Through Client 1, users can interact with the system and log in to their personal accounts, thereby entering the virtual education and training environment.

[0020] Central Processing Module 2 is the core unit in the system responsible for data processing, coordination, and control. This module receives data input from various functional modules, performs analysis, calculations, and decision-making, and sends instructions to other modules to ensure the smooth operation of the entire education and training system and the realization of its various functions.

[0021] The vision module 3 refers to the component in the system used to capture and analyze the user's facial expressions. This module 3 typically includes one or more cameras and an image processing unit, capable of recognizing changes in the user's facial expressions in real time, such as joy, surprise, and confusion, and transmitting these expression data to the central processing module 2 for further processing.

[0022] Modeling module 4 refers to the component in the system used to build and update the user's virtual model. Based on the user's facial expression data identified by vision module 3, module 4 dynamically generates or adjusts the user's image in the virtual environment, enabling the virtual model to reflect the user's real expressions in real time, thereby enhancing the realism of virtual interaction.

[0023] The main features of the above technical solution will be explained in more detail below: This embodiment provides a VR-based education and training system, the core of which lies in enhancing the interactivity and realism of online education and training through the collaborative work of multiple modules. First, client 1 serves as the entry point for user interaction with the system, and its implementation can be diverse. For example, client 1 can be a dedicated VR headset pre-installed with the system application, allowing users to directly launch the system and log in to their accounts. Alternatively, client 1 can be a general-purpose computer or mobile device, accessing the system by installing specific applications or accessing a web-based interface. When logging in, users can manually enter their identity credentials, and after verification, the system allows users to enter the virtual training environment.

[0024] Central Processing Module 2 acts as the data hub in this system. It can be configured as a high-performance server cluster, responsible for receiving, storing, and processing various types of data from Client 1, Vision Module 3, and Modeling Module 4. For example, when a user operates on Client 1, relevant instructions and data are sent to Central Processing Module 2 for parsing and response. Central Processing Module 2 performs logical judgments and calculations on this data and, according to preset rules or algorithms, sends corresponding control signals or data to other modules.

[0025] The vision module 3 is a crucial component for user state perception. This module typically integrates a high-resolution camera capable of capturing real-time images of the user's face. Through built-in image recognition algorithms, the vision module 3 analyzes the captured facial images to identify the user's facial expressions, such as determining whether the user is currently focused, relaxed, or confused. The identified facial expression data is then encapsulated and transmitted to the central processing module 2 via a data interface. For example, the vision module 3 can periodically acquire user facial images and compare them with preset expression templates to determine the current expression state.

[0026] Modeling module 4 is responsible for building and maintaining the user's virtual avatar in the virtual environment. This module 4 works closely with visual module 3, dynamically adjusting the user's virtual model's facial expressions based on the facial expression data identified by visual module 3. For example, modeling module 4 can preset various basic virtual models; when visual module 3 recognizes a "smiling" expression from the user, modeling module 4 immediately drives the virtual model to produce the corresponding smiling expression. This linkage mechanism ensures that the user's avatar in the virtual environment is no longer static but reflects their true emotional state in real time, significantly enhancing the realism and immersion of virtual interaction.

[0027] The following example will provide a more detailed explanation of the above technical solution: Suppose User A wants to participate in a virtual skills training program through this system. First, User A logs into their personal account through Client 1. Client 1 can be the VR headset worn by User A, which has the system's application pre-installed. After successful login, User A enters a virtual training scenario. At this point, Central Processing Module 2 begins coordinating the work of various modules. Visual Module 3 captures User A's facial image in real time using the VR headset's built-in camera. For example, when User A encounters a problem during training, their face may show a "confused" expression. After recognizing this expression, Visual Module 3 immediately transmits the "confused" expression data to Central Processing Module 2. Upon receiving this data, Central Processing Module 2 forwards it to Modeling Module 4. Based on the received "confused" expression data, Modeling Module 4 adjusts the facial expression of User A's virtual model in the virtual scenario in real time, causing the virtual model to also exhibit a "confused" state. Simultaneously, Central Processing Module 2 may also adjust the feedback mechanism in the virtual training scenario based on changes in User A's facial expressions, such as providing additional prompts or adjusting the difficulty of the training content. As a result, user A's true emotional state is mapped onto the virtual environment in real time, allowing user A to more intuitively feel their presence in the virtual world and interact more naturally with other virtual characters or systems. The entire process achieves a closed loop of real-time capture of user expressions, data processing, dynamic updating of the virtual model, and system feedback, greatly enhancing the interactivity and realism of the training.

[0028] Based on the above examples, this system significantly enhances the interactivity and realism of the VR education and training system by introducing a linkage mechanism between the visual module 3 and the modeling module 4. Traditional online education and training systems, such as those mentioned in the background technology, often rely solely on simple motion capture or pre-set virtual avatars for interaction, lacking real-time perception and feedback of the user's deep emotional state. For example, in existing technologies, even if a student is confused, the system struggles to directly detect and make targeted adjustments, limiting the training effectiveness. This system, however, differs. The visual module 3 identifies user A's facial expressions in real time, and the modeling module 4 dynamically updates user A's virtual model, allowing user A's virtual avatar to synchronously reflect their real emotions. This real-time, dynamic expression mapping makes user A's experience in the virtual environment more immersive and realistic, as if they were in a real classroom that understands their emotions. Furthermore, the processing and coordination of this expression data by the central processing module 2 provides a data foundation for subsequent, more intelligent teaching interventions. Compared to the problems of insufficient interaction and poor realism in existing technologies, this system effectively solves these technical challenges through the innovative combination of facial expression recognition and dynamic linkage with virtual models, providing users with a richer and more personalized VR education and training experience.

[0029] In some of the solutions mentioned above in this application, a visual module is proposed to recognize the user's facial expressions. However, the lack of monitoring of the user's body temperature and ambient temperature in this process makes it impossible to fully detect the user's state. For example, the user's physiological changes or the influence of environmental factors are not covered, which affects the authenticity of the training, interactivity, and the effectiveness of preventing AFK behavior.

[0030] In this regard, this application further proposes that it also includes a temperature acquisition module 5, which is used to monitor the user's body temperature and ambient temperature data; the output end of the temperature acquisition module 5 is connected to the central processing module 2, which is used to transmit the user's body temperature and ambient temperature data to the central processing module 2.

[0031] The temperature acquisition module 5 is a hardware unit used to acquire temperature information. Its concept involves converting temperature signals into electrical signals using physical sensors for subsequent processing. One possible implementation is a contact sensor, such as a thermistor, thermocouple, or resistance temperature detector (RTD), which directly contacts the user's skin or is placed in the environment to acquire temperature data. Another possible implementation is a non-contact sensor, such as an infrared temperature sensor, which infers the temperature of an object by measuring its infrared radiation, thus enabling remote monitoring of the user's body or ambient temperature. This module's function is to acquire key indicators of the user's physiological state and external environmental conditions in real time, providing the system with multi-dimensional data input to more comprehensively assess user engagement, comfort, and the presence of any abnormalities. For example, monitoring the user's body temperature can reflect physiological responses such as tension, fatigue, or discomfort; monitoring the ambient temperature can assess whether the learning environment is suitable, preventing user distraction or discomfort due to excessive heat or cold. The output of the temperature acquisition module 5 is connected to the central processing module 2, ensuring that the temperature data acquired by the module is effectively transmitted to the system's core processing unit. The connection can be a physical wired connection, such as through serial communication interfaces like I2C, SPI, and UART, or directly connected to the ADC (analog-to-digital converter) input of the central processing module 2 via analog signal lines.

[0032] Alternatively, wireless connections can be used, such as via Bluetooth, Wi-Fi, or Zigbee, to send data to the central processing module 2 for more flexible deployment. Transmitting user body temperature and ambient temperature data to the central processing module 2 is a crucial step in data integration and comprehensive analysis. Its function is to deliver raw temperature data to the central processing module 2, enabling it to be fused with other data (such as facial expression data recognized by the vision module 3). The transmission process can use a standard data packet format, including data type identifiers, timestamps, and specific temperature values, ensuring data integrity and accuracy. After receiving this data, the central processing module 2 can store, analyze, and compare it with preset thresholds, triggering corresponding system responses based on the analysis results.

[0033] By introducing the temperature acquisition module 5, the system can more comprehensively perceive the user's state. Specifically, the temperature acquisition module 5 is responsible for monitoring the user's body temperature and ambient temperature data in real time. This temperature data is transmitted to the central processing module 2 via a wired or wireless connection through the output of the temperature acquisition module 5. After receiving this temperature data, the central processing module 2 integrates and analyzes it with the user's facial expression data recognized by the vision module 3. For example, when the vision module 3 detects that the user's facial expression is inactive, the central processing module 2 will further combine the user's body temperature and ambient temperature data provided by the temperature acquisition module 5. If the user's body temperature rises or falls abnormally, or the ambient temperature exceeds the comfortable range, the central processing module 2 can more accurately determine whether the user is experiencing physiological discomfort, environmental influences, or simply lack of concentration or being idle. This multi-dimensional data fusion analysis enables the system to make more refined judgments about the user's true state, thereby avoiding misjudgments that may result from relying on a single data source. In this way, the system can not only identify whether users are learning effectively, but also issue timely warnings or adjust teaching content when users experience physical or environmental discomfort, thereby significantly improving the realism and interactivity of education and training, and effectively curbing idle behavior.

[0034] In one specific implementation, the temperature acquisition module 5 may include a non-contact infrared temperature sensor, such as the MLX90614, for measuring the user's forehead body temperature within a certain distance, and a digital temperature sensor, such as the DS18B20, for measuring the ambient temperature of the client 1. Both the infrared and digital temperature sensors are connected to a microcontroller (as part of the temperature acquisition module 5) via an I2C bus or a single-bus protocol. The microcontroller preprocesses and calibrates the acquired raw temperature data, then encapsulates the processed user body temperature data and ambient temperature data into data packets via a UART interface or Wi-Fi module, and periodically sends them to the central processing module 2. After receiving this data, the central processing module 2 can store it in a database and synchronize it with the video stream captured by the vision module 3 for joint analysis. For example, when the central processing module 2 detects through the vision module 3 that the user has been expressionless for an extended period, it will immediately query the corresponding temperature data. If the user's body temperature remains high at this time, the system can infer that the user may be fatigued or unwell, rather than simply hanging up, and adjust subsequent interaction strategies accordingly.

[0035] Through the above technical solution, this system overcomes the limitations of traditional VR education systems in user status detection. By introducing the temperature acquisition module 5, the system can acquire user body temperature and ambient temperature data in real time and transmit them to the central processing module 2 for comprehensive analysis. This allows the central processing module 2 to assess user status not only based on facial expressions recognized by the vision module 3, but also by combining physiological and environmental factors, thus achieving a more comprehensive and accurate evaluation of user status. This multi-dimensional data fusion significantly improves the system's ability to identify user engagement, comfort, and potential idle behavior, effectively avoiding misjudgments caused by a single data source. Ultimately, this solution significantly enhances the realism and interactivity of VR education and training, and more effectively prevents user idle behavior, thereby improving the overall training effect.

[0036] In some of the solutions mentioned above in this application, an account login function is proposed for users to access the system. However, in this process, the account type is singular, lacks multi-level management, and cannot effectively support collaborative work between different roles, resulting in insufficient system interactivity and management efficiency. In response, this application further proposes that accounts include management accounts, teaching accounts, and supervisory accounts, which work together to enable users to log in on client 1.

[0037] In this system, an account serves as the credential for user authentication and access control. Accounts can be implemented in various ways, such as using traditional usernames and passwords, third-party authorization (e.g., OAuth), or biometric technologies (e.g., fingerprint or facial recognition). Management accounts have the highest privileges and are primarily used for system configuration and management. They can be used for global system settings, user permission allocation, data maintenance, and system upgrades. Management accounts can be created and assigned by pre-defined administrators or by specifying the first administrator account during system initialization. Teaching accounts are designed specifically for educational and training activities. Their main functions include participating in learning, engaging in interactive teaching, and creating and managing teaching content. Teaching accounts can be created and assigned to teachers or students by management accounts, or users can register themselves and obtain teaching permissions after system review. Supervisory accounts are used to observe and monitor the educational and training process. Their main function is to view learning progress, user behavior, and training effectiveness evaluations in real time, but they do not directly participate in interactive teaching. Supervisory accounts can be created and assigned to parents, supervisors, or quality assessors by the management account, or temporary access can be granted through a specific link or authorization code. The management account, teaching account, and supervisory account work together to enable user login on client 1. This means that different types of users can securely and effectively access the system through client 1 based on their roles and permissions. Specifically, client 1 can send different login requests to central processing module 2 based on the user's entered account type. Central processing module 2 then verifies permissions and establishes a session based on the account type. Alternatively, client 1 can provide a unified login interface where the user enters credentials, and central processing module 2 identifies the account type and assigns corresponding permissions.

[0038] This application's solution introduces a multi-level account management mechanism, enabling VR-based education and training systems to support refined management and collaborative work among users with different roles. When a user attempts to log in to the system through client 1, their entered account information is transmitted to central processing module 2. Central processing module 2 is responsible for authenticating the account and identifying its type: management, teaching, or supervisory. Based on the identified account type, central processing module 2 assigns the user corresponding system permissions and access scope. For example, management accounts are authorized to access the system configuration interface and user management functions, teaching accounts are authorized to access teaching content and interactive tools, and supervisory accounts are authorized to access real-time monitoring and data reporting functions. This mechanism ensures that each user role can only access functions within their scope of responsibility, thus achieving effective isolation and management of permissions. In this way, the system not only provides basic login functionality but also builds a layered and orderly access control system, allowing client 1 to perform differentiated data interactions and function calls with central processing module 2 based on the user's different identity, greatly improving the system's management efficiency and security.

[0039] In one specific implementation, within a VR-based medical training system, users can log in via client 1. When a system administrator logs in using an administrative account, client 1 displays a system management interface, allowing them to create new VR surgical simulation courses, register new trainees (teaching accounts), or set observation permissions for hospital management (supervisory accounts). These operational instructions are sent from client 1 to central processing module 2 for processing and storage. When a trainee logs in using a teaching account, client 1 presents a VR surgical simulation environment where the doctor can perform virtual surgical procedures and interact with virtual patients. At this time, vision module 3 can recognize the doctor's facial expressions, and modeling module 4 generates a virtual model of the doctor based on their actions and expressions. All this data is coordinated and processed by central processing module 2. Simultaneously, when a hospital administrator logs in using a supervisory account, client 1 displays a real-time monitoring interface, allowing them to observe the trainee's operational process and performance in the VR surgical simulation, as well as the facial expression data recognized by vision module 3 and the user's virtual model image generated by modeling module 4, but without directly intervening in the training process. This multi-account collaborative login mechanism enables users with different roles to securely and effectively access the system through client 1 according to their permissions and responsibilities, and to interact with the central processing module 2 to achieve their respective functional requirements. Similarly, when the system is applied to preschool education, it can effectively capture children's facial expressions and body movements, ensuring the quality of teachers' teaching while ensuring the safety of children.

[0040] Through the aforementioned technical solution, this application effectively addresses the problems of traditional online education and training systems, such as the lack of diverse account types and multi-level management. By introducing management accounts, teaching accounts, and supervisory accounts, the system enables refined permission management and functional division for users with different roles. This not only ensures the security of system operations and prevents unauthorized access but also significantly improves the system's management efficiency and flexibility. Users with different roles can log in and access corresponding functional modules through client 1 according to their responsibilities, thereby achieving more efficient collaborative work. For example, administrators can focus on system maintenance and resource allocation, instructors can focus on course design and teaching interaction, and supervisors can focus on evaluating and providing feedback on training effectiveness. This clearly defined division of labor and clear permissions mechanism allows the entire VR education and training system to operate more orderly, significantly enhancing the system's interactivity and overall management capabilities, thereby effectively improving the quality and efficiency of education and training.

[0041] In some of the solutions mentioned above in this application, a management account is proposed to enable user login. However, in this process, the management account lacks specific configuration functions and cannot set and manage parameters of system modules, resulting in inconvenient system management and the inability to dynamically adjust module parameters to optimize the education and training process.

[0042] In this regard, this application further proposes that the management account is a system configuration account that logs in through client 1 and connects to central processing module 2, and is used to send system configuration instructions 19 to central processing module 2 to set and manage parameters for client 1, central processing module 2, vision module 3 or modeling module 4.

[0043] Specifically, the management account is defined as a system configuration account, which means it is granted the core permissions for system configuration and management. Its main responsibility is to maintain the normal operation of the system and optimize various functional parameters. For example, specific permission identifiers such as "admin" or "system_configurator" can be set for this account in the user database to distinguish its functions from other types of accounts; alternatively, this account can be logged in and operated through an independent management background or console interface, which is only open to accounts with system configuration permissions. The management account logs in through Client 1, ensuring that all management operations are carried out through a unified user interface entry, thereby improving the convenience and security of operations. Client 1 can be an application program interface in a VR headset device, and the administrator logs in by entering credentials through a virtual keyboard or handle in the VR environment; or, Client 1 can also be a PC or mobile application program配套的VR系统, and the administrator remotely logs in and manages the system through this application program. After logging in, the management account establishes a connection with the central processing module 2. This connection establishes a data transmission channel between the management account and the system core processing unit, ensuring that configuration instructions can be conveyed efficiently and accurately. This connection can be a network connection established through the TCP / IP protocol, where Client 1 acts as the client and the central processing module 2 acts as the server for data communication; or it can be asynchronous communication through a message queue mechanism. Client 1 sends configuration messages to the queue, and the central processing module 2 reads and executes them from the queue. The management account is used to send system configuration instructions 19 to the central processing module 2, allowing the administrator to dynamically issue instructions and adjust various settings and parameters of the system in real time. The system configuration instructions 19 can be in a structured data format, such as JSON or XML, which contains instruction types (such as "set parameter", "enable module") and specific parameter values; or, it can also be a predefined API call. Client 1 executes configuration operations by calling the API interface provided by the central processing module 2 and passing the corresponding parameters. These instructions are designed to set and manage parameters for Client 1, the central processing module 2, the vision module 3, or the modeling module 4, thereby specifically optimizing the parameter configuration of each module to ensure the coordinated operation of system components and improve the stability and adaptability of the education and training process. Parameter settings can include adjusting the display resolution of Client 1, the computing resource allocation of the central processing module 2, the facial recognition sensitivity of the vision module 3, the virtual model fineness of the modeling module 4, etc.; management can include enabling or disabling specific module functions, updating module firmware, viewing module operating status, adjusting the communication protocol between modules, etc.

[0044] The proposed solution explicitly defines the management account as a system configuration account and grants it the ability to log in via client 1 and connect to the central processing module 2, thereby enabling it to send system configuration commands 19 to the central processing module 2. The central processing module 2, as the core of the system, receives and processes these commands, thereby setting and managing parameters for client 1, the central processing module 2 itself, the vision module 3, or the modeling module 4. This mechanism transforms the management account from merely a login credential into one capable of fine-grained control and dynamic adjustment of the entire VR education and training system. For example, the administrator can adjust the facial expression recognition algorithm parameters of the vision module 3 in real time based on teaching needs or system operation status to improve recognition accuracy; or adjust the level of detail in the virtual models generated by the modeling module 4 to balance visual effects and system performance. This centralized configuration management approach ensures that all modules of the system can work collaboratively and can be flexibly optimized according to actual application scenarios, significantly improving the system's customizability and management efficiency.

[0045] In one specific implementation, the administrator can wear a VR headset and log in to their management account through a VR client application. This VR client application establishes a secure WebSocket connection with the central processing module 2 deployed on a cloud server. In the VR environment, the administrator operates a virtual control panel using virtual controllers, selects the "System Settings" option, and further selects "Visual Module Parameter Adjustment." At this point, the administrator can adjust a slider to change the facial expression recognition threshold of the visual module 3, for example, from the default value of 0.7 to 0.85, to adapt to different users' facial features or lighting conditions. After the administrator confirms the adjustment, the VR client 1 generates a system configuration instruction 19, which is encapsulated in JSON format, containing the module identifier (e.g., "visual_module_3"), parameter name (e.g., "recognition_threshold"), and the new parameter value (e.g., "0.85"), and sends it to the central processing module 2 via the WebSocket connection. Upon receiving the instruction, the central processing module 2 verifies its validity and updates the corresponding configuration parameters of the visual module 3. Similarly, administrators can also send commands to adjust the texture loading strategy of modeling module 4 or modify the rendering priority settings of client 1 to optimize the user experience.

[0046] Through the above technical solution, this application effectively solves the problems of inconvenient system management and inability to dynamically adjust module parameters caused by the lack of specific configuration functions and the inability to set and manage parameters of system modules in traditional management accounts. The management account is given the role of a system configuration account, enabling it to log in through client 1 and connect to the central processing module 2, thereby sending system configuration commands 19 to set and manage parameters for client 1, central processing module 2, vision module 3, or modeling module 4. This allows system administrators to flexibly and dynamically adjust the operating parameters of each module according to actual needs, significantly enhancing the system's customizability, flexibility, and management efficiency. Ultimately, this helps optimize the VR education and training process, providing a more stable and adaptable user experience.

[0047] In some of the solutions mentioned above in this application, a supervisory account was proposed to cooperate with the management account and the teaching account to realize user login. However, in this process, the supervisory account was not designed as a passive observation role and could not receive and display user facial expression data and virtual model screen in real time, which resulted in the inability to effectively monitor user status and detect idle behavior, thereby affecting the authenticity and interactivity of the training.

[0048] In this regard, this application further proposes that the supervision account is a passive observation account; when the supervision account is logged in through the client 1, the supervision account establishes a data connection with the central processing module 2 to receive and display the user's facial expression data forwarded by the central processing module 2 and identified by the vision module 3, as well as the user's virtual model screen generated by the modeling module 4.

[0049] The supervisory account is defined as a passive observation account, meaning it is primarily used for monitoring and observation and does not have the authority to actively operate or intervene in the training process. For example, when creating such an account, the system can configure it with read-only permissions, restricting any modifications it may make to system settings, user profiles, or teaching content; or, after client 1 detects the supervisory account login, it can only activate the data receiving and display functions, while disabling all interface elements that may lead to active interaction, such as interactive buttons or speaking permissions in a virtual scene.

[0050] This operation is a prerequisite for triggering the monitoring function when logging into the monitoring account through client 1. Specifically, after receiving the login credentials (such as username and password) of the monitoring account, client 1 will verify them through the built-in security authentication module. If the verification is successful, the login process is completed; or, client 1 will send the login request to the central processing module 2, which will verify the validity of the account type and credentials and return a login success instruction to client 1.

[0051] Once logged in successfully, the supervisory account establishes a data connection with the central processing module 2. This data connection can be a persistent network socket connection established by client 1 with the central processing module 2 via the TCP / IP protocol to ensure real-time and stable data transmission; or it can be a bidirectional communication connection established by client 1 with the central processing module 2 using the WebSocket protocol to support low-latency data stream transmission.

[0052] Once the data connection is established, it is primarily used to receive and display data forwarded by the central processing module 2. This data includes user facial expression data identified by the vision module 3 and user virtual model images generated by the modeling module 4. For receiving data, client 1 can have a built-in data parsing module to parse the specific data packet format sent by the central processing module 2, extracting facial expression data and virtual model image data; alternatively, client 1 can subscribe to the data stream service provided by the central processing module 2 to acquire and cache this data in real time. For displaying data, client 1's display interface can include one or more display areas for real-time rendering of the received user virtual model images, displaying the identified facial expression states next to the image or in an overlay format, such as in text form (e.g., "focused," "confused," "happy") or icon form; alternatively, client 1 can use its graphics rendering engine to decode and render the received virtual model image data onto a VR headset or monitor, while simultaneously displaying the user facial expression information analyzed by the vision module 3 through text boxes or graphical indicators.

[0053] The proposed solution sets the supervisory account as a passive observer and establishes a data connection with the central processing module 2 upon login. This allows for real-time reception and display of user facial expression data recognized by the vision module 3 and the user virtual model image generated by the modeling module 4. Specifically, when a supervisor logs into their supervisory account through client 1, the system recognizes the account's passive observation attribute and prompts client 1 to establish a stable data transmission channel with the central processing module 2, which serves as the data processing hub. The central processing module 2 is responsible for aggregating the user facial expression recognition results from the vision module 3 and the real-time user virtual model image from the modeling module 4. The vision module 3 continuously monitors the user's facial features and transmits their expression state data to the central processing module 2; simultaneously, the modeling module 4 updates and generates the user virtual model image in real time based on the user's behavior and system instructions, and also transmits it to the central processing module 2. Upon receiving this multi-dimensional data, the central processing module 2 integrates and forwards it, sending it to client 1, where the supervisory account resides, through the established data connection. After receiving this data, client 1 decodes it and presents it intuitively on the display interface. This allows supervisors to observe the changes in the trainees' facial expressions and the actions of the virtual model in real time and comprehensively, thereby judging the users' focus, emotional state, and whether they are idling. This mechanism enables supervisors to effectively monitor the training process without directly entering the virtual scene for interaction, ensuring the authenticity and effectiveness of the training.

[0054] In one specific implementation, the supervisor can log in using a client 1 equipped with a VR headset or high-resolution display by entering a preset supervisory account and password. After successful system verification, client 1 immediately establishes a real-time data stream connection with the central processing module 2 deployed on a cloud or local server, based on WebRTC or a custom streaming media protocol. The central processing module 2 continuously receives image streams captured by the user's facial camera from the vision module 3 integrated with the user's VR device, and analyzes the user's facial expressions in real time, identifying emotion tags such as "focused," "fatigued," and "confused." Simultaneously, the central processing module 2 also obtains real-time posture and movement images of the current user's virtual model in the virtual scene from the modeling module 4. The central processing module 2 packages these expression tags and virtual model images into a data stream and forwards it to client 1, where the supervisory account is located, through the previously established data connection. Upon receiving the data stream, client 1 decodes it and displays the user's virtual model in the virtual scene in a picture-in-picture or split-screen format on the VR headset or display, updating the user's current facial expression in a corner of the screen or as a text prompt, such as "Current expression: focused." In this way, supervisors can intuitively see users' behavior and emotional reactions in the virtual environment, thus enabling effective supervision.

[0055] Through the above technical solution, this application effectively solves the problem that existing supervisory accounts cannot receive and display user facial expression data and virtual model images in real time, resulting in an inability to effectively monitor user status and detect idle behavior. Specifically, the supervisory account is clearly defined as a passive observation role, avoiding interference from supervisors in the training process and ensuring the purity of the training. Simultaneously, after logging into the supervisory account through client 1, a data connection is established with the central processing module 2, receiving and displaying user facial expression data recognized by the vision module 3 and user virtual model images generated by the modeling module 4. This allows supervisors to obtain real-time, multi-dimensional status information of the trained users. This not only helps supervisors promptly detect decreased user focus, emotional abnormalities, or idle behavior, enabling necessary intervention or recording, but also significantly improves the transparency and authenticity of the training process, thereby enhancing the interactivity and effectiveness of the education and training system. In some of the solutions mentioned above in this application, a supervisory account is proposed to receive and display user facial expression data and virtual model images. However, in its implementation, the teaching account is not clearly defined in terms of how to implement the learning function. Users cannot start learning immediately after the virtual model is built, resulting in a delayed learning process, insufficient interactivity, and affecting the authenticity and efficiency of the training.

[0056] In response, this application proposes a VR-based education and training system, in which the teaching account is a learning account. When a user builds their own virtual model through the management account, a data connection is established with the central processing module 2 to achieve real-time learning.

[0057] Specifically, the teaching account here refers to the role account used by users for learning activities in the VR education and training system. This account can be explicitly designated as a "learner" or "student" role during system registration, with corresponding access permissions for learning functions; alternatively, it can be achieved by upgrading the permissions or assigning roles to existing user accounts, granting them the specific functions and data access capabilities required for learning. This clarifies the core functions and scope of permissions of the account, allowing it to focus on learning tasks and distinguishing it from other management or supervisory roles, thereby optimizing system resource allocation and user experience. Furthermore, the condition "when the user has built their own virtual model through the management account" serves as a trigger mechanism for starting the learning activity, ensuring that the learning environment or resources are ready and preventing users from starting learning without completing the necessary setup, thus improving learning efficiency and experience. The system can set an internal status flag. When the management account completes the construction of the virtual model and performs a save or publish operation, this flag is updated to "Model Ready," allowing the teaching account to start learning. Alternatively, after completing the construction of the virtual model, the management account can send a specific "Model Deployment Complete" command to the central processing module 2. Upon receiving this command, the central processing module 2 will activate the learning functions related to the model for the teaching account to use. Furthermore, "Establishing a data connection with the central processing module 2" is fundamental to achieving real-time data interaction between the teaching account and the system's core processing unit, ensuring that command transmission, data feedback, and scene updates during the learning process can be performed instantly. This connection can be achieved through a TCP / IP-based network socket connection, providing a stable and reliable data transmission channel; alternatively, the WebSocket protocol can be used to establish full-duplex communication between client 1 and central processing module 2 to support more efficient, low-latency real-time interaction. Ultimately, "Achieving real-time learning" means that users can participate in learning activities instantly in the virtual environment, and the system can respond instantly to user operations and inputs, providing immediate feedback, thereby creating an immersive and highly interactive learning experience. Through continuous data connection, the central processing module 2 can receive operation instructions from the teaching account in the virtual scene in real time (such as grabbing, moving, and experimental operations of virtual objects), process these instructions in real time, update the virtual scene status, and feed the updated data back to the client 1. At the same time, combined with the user's facial expression data recognized by the vision module 3 and the virtual model screen generated by the modeling module 4, the central processing module 2 can dynamically adjust the learning content or provide personalized guidance based on the user's real-time behavior and expressions, ensuring the immediacy and interactivity of the learning process.

[0058] This application's solution explicitly defines the teaching account as the learning account and sets "users building their own virtual model through the management account" as a clear prerequisite for starting learning, ensuring that learning activities can begin immediately once the virtual environment is ready. Once the condition is met, the teaching account establishes a stable data connection with the central processing module 2. The central processing module 2, as the core of the system, is responsible for processing all learning interaction data from client 1, including user operations and feedback within the virtual model, as well as real-time updates to the learning content. This mechanism allows users to seamlessly transition from the completed virtual model setup to an immersive, real-time learning experience, avoiding the learning interruptions and delays caused by waiting or manual operation in traditional systems. For example, after the management account completes the setup of a virtual laboratory, the teaching account user can immediately log in and enter the virtual laboratory, interacting in real-time with the central processing module 2 through the data connection, conducting experiments, observing results, and receiving immediate feedback. This tight integration and instant response capability greatly enhances the smoothness and interactivity of the learning process, enabling the VR education and training system to provide a more realistic and efficient learning experience.

[0059] As a specific implementation, in a VR education and training scenario, suppose a teacher (using an administrator account) builds a complex virtual biological anatomy model on client 1 using modeling module 4, and sets corresponding learning tasks and assessment points. Once the teacher completes and saves the model, the system sends a "model ready" notification to central processing module 2. At this time, a student (using a teaching account, i.e., a learning account) logs into client 1. Client 1 detects that the virtual model is ready and automatically establishes a data connection with central processing module 2 based on the WebSocket protocol. Through this connection, the student can immediately enter the virtual anatomy model and perform real-time virtual dissection operations. Central processing module 2 receives the student's operation instructions in real time (e.g., using a virtual scalpel to cut or observe organ structures), processes these instructions instantly, updates the virtual model's status, and transmits the updated image data stream back to client 1 for the student to view in the VR headset. Simultaneously, vision module 3 recognizes the student's facial expressions in real time and sends the expression data to central processing module 2. Central processing module 2 can judge the student's level of understanding or emotional state based on their expressions and adjust the difficulty of the learning content or provide supplementary information accordingly. This instant data interaction and feedback mechanism enables students to have a highly immersive and personalized real-time learning experience.

[0060] Through the above technical solution, this application effectively solves the problem that users cannot start learning immediately after the virtual model is built, eliminating delays and interruptions in the learning process. This instant-start learning mode significantly enhances the interactivity between users and the virtual environment, making the learning experience smoother and more natural. Users can immediately participate in learning activities and receive instant feedback, thereby greatly improving the realism and learning efficiency of VR education and training.

[0061] In some of the solutions mentioned above in this application, a VR-based education and training system was proposed to solve the problems of insufficient interaction, poor realism, and frequent user idling. However, during the deployment and execution of the system, the lack of built-in integration of the system onto electronic devices resulted in poor portability, complex deployment, and inconvenience for users, thus affecting the training effect.

[0062] In response, this application proposes a VR-based education and training system, which is integrated into an electronic device. "Integrated" means that a system or its main functional modules, such as client 1, central processing module 2, vision module 3, modeling module 4, and optional temperature acquisition module 5, are physically integrated into a single hardware carrier, forming a self-contained whole that does not require additional connections to multiple external devices. This integration method aims to simplify the system's structure and deployment. "Electronic device" refers to a hardware platform capable of supporting and running the VR education and training system. This electronic device can be a dedicated VR all-in-one machine, integrating all necessary components such as display, computing, and sensing; or it can be a specially customized smartphone or tablet, achieving VR functionality through external or built-in VR accessories, and storing the system's core processing logic and data internally. Furthermore, this electronic device can also be a portable computer or a dedicated educational terminal, designed to achieve high integration and mobile use.

[0063] This solution physically integrates core functional modules such as Client 1, Central Processing Module 2, Vision Module 3, Modeling Module 4, and Temperature Acquisition Module 5 into a single electronic device, thus constructing a highly integrated VR education and training system. In this structure, Client 1 serves as the user interface, running directly on the electronic device; Central Processing Module 2 handles all data processing and system logic execution, with its computing resources also embedded within the electronic device; Vision Module 3 recognizes user facial expressions, its sensors (such as cameras) are directly integrated into the electronic device, and it transmits data with the built-in Central Processing Module 2; Modeling Module 4 builds the user's virtual model, its processing power also provided by the electronic device's internal computing resources, and it works in conjunction with Vision Module 3. Furthermore, the sensors in Temperature Acquisition Module 5 are also integrated into the electronic device to monitor user body temperature and ambient temperature data, transmitting the data directly to the built-in Central Processing Module 2. This integrated design eliminates the need for complex external connections and deployment processes in the entire VR education and training system. Users do not need to connect multiple devices, configure complex networks, or install multiple software components; they can start and use the system simply by operating a single electronic device. For example, the login and functionality of management, teaching, and supervisory accounts can all be seamlessly integrated on this single electronic device. Management accounts can easily set and manage system parameters; supervisory accounts can directly receive and display user facial expression data and virtual model images; and teaching accounts can also achieve real-time learning on this device. This tight integration not only reduces the physical distance and data transmission latency between system components, improving system response speed and stability, but more importantly, it greatly enhances the system's portability and ease of use. This allows VR education and training to break through the limitations of traditional classrooms, enabling it to be conducted anytime, anywhere, effectively solving the problems of poor portability, complex deployment, and inconvenience for users associated with traditional systems.

[0064] As a specific implementation, the aforementioned VR education and training system can be integrated into a single VR all-in-one device. This VR all-in-one device integrates a high-performance processor (as central processing module 2), memory, storage units, and a high-resolution display and optical system (as the display portion of client 1). The vision module 3 can be implemented using a front-facing camera or eye-tracking camera built into the VR all-in-one device to capture the user's facial expression data. The modeling module 4 utilizes the computing resources within the VR all-in-one device to generate or update the user's virtual model in real time based on the data captured by the vision module 3 and preset algorithms. If the system also includes a temperature acquisition module 5, a temperature sensor can be integrated inside the VR all-in-one device or at a location in contact with the user's skin to monitor the user's body temperature and ambient temperature in real time, transmitting the data to the built-in central processing module 2 for processing. By wearing this VR all-in-one device, users can directly log in to their management, teaching, or supervision accounts to configure the system, learn in real time, or observe, without needing to connect to an external computer or complex cables, achieving a high degree of integration and portability.

[0065] By integrating the VR education and training system into a single electronic device, the aforementioned technical solution significantly enhances the system's portability and ease of deployment. Users can carry and use the device anytime, anywhere for education and training without complex installation and configuration, significantly lowering the barrier to entry and improving the user experience. This integrated design allows all functional modules, including the client 1, central processing module 2, vision module 3, modeling module 4, and temperature acquisition module 5, to work closely together, resulting in higher data transmission efficiency and more stable and reliable system operation. Especially with the integration of multi-level account management, user facial expression recognition, and temperature monitoring, this highly integrated electronic device provides a more immersive, realistic, and efficient VR education and training experience, effectively preventing user idleness and comprehensively improving training effectiveness.

[0066] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

[0067] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A VR-based education and training system, comprising: The client (1) is used to log in to an account; characterized in that it further includes: a central processing module (2) for processing data; a visual module (3) for recognizing user facial expressions and connected to the central processing module; and a modeling module (4) for building a user virtual model and linked with the visual module.

2. The VR-based education and training system as described in claim 1, characterized in that, It also includes a temperature acquisition module (5) for monitoring user body temperature and ambient temperature data; the output of the temperature acquisition module is connected to the central processing module for transmitting the user body temperature and ambient temperature data to the central processing module.

3. The VR-based education and training system as described in claim 1, characterized in that, The account includes a management account, a teaching account, and a supervision account. The management account, the teaching account, and the supervision account work together to enable the user to log in to the client (1).

4. The VR-based education and training system as described in claim 3, characterized in that, The management account is a system configuration account that logs in through the client (1) and connects to the central processing module (2) to send system configuration instructions (19) to the central processing module (2) to set and manage parameters for the client (1), the central processing module (2), the vision module (3) or the modeling module (4).

5. A VR-based education and training system as described in claim 4, characterized in that, The supervision account is a passive observation account; when the supervision account is logged in through the client (1), the supervision account establishes a data connection with the central processing module (2) to receive and display the user's facial expression data forwarded by the central processing module (2) and identified by the vision module (3) as well as the user's virtual model screen generated by the modeling module (4).

6. The VR-based education and training system as described in claim 5, characterized in that, The teaching account is a learning account. When a user builds their own virtual model through the management account, a data connection is established with the central processing module (2) to achieve real-time learning.

7. A VR-based education and training system according to any one of claims 1-6, characterized in that, Its collection is built into electronic devices.

Images