Extended reality (XR) medical learning system based on Miller pyramid medical evaluation model

Abstract

The invention relates to the technical field of medical education and training, in particular to an augmented reality (XR) medical learning system based on a Miller pyramid medical evaluation model. The system comprises a medical professional knowledge and operation theory layer, a medical knowledge application capability layer, a clinical operation presentation layer and a system evaluation and feedback mechanism, and provides a progressive learning path through technical means such as high-precision three-dimensional model generation, multi-person cross-space cooperation, real-time feedback and personalized learning. The system can significantly improve the learning intuition, interactivity and authenticity, solves the problems of limited resources, lack of systematicness and insufficient collaborative learning opportunities in traditional medical education, provides a comprehensive and systematic surgical operation training platform for medical students, and assists the comprehensive improvement of surgical skills and comprehensive quality.

Description

[0001] The present invention belongs to the field of medical education and training technology, and specifically is a system developed using extended reality technology for medical students to perform surgical operations and learn medical knowledge. Background Art

[0002] In the field of medical education and training, extended reality technology has garnered widespread attention for its ability to integrate virtual information with the real world. By creating a highly immersive virtual environment, this technology provides medical students with a safe and controllable surgical simulation platform, demonstrating significant potential in telemedicine, surgical simulation, and medical education. However, traditional medical education still suffers from numerous shortcomings in surgical training. For example, medical students primarily rely on two-dimensional medical images and physical models for learning, making it difficult to gain an intuitive and three-dimensional surgical experience. Existing systems lack a scientific and systematic assessment framework, preventing comprehensive evaluation of students' surgical skills. A lack of interactivity hinders comprehensive training in both virtual and real surgical environments. Furthermore, the limited availability of traditional surgical training resources, such as animals or donated cadavers, makes it difficult to meet the practical needs of a large number of medical students. Furthermore, traditional training methods struggle to simulate the complexities and emergencies of real surgical scenarios, and opportunities for collaborative learning among medical students are limited. These issues collectively hinder the effectiveness of medical education.

[0003] Currently, there are several medical surgical assistance systems based on mixed reality technology on the market, such as the MR telemedicine system developed by Beijing Jincheng Medical Technology Co., Ltd. and the medical surgical simulation system designed by Changzhou Jinse Medical Information Technology Co., Ltd. While these systems achieve certain capabilities, such as 3D visualization and cross-spatial collaboration, their functional modules are fragmented and lack systematic theoretical guidance, making it difficult to form a coherent learning path. Specifically, functional modules such as 3D visualization, cross-spatial collaboration, model manipulation, and testing in existing systems often exist as independent units, lacking deep integration and resulting in a fragmented learning experience. While mixed reality technology theoretically enables cross-spatial collaboration, existing systems' interaction design is still limited to basic operational instructions, lacking deep interaction and immediate feedback mechanisms, which hinders user engagement and learning outcomes. Furthermore, these systems fail to fully utilize learners' personal data and proficiency, failing to provide customized learning paths and difficulty adjustments for each user, thus ignoring the importance of personalized learning. Most systems are only applicable to specific scenarios and lack broad adaptability to diverse learning needs and environments, limiting their widespread and in-depth application in real-world settings.

[0004] In response to the above problems, the present invention aims to construct an extended reality medical learning system based on the Miller Pyramid medical evaluation model.

[0005] The system is designed with three progressive learning levels, from cognitive to behavioral, to provide medical students with solid medical professional knowledge and operational theory, the development of their ability to apply medical knowledge, and practical training in clinical performance. Using MR technology, the system presents human anatomy and surgical procedures in three dimensions, helping medical students gain a deeper understanding of the principles and procedures of surgical procedures. The system also features interactive quizzes and a highly realistic surgical planning simulation, gamifying students' ability to apply their knowledge. It also supports multi-person collaboration to simulate the workflow of a real surgical team. Furthermore, the system features recording and playback capabilities, allowing medical students to review surgical procedures and accumulate practical experience. This systematic learning path not only addresses the shortcomings of traditional medical education but also significantly improves medical students' surgical skills and overall quality, laying a solid foundation for future medical practice. Summary of the Invention

[0006] This paper addresses the challenges of existing medical education and training technologies, including lack of intuitiveness, systematicity, interactivity, limited resources, insufficient realism, and limited collaborative learning opportunities. By developing an extended reality (XR) medical learning system based on the Miller Pyramid medical assessment model, this system utilizes progressive learning pathways, combined with high-precision 3D model generation, multi-person cross-space collaboration, real-time feedback, and personalized learning, to provide medical students with a comprehensive and systematic surgical training platform.

[0007] The present invention provides an extended reality (XR) medical learning system based on the Miller Pyramid medical evaluation model, which includes a medical expertise and operation theory layer, a medical knowledge application capability layer, a clinical operation performance layer, and a system evaluation and feedback mechanism.

[0008] The medical expertise and operational theory layer uses mixed reality technology to construct a virtual operating room and human anatomical model, allowing students to observe three-dimensional anatomical structures and learn the basic knowledge and theories of surgical operations.

[0009] Furthermore, the medical knowledge application ability layer is provided with an interactive test module to examine students' understanding and application ability of medical knowledge in a gamified manner.

[0010] In particular, the clinical operation presentation layer provides a highly simulated surgical planning simulation module, supports multi-person collaboration, and simulates the workflow of a real surgical team.

[0011] In addition, the system evaluation and feedback mechanism records the students' surgical operation process in real time, conducts scientific evaluation based on the Miller Pyramid medical evaluation model, and provides personalized feedback and improvement suggestions.

[0012] The specific implementation of this medical expertise and operational theory layer is as follows: First, CT / MRI scan images are processed using 3D Slicer software to generate 3D models in STL or OBJ format, which are then imported into the Unity environment. Second, Level of Detail (LOD) technology is used to dynamically adjust the model's rendering quality, ensuring high performance at varying viewing distances.

[0013] Furthermore, gesture interaction functionality is configured through the MRTK toolkit, defining input methods for operations such as grabbing, rotating, and scaling. Furthermore, specific voice commands such as "zoom in," "zoom out," and "rotate" are defined and triggered through the SpeechInput Handler. Gesture interaction is managed by the Manipulation Handler component, which is bound to the medical model object and controls the model's movement, rotation, and scaling. Voice interaction is implemented through the Photon PUN plugin, enabling real-time data synchronization in multi-user scenarios.

[0014] The application layer of medical knowledge is implemented through an interactive quiz module. Its specific solution includes a backend database that stores question content, answers, and analysis information, and supports question tag classification and random selection. The frontend sends verification requests to the backend via Ajax or Fetch, providing real-time feedback on user operation results.

[0015] In particular, the system integrates collision detection and raycasting capabilities to identify user-clicked areas and provide instant feedback. Collision detection is achieved through raycasting technology. When a user clicks a specific organ, the system detects the click location and provides feedback through color changes, edge highlighting, or icon markings.

[0016] The core of the clinical operation presentation layer lies in a highly realistic surgical planning simulation module. Its implementation is as follows: First, a network framework is built using the Photon PUN plugin to achieve real-time synchronization between model status and user operations. Second, a color-changing aperture is added to the bottom of the 3D model, with individual color markers assigned based on user ID to distinguish different users' operating states.

[0017] Furthermore, we developed recording and playback features to capture user action data in real time, including gestures, viewpoints, and model interactions, and store this data in JSON format. The recording feature uses the PhotonTransformView component to capture changes in the model's position, rotation, and scale, and transmits these changes in real time to other user devices via an RPC mechanism. The playback feature extracts action records from this stored data and restores the user's actions in timestamp order, facilitating analysis of weaknesses.

[0018] The system's evaluation and feedback mechanism records user operation sequence and timing in a database, identifies weak areas, and generates learning reports. Specifically, the system records the user's click sequence and operation duration on the model, automatically marking organs with repeated errors or areas with prolonged operation time. Secondly, data analysis generates learning reports that showcase the student's performance and identify areas for improvement.

[0019] Furthermore, the system provides personalized feedback and suggestions based on the evaluation results to help students improve their skills in a targeted manner. The beneficial effect of the present invention is that it significantly improves the intuitiveness and interactivity of learning through high-precision 3D model generation and interactive technology, making up for the lack of intuitiveness in traditional teaching.

[0020] In particular, the multi-person cross-space collaboration feature enhances interaction and communication between medical students and between medical students and their instructors, promoting collaborative learning. Real-time feedback and personalized learning mechanisms, through data analysis and technical means, provide each user with a customized learning path and difficulty adjustment, addressing the lack of personalized learning. Furthermore, the system eliminates the need for limited traditional training resources such as animals or donated cadavers, reducing training costs. At the same time, through a highly realistic surgical simulation environment, it helps students accumulate practical experience and better cope with the complexities and emergencies of real surgical scenarios.

[0021] Furthermore, the present invention integrates multiple technical means to form a coherent learning path, progressing from the cognitive level to the behavioral level, providing medical students with a comprehensive and systematic learning platform. In particular, based on the design concept of the Miller Pyramid medical assessment model, the system is not only highly scientific and systematic, but also effectively cultivates students' self-reflection and autonomous learning abilities. Among them, the recording and playback function provides students with the opportunity to review the operation process, helping them to identify deficiencies and improve them, thereby comprehensively improving their surgical skills and overall quality.

[0022] In summary, the present invention solves the problems existing in the existing technology through technical means such as multi-level progressive learning path design, high-precision three-dimensional model generation and interaction, multi-person cross-space collaboration, real-time feedback and personalized learning, and provides medical students with a more intuitive, interactive, realistic and systematic learning environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 Schematic diagram of the overall architecture of the extended reality (XR) medical learning system based on the Miller Pyramid medical assessment model of the present invention;

[0024] Figure 2 A schematic diagram of the process of medical image processing and three-dimensional model generation in the present invention;

[0025] Figure 3 This is a schematic diagram of event synchronization and state management of the multi-person cross-space collaboration function of the present invention;

[0026] Figure 4 This is a schematic diagram of the implementation process of the test function and the recording and playback function of the present invention;

[0027] The accompanying drawings are marked as follows: 1. Medical professional knowledge and operation theory layer; 2. Application ability layer of medical knowledge; 3. Clinical operation performance layer; 4. System evaluation and feedback mechanism; 5. Medical image processing module; 6. 3D Slicer software; 7. ITK and VTK algorithms; 8. Unity environment; 9. LOD optimization module; 10. Gesture interaction module; 11. Voice interaction module; 12. Photon PUN plug-in; 13. PhotonTransformView component; 14. RPC function module; 15. Operation prompt aperture; 16. Back-end database; 17. Real-time feedback system; 18. Collision detection module; 19. Ray detection module; 20. Recording and playback module; 21. JSON data storage module. DETAILED DESCRIPTION

[0028] The present invention provides an extended reality (XR) medical learning system based on the Miller pyramid medical assessment model. Figures 1 to 4 The following content will explain the overall system architecture, medical image processing and 3D model generation process, the implementation of multi-person cross-space collaboration function, and the operating principles of the test function and recording and playback function.

[0029] The overall structure of the present invention is as follows Figure 1 As shown in the figure, it includes the medical professional knowledge and operation theory layer 1, the medical knowledge application ability layer 2, the clinical operation performance layer 3, and the systematic evaluation and feedback mechanism 4. These layers form a coherent learning path through a progressive design, ensuring that medical students gradually master surgical skills and medical knowledge from the cognitive level to the behavioral level.

[0030] The core of the medical expertise and operation theory layer 1 is to use mixed reality technology to build a virtual operating room and human anatomy model. The specific implementation process is as follows: First, the medical image data is processed by the medical image processing module 5.

[0031] like Figure 2As shown, the medical image processing module 5 uses 3D Slicer software 6 to read CT or MRI scan images and converts the 2D image data into a 3D model using ITK and VTK algorithms 7. The generated 3D model is exported in STL or OBJ format and then imported into the Unity environment 8. Within the Unity environment, the LOD optimization module 9 dynamically adjusts the model rendering quality, automatically reducing or increasing the model's level of detail based on the user's viewing distance, ensuring smooth operation of the system on the HoloLens device. The gesture interaction module 10 and the voice interaction module 11 are implemented using the MRTK toolkit and define input methods for operations such as grabbing, rotating, and scaling. The Manipulation Handler component is bound to the medical model object, allowing users to control the model's position, angle, and size through gestures. Voice interaction uses the SpeechInput Handler to trigger specific commands such as "zoom in," "zoom out," and "rotate," further enhancing interactive flexibility. The design goal of the medical knowledge application ability layer 2 is to assess students' understanding and application of medical knowledge through interactive quizzes.

[0032] The back-end database 16 stores the question content, answers and analysis information, and supports question label classification and random question extraction. The front-end real-time feedback system 17 sends a verification request to the back-end via Ajax or Fetch, and displays the user operation results in real time.

[0033] For example, when a student completes a multiple-choice question, the system immediately returns the judgment result and displays a detailed explanation. For simulation operation questions, the collision detection module 18 and the ray detection module 19 are used to identify the user's click area and provide instant feedback.

[0034] like Figure 4 As shown in the figure, when a user clicks a specific organ, Raycasting technology detects the click location and provides feedback through color changes, edge highlighting, or icon markings. This instant feedback mechanism helps students clarify their operational objectives and deepen their understanding. The core of Clinical Operation Presentation Layer 3 lies in the highly realistic surgical planning simulation module.

[0035] like Figure 3 As shown, the Photon PUN plugin 12 is used to build a network framework, enabling event synchronization and state management in multi-user scenarios. The PhotonTransformView component 13 is bound to the 3D model, capturing changes in the model's position, rotation, and scale, and transmitting these changes to other user devices in real time. The RPC function module 14 is used for remote procedure calls, ensuring that key user operations are executed synchronously across all devices.

[0036] For example, when the main operator rotates the model, the devices of other users will synchronously display the status of the rotated model. In order to distinguish the operating status of different users, a color-changing aperture 15 is added to the bottom of the 3D model. Each user is assigned an independent color identifier when entering the session. When the user operates the model, the color of the aperture changes dynamically, intuitively displaying the current operating status. The recording and playback module 20 captures user operation data in real time, including gestures, perspectives, and model interaction operations, and stores this data in JSON format to the JSON data storage module 21. The playback function extracts operation records from the stored data and restores the user operation process in timestamp order to facilitate the analysis of weak links. The system evaluation and feedback mechanism 4 records user operation sequence, time and other information through the database, generates learning reports and provides personalized feedback.

[0037] The specific implementation process is as follows: The system records the user's click sequence and operation time on the model, and automatically marks organs that have been repeatedly incorrectly clicked or areas that take a long time to operate. The data analysis module compiles statistics on user operation data and generates a learning report that shows the student's performance and points out areas for improvement.

[0038] For example, if the system detects that a student repeatedly mistakenly clicks on the liver area during a simulated surgery, it will mark that area as a weak link in the learning report and recommend that the student strengthen their study of the relevant knowledge. Furthermore, the system adjusts the learning path and difficulty level based on the assessment results, providing each user with a customized training plan.

[0039] For example, for students who are proficient in operation, the system adds more complex tasks; while for students who are slow in operation, the system recommends basic exercises to consolidate their skills. The actual application scenarios of the present invention include daily learning and team collaboration training for medical students.

[0040] For example, in a virtual operating room, multiple students access the same session using HoloLens devices to perform a virtual surgery together. The primary operator is responsible for key steps, while other students assist with auxiliary tasks. The operation prompt aperture 15 clearly displays each student's operating status, reducing conflicts and misunderstandings during collaboration. After the surgery is completed, the system automatically generates a learning report detailing each student's operation process and performance, and providing suggestions for improvement. Furthermore, the recording and playback function allows students to review the entire surgical process, analyze their own shortcomings, and make improvements.

[0041] This invention integrates multiple technologies to form a coherent learning path, progressing from the cognitive level to the behavioral level, providing medical students with a comprehensive and systematic learning platform. Based on the design concept of the Miller Pyramid medical assessment model, the system is not only highly scientific and systematic, but also effectively cultivates students' self-reflection and autonomous learning abilities. High-precision three-dimensional model generation and interactive technology significantly improve the intuitiveness and interactivity of learning, compensating for the lack of intuitiveness in traditional teaching.

[0042] The multi-person cross-space collaboration feature enhances interaction and communication among medical students and between medical students and their instructors, promoting collaborative learning. Real-time feedback and personalized learning mechanisms leverage data analysis and technical means to provide each user with a customized learning path and difficulty adjustment, addressing the lack of personalized learning. Furthermore, the system eliminates the need for limited traditional training resources such as animals or donated cadavers, reducing training costs. The highly realistic surgical simulation environment helps students accumulate practical experience and better cope with the complexities and emergencies of real-world surgical scenarios.

[0043] In summary, the present invention solves the problems existing in the existing technology through technical means such as multi-level progressive learning path design, high-precision three-dimensional model generation and interaction, multi-person cross-space collaboration, real-time feedback and personalized learning, and provides medical students with a more intuitive, interactive, realistic and systematic learning environment.

Claims

1. An extended reality medical learning system based on the Miller pyramid medical assessment model, characterized by It includes medical professional knowledge and operation theory layer (1), medical knowledge application ability layer (2), clinical operation performance layer (3) and system evaluation and feedback mechanism (4). The medical expertise and operation theory layer (1) generates a virtual operating room and human anatomy model through mixed reality technology. The medical knowledge application ability layer (2) sets up an interactive test module to examine knowledge application ability. The clinical operation performance layer (3) provides a surgical planning simulation module to support multi-person collaboration. The system evaluation and feedback mechanism (4) records user operation data and generates a learning report.

2. The extended reality medical learning system according to claim 1, characterized in that The medical expertise and operation theory layer (1) also includes a medical image processing module (5) for converting CT or MRI scan images into three-dimensional model files and outputting the three-dimensional model in STL or OBJ format through 3D Slicer software (6).

3. The extended reality medical learning system according to claim 2, characterized in that It further includes an LOD optimization module (9) for dynamically adjusting the rendering quality of the three-dimensional model according to the user's viewing distance.

4. The extended reality medical learning system according to claim 1, characterized in that The medical knowledge application capability layer (2) also includes a back-end database (16) that stores question content, answers and analysis information, and supports a random question selection function.

5. The extended reality medical learning system according to claim 4, characterized in that It further includes a collision detection module (18) and a ray detection module (19) for identifying the user's clicked position and immediately feeding back the result.

6. The extended reality medical learning system according to claim 1, characterized in that The clinical operation presentation layer (3) further includes a Photon PUN plug-in (12) for implementing event synchronization and state management in a multi-user scenario.

7. The extended reality medical learning system according to claim 6, characterized in that It further includes a color-variable aperture (15) for distinguishing the operation status of different users and displaying the operation progress through independent color identification.

8. The extended reality medical learning system according to claim 1, characterized in that The system evaluation and feedback mechanism (4) further includes a recording and playback module (20) for capturing user operation data and storing the data in a JSON data storage module (21) in a JSON format.

9. The extended reality medical learning system according to claim 8, characterized in that It further includes a playback function for extracting operation records from the JSON data storage module (21) and restoring the user operation process in timestamp order.

10. The extended reality medical learning system according to claim 1, characterized in that The gesture interaction module (10) is bound to the medical model object through the Manipulation Handler component, and supports the user to control the position, angle and size of the model through gestures.

Images