An augmented reality surgery simulation training method and terminal based on gesture recognition

Abstract

The application discloses an augmented reality surgery simulation training method for gesture recognition, responds to a selection operation of a user, loads surgery simulation data corresponding to the selection operation from a surgery simulation database, generates an augmented reality scene according to the loaded surgery simulation data, collects hand operation data of the user, identifies the hand operation data by using a gesture recognition model to determine operation intention data, adjusts the augmented reality scene according to the operation intention data to simulate surgery interactive operation, and monitors the hand operation data of the user in real time, and when it is detected that the hand operation data of the user does not match preset standard surgery operation, visual warning information and real-time guidance information are provided in the augmented reality scene, surgery simulation and interactive experience without an additional controller can be realized, and the authenticity and efficiency of simulation are improved.

Description

Technical Field

[0001] This invention relates to the field of medical training technology, and in particular to an augmented reality surgical simulation training method and terminal based on gesture recognition. Background Technology

[0002] In the medical field, surgical training is a crucial step in ensuring that doctors possess the necessary skills and knowledge. Traditional surgical training methods primarily rely on expert guidance, textbook study, and practice on actual patients. However, with technological advancements, simulation training systems have gradually become an important component of surgical training. Currently, medical surgical training mainly relies on the following methods: 1. Traditional model-based surgical training system: This method uses physical models for surgical training, where trainees simulate surgery on the physical model by operating real medical instruments.

[0003] 2. Virtual Reality (VR) Based Surgical Simulation System: Trainees wear VR devices to conduct surgical training in a completely virtual environment, simulating the operation of surgical tools through a specially designed controller.

[0004] 3. Computer-based surgical simulation software: Operates on a computer screen using a mouse and keyboard to simulate the surgical procedure.

[0005] 4. Mixed reality surgical training system: combines real models with digital projections, but usually requires interaction with specific controllers or markers rather than directly recognizing the doctor's gestures.

[0006] However, while these technologies have improved the efficiency and safety of surgical training to some extent, they each have some significant drawbacks: 1. Traditional physical model training systems lack flexibility and realism, cannot simulate the special circumstances of different cases, and cannot provide real-time data feedback and evaluation.

[0007] 2. Although VR systems offer a strong sense of immersion, they suffer from insufficient tactile feedback, and prolonged use of VR devices can easily cause dizziness, which does not meet the needs of doctors for long-term training.

[0008] 3. Computer-based simulation software uses mouse and keyboard operation, which is too different from the actual surgical operation and cannot effectively train the doctor's fine hand movements.

[0009] 4. Existing mixed reality systems require specialized controllers or markers, which increases the learning cost and cannot realistically reflect the natural hand gestures of doctors in actual surgery.

[0010] 5. Existing technologies generally lack the ability to recognize and interact with natural gestures during surgery, and cannot fully simulate the gesture operation experience of doctors in a real surgical environment.

[0011] 6. Existing systems lack sufficient simulation of surgical navigation procedures, making it difficult to provide specialized training for preoperative planning and intraoperative navigation. Summary of the Invention

[0012] The technical problem to be solved by the present invention is to provide an augmented reality surgical simulation training method and terminal based on gesture recognition, which can realize surgical simulation and interactive experience without additional controllers, and improve the realism and efficiency of simulation.

[0013] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: An augmented reality surgical simulation training method based on gesture recognition includes the following steps: S1. In response to the user's selection operation, load the surgical simulation data corresponding to the selection operation from the surgical simulation database, and generate an augmented reality scene based on the loaded surgical simulation data; S2. Collect the user's hand operation data, use a gesture recognition model to identify the hand operation data to determine operation intention data, and adjust the augmented reality scene according to the operation intention data to simulate surgical interaction operation; S3. Monitor the user's hand operation data in real time. When the user's hand operation data does not match the preset standard surgical operation, provide visual warning information and real-time guidance information in the augmented reality scene.

[0014] To solve the above-mentioned technical problems, another technical solution adopted by the present invention is as follows: An augmented reality surgical simulation training terminal based on gesture recognition includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the various steps of the aforementioned augmented reality surgical simulation training method based on gesture recognition.

[0015] The beneficial effects of this invention are as follows: This invention provides an augmented reality surgical simulation training method and terminal based on gesture recognition. Responding to user selection operations, it loads surgical simulation data corresponding to the selected operation from a surgical simulation database, generates an augmented reality scene based on the loaded surgical simulation data, and utilizes augmented reality technology to provide a more realistic and intuitive environment for surgical simulation training. By collecting user hand operation data and using a gesture recognition model to identify the hand operation data to determine the operation intention data, the augmented reality scene is adjusted according to the operation intention data to simulate surgical interaction operations, achieving surgical simulation and interactive experience without the need for an additional controller, thus improving the interactivity and realism of the simulation training. Furthermore, it monitors the user's hand operation data in real time. When it detects that the user's surgical operation does not match the preset standard surgical operation, it provides visual warning information and real-time guidance information in the augmented reality scene, helping users to promptly identify and correct risks and non-standard behaviors in surgical operations, thereby improving the effectiveness and safety of surgical simulation training. Attached Figure Description

[0016] Figure 1 This is a flowchart of an augmented reality surgical simulation training method based on gesture recognition, according to an embodiment of the present invention. Figure 2 This is an architecture diagram of an augmented reality surgical simulation training method based on gesture recognition, according to an embodiment of the present invention. Figure 3 This is another flowchart of an augmented reality surgical simulation training method based on gesture recognition according to an embodiment of the present invention; Figure 4 This is a flowchart illustrating the gesture recognition process in an augmented reality surgical simulation training method based on gesture recognition, according to an embodiment of the present invention. Figure 5 This is a schematic diagram of an augmented reality surgical simulation training terminal based on gesture recognition according to an embodiment of the present invention; Label Explanation: 1. An augmented reality surgical simulation training terminal based on gesture recognition; 2. Memory; 3. Processor. Detailed Implementation

[0017] To explain in detail the technical content, objectives, and effects of the present invention, the following description is provided in conjunction with the embodiments and accompanying drawings.

[0018] Please refer to Figure 1 This invention provides an augmented reality surgical simulation training method based on gesture recognition, comprising the following steps: S1. In response to the user's selection operation, load the surgical simulation data corresponding to the selection operation from the surgical simulation database, and generate an augmented reality scene based on the loaded surgical simulation data; S2. Collect the user's hand operation data, use a gesture recognition model to identify the hand operation data to determine operation intention data, and adjust the augmented reality scene according to the operation intention data to simulate surgical interaction operation; S3. Monitor the user's hand operation data in real time. When the user's hand operation data does not match the preset standard surgical operation, provide visual warning information and real-time guidance information in the augmented reality scene.

[0019] As can be seen from the above description, the beneficial effects of the present invention are as follows: By responding to the user's selection operation, surgical simulation data corresponding to the selection operation is loaded from the surgical simulation database, and an augmented reality scene is generated based on the loaded surgical simulation data, providing a more realistic and intuitive environment for surgical simulation training using augmented reality technology; By collecting the user's hand operation data, using a gesture recognition model to identify the hand operation data to determine the operation intention data, and adjusting the augmented reality scene according to the operation intention data to simulate surgical interaction operations, a surgical simulation and interactive experience without the need for an additional controller is achieved, improving the interactivity and realism of simulation training; The user's hand operation data is monitored in real time, and when it is detected that the user's surgical operation does not match the preset standard surgical operation, visual warning information and real-time guidance information are provided in the augmented reality scene, which helps the user to promptly discover and correct risks and non-standard behaviors in the surgical operation, improving the effectiveness and safety of surgical simulation training.

[0020] Furthermore, the process of collecting user hand gesture data and using a gesture recognition model to identify the hand gesture data to determine the operation intention data includes: Collect the three-dimensional coordinates of preset key points on the user's hand, calculate the curvature and relative position of each finger joint based on the three-dimensional coordinates, and generate a gesture feature vector; A deep learning model is used to classify and recognize the gesture feature vector. Based on the classification result, gesture data of the corresponding category is obtained from the gesture database. The gesture feature vector is matched with the gesture data to determine the user's current surgical gesture. The current surgical stage of the user is determined based on the sequence of surgical operations already performed by the user. Contextual analysis is then performed based on the user's current surgical gestures, the current surgical stage, and the surgical expert action database to determine the operational intent data corresponding to the user's current surgical gestures.

[0021] As described above, by acquiring the three-dimensional coordinates of key points on the hand and calculating the curvature and phase position relationship of finger joints, the system accurately captures the details of hand movements, thereby improving the accuracy of gesture recognition. Using a deep learning model for gesture classification and recognition enables rapid processing of hand operation data and real-time gesture recognition. Based on the user's surgical operation sequence, the system determines the current surgical stage. Based on the current surgical gestures, the current surgical stage, and the surgical expert's action database, the system can better understand the meaning of the user's gestures in a specific surgical environment, improving the accuracy of inferring the operation intent.

[0022] Furthermore, adjusting the augmented reality scene based on the operational intent data to simulate surgical interaction includes: If the operation intent data is to obtain a virtual surgical tool, then the corresponding virtual surgical tool is determined and generated based on the user's current surgical gesture, and the virtual surgical tool is added to the augmented reality scene; If the operation intent data is feedback information for obtaining the current surgical gesture, then the augmented reality scene is adjusted according to the user's current surgical gesture to generate surgical operation visual data corresponding to the user's current surgical gesture.

[0023] As described above, based on the user's operational intent data, corresponding virtual surgical tools or feedback information on surgical gestures are generated in augmented reality scenarios. This adapts to various complex surgical types and special situations, improves the flexibility of surgical simulation, and simulates a real surgical interaction experience without worrying about posing risks to real patients, thereby improving surgical safety and success rate.

[0024] Furthermore, step S3 also includes: Record the user's surgical simulation training process, generate a training evaluation report for the user based on the recording results, and find and display the corresponding training content and improvement suggestions based on the training evaluation report.

[0025] As described above, by recording the user's surgical simulation training process and objectively evaluating the user's surgical simulation training based on the recorded results, the accuracy of the training evaluation report is improved. Based on the training evaluation report, the corresponding training content and improvement suggestions are found and displayed, making the training feedback more targeted and practical, avoiding blind practice, and thus improving training efficiency.

[0026] Furthermore, generating the user's training evaluation report based on the recorded results includes: The user's skill proficiency is scored based on the surgical simulation training time recorded in the results. The hand manipulation data in the recorded results were compared with the surgical expert action database using a standard difference analysis to obtain the standard difference analysis results. A training evaluation report for the user is generated based on the user's skill proficiency score and the results of the standard difference analysis.

[0027] As described above, by objectively quantifying the time spent by users on surgical simulation training, the evaluation results are more fair and accurate. By comparing the user's hand operation data with the surgical expert action database, the system can identify the user's strengths and weaknesses in surgical operations. Combined with skill proficiency scores and standard difference analysis results, a training evaluation report is generated, which helps users discover their problems and take corresponding measures to improve, thereby improving their surgical skills more quickly.

[0028] Please refer to Figure 5 Another embodiment of the present invention provides an augmented reality surgical simulation training terminal based on gesture recognition, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the various steps of the above-described augmented reality surgical simulation training method based on gesture recognition.

[0029] The above-described augmented reality surgical simulation training method and terminal based on gesture recognition of the present invention are applicable to surgical simulation training scenarios, and can realize surgical simulation and interactive experience without the need for an additional controller, thereby improving the realism and efficiency of the simulation.

[0030] Please refer to Figures 1 to 4 Embodiment 1 of the present invention is as follows: An augmented reality surgical simulation training method based on gesture recognition includes the following steps: S1. In response to the user's selection operation, load the surgical simulation data corresponding to the selection operation from the surgical simulation database, and generate an augmented reality scene based on the loaded surgical simulation data.

[0031] In this embodiment, facial recognition technology is used to automatically identify the operator's identity, load their historical training data and personalized settings, and load corresponding surgical simulation data from a preset surgical simulation database according to the user's selection. The user's selection includes surgical type, difficulty level, and training mode (basic training mode, surgical navigation model, and surgical evaluation mode). The surgical simulation data includes different types of surgical scenarios, patient anatomical structure models, and surgical navigation paths. Augmented reality scenes are generated based on the loaded surgical simulation data, and augmented reality technology is used to provide a more realistic and intuitive environment for surgical simulation training.

[0032] Furthermore, in this embodiment, the surgical simulation training process requires equipment calibration, including the position calibration of the depth camera and the display calibration of the AR glasses. Adaptive adjustments are made based on the detected lighting conditions of the training environment to ensure the accuracy of subsequent surgical gesture capture. Specifically, the processes for depth camera position calibration and AR glasses display calibration are as follows: 1. Depth camera position calibration process: (1) Reference marker recognition: A standard calibration board with specific geometric features is pre-placed in the training space. The calibration board includes precisely calibrated geometric patterns (such as checkerboard, dot array, etc.). The image of the calibration board is captured by a depth camera. Computer vision algorithms (such as corner detection, contour recognition) are used to identify the image to determine the position of the marker in the image. By comparing the position of the marker in the image with its known actual physical position, the spatial transformation matrix between the depth camera and the marker is calculated. (2) Establishment of spatial coordinate system: Based on the identified reference marker positions, establish a global coordinate system, map any point in the training space to the system coordinate system, calculate the intrinsic parameters (such as focal length, principal point offset, distortion coefficient) and extrinsic parameters (rotation matrix and translation vector) of the depth camera, and store the intrinsic and extrinsic parameters for subsequent depth data processing and spatial mapping; (3) Depth accuracy calibration: Use multiple reference objects at known distances to test the accuracy of depth measurement. For different distance ranges, establish a depth measurement error compensation model and automatically adjust the exposure parameters and depth threshold of the depth camera according to the ambient light conditions and reflection characteristics. (4) Dynamic recalibration mechanism: Continuously monitor environmental changes (such as changes in light and slight camera movements) during surgical simulation training. When a potential calibration offset is detected, the system will trigger a rapid recalibration mechanism. By comparing the known user hand size with the detected hand size, the calibration will be finely adjusted in real time based on the comparison results.

[0033] 2. AR Glasses Display Calibration Process (1) Collect user visual parameters: guide the user to complete the pupil distance measurement process, capture the user's pupil position through the eyeglass tracking camera, and measure the user's visual preferences and characteristics by having the user align with a specific visual target. Adjust the corresponding display parameters (such as focal plane position) according to whether the user is wearing corrective glasses. (2) Spatial mapping and alignment: The tracking camera built into the AR glasses captures environmental feature points and matches them with the spatial coordinate system established by the depth camera. The system calculates the precise position and orientation of the AR glasses relative to the global coordinate system and updates the position data of the AR glasses in space in real time through SLAM (Simultaneous Localization and Mapping) technology. (3) Display parameter optimization: The system adjusts the brightness, contrast and color of the AR display content to adapt to the current ambient light conditions, and optimizes the holographic projection angle and focal length to reduce "ghost images" and visual fatigue. The system dynamically adjusts the content rendering angle according to the user's field of vision to ensure a natural perspective effect. (4) Interactive calibration test: The system displays specific virtual calibration points and requires the user to touch these points with gestures. By comparing the deviation between the user's actual finger position and the finger position detected by the system, the system calculates and compensates for the spatial alignment error, allowing the user to complete a series of precise alignment tasks. The system optimizes the accuracy of touch interaction accordingly. (5) Fusion accuracy verification: The system places physical reference objects (such as surgical tools) at key locations in the training space, and the AR glasses overlay the corresponding virtual models on the reference objects. The user confirms the alignment degree between the two, and the system performs final fine-tuning and calibration based on user feedback and automatically detected alignment errors.

[0034] In this embodiment, a personalized calibration profile is created for each user, recording their visual parameters and preference settings. The calibration results for each user are encrypted and stored in the system database so that they can be quickly loaded the next time the user uses the system. The system also periodically prompts users to perform recalibration to adapt to possible physiological changes and equipment wear and tear.

[0035] S2. Collect the user's hand operation data, use a gesture recognition model to identify the hand operation data to determine the operation intention data, and adjust the augmented reality scene according to the operation intention data to simulate surgical interaction operation, specifically including S2.1-S2.2.

[0036] S2.1 The step of collecting the user's hand operation data and using a gesture recognition model to identify the hand operation data to determine the operation intention data includes: Collect the three-dimensional coordinates of preset key points on the user's hand, calculate the curvature and relative position of each finger joint based on the three-dimensional coordinates, and generate a gesture feature vector; A deep learning model is used to classify and recognize the gesture feature vector. Based on the classification result, gesture data of the corresponding category is obtained from the gesture database. The gesture feature vector is matched with the gesture data to determine the user's current surgical gesture. The current surgical stage of the user is determined based on the sequence of surgical operations already performed by the user. Contextual analysis is then performed based on the user's current surgical gestures, the current surgical stage, and the surgical expert action database to determine the operational intent data corresponding to the user's current surgical gestures.

[0037] Please refer to Figure 4In this embodiment, by acquiring the three-dimensional coordinates of key points on the hand and calculating the curvature and phase position relationship of the finger joints, the details of hand movements are accurately captured, thereby improving the accuracy of gesture recognition. Using a deep learning model for gesture classification and recognition can quickly process hand operation data and achieve real-time gesture recognition. The current surgical stage is determined based on the user's surgical operation sequence. Based on the current surgical gestures, the current surgical stage, and the surgical expert action database, the system can better understand the meaning of the user's gestures in a specific surgical environment and improve the accuracy of operation intention inference.

[0038] Furthermore, in this embodiment, the user's hand operation data is collected, specifically as follows: 1. Raw data collection Spatial location data: The three-dimensional coordinates (x, y, z) of 21 key points on the hand are captured by a depth camera, with a sampling frequency of 60Hz; Finger joint angles: Calculate the bending angle and relative position of each finger joint to form 15 joint angle features; Dynamic trajectory information: Records the hand movement trajectory within a time window of 0.5-2 seconds, represented by a time-series vector; 2. Feature Engineering and Data Augmentation Skeleton distance features: Calculate the Euclidean distance and geodesic distance between key points to construct a hand skeleton feature map; Motion speed characteristics: Extract the instantaneous speed, acceleration, and tremor frequency of hand movements; Pose normalization: Eliminates hand position and orientation differences through Spatial Transformation Network (STN), focusing on the gesture itself; Data augmentation: Expanding the training samples by adding Gaussian noise, random rotation, scaling, and time window sliding.

[0039] Furthermore, in this embodiment, the surgical specific gesture recognition process is as follows: 1. Surgical Action Classification System: A hierarchical classification system is constructed, which includes 50 basic surgical actions and 30 compound surgical actions. The basic actions include precise grasping, blunt dissection, sharp cutting, suture knotting, and fine suction, while the compound actions include needle holding and suturing, vascular clamping, tissue separation, and hemostat operation. 2. Context-aware recognition: The prior probability of gesture recognition is adjusted by combining information of the current surgical stage (such as craniotomy, tumor resection, suturing, etc.). The temporal dependency between surgical actions is captured by the Conditional Random Field (CRF) model. The system memorizes the previous action sequence and predicts the next possible surgical action, thereby improving the recognition accuracy. 3. Fine-grained metric learning: By using a contrastive learning method, the feature similarity of similar surgical actions is maximized and the feature distance between different actions is minimized. A triplet loss function is used to enhance the feature discrimination ability for similar surgical actions that are difficult to distinguish (such as different types of grasping), so as to achieve the recognition of sub-millimeter hand micro-movements, such as key surgical skill indicators such as slight finger tremors and slight rotation.

[0040] Furthermore, in this embodiment, the deep learning model architecture adopts a hierarchical multi-stream network architecture, specifically as follows: 1. Skeleton Feature Extraction Network: A Graph Convolutional Network (GCN) is used to process hand skeleton data. Each joint of the hand is used as a node in the graph, and the bone connections are used as edges. Each layer of the GCN contains multiple residual connection modules to extract the relative position and geometric structure features between the joints of the hand. The attention mechanism is used to highlight the joint information that is highly related to specific surgical actions. 2. Temporal Dynamic Modeling Network: Combining Bidirectional Long Short-Term Memory Network (Bi-LSTM) and Temporal Convolutional Network (TCN) to capture the temporal features of gestures, a multi-scale temporal pyramid structure is adopted to simultaneously capture short-term fine movements (such as finger bending) and long-term action sequences (such as the entire suturing process), so as to realize the recognition of surgical actions under different time windows and handle the variability of the duration of different surgical actions. 3. Surgical tool interaction recognition module: It integrates hand posture and surrounding environment information to recognize the interaction intention between the hand and the virtual surgical tool. It predicts the tool operation effect caused by the gesture through the physical simulation module, and combines a 3D convolutional neural network with attention mechanism to identify the contact point between the finger and the tool and the direction of force. 4. Multimodal feature fusion network: An adaptive weighting mechanism is used to fuse skeleton features, temporal features and tool interaction features. Information exchange between different feature domains is realized through a cross-attention mechanism. An expert knowledge distillation module is introduced to integrate the experience of surgical experts into the model decision-making process.

[0041] The training and optimization process of deep learning models is as follows: 1. Construction of expert demonstration database: Collect 5,000 hours of surgical action data from 50 surgeons with different experience levels. Each surgical action includes samples from at least 100 different performers, covering different hand shapes, operating habits and skill variations. The action quality is labeled and reviewed by a team of senior surgical experts to establish a gold standard dataset. 2. Multi-stage training strategy: In the first stage, the base network is pre-trained using a large-scale general gesture dataset; in the second stage, supervised fine-tuning is performed using labeled surgical gesture data; in the third stage, reinforcement learning is used to optimize the accuracy and timeliness of gesture recognition, rewarding accurate and timely recognition results. 3. Few-shot learning and continuous adaptation: Based on the prototype network, the system achieves few-shot learning of new surgical actions, requiring only 5-10 examples to recognize new actions. During use, the system continuously collects user feedback and performs online fine-tuning through comparative learning methods. The meta-learning strategy enables the model to quickly adapt to the individual operating styles and habits of different doctors.

[0042] Real-time inference and application optimization of deep learning models, specifically: 1. Model Compression and Acceleration: The number of model parameters is reduced by 80% through knowledge distillation and network pruning to ensure real-time performance on edge devices. Hybrid precision computing and model quantization techniques are used to optimize the utilization of computing resources and achieve a single-frame inference latency of less than 15ms, ensuring real-time response of gesture recognition. 2. Confidence assessment and error correction mechanism: The system calculates the confidence score of each recognition result and displays a visual reliability index. When the confidence score is lower than the threshold, the system retains multiple possible gesture candidates and waits for more time-series information before making a determination. At the same time, a recursive error correction mechanism is introduced to correct occasional recognition errors based on the consistency constraints of consecutive frames. 3. Surgical procedure adaptive adjustment: The system automatically detects the current surgical stage and dynamically adjusts the sensitivity and accuracy of gesture recognition. For key surgical steps (such as vascular manipulation), the system increases the recognition accuracy requirements and enables multiple verifications. Based on the user's historical operating habits, the system personalizes the recognition parameters and thresholds of specific gestures.

[0043] S2.2 Adjusting the augmented reality scene according to the operation intent data to perform model surgical interaction operations, including: If the operation intent data is to obtain a virtual surgical tool, then the corresponding virtual surgical tool is determined and generated based on the user's current surgical gesture, and the virtual surgical tool is added to the augmented reality scene; If the operation intent data is feedback information for obtaining the current surgical gesture, then the augmented reality scene is adjusted according to the user's current surgical gesture to generate surgical operation visual data corresponding to the user's current surgical gesture.

[0044] In this embodiment, a three-dimensional anatomical model is reconstructed based on the patient's medical imaging data (such as CT and MRI) and projected into the training space through AR glasses. The system overlays surgical navigation information, including the optimal approach, key anatomical landmarks, and safety boundaries, into the AR field of view to generate an augmented reality scene. The AR projection content is adjusted in real time according to the user's perspective and positional movement to maintain visual stability and accuracy. When the system recognizes the user's surgical gestures, it presents corresponding virtual surgical tools or operational visual data in the AR scene, simulating the physical interaction between the surgical tools and the patient's tissues, including tissue deformation, incision, and suturing effects. Based on the accuracy of the gestures and the operational path, the system calculates and presents real-time feedback on the surgical operation, such as bleeding volume and the degree of tissue damage.

[0045] Furthermore, in this embodiment, obtaining surgical navigation information includes the following aspects, specifically: 1. Medical image data preprocessing and 3D reconstruction (1) Multimodal medical image acquisition The system receives multimodal medical imaging data from patients, including CT, MRI, angiography, PET, etc., and selects the most suitable imaging method for different tissue characteristics: CT is used for bone structure, MRI is used for soft tissue analysis, and angiography is used for the vascular system. The system receives and processes this medical imaging data using the DICOM standard format. (2) Medical image segmentation and annotation The system automatically segments key anatomical structures in images using a multi-level cascaded U-Net deep learning network. For special regions such as brain tumors, a densely connected network with an attention mechanism is used to improve segmentation accuracy. The system can identify and label more than 40 key anatomical structures, including blood vessels, nerves, functional areas, and lesion areas. The segmentation results are continuously optimized by a semi-supervised learning algorithm. As the system is used more frequently, the segmentation accuracy continues to improve. (3) High-precision three-dimensional model reconstruction Based on the segmentation results, an initial 3D mesh model is generated using an improved Marching Cubes algorithm. A mesh optimization algorithm is applied to reduce the number of polygons in the model while preserving key anatomical details. Physical properties (such as elasticity and density) are assigned to different tissues through physical simulation, laying the foundation for subsequent surgical simulation. Layered rendering technology is used to enable interactive dissection and perspective of the reconstructed model.

[0046] 2. Surgical pathway planning and optimization (1) Automatic generation of the best entry path Based on more than 5,000 successful surgical records in the expert surgical database, the system has established an approach evaluation neural network. When the patient's specific anatomical structure and lesion location are input, the system automatically calculates multiple possible approach options and scores each approach in multiple dimensions, including: tissue damage risk, operative difficulty, and quality of exposure. The system continuously optimizes the approach recommendation through reinforcement learning algorithms and incorporates clinical feedback into the evaluation model. (2) Safety boundary calculation and visualization Based on the minimum safe distance requirements between dangerous structures (such as large blood vessels and important nerve bundles) and surgical instruments, the system generates dynamic safety boundaries. It adopts an adaptive mesh deformation algorithm to accurately calculate the three-dimensional safety area around irregular anatomical structures. According to the sensitivity of different tissues, it sets multi-level safety boundaries and presents them in the AR field of view with different colors and transparency. When the surgical instruments approach the safety boundary, the system provides progressive visual and tactile feedback warnings. (3) Functional area mapping and protection By combining the patient's preoperative functional MRI data, the system marks brain functional areas (such as motor areas and language areas), and generates personalized functional area distribution maps by fusing probability maps and individual patient data. Special protective boundaries are set around the functional areas, and functional risk assessments related to the areas are provided. During the training process, the clinical consequences that may be caused by damage to the functional areas are simulated, enhancing the doctor's risk awareness.

[0047] 3. Intelligent navigation information generation (1) Intelligent recognition of key anatomical landmarks The system analyzes the spatial relationships between anatomical structures through graph neural networks, automatically identifies key surgical points and landmark structures, and dynamically adjusts the displayed anatomical markers based on preoperative planning and surgical progress to avoid information overload. For easily confused anatomical structures (such as similar vascular branches), the system provides enhanced recognition markers and additional information prompts. Combined with eye-tracking technology, the system can sense the doctor's focus of attention and intelligently highlight detailed information about the area of ​​current attention. (2) Surgical procedure guidance information The system breaks down complex surgeries into multiple key steps, generates detailed operation guidelines for each step, and automatically displays suggested operations and precautions for the next step based on the currently detected surgical stage. Through augmented reality arrows, outline highlighting, and animation demonstrations, it intuitively guides doctors to perform standardized operations. The system adjusts the level of detail of the guidance information according to the doctor's experience level to avoid interfering with the expert. (3) Dynamic risk assessment and early warning The system analyzes the positional relationship between surgical tools and key structures in real time, calculates the potential risk index, and predicts the tissue deformation and organ displacement that the current operation may cause based on physical simulation. When a high-risk operation is detected (such as approaching an important blood vessel), the system generates multi-level warning information and extracts common risk patterns from historical surgical data through machine learning algorithms to provide early warning of possible complications.

[0048] 4. AR presentation technology for surgical navigation information (1) Multi-level information display architecture The system adopts a layered display architecture, dividing navigation information into three layers according to importance and urgency: Core layer: key information that is always displayed (such as safety boundaries and current tool position); Context layer: auxiliary information that is dynamically displayed according to the surgical progress (such as next step guidance); Details layer: detailed reference information that users can access through specific gestures (such as detailed diagrams of vascular structures). Users can control the display and hiding of information in each layer through simple gestures, so as to obtain information on demand. (2) Visual encoding and enhanced perception Use a consistent visual language to encode different types of navigation information. For example, color coding: red indicates danger zones, green indicates safety passages, and yellow indicates areas of concern; transparency coding: the more important the structure, the less transparent it is; pulsation effect: add a pulsation effect to parts that require immediate attention to enhance perception priority; and use a visual saliency optimization algorithm to ensure that key navigation information is easily identifiable in complex backgrounds. (3) Context-aware information adaptation The system automatically adjusts the brightness and contrast of the AR display content according to the current ambient light conditions. Based on the user's line of sight and focus of attention, it intelligently adjusts the position and level of detail of the information. When an emergency is detected, the system will temporarily simplify the display of non-critical information, highlight warning information, and dynamically adjust the information density based on the user's fatigue assessment (such as operation time and hand stability) to avoid cognitive overload.

[0049] 5. Real-time update mechanism for navigation information (1) Dynamic anatomical structure tracking Using a physics-based tissue deformation model, the position and shape of anatomical structures are updated in real time during the operation. Real-time data captured by a depth camera is used to correct the simulated deformation, ensuring that the model is consistent with reality. For soft tissue deformation, nonlinear finite element analysis is used to improve the accuracy of deformation prediction. The system can handle topological changes caused by tissue resection, separation and other operations. (2) Adaptive navigation path update The system continuously evaluates the current surgical progress and automatically recalculates the optimal path when a deviation from the plan is detected. Based on tissue deformation and exposure, it dynamically adjusts the recommended surgical approach and operation sequence. When unexpected situations occur (such as simulated bleeding), the system can quickly generate emergency treatment plans. Through a parallel computing architecture, it ensures the real-time nature of path updates, with update latency controlled within 100ms. (3) Collaborative knowledge base and real-time consultation The system connects to a medical knowledge graph, allowing it to retrieve relevant medical literature and cases based on the current surgical situation. During training, it can simulate remote expert consultations, providing AI-based expert advice. The system records and analyzes collective wisdom, integrating the experience of multiple experts into navigation suggestions. Through a federated learning architecture, it continuously optimizes the navigation algorithm while protecting patient privacy.

[0050] S3. Monitor the user's hand operation data in real time. When the user's hand operation data does not match the preset standard surgical operation, provide visual warning information and real-time guidance information in the augmented reality scene.

[0051] In this embodiment, the system monitors the user's surgical operations in real time. When potential risks or non-standard operations are detected, visual warning information and real-time guidance information are provided through the AR interface. This helps the user to promptly identify and correct risks and non-standard behaviors in the surgical operation, thereby improving the effectiveness and safety of surgical simulation training.

[0052] Furthermore, in this embodiment, step S3 further includes: recording the user's surgical simulation training process, generating a training evaluation report for the user based on the recording results, and finding and displaying corresponding training content and improvement suggestions based on the training evaluation report. The system records the user's entire surgical operation process, including gesture data, operation path, time consumption, etc. After training, the system generates a detailed evaluation report, including skill proficiency scores, difference analysis from expert standards, progress trends, etc., and uploads the training data to a cloud database for long-term skill development tracking and training program optimization.

[0053] Furthermore, in this embodiment, generating the user's training evaluation report based on the recorded results includes: scoring the user's skill proficiency based on the surgical simulation training time consumed in the recorded results; performing a standard difference analysis (SDA) on the hand operation data in the recorded results and the surgical expert action database to obtain the SDA results; and generating the user's training evaluation report based on the user's skill proficiency score and the SDA results. By objectively quantifying the surgical simulation training time spent by the user, the evaluation results are made more fair and accurate. By comparing the user's hand operation data with the surgical expert action database, the system can identify the user's strengths and weaknesses in surgical operations. The training evaluation report generated by combining the skill proficiency score and the SDA results helps the user identify their problems and take corresponding measures to improve, thereby improving their surgical skills more quickly.

[0054] Furthermore, in this embodiment, during surgical simulation training, it can distinguish the different functions of the left and right hands, and simultaneously support the recognition of complex gestures involving coordinated two-handed operation. The implementation process includes: 1. Asymmetrical two-handed character modeling This system innovatively introduces "surgical role modeling" technology, which distinguishes the different functions of the left and right hands in the following ways: Dynamic dominant and auxiliary hand recognition: The system does not pre-determine which hand is the dominant or auxiliary hand, but dynamically identifies the current dominant and auxiliary hand relationship by analyzing the complexity and accuracy of the gesture; Role-specific feature extraction: Design different feature extraction pipelines for the main operator and the assistant operator. The main operator focuses on fine-grained operation and tool control, while the assistant operator focuses on organizational guidance and scene maintenance. Functional differentiation recognition threshold: Automatically adjust the recognition sensitivity and accuracy requirements of the left and right hands according to the type of surgery, such as improving the accuracy requirements of the dominant hand in microsurgery.

[0055] 2. Hand-to-hand collaborative semantic understanding network: Spatial Relationship Encoder: Captures the relative position, distance, and orientation of the left and right hands to construct a spatial relationship feature map of both hands; Cooperative motion parser: Recognizes the coordinated motion patterns of the left and right hands, such as the coordinated work of one hand fixing tissue and the other hand performing cutting. Temporal dependency graph network: Establish temporal dependency graphs between left and right hand movements to understand the temporal coordination relationship between the two hands; The Bilateral Cooperative Semantic Understanding Network treats both hands as a unified operating system, rather than two independent gesture sources, fundamentally changing the mode of gesture recognition.

[0056] 3. Multi-perspective integrated conflict resolution mechanism Active occlusion detection: The system detects mutual occlusion of hands in real time and estimates the possible location of the occluded parts. Predictive viewpoint switching: Before detecting potential occlusion, the system predictively adjusts the focus viewpoint to ensure the continuity of gesture capture. Partial observation completion: Inferring the position of the obscured finger based on the physical constraints of the hand skeleton and the continuity of previous movements. Multi-view data fusion: When using multiple depth cameras, the system fuses observations from different perspectives using a probabilistic model to overcome the limitations of a single perspective. This mechanism solves the recognition failure problem of traditional systems when both hands are crossed, providing necessary technical support for precision surgical training.

[0057] 4. Surgical procedure awareness context recognition engine Automatic surgical stage identification: The system can automatically identify which surgical stage is currently in based on the sequence of operations already performed; Stage-specific gesture library: Establish specialized gesture recognition models for different surgical stages (such as incision, separation, suturing, etc.); Surgical rule-constrained reasoning: Introducing a reasoning mechanism based on surgical rules to exclude unreasonable gesture interpretations in the current context; Experience pattern matching: Based on an expert surgical database, observed bimanual coordination patterns are matched with standard operating procedures; This technology significantly improves the accuracy of gesture recognition in complex surgical environments, far exceeding the performance of general gesture recognition systems.

[0058] This embodiment also provides the following specific application scenarios: During neurosurgical navigation training, Dr. Wang needed to practice navigation techniques for brain tumor resection. He entered the hospital's simulation training room, put on the system's AR glasses, and stood in front of the training table, facing a pre-set depth camera.

[0059] After the system starts, it automatically identifies Dr. Wang and loads his previous training records and personalized settings. The system prompts Dr. Wang to select a training mode, and he selects the "Neurosurgical Navigation Surgery - Brain Tumor Resection" module via gesture. The system immediately loads the brain MRI and CT scan data of a virtual patient from the database and projects a three-dimensional anatomical model of the patient's head onto the training table through AR glasses, including the precise location of the skin, skull, brain tissue, and tumor.

[0060] Dr. Wang began the planning phase before training. He extended his right hand, making a "grabbing" gesture, which the system recognized and generated a virtual surgical planning pen in the AR field of view. He used this virtual pen to mark the expected incision location on the patient's head, while the system simultaneously displayed the anatomical structures below the incision in real time. When Dr. Wang's marking deviated from the optimal approach, the system displayed a red warning mark through the AR interface and provided suggestions for improvement.

[0061] Next, the navigation and positioning phase began. Dr. Wang made a "confirm" gesture (extending his thumb), and the system switched to navigation mode. He used his left hand to make a "zoom" gesture and his right hand to make a "rotate" gesture, flexibly adjusting the size and angle of the patient's head model to better observe the surgical path. The system recognized these gestures and responded in real time, causing the virtual model to change according to the doctor's intentions.

[0062] When Dr. Wang needed to measure the depth of a tumor, he made a specific "measurement" gesture (extending and separating his index and middle fingers). The system immediately created a measurement line between the two fingers and displayed the precise distance value. By moving his fingers, he measured the distance between the tumor and important nerves and blood vessels from different angles, and the system updated the measurement results in real time.

[0063] During the simulated surgical procedure, Dr. Wang selected different virtual surgical tools, such as scalpels, drills, and micro-forceps, using specific gestures. The system simulated tissue reactions in real time based on the tool type and the precision of the operation. For example, when he used virtual micro-forceps to grasp tumor tissue, the system displayed appropriate tissue deformation and bleeding effects, providing realistic visual feedback.

[0064] During training, the system continuously recorded Dr. Wang's gesture data, operation trajectory, and decision-making time. When his gestures during a crucial step were not precise enough, the system displayed guiding prompts of standard gesture movements through the AR interface to help him correct technical details. Simultaneously, based on its built-in expert operation database, the system evaluated his operational efficiency and safety in real time, displaying a real-time score in the corner of the AR field of view.

[0065] After completing an hour of training, Dr. Wang made a "finish" gesture (crossing his hands). The system generated a detailed training report, including multi-dimensional assessments of the rationality of the surgical path selection, the stability of the hand gestures, and the quality of key steps. The report also compared Dr. Wang's training with previous records, highlighting areas of improvement and areas that still need to be addressed. Based on the identified weaknesses, the system also recommended targeted training content, such as specialized training modules on "microvascular identification and protection."

[0066] The training data was automatically saved to the hospital's training management system. The director of teaching can use the data analysis to evaluate Dr. Wang's surgical skill development and provide personalized guidance. Simultaneously, anonymized data from all training physicians was used to improve the system's gesture recognition algorithm and optimize the training content.

[0067] Please refer to Figure 5 Embodiment 2 of the present invention is: an augmented reality surgical simulation training terminal based on gesture recognition, including a memory 2, a processor 3, and a computer program stored on the memory 2 and executable on the processor 3. When the processor 3 executes the computer program, it implements the various steps of the augmented reality surgical simulation training method based on gesture recognition in Embodiment 1.

[0068] In summary, the present invention provides an augmented reality surgical simulation training method and terminal based on gesture recognition. Responding to user selection operations, it loads surgical simulation data corresponding to the selected operation from a surgical simulation database, generates an augmented reality scene based on the loaded surgical simulation data, and utilizes augmented reality technology to provide a more realistic and intuitive environment for surgical simulation training. By collecting the three-dimensional coordinates of key hand points and calculating the curvature and phase position relationship of finger joints, and using a deep learning model for gesture classification and recognition, it can quickly process hand operation data, determine the user's current surgical gesture, determine the current surgical stage based on the user's surgical operation sequence, and understand the meaning of the user's gesture in a specific surgical environment based on the current surgical gesture, the current surgical stage, and a surgical expert action database, improving the accuracy of operation intent inference. Based on the user's operation intent data, it generates corresponding virtual surgical tools or provides feedback information on surgical gestures in the augmented reality scene, improving the flexibility and realism of the surgical simulation. Real-time monitoring of the user's hand operation data, when a mismatch is detected between the user's surgical operation and a preset standard surgical operation, provides visual warning information and real-time guidance information in the augmented reality scene, improving the effectiveness and safety of surgical simulation training. In addition, the system records the user's surgical simulation training process. By objectively quantifying the time spent on surgical simulation training and comparing the user's hand operation data with the surgical expert's action database, it obtains skill proficiency scores and standard difference analysis results to generate training evaluation reports. This helps users identify their problems and take corresponding measures to improve, thereby improving their surgical skills more quickly.

[0069] Therefore, the embodiments of the present invention can achieve the following beneficial effects: 1. Natural interactive experience: By recognizing the doctor's natural surgical gestures, no additional controllers or markers are needed, which greatly reduces the learning cost and enables doctors to train in a way that is closest to real surgery, shortening the skill conversion time.

[0070] 2. Precise gesture recognition: Utilizing deep learning algorithms and a high-precision depth camera, it can accurately recognize doctors' complex surgical gestures, achieving sub-millimeter-level operational precision to meet the needs of fine surgical training.

[0071] 3. Augmented Reality Immersive Experience: By overlaying virtual anatomical structures and surgical information into the real space using AR technology, the tactile and spatial sensations of the real environment are preserved while avoiding the dizziness that VR systems may cause, making it more suitable for long-term training.

[0072] 4. Personalized training plans: Based on the user's training history and skill level, the training content and difficulty are automatically adjusted to provide targeted improvement plans, greatly improving training efficiency.

[0073] 5. Objective skills assessment: By digitally recording and analyzing every detail of the doctor's operation, objective and comprehensive skills assessment is provided, making up for the shortcomings of inconsistent evaluation standards in traditional apprenticeship training.

[0074] 6. Reduced training costs: Compared with traditional physical models or animal experiments, this system does not require physical materials, can be reused, and the scheme can be flexibly adjusted, which greatly reduces the training costs of hospitals.

[0075] 7. Abundant navigation training scenarios: The system has multiple pre-set surgical navigation scenarios and case variations, which can simulate the handling of various complex situations and unexpected circumstances, and improve doctors' ability to deal with complex cases.

[0076] 8. Data accumulation and continuous optimization: The training data collected by the system can be used for medical education research and algorithm optimization. As the usage time increases, the system's recognition accuracy and training content will continue to improve.

[0077] 9. Remote collaboration capability: The system supports remote experts to access the same AR environment and guide trainees to operate through gestures, breaking through geographical limitations and realizing the effective transfer of valuable surgical experience.

[0078] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent modifications made based on the content of the present invention specification and drawings, or direct or indirect applications in related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. An augmented reality surgical simulation training method based on gesture recognition, characterized in that, Including the following steps: S1. In response to the user's selection operation, load the surgical simulation data corresponding to the selection operation from the surgical simulation database, and generate an augmented reality scene based on the loaded surgical simulation data; S2. Collect the user's hand operation data, use a gesture recognition model to identify the hand operation data to determine operation intention data, and adjust the augmented reality scene according to the operation intention data to simulate surgical interaction operation; S3. Monitor the user's hand operation data in real time. When the user's hand operation data does not match the preset standard surgical operation, provide visual warning information and real-time guidance information in the augmented reality scene.

2. The augmented reality surgical simulation training method based on gesture recognition according to claim 1, characterized in that, The process of collecting user hand gesture data and using a gesture recognition model to identify the hand gesture data to determine the operation intention data includes: Collect the three-dimensional coordinates of preset key points on the user's hand, calculate the curvature and relative position of each finger joint based on the three-dimensional coordinates, and generate a gesture feature vector; A deep learning model is used to classify and recognize the gesture feature vector. Based on the classification result, gesture data of the corresponding category is obtained from the gesture database. The gesture feature vector is matched with the gesture data to determine the user's current surgical gesture. The current surgical stage of the user is determined based on the sequence of surgical operations already performed by the user. Contextual analysis is then performed based on the user's current surgical gestures, the current surgical stage, and the surgical expert action database to determine the operational intent data corresponding to the user's current surgical gestures.

3. The augmented reality surgical simulation training method based on gesture recognition according to claim 2, characterized in that, Adjusting the augmented reality scene based on the operational intent data to simulate surgical interaction operations includes: If the operation intent data is to obtain a virtual surgical tool, then the corresponding virtual surgical tool is determined and generated based on the user's current surgical gesture, and the virtual surgical tool is added to the augmented reality scene; If the operation intent data is feedback information for obtaining the current surgical gesture, then the augmented reality scene is adjusted according to the user's current surgical gesture to generate surgical operation visual data corresponding to the user's current surgical gesture.

4. The augmented reality surgical simulation training method based on gesture recognition according to claim 1, characterized in that, Step S3 further includes: Record the user's surgical simulation training process, generate a training evaluation report for the user based on the recording results, and find and display the corresponding training content and improvement suggestions based on the training evaluation report.

5. The augmented reality surgical simulation training method based on gesture recognition according to claim 4, characterized in that, The step of generating the user's training evaluation report based on the recorded results includes: The user's skill proficiency is scored based on the surgical simulation training time recorded in the results. The hand manipulation data in the recorded results were compared with the surgical expert action database using a standard difference analysis to obtain the standard difference analysis results. A training evaluation report for the user is generated based on the user's skill proficiency score and the results of the standard difference analysis.

6. An augmented reality surgical simulation training terminal based on gesture recognition, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it performs the following steps: S1. In response to the user's selection operation, load the surgical simulation data corresponding to the selection operation from the surgical simulation database, and generate an augmented reality scene based on the loaded surgical simulation data; S2. Collect the user's hand operation data, use a gesture recognition model to identify the hand operation data to determine operation intention data, and adjust the augmented reality scene according to the operation intention data to simulate surgical interaction operation; S3. Monitor the user's hand operation data in real time. When the user's hand operation data does not match the preset standard surgical operation, provide visual warning information and real-time guidance information in the augmented reality scene.

7. The augmented reality surgical simulation training terminal based on gesture recognition according to claim 6, characterized in that, The process of collecting user hand gesture data and using a gesture recognition model to identify the hand gesture data to determine the operation intention data includes: Collect the three-dimensional coordinates of preset key points on the user's hand, calculate the curvature and relative position of each finger joint based on the three-dimensional coordinates, and generate a gesture feature vector; A deep learning model is used to classify and recognize the gesture feature vector. Based on the classification result, gesture data of the corresponding category is obtained from the gesture database. The gesture feature vector is matched with the gesture data to determine the user's current surgical gesture. The current surgical stage of the user is determined based on the sequence of surgical operations already performed by the user. Contextual analysis is then performed based on the user's current surgical gestures, the current surgical stage, and the surgical expert action database to determine the operational intent data corresponding to the user's current surgical gestures.

8. The augmented reality surgical simulation training terminal based on gesture recognition according to claim 7, characterized in that, Adjusting the augmented reality scene based on the operational intent data to simulate surgical interaction operations includes: If the operation intent data is to obtain a virtual surgical tool, then the corresponding virtual surgical tool is determined and generated based on the user's current surgical gesture, and the virtual surgical tool is added to the augmented reality scene; If the operation intent data is feedback information for obtaining the current surgical gesture, then the augmented reality scene is adjusted according to the user's current surgical gesture to generate surgical operation visual data corresponding to the user's current surgical gesture.

9. The augmented reality surgical simulation training terminal based on gesture recognition according to claim 6, characterized in that, Step S3 further includes: Record the user's surgical simulation training process, generate a training evaluation report for the user based on the recording results, and find and display the corresponding training content and improvement suggestions based on the training evaluation report.

10. An augmented reality surgical simulation training terminal based on gesture recognition according to claim 9, characterized in that, The step of generating the user's training evaluation report based on the recorded results includes: The user's skill proficiency is scored based on the surgical simulation training time recorded in the results. The hand manipulation data in the recorded results were compared with the surgical expert action database using a standard difference analysis to obtain the standard difference analysis results. A training evaluation report for the user is generated based on the user's skill proficiency score and the results of the standard difference analysis.

Images