Surgical skill training system based on AI visual judgment

Abstract

The invention discloses a surgical skill training system based on AI visual judgment, and relates to the technical field of surgical skill training and evaluation, and the system comprises a high-precision visual perception module which employs an instance segmentation algorithm based on deep learning, supports the millisecond-level recognition of 15 types of common surgical instruments, and provides a high-precision visual perception module. The output parameters comprise three-dimensional space coordinates, an instrument opening and closing angle and a tissue contact state; identifying different tissue types through multispectral imaging, detecting tissue abnormal change in real time, positioning a bleeding point, calculating traction deformation, and marking a thermal damage area; the operation behavior analysis module is used for performing action atomization processing, decomposing continuous operation into 200 + type basic action units, and establishing an action relation graph by adopting a space-time convolutional network; identifying an operation stage, and defining a standardized process node of a typical operation; and judging the current stage through the LSTM time sequence model. According to the invention, the efficiency and effect of surgical training are improved.

Description

Technical Field

[0001] This invention relates to the field of surgical skills training and assessment technology, and in particular to a surgical skills training system based on AI visual judgment. Background Technology

[0002] Efficient and standardized training in surgical skills is a core element in improving medical quality and patient safety. Traditional skills assessment primarily relies on subjective evaluation by senior surgical experts through direct observation (e.g., in the operating room or training laboratory) using standardized rating scales. This approach has significant drawbacks:

[0003] 1. High degree of subjectivity and variability: Different reviewers have different understandings and grasps of the scoring criteria, resulting in a lack of consistency and comparability in the evaluation results. Reviewers' experience, fatigue, and even personal preferences can all affect the scores.

[0004] 2. Resource-intensive and not scalable: Relying on expert on-site observation and scoring is time-consuming and labor-intensive, making it difficult to meet the needs of large-scale and frequent training and assessment, especially when senior doctors have limited time, thus becoming a bottleneck for the expansion of the training system.

[0005] 3. Delayed and non-quantitative feedback: Evaluation results are usually summative and not real-time, preventing trainees from immediately learning about errors and optimization suggestions during the operation. Feedback is mostly qualitative description, lacking detailed and objective quantitative data to guide specific improvement directions.

[0006] 4. Difficulty in capturing microscopic kinematic characteristics: The human eye has difficulty accurately and continuously tracking and quantifying the subtle movement trajectories (such as path length, speed changes, shaking frequency, hand coordination) and operational efficiency of surgical instruments.

[0007] Therefore, there is an urgent need for a surgical skills training system based on AI visual judgment to change this situation. Summary of the Invention

[0008] The purpose of this invention is to address the shortcomings of existing technologies by proposing an AI-based visual judgment-based surgical skills training system. Its advantages lie in improving the efficiency and effectiveness of surgical training.

[0009] To achieve the above objectives, the present invention adopts the following technical solution:

[0010] The AI-based visual judgment surgical skills training system includes a high-precision visual perception module, which adopts a deep learning-based instance segmentation algorithm to support millisecond-level recognition of 15 common surgical instruments. The output parameters include three-dimensional spatial coordinates, instrument opening and closing angles, and tissue contact status. It identifies different tissue types through multispectral imaging, detects abnormal tissue changes in real time, locates bleeding points, calculates traction deformation, and marks thermal damage areas.

[0011] The surgical behavior analysis module atomizes actions, decomposing continuous operations into 200+ basic action units and using a spatiotemporal convolutional network to establish an action relationship graph; it identifies operation stages by defining standardized process nodes for typical surgeries; and it uses an LSTM temporal model to determine the current stage.

[0012] Multi-dimensional skills assessment module: kinematic analysis, path efficiency, ratio of actual path to ideal path; hand tremor, Fourier analysis of high-frequency instrument vibration; bimanual coordination, phase difference analysis of left and right hand movements; suture quality inspection, stitch spacing uniformity, knot firmness, tissue treatment evaluation, unnecessary traction count, and protection integrity of critical structures.

[0013] The real-time interactive feedback module features 3D navigation guidance, overlaying virtual operation paths onto a real-world scene, real-time highlighting of operational errors, a progress bar, display of remaining time for the current step and overall completion status, and a multi-channel feedback mechanism.

[0014] This invention is further configured to: construct an AI evaluation system that does not rely on human intervention, automatically outputting objective and reproducible quantitative results; and construct a multi-dimensional quantitative indicator system, establishing a computable indicator set aligned with clinical evaluation standards.

[0015] \begin{align*};

[0016] \text{Kinematic parameters:}&\quad\text{Path length};

[0017] L=\int_{t_0}^{t_1}\sqrt{v_x^2+v_y^2+v_z^2}dt\\

[0018] &\quad\text{motion smoothness};

[0019] F=1-\frac{\sigma_a}{\mu_a}\quad(a:\text{Acceleration})\\

[0020] \text{Operative metrics:}&\quad\text{Tissue damage rate};

[0021] D = \frac{T_{\text{err}}}{T_{\text{total}}}\times100\%\\&\quad\text{stitching precision};

[0022] P=\frac{1}{n}\sum_{i=1}^ne^{-\lambda|d_i-d_{\text{targe t}}|}\quad(d_i:\text{needle pitch});

[0023] \end{align*}.

[0024] The present invention is further configured as follows: robust visual feature extraction, surgical scene parsing model (based on improved U-Net++):

[0025] \begin{align*};

[0026] \text{Instrument segmentation:}&\quad\mathcal{L}_{\text{seg}}=\alpha

[0027] \mathcal{L}_{\text{Dice}}+(1-\alpha)\mathcal{L}_{\text{CE}}\\;

[0028] Attitude estimation:

[0029] &\quad\hat{\theta}=\arg\min_{\theta}\sum\|

[0030] \Phi(I;\theta)-K_{\text{3D}}\|_2+\lambdaR(\theta)

[0031] \end{align*};

[0032] in:

[0033] $\Phi$: Pose estimation network (HRNet variant); $K_{\text{3D}}$: True coordinates of the machine keypoints; $R(\theta)$: Motion smoothing regularization term.

[0034] The present invention is further configured to construct a temporal action segmentation algorithm:

[0035] y_t=\text{Softmax}\left(\text{Conv1D}(\text{ReLU}(W*x_{tk:t+k})\right);

[0036] Output: Action label at time $t$: $y_t\in{\text{holding needle, piercing, thread,...}}$.

[0037] This invention is further configured as a skills assessment model (core innovation) and a multimodal feature fusion assessment framework:

[0038] \text{Score}=

[0039] f\left(\underbrace{\mathbf{F}_{\text{kinematic}}}_{\text{motion characteristics}};

[0040] \underbrace{\mathbf{F}_{\text{context}}}_{\text{scene semantics}};

[0041] \underbrace{\mathbf{F}_{\text{task}}}_{\text{task parameters}}\right);

[0042] Output: 6-dimensional OSATS score vector $\in[0,5]^6$.

[0043] The present invention is further configured with real-time performance assurance technology and a lightweight inference scheme:

[0044] \begin{array}{c|c};

[0045] \text{module}&\text{delay}\\\hline;

[0046] \text{device tracking}&<15\text{ms}\\;

[0047] \text{action recognition}&<30\text{ms}\\;

[0048] \text{rating generation}&<50\text{ms}\\;

[0049] \end{array};

[0050] Key technologies:

[0051] Model pruning (Layer pruning ratio $\eta = 0.4$);

[0052] Quantitative deployment (FP16→INT8).

[0053] The present invention is further configured such that the multi-channel feedback mechanism includes visual alerts, tactile reminders, and voice guidance.

[0054] The invention is further configured such that: the visual warning flashes red light when the instrument approaches a dangerous area; the tactile reminder vibrates the fingertips of the glove when the operating force exceeds the limit; and the voice guidance reminds users of any missed key steps.

[0055] The beneficial effects of this invention are as follows:

[0056] 1. This AI-based visual judgment surgical skills training system improves the objectivity of assessment, completely eliminates the subjective bias of human scoring, and the scoring results are 0.93±0.02 with the expert group's ICC value. The difference in repeated assessments of the same operation is <2%. Training efficiency is optimized, the time for trainees to meet the standards is shortened by 40-45%, the rate of standardized operation is increased by 60-65%, and the pass rate of assessment is increased by 25-30%.

[0057] 2. This AI-based surgical skills training system boasts breakthroughs in perception capabilities. Multispectral fusion enables instrument recognition to reach 99.2%, tissue condition detection accuracy to reach 0.3mm, and anti-interference capabilities to be improved by 3 times. Its intelligent assessment advantages support the automatic decomposition of 200+ atomic actions and can quantify and assess 8 clinical dimensions. New surgical procedures can be adapted with only 10 samples.

[0058] 3. This AI-based visual judgment surgical skills training system reforms teaching, standardizes and digitizes training, reduces expert guidance costs by 85%, and supports remote assessment and guidance; it also ensures safety by providing risk operation warnings 2-3 seconds in advance, reducing training accidents by 78%, and fully records the operation process for future reference.

[0059] 4. This AI-based visual judgment surgical skills training system directly saves costs, reducing consumable usage by 30-40%, reducing the number of repetitive training sessions by 50%, and achieving a return on investment period of less than 1.5 years. Indirect benefits include shortening the physician training cycle by 6-8 months, increasing operating room turnover by 15%, and reducing medical error-related compensation.

[0060] 5. This AI-based visual judgment surgical skills training system is hardware compatible, supports five mainstream training devices, can connect to various surgical robots, and is compatible with various VR / AR / MR terminals; the algorithm has evolved, shortening the model iteration cycle to two weeks, supports federated learning and collaborative optimization, and the evaluation index library can be dynamically expanded. Attached Figure Description

[0061] Figure 1 This is a schematic diagram of the overall equipment of the surgical skills training system based on AI visual judgment proposed in this invention;

[0062] Figure 2 This is a schematic diagram of the hardware system of the surgical skills training system based on AI visual judgment proposed in this invention;

[0063] Figure 3 This is a schematic diagram of the software control interface of the surgical skills training system based on AI visual judgment proposed in this invention. Detailed Implementation

[0064] The technical solution of this patent will be further described in detail below with reference to specific embodiments.

[0065] The embodiments of this patent are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this patent, and should not be construed as limiting this patent.

[0066] Reference Figure 1-3 The AI-based visual judgment surgical skills training system includes a high-precision visual perception module, which adopts a deep learning-based instance segmentation algorithm to support millisecond-level recognition of 15 common surgical instruments. The output parameters include three-dimensional spatial coordinates, instrument opening and closing angles, and tissue contact status. It identifies different tissue types through multispectral imaging, detects abnormal tissue changes in real time, locates bleeding points, calculates traction deformation, and marks thermal damage areas.

[0067] The surgical behavior analysis module atomizes actions, decomposing continuous operations into 200+ basic action units and using a spatiotemporal convolutional network to establish an action relationship graph; it identifies operation stages by defining standardized process nodes for typical surgeries; and it uses an LSTM temporal model to determine the current stage.

[0068] Multi-dimensional skills assessment module: kinematic analysis, path efficiency, ratio of actual path to ideal path; hand tremor, Fourier analysis of high-frequency instrument vibration; bimanual coordination, phase difference analysis of left and right hand movements; suture quality inspection, stitch spacing uniformity, knot firmness, tissue treatment evaluation, unnecessary traction count, and protection integrity of critical structures.

[0069] The real-time interactive feedback module features 3D navigation guidance, overlaying virtual operation paths onto a real-world scene, real-time annotation of operation error points, a progress bar, display of remaining time for the current step and overall completion status, and a multi-channel feedback mechanism.

[0070] The multi-channel feedback mechanism includes visual warnings, tactile reminders, and voice guidance; visual warnings are triggered when the instrument approaches a dangerous area, with the target area flashing red; tactile reminders are triggered when the operating force is excessive, with the fingertips of the glove vibrating; and voice guidance is used to remind users of missed key steps.

[0071] In this embodiment, an AI-based assessment system that does not rely on human intervention is constructed to automatically output objective and reproducible quantitative results; a multi-dimensional quantitative indicator system is built to establish a set of computable indicators aligned with clinical assessment standards.

[0072] \begin{align*};

[0073] \text{Kinematic parameters:}&\quad\text{Path length};

[0074] L=\int_{t_0}^{t_1}\sqrt{v_x^2+v_y^2+v_z^2}dt\\

[0075] &\quad\text{motion smoothness};

[0076] F=1-\frac{\sigma_a}{\mu_a}\quad(a:\text{Acceleration})\\

[0077] \text{Operative metrics:}&\quad\text{Tissue damage rate};

[0078] D = \frac{T_{\text{err}}}{T_{\text{total}}}\times100\%\\&\quad\text{stitching precision};

[0079] P=\frac{1}{n}\sum_{i=1}^ne^{-\lambda|d_i-d_{\text{targe t}}|}\quad(d_i:\text{needle pitch});

[0080] \end{align*}.

[0081] Robust visual feature extraction, surgical scene parsing model (based on improved U-Net++):

[0082] \begin{align*};

[0083] \text{Instrument segmentation:}&\quad\mathcal{L}_{\text{seg}}=\alpha

[0084] \mathcal{L}_{\text{Dice}}+(1-\alpha)\mathcal{L}_{\text{CE}}\\;

[0085] Attitude estimation:

[0086] &\quad\hat{\theta}=\arg\min_{\theta}\sum\|

[0087] \Phi(I;\theta)-K_{\text{3D}}\|_2+\lambdaR(\theta)

[0088] \end{align*};

[0089] in:

[0090] $\Phi$: Pose estimation network (HRNet variant); $K_{\text{3D}}$: True coordinates of the machine keypoints; $R(\theta)$: Motion smoothing regularization term.

[0091] Anti-interference design:

[0092] An Occlusion Awareness Module (OAM) is introduced to handle tissue occlusion;

[0093] Multispectral fusion is used to resist blood / smoke interference.

[0094] Hierarchical Action Recognition Framework

[0095] Constructing a temporal action segmentation algorithm:

[0096] y_t=\text{Softmax}\left(\text{Conv1D}(\text{ReLU}(W*x_{tk:t+k})\right);

[0097] Output: Action label at time $t$: $y_t\in{\text{holding needle, piercing, thread,...}}$.

[0098] Skills assessment model (core innovation), multimodal feature fusion assessment framework:

[0099] \text{Score}=

[0100] f\left(\underbrace{\mathbf{F}_{\text{kinematic}}}_{\text{motion characteristics}};

[0101] \underbrace{\mathbf{F}_{\text{context}}}_{\text{scene semantics}};

[0102] \underbrace{\mathbf{F}_{\text{task}}}_{\text{task parameters}}\right);

[0103] Output: 6-dimensional OSATS score vector $\in[0,5]^6$.

[0104] Real-time performance guarantee technology, lightweight inference solution:

[0105] \begin{array}{c|c};

[0106] \text{module}&\text{delay}\\\hline;

[0107] \text{device tracking}&<15\text{ms}\\;

[0108] \text{action recognition}&<30\text{ms}\\;

[0109] \text{rating generation}&<50\text{ms}\\;

[0110] \end{array};

[0111] Key technologies:

[0112] Model pruning (Layer pruning ratio $\eta = 0.4$);

[0113] Quantitative deployment (FP16→INT8).

[0114] The above algorithm, through a technical chain of quantitative index definition → robust feature extraction → hierarchical action understanding → multimodal evaluation modeling, achieves industrial-grade standardization of surgical skill assessment for the first time.

[0115] The system deployment involves installing a trinocular vision module (containing two visible light cameras and one near-infrared camera) and a depth sensor on the top of a standard surgical training box, connected to an edge computing host (NVIDIA Jetson AGX Xavier). Multi-camera calibration is performed using a checkerboard pattern, with spatial positioning errors controlled within ±0.15mm.

[0116] Once the training is initiated and a training exercise (such as laparoscopic suturing) is selected, the system will automatically load it.

[0117] Three-dimensional anatomical model, expert operation database. During real-time training: 60fps video stream undergoes visual processing (18ms) → motion analysis (22ms) → feedback generation (10ms). When the instrument is less than 2mm from the danger zone, an AR red alarm and vibration alert are triggered. A yellow correction guide is displayed when the suture stitch distance difference is greater than 1.2mm.

[0118] The system is evaluated and maintained, and a 3D report (including an operation heatmap and 8-dimensional scoring) is generated after training, supporting historical data comparison. Monthly system calibration is performed, and the algorithm model is updated through federated learning.

[0119] This implementation method can be deployed in 30 minutes, achieving: a 63% increase in operational standardization rate, a 40% reduction in training cycle, a feedback delay of <50ms, and a 95% reduction in subjective evaluation bias.

[0120] Working Principle: Based on multimodal visual perception and artificial intelligence algorithms, a full-stack quantitative assessment and feedback closed-loop system for surgical operations is constructed. Surgical scenes are captured by multispectral cameras, and instruments / tissues are identified in real time using an anti-interference segmentation network (such as an improved U-Net). Instrument kinematic data and scene semantics are fused (3D-CNN + Transformer) to decompose the operation into atomic actions. A mapping model from multi-dimensional indicators to clinical scores is established (dual-stream fusion architecture). AR guidance (such as trajectory correction and hazard warning) is generated based on a rule engine with a latency of <50ms. By replacing human observation with computer vision and expert experience with AI algorithms, this system overcomes the triple bottlenecks of subjectivity, efficiency, and accuracy in traditional training, providing a standardized intelligent infrastructure for surgical education.

[0121] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A surgical skills training system based on AI visual judgment, characterized in that, It includes a high-precision visual perception module that uses a deep learning-based instance segmentation algorithm to support millisecond-level recognition of 15 common surgical instruments. The output parameters include three-dimensional spatial coordinates, instrument opening and closing angles, and tissue contact status. It can identify different tissue types through multispectral imaging, detect abnormal tissue changes in real time, locate bleeding points, calculate traction deformation, and mark thermal damage areas. The surgical behavior analysis module atomizes actions, decomposing continuous operations into 200+ basic action units and using a spatiotemporal convolutional network to establish an action relationship graph; it identifies operation stages by defining standardized process nodes for typical surgeries; and it uses an LSTM temporal model to determine the current stage. Multi-dimensional skills assessment module, kinematic analysis, path efficiency, ratio of actual path to ideal path; Hand tremor: Fourier transform analysis of high-frequency instrument vibration; Bimanual coordination: phase difference analysis of left and right hand movements; Suture quality inspection: stitch spacing uniformity, knot strength, tissue treatment evaluation, unnecessary traction counts, and protection integrity of critical structures. The real-time interactive feedback module features 3D navigation guidance, overlaying virtual operation paths onto a real-world scene, real-time highlighting of operational errors, a progress bar, display of remaining time for the current step and overall completion status, and a multi-channel feedback mechanism.

2. The surgical skills training system based on AI visual judgment according to claim 1, characterized in that, Build an AI-powered assessment system that does not rely on human intervention, automatically outputting objective and reproducible quantitative results; construct a multi-dimensional quantitative indicator system, establishing a computable indicator set aligned with clinical assessment standards: \begin{align*}; \text{Kinematic parameters:}&\quad\text{Path length}; L=\int_{t_0}^{t_1}\sqrt{v_x^2+v_y^2+v_z^2}dt\\ &\quad\text{motion smoothness}; F=1-\frac{\sigma_a}{\mu_a}\quad(a:\text{Acceleration})\\ \text{Operative metrics:}&\quad\text{Tissue damage rate}; D = \frac{T_{\text{err}}}{T_{\text{total}}}\times100\%\\&\quad\text{stitching precision}; P=\frac{1}{n}\sum_{i=1}^ne^{-\lambda|d_i-d_{\text{targe t}}|}\quad(d_i:\text{needle pitch}); \end{align*}.

3. The surgical skills training system based on AI visual judgment according to claim 2, characterized in that, Robust visual feature extraction, surgical scene parsing model (based on improved U-Net++): \begin{align*}; \text{Instrument segmentation:}&\quad\mathcal{L}_{\text{seg}}=\alpha \mathcal{L}_{\text{Dice}}+(1-\alpha)\mathcal{L}_{\text{CE}}\\; Attitude estimation: &\quad\hat{\theta}=\arg\min_{\theta}\sum\| \Phi(I;\theta)-K_{\text{3D}}\|_2+\lambdaR(\theta) \end{align*}; in: $\Phi$: Pose estimation network (HRNet variant); $K_{\text{3D}}$: True coordinates of the machine keypoints; $R(\theta)$: Motion smoothing regularization term.

4. The surgical skills training system based on AI visual judgment according to claim 3, characterized in that, Constructing a temporal action segmentation algorithm: y_t=\text{Softmax}\left(\text{Conv1D}(\text{ReLU}(W*x_{tk:t+k})\right); Output: Action label at time $t$: $y_t\in{\text{holding needle, piercing, thread,...}}$.

5. The surgical skills training system based on AI visual judgment according to claim 4, characterized in that, Skills assessment model (core innovation), multimodal feature fusion assessment framework: \text{Score}= f\left(\underbrace{\mathbf{F}_{\text{kinematic}}}_{\text{motion characteristics}}; \underbrace{\mathbf{F}_{\text{context}}}_{\text{scene semantics}}; \underbrace{\mathbf{F}_{\text{task}}}_{\text{task parameters}}\right); Output: 6-dimensional OSATS score vector $\in[0,5]^6$.

6. The surgical skills training system based on AI visual judgment according to claim 5, characterized in that, Real-time performance guarantee technology, lightweight inference solution: \begin{array}{c|c}; \text{module}&\text{delay}\\\hline; \text{device tracking}&<15\text{ms}\\; \text{action recognition}&<30\text{ms}\\; \text{rating generation}&<50\text{ms}\\; \end{array}; Key technologies: Model pruning (Layer pruning ratio $\eta = 0.4$); Quantitative deployment (FP16→INT8).

7. The surgical skills training system based on AI visual judgment according to claim 1, characterized in that, The multi-channel feedback mechanism includes visual alerts, tactile reminders, and voice guidance.

8. The surgical skills training system based on AI visual judgment according to claim 7, characterized in that, The visual warning flashes red when the equipment approaches a dangerous area; the tactile alert vibrates at the fingertips of the glove when the operating force is excessive; and the voice guidance reminds users of any missed steps.

Images