A method and system for surgical training and assessment based on mixed reality technology
By establishing a unified coordinate system and a composite structure locator using mixed reality technology, the problem of easy detachment of positioning markers during surgical training was solved, achieving stable recognition and quantitative evaluation under multiple postures, and improving the safety and efficiency of training.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHANGHAI QUANSHI INTELLIGENT SENSE TECHNOLOGY CO LTD
- Filing Date
- 2026-03-18
- Publication Date
- 2026-06-12
AI Technical Summary
Existing surgical skills training systems often have positioning markers that easily fall out of the recognition range during multi-pose operations, leading to unstable pose calculations that are difficult to meet the needs of complex surgical operations. Furthermore, the lack of objective quantification in training evaluation affects training effectiveness and safety.
The system, based on mixed reality technology, establishes a unified coordinate system through a calibration board. It combines a composite structure locator and a binocular camera to identify the position and pose of surgical instruments, achieving stable identification of surgical instruments in multiple poses. The training process is broken down into multiple steps for quantitative evaluation, generating improvement suggestions.
It improves the operational consistency and safety of surgical training, ensures that virtual guidance corresponds accurately to the physical model, reduces human error, provides objective quantitative evaluation results, and helps trainees identify weaknesses and optimize training.
Smart Images
Figure CN122199226A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of medical information technology, and in particular to a surgical training and evaluation method and system based on mixed reality technology. Background Technology
[0002] Surgical skills training is a crucial component of medical education and clinical competence development, and its effectiveness directly impacts surgical safety and medical quality. To enhance the intuitiveness and repeatability of training, some existing technologies incorporate computer vision and navigation techniques. These technologies use cameras to capture the positional information of instruments and models, and display instrument deviations, standard paths, or operational prompts on external display devices.
[0003] However, the above training methods usually rely on a single display terminal for information presentation, which still makes it difficult to avoid the problem of eye-switching. At the same time, their positioning methods mostly use planar markers or simple three-dimensional structures. When the surgical instruments are tilted or flipped at a large angle, the positioning markers are easy to fall out of the recognition range, resulting in unstable or even interrupted pose calculations, which makes it difficult to meet the training needs of multi-pose surgical operations.
[0004] Therefore, this invention proposes a surgical training and evaluation method and system based on mixed reality technology. The information disclosed in the background section is only for enhancing understanding of the background of this disclosure and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention
[0005] The purpose of this invention is to address the shortcomings of existing technologies by providing a surgical training and evaluation method and system based on mixed reality technology, thereby resolving the technical problems mentioned in the background section.
[0006] To achieve the above objectives, the present invention provides the following technical solution: A surgical training and evaluation method based on mixed reality technology includes the following steps: S1. Based on the calibration board with positioning markers, obtain the coordinate transformation relationship between the binocular camera and the mixed reality glasses relative to the world coordinate system, and calculate the coordinate transformation relationship between the mixed reality glasses and the binocular camera to achieve coordinate system unification; S2. Rigidly connect the teaching model to the model locator, rigidly connect the surgical instrument to the composite structure locator, and pre-calibrate the fixed coordinates of the instrument tip in the locator coordinate system. S3. The binocular camera identifies the positioning marks of the model locator and the composite structure locator and calculates the pose. The pose is unified to the world coordinate system and sent synchronously to the mixed reality glasses and evaluation processing unit. The position of the instrument tip in the world coordinate system is calculated from the locator pose. S4. Preset virtual coordinates of registration points in the virtual model coordinate system, collect the corresponding entity coordinates of registration points, calculate the coordinate transformation matrix from the virtual model to the world coordinate system, and superimpose the virtual model and the entity teaching model accordingly. S5. The mixed reality glasses display standard operating instructions from a first-person perspective; the training process is broken down into multiple operation steps and the boundaries of each step are automatically determined. Evaluation parameters are calculated based on real-time pose data and compared with thresholds to score each step. An evaluation report containing weak points and improvement suggestions is output.
[0007] S1 specifically includes: obtaining the coordinate transformation matrix of the calibration plate relative to the world coordinate system and the coordinate transformation matrix from the world coordinate system to the coordinate system of the mixed reality glasses; obtaining the coordinate transformation matrix of the calibration plate relative to the camera coordinate system from the binocular camera; and calculating the coordinate transformation matrix of the camera relative to the world coordinate system and the coordinate transformation matrix of the mixed reality glasses relative to the camera coordinate system based on these matrixes.
[0008] S2 specifically includes: establishing a locator coordinate system on the composite structure locator, and setting two types of positioning marks: a cylindrical structure and a planar structure. Multiple sets of positioning marks are arranged circumferentially on the outer side of the cylindrical structure, and the planar structure is located within a preset height range above the bottom surface of the cylindrical structure. The recognition confidence of the two types of positioning marks is calculated separately, and the pose calculation results are selected or fused when the threshold is met.
[0009] S3 specifically includes: calculating the coordinate transformation matrix from the teaching model to the camera coordinate system and the coordinate transformation matrix from the locator to the camera coordinate system based on the coordinates of the two-dimensional feature points of the positioning identifier and the known three-dimensional coordinates; selecting or fusing the pose calculation results of the composite structure locator according to the recognition confidence and error index; unifying the coordinate transformation matrix to the world coordinate system and encapsulating it into a data packet containing a timestamp, recognition confidence, error index and valid flag bit for transmission.
[0010] S4 specifically includes: sampling the entity coordinates of the registration point based on the position of the instrument tip in the world coordinate system when the touch judgment condition is met, solving the coordinate transformation matrix from the virtual model to the world coordinate system, and determining whether to trigger re-acquisition or add registration points based on the root mean square of the registration residual.
[0011] S5 specifically includes: calculating evaluation parameters based on one or more of positional deviation, angular deviation, path deviation, and operation time; scoring each operation step using threshold deduction; and identifying weak links and generating improvement suggestions based on the step scores or the distribution of out-of-limit events.
[0012] In step S3, when the confidence level of the positioning identifier is lower than a preset threshold or the pose calculation error index exceeds a preset threshold, the corresponding frame pose data is marked as invalid and does not participate in the virtual-real overlay display and scoring calculation. Furthermore, the pose data of consecutive valid frames is smoothed, and when the number of consecutive invalid frames exceeds a preset threshold, the user is prompted to adjust the camera view or re-execute the coordinate system calibration of step S1.
[0013] A surgical training and evaluation system based on mixed reality technology includes: a calibration board, a binocular camera, mixed reality glasses, a teaching model, a model locator, a composite structure locator, and an evaluation processing unit.
[0014] The beneficial effects of this invention are as follows: This invention uses mixed reality glasses to directly overlay standard operating instructions onto the physical surgical area from a first-person perspective. Trainees do not need to frequently switch their gaze between the operating area and the external display device during training, allowing them to maintain focus on hand-eye coordination, reducing distraction risks, and improving operational continuity and training safety. A unified spatial reference coordinate system is established using a calibration board, unifying the coordinate systems of the binocular camera and the mixed reality glasses into the same world coordinate system. This effectively avoids the misalignment issues caused by independent coordinate systems of different devices, ensuring precise spatial correspondence between virtual instructions and the physical teaching model, and improving the stability and reliability of the virtual-real overlay.
[0015] This invention employs a composite structure locator incorporating both cylindrical and planar structures. It combines recognition confidence and error indices for pose selection or fusion, enabling surgical instruments to maintain stable recognition and continuous pose calculation during tilting, flipping, and other multi-pose operations. This overcomes the problem of traditional planar positioning methods failing under complex operating angles. By pre-setting registration points in the virtual model and setting corresponding groove registration points on the physical teaching model, registration point acquisition is performed based on the instrument tip position. Combined with repeated sampling and registration quality judgment mechanisms, high-precision registration between the virtual and physical teaching models is achieved, reducing the impact of human error on the registration results and improving the repeatability of the registration process.
[0016] This invention breaks down the surgical training process into multiple operational steps and calculates quantitative evaluation parameters such as positional deviation, angular deviation, path deviation, and operation time for each step. This avoids relying solely on overall scoring or subjective evaluation, making the training assessment results more objective and quantifiable, and better reflecting the trainee's operational level. By analyzing the scoring results and deviation distribution of each operational step, weak operational steps are automatically identified, and corresponding improvement suggestions are generated based on the deviation type. This allows trainees to clearly identify their problems and conduct targeted training, reducing ineffective repetitive practice and improving overall training efficiency. Attached Figure Description
[0017] Figure 1 This is a schematic diagram of a surgical training and evaluation method based on mixed reality technology according to the present invention. Detailed Implementation
[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0019] Example 1: As Figure 1 As shown, this embodiment provides a surgical training and evaluation method based on mixed reality technology, including the following steps: S1. Based on the calibration board with positioning markers, obtain the coordinate transformation relationship between the binocular camera and the mixed reality glasses relative to the world coordinate system, and calculate the coordinate transformation relationship between the mixed reality glasses and the binocular camera to achieve coordinate system unification, as follows: In this embodiment, in order to realize coordinate system one between the binocular camera and the mixed reality glasses, the coordinate system one calibration step is first performed to establish a unified spatial reference for each device in the training scene.
[0020] (a) Establishment of calibration plate and coordinate system: A calibration board with positioning markers is fixedly set up in the training scene. The calibration board remains stationary or in motion during the training process, and can be stably tracked by the synchronous positioning and mapping algorithm.
[0021] Establish a calibration plate coordinate system on the calibration plate. (A coordinate system fixed to the calibration plate), for example, establishing a three-dimensional coordinate axis with one corner of the calibration plate as the origin.
[0022] Mixed reality glasses construct a 3D map of the training scene based on the Simultaneous Localization and Mapping (SLAM) algorithm, and establish a world coordinate system using this map. (Global coordinate system of the training scene). The mixed reality glasses calculate the coordinate system of the calibration board by recognizing the positioning marks on the calibration board. Relative to the world coordinate system coordinate transformation matrix (Coordinate transformation matrix from calibration plate to world coordinate system).
[0023] Meanwhile, mixed reality glasses are based on their output relative world coordinate system. The pose information is used to calculate the world coordinate system. To the coordinate system of mixed reality glasses coordinate transformation matrix (This can be obtained by inverting the pose matrix of the mixed reality glasses in the world coordinate system).
[0024] (II) Binocular camera calibration and camera coordinate system establishment: Before the coordinate system is calibrated, the stereo camera is calibrated to obtain its imaging parameters and geometric relationships. The stereo camera calibration includes: obtaining the intrinsic parameter matrices of the left and right cameras (used to describe the focal length and principal point position), distortion parameters (used to describe lens distortion), and the coordinate transformation matrix of the right camera relative to the left camera (used to describe the stereo baseline relationship).
[0025] After completing the stereo camera calibration, establish the stereo camera coordinate system. (Binocular camera coordinate system). The binocular camera identifies the positioning marks on the calibration plate and calculates the calibration plate coordinate system. Relative to camera coordinate system coordinate transformation matrix (Coordinate transformation matrix from calibration board to camera coordinate system).
[0026] (III) Solving the relationship between the camera and world coordinates: Obtained from mixed reality glasses and the data obtained by binocular cameras Calculate the camera coordinate system Relative to the world coordinate system coordinate transformation matrix The calculation relationship is as follows: ; in, for The inverse matrix (the coordinate transformation matrix from camera to calibration board) is thus obtained. This completes the unification between the stereo camera coordinate system and the world coordinate system.
[0027] (iv) Eyeglasses and Camera Coordinate System 1: In obtaining and Then, calculate the coordinate system of the mixed reality glasses. With the binocular camera coordinate system The coordinate relationship between them, and its calculation relationship are as follows: ; in, This represents the coordinate transformation matrix from the camera coordinate system to the mixed reality glasses coordinate system, used to uniformly map the pose data obtained from the binocular camera to the display space of the mixed reality glasses.
[0028] (V) Multi-frame sampling and robust calibration: To improve the stability and accuracy of coordinate system calibration, at least M frames of calibration data are continuously collected during the calibration process, where M ≥ 10.
[0029] Calculate the reprojection error for each frame of calibration data. (Reprojection error of the calibration result of the j-th frame), when When the value exceeds a preset threshold, the corresponding frame calibration result is judged as abnormal and discarded; the pose results of the remaining valid frames are optimized to obtain the final result. and .
[0030] Simultaneously, the calibration quality index E (coordinate system calibration quality index) is calculated. When the calibration quality index meets the preset conditions, the coordinate system calibration is determined to be successful; otherwise, the calibration process is prompted to be re-executed.
[0031] Where j is the index of the valid calibration frame ( ); i is the index of the registration point ( ).
[0032] (vi) Time Synchronization and Data Consistency Processing: Both the binocular camera and the mixed reality glasses carry timestamps when outputting pose data and are synchronized based on a unified time reference. When the time difference between the camera pose data and the glasses pose data exceeds a preset threshold, the pose data is interpolated and aligned before being used for virtual-real overlay display; when reliable interpolation and alignment cannot be achieved, the corresponding frame data is marked as invalid and does not participate in subsequent display and evaluation processing to avoid jitter or misalignment in virtual-real overlay.
[0033] The specific implementation of synchronization based on a unified time base is as follows: During the system initialization phase, the binocular camera and the mixed reality glasses are connected to the same network time protocol server through a local area network to synchronize their internal clocks; or, the evaluation and processing unit periodically sends hardware synchronization trigger pulse signals containing global timestamps to the binocular camera and the mixed reality glasses to eliminate clock drift between independent hardware devices.
[0034] (vii) Drift detection and recalibration triggering: During training, if the registration residual between the virtual model and the physical model exceeds the preset threshold for multiple consecutive frames, or if the pose offset exceeds the allowable range when the calibration board re-enters the field of view, the coordinate system recalibration process is triggered to suppress the cumulative error caused by synchronous positioning and mapping drift.
[0035] S2. Rigidly connect the teaching model to the model locator, and rigidly connect the surgical instrument to the composite structure locator, and pre-calibrate the fixed coordinates of the instrument tip in the locator coordinate system, as follows: After completing the coordinate system calibration, the rigid connection steps of the teaching model, surgical instruments and positioner are performed to establish a stable spatial relationship between the instruments and the model under a unified coordinate system.
[0036] (a) Rigid connection between the teaching model and the model positioner In this embodiment, the teaching model is fixedly installed on the training platform or support base, and the model locator is rigidly connected to the teaching model by bolts, buckles or locking mechanisms to restrict the translational and rotational degrees of freedom of the teaching model during the training process.
[0037] The rigid connection ensures that the model locator and the teaching model maintain a fixed relative position during training. Their relative displacement and relative rotation are both limited by a preset threshold, thereby ensuring that the pose of the teaching model can be stably represented by the pose of the model locator.
[0038] Establish a teaching model coordinate system on the model locator. (Teaching Model Coordinate System), which maintains a fixed relationship with the model locator and is used to represent the spatial pose of the teaching model in the world coordinate system.
[0039] (ii) Rigid connection between surgical instruments and composite structure positioner: The surgical instruments and composite structure positioner are rigidly connected through an adapter. The adapter includes a clamping hole or a sleeve structure and is equipped with an anti-rotation key, a positioning pin or a coaxial limiting structure to ensure that the axial direction and angle of the composite structure positioner and the surgical instruments are consistent after installation, so as to avoid relative rotation or loosening during training.
[0040] The rigid connection ensures that the composite structure locator and the surgical instrument maintain a fixed relative position during training. Their relative displacement and relative rotation angle are both limited by preset thresholds, thereby ensuring that the posture of the surgical instrument can be stably characterized by the posture of the composite structure locator.
[0041] Establish a locator coordinate system on the composite structure locator. (Positioner coordinate system), and complete the geometric calibration between the positioner coordinate system and the surgical instruments before training.
[0042] (III) Coordinate binding of the instrument tip: in the positioner coordinate system Below, the tip of the surgical instrument is pre-calibrated. (The actual operating tip of the surgical instrument) in the fixed coordinates of the locator coordinate system (The fixed coordinates of the tip in the locator coordinate system).
[0043] During training, the pose transformation matrix of the composite structure locator in the world coordinate system can be combined with the fixed coordinates. The real-time position of the tip of the surgical instrument in the world coordinate system is calculated for subsequent path guidance, error calculation and scoring evaluation.
[0044] (iv) Structural composition and spatial layout of the composite structure positioner: The composite structure positioner consists of two parts: a cylindrical structure and a planar structure, wherein: The cylindrical structure is arranged along the axis of the surgical instrument, and multiple sets of identifiable positioning marks are arranged at circumferential intervals on its outer surface. The planar structure is located within a preset height range above the bottom surface of the cylindrical structure, and the height is denoted as h (the height of the planar structure relative to the bottom surface of the cylinder, in centimeters), where h is within the range of 6–15 cm; The normal direction of the planar structure forms a preset angle with the axis of the cylindrical structure, so that the planar structure can still be effectively observed by the binocular camera when the surgical instruments are tilted or flipped.
[0045] Through the above structural layout, the composite structure locator ensures that at least one set of positioning markers is within the identifiable field of view of the binocular camera during multi-angle operation.
[0046] (v) Positioning Marker Recognition and Pose Output Strategy: The binocular camera recognizes the positioning marks on the cylindrical and planar structures respectively, and calculates the recognition confidence C (Chinese definition: positioning mark recognition confidence) for each group of positioning marks.
[0047] When the recognition confidence of a certain set of location markers is not lower than a preset threshold When the minimum recognition confidence threshold is reached, the group of positioning identifiers is determined to be a valid identifier, and the corresponding pose calculation result is output based on the group of positioning identifiers.
[0048] When the positioning markers of both the cylindrical structure and the planar structure meet the recognition confidence threshold, the system can select or fuse the two sets of pose results based on the recognition confidence level or the pose calculation error to improve the stability of pose calculation; when only one set of positioning markers meets the recognition conditions, the pose result is output based on that set of positioning markers.
[0049] (vi) Connection Status Verification and Training Preparation: Before training begins, the system executes a connection status verification process to confirm the rigid connection status between the teaching model, surgical instruments, and the corresponding positioners. If a loose connection is detected, the recognition confidence level remains below the threshold, or the positioning marker is unrecognizable, the system prompts the user to reinstall or adjust the positioner before proceeding with the training process.
[0050] S3. The binocular camera identifies the positioning markers of the model locator and the composite structure locator, calculates the pose, unifies the pose to the world coordinate system, and synchronously sends it to the mixed reality glasses and evaluation processing unit. The position of the instrument tip in the world coordinate system is then calculated from the locator's pose, as follows: After completing the coordinate system calibration and the rigid connection between the teaching model and the surgical instruments, the real-time pose capture and transmission steps are performed to acquire the spatial pose of the teaching model and surgical instruments in the world coordinate system in real time, and synchronize the pose data to the mixed reality glasses and the evaluation and processing unit.
[0051] (a) Real-time image acquisition and identification: The binocular camera continuously acquires image frames containing the model locator and the composite structure locator at a preset sampling frame rate. The image frames are used to detect and identify the positioning marks to obtain the coordinates of the two-dimensional feature points of the positioning marks.
[0052] The two-dimensional feature points include, but are not limited to, the corner points of the positioning marker, the center point of the circle, or the key points of the predefined coded pattern.
[0053] To ensure identification reliability, an identification confidence score C (location identifier identification confidence score) is calculated for each set of location identifiers and compared with the lowest identification confidence score threshold. The lowest confidence threshold is compared to the value of the lowest recognition threshold for subsequent pose output determination.
[0054] (II) Pose Calculation Inputs, Outputs, and Error Indicators: The inputs for pose calculation must include at least: The coordinates of the two-dimensional feature points; The known three-dimensional coordinates of the positioning mark in the locator coordinate system; Imaging and geometric parameters of a binocular camera, including intrinsic parameters, distortion parameters, and binocular extrinsic parameters.
[0055] Based on the above input, the coordinate transformation matrix of the locator relative to the camera coordinate system is calculated: The output of the model locator corresponding to the teaching model (Coordinate transformation matrix from teaching model to camera coordinate system); Output of the composite structure positioner corresponding to the surgical instrument (Coordinate transformation matrix from locator to camera coordinate system).
[0056] Simultaneously output pose calculation error index (Pose calculation error index), the error index can be a reprojection error or an equivalent error measure, used to determine whether the pose of the frame is reliable.
[0057] when or If the pose of the frame exceeds the preset threshold, the frame will be marked as invalid and will not be included in subsequent display and evaluation.
[0058] (III) Multi-marker selection and fusion for composite structure locators: For composite structure locators, the binocular camera performs pose calculations on the cylindrical structure locator and the planar structure locator respectively, obtaining: (Coordinate transformation matrix from locator to camera coordinate system obtained based on cylindrical structure identifier); (Coordinate transformation matrix from locator to camera coordinate system obtained based on planar structure identification).
[0059] When the cylindrical structure marking meets the valid conditions for recognition confidence and error indices, it should be used preferentially. ; When the cylindrical structure marking is invalid while the planar structure marking is valid, the following applies: ; When both are effective, one can be selected based on the confidence level or the error index, or they can be fused according to a preset strategy to obtain the final result. This improves the stability of the instrument's pose calculation during tilting, flipping, and other multi-posture operations.
[0060] As a preferred fusion strategy, when both sets of coordinate transformation matrices calculated based on the cylindrical and planar structures are valid, a weighted fusion method based on pose calculation error indices is adopted. The reprojection errors of both matrices are obtained separately; the smaller the error, the greater the computational weight assigned. The translation components of the two sets of transformation matrices are linearly weighted and summed according to this weight. The rotation components of the two sets of transformation matrices are converted to quaternions and then fused using spherical linear interpolation according to this weight, thereby calculating the final fused coordinate transformation matrix.
[0061] (iv) Pose unification to the world coordinate system: Based on the relationship between the camera and the world coordinate system obtained by S1 (Coordinate transformation matrix from camera to world coordinate system), unifying the poses of the teaching model and the locator to the world coordinate system, yields: ; ; in, The coordinate transformation matrix from the teaching model to the world coordinate system; : The coordinate transformation matrix from the locator to the world coordinate system.
[0062] (v) Calculation of the position of the instrument tip in the world coordinate system: based on the fixed coordinates of the tip pre-calibrated in S2. (The fixed coordinates of the tip in the locator coordinate system), by Calculate the position of the apex in the world coordinate system: ; in The position coordinates of the apex in the world coordinate system. Used for the calculation and evaluation of subsequent parameters such as incision path, operation depth, and deviation.
[0063] (vi) Data packet encapsulation, synchronous transmission, and out-of-order processing: The output of each frame is encapsulated into a pose data packet and sent to the mixed reality glasses and evaluation processing unit in real time. The pose data packet includes at least: 1) Timestamp; 2) , 3) Location of the apex ;4) Identification confidence level C;5) Pose calculation error index 6) Valid flag V (Valid flag, used to indicate whether the pose of this frame is valid).
[0064] When data is out of order or late, the receiver rearranges the pose data packets according to the timestamp; when the delay exceeds the preset time limit, the late data packets are discarded to avoid time mismatch between superimposed display and evaluation.
[0065] (vii) Frame loss degradation processing and smoothing to suppress jitter: When the valid flag bit V of a certain frame pose data packet is invalid, the mixed reality glasses and the evaluation processing unit do not use the frame pose for virtual-real superposition and scoring calculation, and can use the valid pose of the previous frame as a short-term hold to reduce display jumps caused by occasional occlusion.
[0066] Set a threshold L for consecutive invalid frames. When the number of consecutive invalid frames exceeds L, the system will prompt you to adjust the camera view, clear the obstruction, or re-execute the coordinate system calibration steps.
[0067] To reduce pose jitter, continuous effective pose data is smoothed, including low-pass smoothing of translation components and interpolation smoothing of rotation components, thereby improving the stability of virtual-real superposition and the usability of evaluation data.
[0068] Specifically, the translation component can be smoothed using a low-pass filter algorithm; the rotation component can be smoothed by interpolation by converting the rotation matrix of adjacent valid frames into a quaternion expression, calculating the smooth transition quaternion using a spherical linear interpolation algorithm, and finally restoring it to a rotation matrix, thereby avoiding gimbal lock or three-dimensional deformation during the rotation interpolation process.
[0069] S4. Preset the virtual coordinates of the registration points in the virtual model coordinate system, collect the corresponding entity coordinates of the registration points, calculate the coordinate transformation matrix from the virtual model to the world coordinate system, and use this matrix to overlay the virtual model with the entity teaching model, as detailed below: After completing the real-time pose capture and transmission, the virtual model and the physical teaching model are registered and superimposed to establish a precise mapping relationship between the virtual model coordinate system and the world coordinate system, thereby achieving a consistent spatial superposition of the virtual and physical models.
[0070] (I) Registration point design and virtual model preparation: In the virtual 3D model coordinate system (Virtual model coordinate system) has n preset registration points. The registration points are predefined geometric feature points on the virtual model, such as the center point of the groove on the teaching model, where n≥3.
[0071] To ensure a one-to-one correspondence and repeatability of the registration points, the registration points are distributed in different spatial locations in the virtual model according to their numerical order, and the distance between any two registration points is not less than a preset threshold to avoid accidental touches and confusion during the registration process. Correspondingly, corresponding groove registration points are set on the physical teaching model.
[0072] (II) Acquisition and Touch Judgment of Entity Registration Points: Coordinates of Entity Registration Points (Coordinates of the i-th entity registration point in the world coordinate system) World coordinates of the tip point output by S3 The sample was obtained at the moment when the touch determination was made.
[0073] Touch detection must meet at least one of the following conditions or a combination thereof: 1) The distance from the tip to the center of the corresponding groove is not greater than the touch distance threshold. (Touch distance threshold, in millimeters); 2) The instantaneous velocity of the tip within the contact area does not exceed the contact velocity threshold. (Touch speed threshold); 3) The above conditions must be maintained for a duration no less than the touch holding time threshold. (Touch hold time threshold, in milliseconds).
[0074] When the touch detection condition is met, record the world coordinates of the tip at that moment. As candidate entity registration points.
[0075] (III) Repeat sampling and outlier removal: In order to reduce the impact of factors such as hand tremors and slippage on the registration accuracy, each registration point is repeated sampled no less than K times, where K≥3 (number of repeated samplings).
[0076] Multiple candidate coordinates obtained from the same registration point Outlier sampling points that deviate from the mean or median by more than a preset threshold are removed, and the mean or median of the remaining sampling points is used as the final entity registration point coordinates. , where k is the index of the number of repeated samples, and .
[0077] (iv) Solving the virtual-to-world coordinate mapping: based on the coordinates of the registration points in the virtual model Coordinates of the corresponding entity registration points Solving the coordinate system of the virtual model To the world coordinate system The rigid body transformation is performed to minimize the following objective function: ; Where: R is a three-dimensional rotation matrix that satisfies orthogonality constraints and has a determinant of 1; Let R be a three-dimensional translation vector. Then, calculate R and... Combined into a homogeneous coordinate transformation matrix (Coordinate transformation matrix from virtual model to world coordinate system).
[0078] (v) Registration quality assessment and reacquisition strategy: Calculate the root mean square of the registration residuals (Root mean square of registration residuals), which is calculated as follows: ; When the root mean square of the registration residual When the preset threshold is exceeded, the system prompts the user to re-collect entity registration points or increase the number of registration points n until the registration accuracy requirements are met, and then proceed to the subsequent training process.
[0079] (vi) Superposition and display of virtual and real models: Mixed reality glasses are based on the obtained world-to-glasses coordinate transformation matrix. And the transformation matrix from virtual model to world coordinates Calculate the transformation matrix from the virtual model to the glasses coordinate system: ; For any point in the virtual model (Virtual model point coordinates), its position in the mixed reality glasses coordinate system is: ; in The virtual model points are positioned in the glasses' coordinate system. The mixed reality glasses are based on the aforementioned... The virtual model is rendered and overlaid to make it consistent with the physical teaching model in spatial position.
[0080] S5. Standard operating guidelines for first-person perspective overlay display of mixed reality glasses; the training process is broken down into multiple operational steps and the boundaries of each step are automatically determined. Evaluation parameters are calculated based on real-time pose data and compared with thresholds to score each step. An evaluation report containing weak points and improvement suggestions is output, as follows: After completing the registration and overlay of the virtual and real models, the first-person perspective teaching and process breakdown evaluation steps are executed. Mixed reality glasses are used to guide trainees in real time from the first-person perspective, and the training process is evaluated in a process-by-process quantitative manner based on the pose data collected in real time, and an evaluation report containing weak points and improvement suggestions is output.
[0081] (a) First-person perspective teaching overlay display The mixed reality glasses render and overlay virtual guidance information in the glasses' coordinate system, and the virtual guidance information includes at least: 1) Standard instrument posture guidance, used to indicate the target direction of the instrument axis; 2) Standard path guidance, used to indicate the movement path of the instrument tip or the incision outline; 3) Key point guidance, used to indicate the boundaries of entry points, restricted areas, or target areas; 4) Step prompts, which are used to remind you of the key points and precautions for the current step.
[0082] The spatial correspondence between the virtual guidance information and the physical surgical area is automatically established based on the virtual-real registration result formed by S4, thereby enabling the trainee to continuously focus their gaze on the surgical area to complete the training operation without having to frequently check the external display device.
[0083] (II) Rules for Decomposing and Determining the Process The surgical training process is broken down into multiple operational steps, and entry and exit conditions are set for each operational step, enabling the system to automatically determine the current step.
[0084] The entry and termination conditions include at least one or more of the following decision elements: 1) Position of the instrument tip in the world coordinate system Spatial relationship with the target area; 2) Whether the deviation between the instrument posture and the standard posture meets the preset range; 3) Whether the speed of the tip is lower than the preset threshold and remains at the preset time (used to determine "entry point / click / stay"); 4) Time threshold (used to limit the shortest / longest duration of a process).
[0085] In this embodiment, examples of steps include, but are not limited to: preoperative preparation / disinfection, surgical area location, instrument entry point, path execution, and end treatment; the number and order of steps can be pre-configured.
[0086] (III) Evaluation Parameter Calculation and Data Sources The evaluation processing unit automatically calculates evaluation parameters based on real-time acquired pose data, requiring no manual intervention. The pose data includes at least: S3 output instrument tip world coordinates With timestamp; S3 outputs the positioner or instrument pose (used to obtain the instrument axis direction). The virtual-real registration result output by S4 (used to map the standard path and target area to the world coordinate system).
[0087] For each operational step, calculate at least the following types of evaluation parameters: 1) Position deviation: Let the standard entry point or standard target point be... (The position coordinates of the standard point in the world coordinate system), then the position deviation is: ; in Position deviation, in millimeters.
[0088] 2) Angular deviation: Let the unit vector in the direction of the instrument axis be... (Unit vector along the instrument axis), unit vector along the standard posture direction is (Unit vector of standard attitude direction), then the angle deviation is: ; in : Angular deviation, in degrees.
[0089] 3) Path Deviation: In the path execution phase, the standard path is represented by a set of standard path points. (Standard path point set), then the position of the tip at the sampling time. Calculate the minimum distance from the standard path as the path deviation: ; in Path deviation, in millimeters, where P is the set of standard path points. Any reference point in the system.
[0090] 4) Operation duration of each step: Let the start timestamp of the step be... (Start timestamp of the process), end timestamp is (If the end timestamp of the process is given), then the process duration is: ; in : Operation time of each step, in seconds.
[0091] (iv) Threshold determination and scoring rules: Set quantitative evaluation thresholds for each operation step, including at least one or more of the following: position deviation threshold, angle deviation threshold, path deviation threshold and duration threshold.
[0092] When the evaluation parameter exceeds the corresponding threshold, it is determined that an over-limit event has occurred in that step, and points are deducted from that step.
[0093] In this embodiment, the stage score S (stage score) ranges from 0 to 100 points, and the scoring rule can adopt a threshold deduction method: The initial score is 100 points; For each instance of exceeding the limit, a preset score will be deducted according to the event type. When multiple evaluation parameters exceed the limit simultaneously, the deductions are accumulated. The minimum score for each stage is truncated to 0.
[0094] At the same time, the deviation type markers of each step (such as position deviation exceeding the limit, angle deviation exceeding the limit, path deviation exceeding the limit, and time exceeding the limit) and their corresponding deviation values are recorded for subsequent weak point identification and suggestion generation.
[0095] (V) Identification of Weak Links and Generation of Improvement Suggestions: After training, the system identifies weak links based on link scores and deviation distribution. The rules for identifying weak links include one or a combination of the following methods: 1) The operational step with the lowest score; 2) The operational steps with the most instances of exceeding limits; 3) Deviation amount (e.g.) , , , The largest and longest-lasting operation.
[0096] Improvement suggestions are automatically generated based on a mapping relationship of "deviation type - suggested action", which includes at least the following: When the position deviation exceeds the limit, the system prompts you to adjust the entry point position and align it with the standard point. When the angle deviation exceeds the limit, it prompts you to adjust the grip and the direction of the instrument axis to approach the standard posture; When the path deviation exceeds the limit, a prompt will appear indicating that the path should be moved along the standard path outline superimposed from the first-view perspective. When the time limit is exceeded, prompts will be given to optimize the sequence of actions and reduce unnecessary pauses or back-and-forth actions.
[0097] The above improvement suggestions, along with the corresponding deviation types, time periods, and key values, should be included in the evaluation report.
[0098] (vi) Evaluation Report Output and Credibility Instructions An evaluation report is generated after training, and the evaluation report includes at least the following: 1) Scores for each operational step and the overall score; 2) The type, value, and time period of deviation in each stage; 3) Weak links and their ranking; 4) Suggestions for improvement regarding weak points; 5) Data credibility information.
[0099] The data credibility prompt is generated based at least on the valid flag bits output by S3 and the statistical results of consecutive invalid frames. When consecutive invalid frames exceed the threshold or the pose error index continues to exceed the limit during training, the evaluation report will indicate that the credibility of the score for that segment has decreased, and it will be recommended to retrain or adjust the viewpoint before re-evaluating.
[0100] Example 2: This example proposes a surgical training and evaluation system based on mixed reality technology, including: A calibration plate, wherein the calibration plate is provided with positioning marks; A binocular camera is used to acquire images containing teaching models and surgical instruments and to identify the positioning markers in order to calculate the pose of the teaching model and the corresponding positioning device of the surgical instruments relative to the camera coordinate system. Mixed reality glasses are used to establish a world coordinate system based on simultaneous localization and mapping algorithms and output the pose of the glasses relative to the world coordinate system, and overlay standard operation guidance information in a first-person perspective. Teaching model; A model locator is rigidly connected to the teaching model and is equipped with a positioning marker. A composite structure locator is rigidly connected to the surgical instrument and is equipped with positioning marks. The composite structure locator includes a cylindrical structure and a planar structure. Multiple sets of positioning marks are arranged circumferentially on the outer side of the cylindrical structure. The planar structure is set within a preset height range above the bottom surface of the cylindrical structure. The evaluation processing unit, communicatively connected to the binocular camera and the mixed reality glasses, is used for: obtaining the coordinate transformation matrix of the calibration board relative to the world coordinate system, the coordinate transformation matrix from the world coordinate system to the mixed reality glasses coordinate system, and the coordinate transformation matrix of the calibration board relative to the camera coordinate system based on the calibration board; calculating the coordinate transformation matrix of the camera relative to the world coordinate system and the coordinate transformation matrix of the mixed reality glasses relative to the camera coordinate system to achieve coordinate system unification; unifying the poses of the model locator and the composite structure locator to the world coordinate system and synchronizing them to the mixed reality glasses; presetting virtual coordinates of registration points in the virtual model coordinate system, and obtaining the corresponding physical coordinates of registration points based on the position sampling of the surgical instrument tip in the world coordinate system to solve the coordinate transformation matrix from the virtual model to the world coordinate system, thereby achieving the superposition of the virtual model and the teaching model; decomposing the training process into multiple operation steps and automatically determining the step boundaries; calculating evaluation parameters based on real-time pose data and comparing them with thresholds to score the steps, generating an evaluation report containing weak points and improvement suggestions.
[0101] The above formulas are all dimensionless calculations. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters and thresholds in the formulas are set by those skilled in the art according to the actual situation.
[0102] To facilitate better implementation of the present invention by those skilled in the art, this embodiment provides preferred setting ranges for the aforementioned key preset thresholds: the minimum recognition confidence threshold is preferably 75% to 85%; the consecutive invalid frame threshold is preferably 5 to 15 frames. In the touch determination conditions of step S4, the touch distance threshold is preferably 1.0 to 3.0 mm, the touch speed threshold is preferably 2.0 to 5.0 mm per second, and the touch holding time threshold is preferably 300 to 800 milliseconds. The above preferred ranges can provide a good virtual-real overlay effect while taking into account both registration response speed and anti-mistouch stability.
[0103] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0104] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A surgical training and assessment method based on mixed reality technology, characterized in that, Includes the following steps: S1. Based on the calibration board with positioning markers, obtain the coordinate transformation relationship between the binocular camera and the mixed reality glasses relative to the world coordinate system, and calculate the coordinate transformation relationship between the mixed reality glasses and the binocular camera. S2. Rigidly connect the teaching model to the model locator, rigidly connect the surgical instrument to the composite structure locator, and pre-calibrate the fixed coordinates of the instrument tip in the locator coordinate system. S3. The binocular camera identifies the positioning marks of the model locator and the composite structure locator and calculates the pose. The pose is unified to the world coordinate system and sent synchronously to the mixed reality glasses and evaluation processing unit. The position of the instrument tip in the world coordinate system is calculated from the locator pose. S4. Preset virtual coordinates of registration points in the virtual model coordinate system, collect the corresponding entity coordinates of registration points, and calculate the coordinate transformation matrix from the virtual model to the world coordinate system. Based on this, the virtual model and the entity teaching model can be superimposed. S5, a first-person perspective overlay display of mixed reality glasses, provides standard operating instructions, breaks down the training process into multiple operational steps and automatically determines the boundaries of each step. It calculates evaluation parameters based on real-time pose data and compares them with thresholds to score each step, outputting an evaluation report that includes weak points and improvement suggestions.
2. The surgical training and evaluation method based on mixed reality technology according to claim 1, characterized in that, S1 specifically includes: The coordinate transformation matrix of the calibration plate relative to the world coordinate system and the coordinate transformation matrix from the world coordinate system to the coordinate system of the mixed reality glasses are obtained from the mixed reality glasses. The coordinate transformation matrix of the calibration plate relative to the camera coordinate system is obtained from the binocular camera. Based on these, the coordinate transformation matrix of the camera relative to the world coordinate system and the coordinate transformation matrix of the mixed reality glasses relative to the camera coordinate system are calculated.
3. The surgical training and evaluation method based on mixed reality technology according to claim 1, characterized in that, S2 specifically includes: A coordinate system for the locator is established on the composite structure locator, and two types of positioning marks, namely cylindrical structure and planar structure, are set. Multiple sets of positioning marks are arranged circumferentially on the outer side of the cylindrical structure, and the planar structure is located within a preset height range above the bottom surface of the cylindrical structure. The recognition confidence of the two types of positioning marks is calculated separately, and the pose calculation results are selected or fused when the threshold is met.
4. The surgical training and evaluation method based on mixed reality technology according to claim 1, characterized in that, S3 specifically includes: Based on the coordinates of two-dimensional feature points and three-dimensional known coordinates of the positioning identifier, the coordinate transformation matrix from the teaching model to the camera coordinate system and the coordinate transformation matrix from the locator to the camera coordinate system are obtained. The pose calculation results of the composite structure locator are selected or fused according to the recognition confidence and error index.
5. The surgical training and evaluation method based on mixed reality technology according to claim 4, characterized in that, The coordinate transformation matrix is unified to the world coordinate system and encapsulated into a data packet containing a timestamp, identification confidence level, error index, and valid flag bit for transmission.
6. The surgical training and evaluation method based on mixed reality technology according to claim 1, characterized in that, S4 specifically includes: Based on the position of the instrument tip in the world coordinate system, the entity coordinates of the registration point are obtained by sampling when the touch judgment condition is met. The coordinate transformation matrix from the virtual model to the world coordinate system is solved, and the root mean square of the registration residual is used to determine whether to trigger re-acquisition or add registration points.
7. The surgical training and evaluation method based on mixed reality technology according to claim 1, characterized in that, S5 specifically includes: Evaluation parameters are calculated based on one or more of the following: positional deviation, angle deviation, path deviation, and operation time. Threshold deductions are used to score each operation step, and weak links are identified and improvement suggestions are generated based on the step scores or the distribution of out-of-limit events.
8. A surgical training and evaluation method based on mixed reality technology according to claim 5, characterized in that, In step S3, when the confidence level of the positioning identifier is lower than a preset threshold or the pose calculation error index exceeds a preset threshold, the corresponding frame pose data is marked as invalid and does not participate in the virtual-real overlay display and scoring calculation. Furthermore, the pose data of consecutive valid frames is smoothed, and when the number of consecutive invalid frames exceeds a preset threshold, the user is prompted to adjust the camera view or re-execute the coordinate system calibration of step S1.
9. A surgical training and evaluation system based on mixed reality technology, employing the surgical training and evaluation method based on mixed reality technology according to any one of claims 1-8, characterized in that, include: The system includes a calibration board, a binocular camera, mixed reality glasses, a teaching model, a model locator, a composite structure locator, and an evaluation processing unit.