A medical education simulation method and apparatus

Abstract

The present application relates to medical education simulation technology, and provides a medical education simulation method and device, which are used for solving the problems of virtual-real combination training system, such as virtual-real misplacement caused by sensing delay and lack of process quantitative evaluation. The method obtains multi-source sensing data of a simulation medical instrument and maps the data to a virtual coordinate system; adaptive filtering and delay compensation are performed on the data, time-synchronized digital twin pose and confidence scalar are obtained through forward prediction; pose and shape data are combined to solve the problem of position dynamics with flexibility, and the bending shape and force under pressure are calculated to drive physical feedback; the features of the above data are extracted, and the evaluation is generated by introducing a penalty mechanism, an elastic alignment algorithm and expert sequence comparison. The present application realizes the high space-time synchronization of physical feeling and vision and the quantitative correction of the whole operation process.

Description

Technical Field

[0001] This invention relates to medical education simulation technology, and in particular to a medical education simulation method and apparatus. Background Technology

[0002] With the deep integration of information technology and medical education, virtual simulation training is playing an increasingly prominent role in the training of general practitioners, surgical planning, and interventional therapy teaching. The aim is to develop operational skills and muscle memory through high-fidelity simulations without contact with real patients, thereby reducing preventable medical errors. To achieve this goal, medical operation training has generally developed along two technological paths. One relies on purely physical mannequin models, which can provide some realistic tactile feedback. However, due to mechanical limitations, it is difficult to intuitively present the dynamic changes in the internal anatomical structures of the human body during operation, and the cost of consumables is high, with a limited range of cases. The other relies on purely software-based virtual reality or augmented reality systems, which can overcome geographical limitations and switch between various cases at a lower cost, providing an immersive three-dimensional visualization environment. However, it lacks realistic physical force feedback, and trainees do not receive the necessary resistance when operating delicate instruments such as guidewires, endoscopes, and puncture needles, making it difficult to develop effective muscle memory.

[0003] To combine the advantages of both, the industry is gradually shifting towards a technical approach that organically integrates virtual and physical models. This involves using sensors to acquire the depth and spatial pose of instruments in real time, simultaneously displaying the instrument's state within a virtual human body on a screen, thus enabling training and assessment of operational skills. However, further analysis of the underlying software logic and system architecture of such hybrid virtual-physical training systems reveals that the core obstacle to their clinical application lies in the interdependent link between acquiring and utilizing real, continuous operational data. Spatiotemporal calibration between physical spatial perception and virtual engine rendering is usually directly mapped using only simple linear filtering and rigid body coordinate translation. However, interventional instruments such as catheters and guidewires are highly flexible and undergo complex bending and deformation due to stress when entering tortuous anatomical cavities. Rigid body coordinate mapping cannot restore their three-dimensional topological shape, causing the instruments in the virtual image to penetrate the virtual tissue. At the same time, there is an inherent time difference between sensor sampling, bus transmission and graphics rendering pipeline. During rapid puncture or advancement, the position of the instrument in the virtual image lags significantly behind the operator's physical sense, disrupting the crucial hand-eye coordination in training and making it difficult for the system to obtain high-confidence operation timing data without delay or distortion. Even with operational data acquired, existing assessment logic remains relatively flat, relying heavily on static results or single-state triggers. For example, success or failure is determined solely by whether the instrument's tip reaches the preset target coordinates within a specified time. This fails to capture dynamic micro-defects throughout the entire operation, such as angular deviations, timely deceleration in narrow areas, uniform force application, and the degree of conformity between the overall trajectory and the standard trajectory. Furthermore, varying proficiency levels among trainees result in different durations of action sequences, making conventional Euclidean distance comparisons inadequate for handling alignment issues caused by time axis scaling. Consequently, precise process-based correction guidance is lacking. The former determines whether the system can acquire high-confidence, realistic operational time-series trajectories, while the latter requires such high-precision physical-digital twin data for implementation. These two aspects are interdependent and jointly constrain the practical application of a virtual-real integrated medical training system. Summary of the Invention

[0004] To address the technical problems in existing virtual-real integrated medical training systems, such as spatiotemporal delays and rigid body mapping defects between physical perception and graphics rendering, which lead to virtual instruments clipping, physical feel and visual misalignment, and difficulty in accurately quantifying the microscopic process of operation sequences for time-axis scaling, the present invention aims to provide a medical education simulation method and device.

[0005] This invention provides a medical education simulation method and apparatus, comprising the following steps: acquiring multi-source sensor observation data of a simulated medical device during operation, the multi-source sensor observation data including pose observation data and morphological observation data of the simulated medical device, and mapping the multi-source sensor observation data to a preset three-dimensional virtual world coordinate system; performing adaptive filtering and delay compensation on the mapped pose observation data to extract the dynamic adjustment process noise and observation noise of the state switching characteristics between the simulated medical device in an in vitro free state and an in vivo constrained state, and performing forward prediction by combining the online estimated link delay and the fusion coefficient that adaptively shrinks with the degree of motion stability, to obtain the time-synchronized digital twin pose state and frame-by-frame confidence scalar; based on the digital twin pose state and the morphological observation data, performing position dynamics with flexibility on the multi-mass model and the preloaded anatomical and physiological model of the simulated medical device, back-calculating the pressure bending morphology of the simulated medical device conforming to the anatomical cavity curve and the force applied to the virtual tissue, and generating corresponding physical damping feedback commands; After the operation is completed, a temporal feature matrix containing the digital twin's pose state, scalar velocity, the applied force, and the frame-by-frame confidence scalar is extracted. The local cost is weighted using the frame-by-frame confidence scalar, and the temporal feature matrix is ​​compared with a standard expert sequence using an elastic alignment algorithm incorporating stagnation and jump penalties to generate a quantitative evaluation result. This method effectively eliminates the spatiotemporal delay and distortion between physical operation feel and virtual vision, achieves a high degree of synchronization between physical and digital twins, and provides procedural quantitative correction accurate to specific anatomical nodes.

[0006] As a preferred embodiment of the present invention, the adaptive filtering of the mapped pose observation data to extract the state switching characteristics of the simulated medical device between the in vitro free state and the in vivo constrained state, and dynamically adjusting the process noise and observation noise, includes: calculating the difference quotient based on the posterior velocity estimate of the filter to obtain the linear acceleration, and deriving the jerk vector from the difference quotient of the linear acceleration; averaging the amplitude of the jerk vector within a preset sliding window to obtain a jerk amplitude scalar used to characterize the smoothness of motion; determining the state of the simulated medical device based on the jerk amplitude scalar and a preset dual-threshold hysteresis mechanism: when the trigger condition is met and the device is in the in vitro free state, increasing the process noise matrix in the filter to improve the sensitivity to observation; when the trigger condition is met and the device is in the in vivo constrained state, decreasing the process noise matrix and increasing the observation noise matrix to filter out high-frequency jitter in complex media. This scheme enables the system to maintain a sensitive response to operation in vitro and effectively suppress high-frequency jitter caused by strong magnetic fields or complex media in vivo, avoiding repeated twitching and distortion of the virtual device within tissues.

[0007] As a preferred embodiment of the present invention, the step of combining the online estimated link delay with a fusion coefficient that adaptively shrinks with the smoothness of motion for forward prediction to obtain the time-synchronized digital twin pose state and frame-by-frame confidence scalar includes: using data frame timestamps to statistically analyze the instantaneous end-to-end delay from acquisition to rendering, and obtaining the online estimated link delay through recursive smoothing; substituting the posterior estimated state into a state transition function to advance the duration corresponding to the link delay, thereby obtaining the advanced prediction state; determining the fusion coefficient based on the jerk amplitude scalar, wherein the fusion coefficient shrinks exponentially with the increase of the jerk amplitude scalar and is limited by a preset fusion lower limit; and using the fusion coefficient to perform a weighted summation of the posterior estimated state and the advanced prediction state to compensate for inherent delay and suppress visual overshoot, thereby obtaining the digital twin pose state for rendering. The position sub-block trace and attitude quaternion sub-block trace of the posterior covariance are respectively dimensionless according to the corresponding reference variance, and the frame-by-frame confidence scalar characterizing the observation reliability is jointly constructed by combining the dimensionless filtered innovation quantity in an exponential form.

[0008] As a preferred embodiment of the present invention, the step of performing positional dynamics solution with flexibility on the multi-mass model of the simulated medical device and the preloaded anatomical and physiological model based on the digital twin pose state and the morphological observation data includes: establishing distance constraints between adjacent masses and establishing bending constraints between three adjacent masses in the multi-mass model, wherein the reference bending angle of the bending constraint is obtained by conversion based on the morphological observation data, and the flexibility parameters of the distance constraint and the bending constraint are set according to the tensile stiffness and bending stiffness of the corresponding device segment; establishing distance constraints and volume constraints within the deformable tissue mesh formed by the anatomical and physiological model, wherein the constraint flexibility parameters of the deformable tissue mesh are tuned according to the Young's modulus of the target tissue. Continuous collision detection is performed on the surfaces of the multi-mass model and the deformable tissue mesh. When collision penetration occurs, inequality contact constraints are constructed along the outer normal of the penetrated surface. The distance constraint, the bending constraint, the volume constraint, and the inequality contact constraint are sequentially projected in the same Gaussian-Seidel iterative loop. The position correction amount is allocated according to the inverse mass weighting of each mass, so that the pressure bending morphology of the simulated medical device and the pressure deformation of the anatomical and physiological model converge self-consistently in the same loop.

[0009] As a preferred embodiment of the present invention, the step of weighting the local cost using the frame-by-frame confidence scalar and comparing the temporal feature matrix with the standard expert sequence by introducing an elastic alignment algorithm with stagnation penalty and jump penalty to generate a quantitative evaluation result includes: standardizing the temporal feature matrix and the standard expert sequence in the dimensions of position and scalar velocity, and calculating the standardized position difference, standardized scalar velocity difference, and dimensionless quaternion pose difference to construct the tracking cost; locating the anatomical partition to which the device belongs within the anatomical and physiological model based on the current physical position of the device, obtaining the corresponding physical safety threshold, constructing a nonlinear safety penalty for the portion of the force exceeding the physical safety threshold, and merging the tracking cost and the nonlinear safety penalty to obtain the local cost; constructing a reliability factor using the frame-by-frame confidence scalar, and multiplying the local cost by the reliability factor to obtain the weighted local cost, so as to reduce the contribution of low-confidence frames to the cumulative distance; The cumulative distance matrix is ​​solved using a dynamic programming algorithm. A stagnation penalty is added during vertical transitions, and a jump penalty is added during horizontal transitions. The solution path is restricted to a global band that expands adaptively with the difference in length between the two sequences. A comprehensive evaluation score is calculated based on the average cost mapping on the optimal alignment path. The three-dimensional coordinates corresponding to the time nodes when the weighted local cost reaches a peak and the confidence exceeds the limit are displayed as warnings on the interface.

[0010] This invention also provides a medical education simulation device, comprising: a multimodal fusion sensing and analysis module, used to acquire multi-source sensor observation data of a simulated medical device during operation, the multi-source sensor observation data including pose observation data and morphology observation data of the simulated medical device, and mapping the multi-source sensor observation data to a preset three-dimensional virtual world coordinate system; and a spatiotemporal synchronization and prediction solution module, used to perform adaptive filtering and delay compensation on the mapped pose observation data to extract the dynamic adjustment process noise and observation noise of the state switching characteristics between the simulated medical device in the external free state and the internal constrained state, and to perform forward prediction by combining the online estimated link delay and the fusion coefficient that adaptively shrinks with the degree of motion stability to obtain the time-synchronized digital twin pose state and frame-by-frame confidence scalar. The flexible body and biomechanical coupling module is used to perform positional dynamics calculations with flexibility on the multi-mass model and pre-loaded anatomical and physiological model of the simulated medical device based on the digital twin pose state and the morphological observation data. It back-calculates the pressure bending morphology of the simulated medical device conforming to the anatomical cavity curve and the force applied to the virtual tissue, and generates corresponding physical damping feedback commands. The multi-dimensional dynamic trajectory evaluation module is used to extract the temporal feature matrix containing the digital twin pose state, scalar velocity, the force, and the frame-by-frame confidence scalar after the operation is completed. It uses the frame-by-frame confidence scalar to weight the local cost, and compares the temporal feature matrix with the standard expert sequence by introducing an elastic alignment algorithm with stagnation penalty and jump penalty to generate a quantitative evaluation result.

[0011] As a preferred embodiment of the present invention, the multimodal fusion sensing and analysis module is configured to perform affine transformation calibration before mapping the multi-source sensor observation data to a preset three-dimensional virtual world coordinate system: It acquires multiple sets of physical coordinate sampling points collected from fixed anatomical landmarks on the surface of a physical human body model in the physical coordinate system, and extracts multiple sets of virtual coordinate known points in the corresponding virtual three-dimensional scene; it performs cinnabar removal on the physical coordinate sampling points and the virtual coordinate known points respectively, and constructs a cross-covariance matrix; it then obtains the rotation matrix and translation vector by performing singular value decomposition on the cross-covariance matrix; it establishes a mapping relationship between the physical sensing coordinate system and the three-dimensional engine world coordinate system using the rotation matrix and translation vector, and triggers recalibration when the cumulative offset of the fixed reference point exceeds the limit or the filtered information exhibits a one-sided systematic bias.

[0012] As a preferred embodiment of the present invention, the spatiotemporal synchronization and prediction solution module includes an adaptive extended Kalman filter. The adaptive extended Kalman filter is configured to: construct a continuous state vector using position, linear velocity, attitude quaternions, and angular velocity components; input linear acceleration and angular acceleration into the process noise in the form of zero-mean white noise; construct the process noise matrix recursively derived from the prior covariance; recursively match the observation noise matrix with a forgetting factor using the sample covariance of the a priori residuals to achieve adaptive adjustment of the sensor signal-to-noise ratio as the medium changes; when a sensor experiences instantaneous packet loss or an abnormal increase in information causing a decrease in the frame-by-frame confidence scalar and no valid observation is received within a set timeout, skip the state update step and only execute the prediction step, and reduce the prediction confidence by attenuating the prior covariance according to the amplification factor; freeze the rendered pose and output an alarm when the number of pure prediction frames exceeds the limit.

[0013] As a preferred embodiment of the present invention, the device further includes a holographic rendering and physical feedback driving module. The flexible body and biomechanical coupling module and the holographic rendering and physical feedback driving module are configured as follows: The flexible body and biomechanical coupling module calculates the normal drag based on the normal deformation displacement of the collision contact point and the stiffness coefficient of the contact anatomy partition. Simultaneously, it uses a Stribek relationship to ensure a continuous transition of the friction coefficient between static and dynamic friction with the tangential relative velocity, thus smoothly constraining tangential correction. It then calculates the tangential drag in conjunction with the normal drag. The holographic rendering and physical feedback driving module converts the synthesized normal drag and tangential drag into a target torque, generates a pulse width modulation signal, and after upper limit saturation and rate of change limiting processing, issues the physical damping feedback command to the force feedback servo component. This design achieves a smooth presentation of visual and force perception based on unified mechanical constitutive parameters and effectively suppresses numerical oscillations and state jumps.

[0014] As a preferred embodiment of the present invention, the device is further configured to perform a physical safety response based on digital twin determination: the flexible body and biomechanical coupling module detects in real time the penetration trend and local force application of the simulated medical device in the virtual tissue. When it is determined that the simulated medical device has mistakenly entered a non-operational anatomical area or the physical force applied to a dangerous area exceeds the mapped safety threshold and this over-limit state is maintained for a preset number of frames, an over-limit interruption signal is generated. In response to the over-limit interruption signal, the device flashes a warning on the 3D rendering interface and sends an interruption and reverse-bias command directly to the servo side via the communication channel, bypassing the conventional feedback path, to cut off the free sliding enable of the torque motor and trigger the electromagnetic locking device to lock the device feed until it is determined that the simulated medical device has retreated to a safe position in the reverse direction and the local force has fallen back below the safety threshold, at which point the locking is released. This solution can bypass conventional graphical feedback and directly force the establishment of a safety boundary in the physical world, ensuring the compliance and forced correction effect of the operation training process.

[0015] Compared with the prior art, the present invention has the following beneficial effects: To address the need to eliminate spatiotemporal delays and distortions between physical manipulation feel and virtual vision, this invention introduces an adaptive extended Kalman filter and combines it with flexible body position dynamics: By real-time monitoring of the jerk amplitude derived from posterior velocity estimation, process noise and observation noise are dynamically adjusted between the free state outside the body and the constrained state inside the body using a dual-threshold hysteresis approach. This ensures the system remains sensitive to observations outside the body and suppresses high-frequency jitter in strong magnetic fields or complex media inside the body, preventing the virtual instrument from repeatedly twitching and penetrating the tissue. Forward prediction is performed using link delay estimated online by timestamps and a fusion coefficient that adaptively contracts with the smoothness of motion. This compensates for inherent delays while suppressing visual overshoot during sudden stops and advances, bringing the equivalent asynchronous presentation of relative tactile feel to the single-frame level. Furthermore, the instrument is discretized into multiple mass points, and the compressive bending morphology is back-calculated using position dynamics constraints with flexibility. This allows the virtual instrument to conform to the anatomical cavity curves without penetrating the tissue. The reaction force is back-calculated based on the deformation and stiffness at the compression point to drive physical feedback, ensuring that physical feel and 3D vision remain consistent in time and space. To address the need for procedural quantitative evaluation of operation sequences that require time-axis scaling, this invention constructs a local cost using a weighted heterogeneous distance that includes standardized positional differences, posture differences, scalar velocity differences, and nonlinear safety penalties. Stagnation and jump penalties are introduced into dynamic programming, enabling flexible alignment of action sequences with varying time lengths. This distinguishes between reasonable deceleration for safety and non-compliant pauses or overreaching, while providing deviations along the optimal alignment path down to specific anatomical nodes and times. This transforms the assessment from a binary judgment of whether the endpoint is met to a quantitative correction of the entire operation process. Since the safety threshold is determined based on the anatomical region where the instrument is located in each frame, the evaluation can apply different safety standards to different anatomical sites such as the glottis, trachea, esophageal inlet, and vascular bifurcation. Furthermore, the confidence scalar derived frame-by-frame from the anti-delay synchronization stage is integrated into the evaluation stage. A reliability factor is used to weight the local cost, excluding low-confidence nodes from the error peak determination, preventing occasional sensor degradation from being misjudged as student operational errors and improving the reliability of the evaluation conclusions. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the system logic structure of a preferred embodiment of the medical education simulation device of the present invention; Figure 2 This is a flowchart illustrating a preferred embodiment of the medical education simulation method of the present invention; Figure 3 This is a schematic diagram of the working timing of multi-threaded asynchronous collaboration in a preferred embodiment of the present invention.

[0017] In the diagram: 100, Multimodal fusion perception and analysis module; 101, Spatiotemporal synchronization and prediction calculation module; 102, Flexible body and biomechanical coupling module; 103, Holographic rendering and physical feedback driving module; 104, Multidimensional dynamic trajectory evaluation module. Detailed Implementation

[0018] The technical solution of the present invention will be described in more detail below with reference to the accompanying drawings. The model order, sampling frequency, threshold value, weight configuration, constraint form and formula expression given in the following embodiments are illustrative and not limiting. Equivalent substitutions, parameter tuning and formal transformations made by those skilled in the art without departing from the concept of the present invention should be covered within the protection scope of the present invention.

[0019] The medical education simulation device provided in this embodiment comprises a physical human body model equipped with a sensor array, a simulated medical device, a force feedback servo component, a lower-level motherboard responsible for signal acquisition and protocol conversion, and an upper-level computing terminal running the core calculation logic. The simulated medical device has a six-axis miniature electromagnetic tracking coil implanted at its tip to acquire six degrees of freedom of pose, and a fiber Bragg grating sensor array embedded along its flexible rod to acquire curvature information distributed along the arc length. Both types of sensor signals are synchronously acquired by the lower-level computer and then transmitted to the upper-level computer via a high-speed serial bus or gigabit Ethernet. The lower-level motherboard assigns a unified hardware timestamp to each sensor channel and encapsulates the electromagnetic positioning message and the grating demodulation message within the same data frame. The frame header carries the acquisition time, channel number, and verification field, enabling the upper-level computer to obtain multi-source observations with a common time reference during the unpacking stage, providing a time-aligned basis for subsequent filtering, coupling, and evaluation. The technical contribution of the host computer lies in the intermediate solution layer implanted between physical perception and graphics rendering. This layer allows the raw sensor data to undergo calibration mapping, state observation and delay compensation, flexible body dynamics coupling, and expert trajectory elastic regularization before entering the rendering pipeline. This creates a sequentially connected and closed data loop between physical multidimensional perception, digital twin synchronization, dynamic mechanics solution, and expert trajectory regularization, instead of the crude method of directly connecting the underlying readings to the renderer.

[0020] See Figure 1The system logic structure shown is divided into several independent yet highly collaborative functional units in the host computer software. The multimodal fusion sensing and analysis module 100 communicates with the lower-level computer and polls the hardware interface at approximately 1000Hz, unpacking discrete messages into a continuous physical data stream. This stream includes the three-dimensional absolute coordinates and attitude of several key nodes of the instrument tip and rod, as well as the axial and tangential resistance transmitted back from the pressure channel. During the analysis phase, the attitude in Euler angle form is uniformly converted into quaternions to avoid gimbal lock-up. Simultaneously, the local curvature at each grating measurement point is inverted from the grating wavelength drift, and the centerline shape of the instrument rod is reconstructed through arc-length integration, serving as one of the observation constraints for subsequent flexible body calculations. The spatiotemporal synchronization and prediction calculation module 101 runs adaptive extended Kalman filter logic at approximately 240Hz, filtering, resampling, and forward compensating the noisy and lagging six-degree-of-freedom data stream, outputting a digital twin state that is moderately ahead in time and accompanied by a frame-by-frame confidence scalar. The flexible body and biomechanical coupling module 102 incorporates an anatomical and physiological model reconstructed from image data in three dimensions. It performs continuous collision detection and position-based dynamics solving on the predicted pose and virtual tissue mesh at approximately 120Hz, inversely calculating the bending morphology of the device and determining the normal and tangential reaction forces. The holographic rendering and physical feedback driving module 103 pushes images to the display or head-mounted display at a frequency of at least 60Hz, while simultaneously converting the reaction force scalar into a pulse-width modulation signal and sending it to the servo motor. The multidimensional dynamic trajectory evaluation module 104 maintains a timing buffer pool and asynchronously calls the elastic warping logic after a complete operation, comparing it with a standard sequence in the cloud and generating a microscopic quantification report. The high-frequency data exchange between the aforementioned modules is accomplished using two types of memory structures. The first is a lock-free circular buffer for decoupling the acquisition and filtering threads. Its capacity is reserved for worst-case scheduling jitter in the filtering thread; the acquisition side only writes the tail pointer, and the calculation side only moves the head pointer, thus avoiding blocking caused by lock contention. The second is a double-buffered shared memory for reading by the filtering, mechanics, and rendering threads. The writer atomically flips the buffer pointer after updating the status of the background buffer, ensuring that the reader always accesses a consistent completed frame, preventing fragmented reads. The time-series matrix maintained by the multi-dimensional dynamic trajectory evaluation module 104 is stacked sequentially with fixed-length feature frames. Each frame, in addition to kinematic and force components, records the frame confidence for data quality assessment during the evaluation phase.

[0021] See Figure 2 The illustrated method flow, in the system initialization phase, executes step S100, first establishing an affine transformation relationship between the physical operating platform and the virtual 3D scene. A probe is then used to sequentially touch several fixed anatomical landmarks on the surface of the dummy human body, such as the sternal notch, xiphoid process, and bilateral anterior superior iliac spines, to obtain a set of sampling points in the physical coordinate system. The corresponding virtual model marker points are known to be denoted as follows: To suppress single-point touch errors, the number of sampling points is set to be an overdetermined set greater than the minimum solution requirement, and the optimal rotation and translation are obtained using rigid body registration. Specifically, the centroids of the two point sets are first calculated separately. and The centroids of each sampling point are subtracted to remove the centroids, and then the cross-covariance matrix is ​​constructed using the removed point pairs. Perform singular value decomposition on it. Then the rotation matrix is ​​taken

[0022] The determinant terms in the diagonal matrix are used to exclude reflection solutions to ensure... This is a normal orthogonal rotation. After solving, the root mean square error of registration is calculated. If the error exceeds a preset threshold, the sampling point with the largest residual is discarded and the solution is recalculated until the error converges to the allowable range, thus ensuring that the physical sensing coordinate system and the 3D engine world coordinate system are precisely aligned. All poses acquired by the sensor thereafter are then processed... A unified mapping is performed to the virtual world coordinate system, forming the original observation vector flow. During operation, the system continuously monitors the drift of the registration reference. When the cumulative offset of the fixed reference point position observed by the persistent reference coil relative to the calibration time exceeds a threshold, or when the filtered information exhibits a unilateral systematic bias within a long time window, coordinate drift is detected and recalibration is prompted to counteract slow mismatches caused by electromagnetic field distortion or dummy displacement. At the same stage, the engine loads the 3D mesh of the selected case, the biophysical parameters of each tissue, and the corresponding expert operation timing template into high-speed memory. The Young's modulus of the vocal cord mucosa is approximately... Tracheal cartilage ring removal This is used to support subsequent mechanical back-calculation and safety assessment.

[0023] Anti-delay synchronization is performed in step S200, serving as the first line of defense against virtual-real misalignment. A continuous state vector of the system is defined.

[0024] in , These represent three-dimensional position and linear velocity, respectively. For attitude quaternions, Let be the angular velocity. The translational motion of the instrument within adjacent sampling intervals can be sufficiently approximated by a uniform velocity term. However, the acceleration uncertainty caused by friction and variations in handling during tube placement is incorporated into the process noise. Therefore, linear acceleration is not treated as an independent state component to avoid introducing unknown quantities with neither state nor observation sources into the one-step prediction, which could prevent prediction closure. Accordingly, the state one-step prediction takes... Its position and velocity components are updated according to the uniform velocity model.

[0025] Attitude and angular velocity components are updated according to the constant angular velocity model, i.e. and Incremental quaternions , This represents quaternion multiplication. Linear and angular accelerations not explicitly modeled enter the process noise as zero-mean white noise, with their covariance constructed separately for translational and rotational components. The translational component has three independent and identically distributed coordinate axes. For any coordinate axis, the noise gain of the two-dimensional sub-state composed of position and velocity is... The single-axis process noise block is

[0026] in The variance of the linear acceleration modeled as zero-mean white noise is given; this is the discrete white noise acceleration model. The three axes are expanded by sorting the states with the position component first and the velocity component last, resulting in a 6x6 translational process noise block that strictly matches the six-dimensional translational substate.

[0027] It is a third-order identity matrix. Attitude and angular velocity together occupy seven dimensions, and attitude evolution follows nonlinear quaternion kinematics. The process noise is not constructed using the scalar formula of translation, but is obtained by mapping angular acceleration white noise through kinematic sensitivity: Quaternion differential equation The four-by-three coefficient matrix is

[0028] Within one transfer step, the angular acceleration noise causes the angular velocity component to gain a gain of . The perturbation simultaneously causes the rotation increment to receive a perturbation equal to half the square of the time step, the latter being... Mapping to quaternion components, the equivalent gain to the quaternion is: Based on this, a 7x3 rotational noise gain is assembled with quaternions first and angular velocity second, resulting in a 7x7 rotational noise block.

[0029] in The variance of angular acceleration modeled as zero-mean white noise. It contains the cross-covariance between quaternions and angular velocity, and is recalculated frame by frame with the current attitude estimation; the noise of the whole state process is... and Concatenate along the main diagonal to form a block diagonal matrix Its order is strictly consistent with the thirteen-dimensional state, and the prior covariance is recursively derived. This results in a complete closure in terms of dimensions, where The transfer function is the Jacobian at the current estimate. The electromagnetic tracking coil directly observes the position and attitude, and the observation function... It approximates a selection mapping on the corresponding components, and its Jacobian is... The position and posture form a unit structure; from this, new information is obtained. New information covariance Gain The posterior estimate and covariance are and The attitude components are normalized after each prediction and update. To maintain the unit quaternion constraint; the unit norm constraint ensures that the quaternion covariance does not carry information along the norm direction. This embodiment uses stepwise normalization to handle this, which can also be equivalently rewritten as a multiplicative filtering expression with a three-dimensional rotating vector as the error state. Both expressions are functionally equivalent to this scheme.

[0030] When instruments move rapidly through the air, hand tremors are significant. However, upon entering tissue, their speed decreases sharply and their trajectory becomes smoother due to the encapsulation of the tube wall. These two scenarios require opposite confidence orientations from the filter. Therefore, the jerk amplitude derived from the posterior estimate of the filter is used as the environmental criterion: first, the difference between the posterior velocity estimates of adjacent frames is used to approximate the linear acceleration. Then, take the difference quotient of the linear acceleration to obtain the acceleration vector. Its dimension is length cubed seconds, and after taking its Euclidean norm, it is in a length of The scalar criterion is obtained by averaging the frames within a sliding window.

[0031] in To pre-determine the window length, the first step is to take the modulus and then average it to avoid the positive and negative components of the tremor-like sudden movement canceling each other out in the vector average and thus missing detection. The estimated value derived from the filter output is used only to select the tuning levels of process noise and observation noise between the in vitro free state and the in vivo constrained state. It does not participate in the one-step state prediction and therefore does not change the Markov dependence of the filter recursion on the previous and subsequent states. Increasing process noise when in the in vitro free state... To make the system more reliant on real-time observations to maintain sensitivity, and to reduce [the impact of] being in a constrained state within the system. Increase observation noise This makes the system rely more heavily on smooth predictions, thereby filtering out high-frequency jitter in strong magnetic fields or complex media, and preventing the virtual catheter from repeatedly twitching and puncturing within the tissue. To prevent Fluctuations around a single threshold cause frequent state jumps. A dual-threshold hysteresis mechanism with additional dwell count is used, and a higher threshold is set. Low threshold and Only when And velocity amplitude Below the threshold Continuous maintenance The frame transitions from a free state to a constrained state, and vice versa only when... Continuous maintenance The frame transitions from a constrained state to a free state, and fluctuations between the two thresholds do not trigger switching. Window averaging, combined with hysteresis decision-making, prevents the inherent noise sensitivity of the difference quotient from causing erroneous switching. Furthermore, observation noise is further self-corrected online based on residual statistics, using a posteriori residuals... Sample covariance Perform recursive matching with forgetting factor Forgetting weight From the forgetting coefficient The decision enables the filter to adaptively adjust its confidence in the observation as the sensor signal-to-noise ratio changes with the medium.

[0032] Link latency is estimated online instead of based on fixed assumptions. Using the timestamps carried in each message and processing stage, the system obtains an instantaneous end-to-end latency sample from the acquisition time to the renderable time. First, outliers are removed within a sliding window using the median absolute deviation, then... Recursive smoothing Take the smaller value to obtain a stable delay estimate. After obtaining the posterior estimate, substitute it into the transfer function to advance. Obtaining an advanced state Considering that extrapolation of uniform and constant angular velocities will amplify errors and cause visual overshoot during sharp changes of direction, this embodiment does not unconditionally use full extrapolation, but introduces a fusion coefficient that adapts to the smoothness of motion.

[0033] in This means that the first parameter is truncated to the interval given by the last two parameters. The overshoot suppression sensitivity coefficient is positive, and its dimension is taken as the reciprocal of the accelerometer amplitude so that the exponential term is dimensionless. The preset lower limit of fusion is between zero and one, ensuring that minimal delay compensation is retained even under drastic operations, so as not to completely revert to a lagging display; and After offline playback and adjustment, the goal is to strike a balance between the visual overshoot amplitude in emergency stop conditions and the residual delay in steady-state conditions. For example, taking... The reciprocal of the typical amplitude of the jerk during rapid external oscillation. Take values ​​from 0.2 to 0.4. Based on this, obtain the rendering status to be sent to the next module.

[0034] When the technique is steady and the acceleration is small Implementing near-full delay compensation, when sudden stops and starts occur... The process involves compression, prioritizing minimal equivalent residual delay overshoot. This reduces the equivalent asynchrony between perception and presentation to the level of a single frame.

[0035] In addition to the output state, this step synchronously derives a confidence scalar characterizing the observation reliability of the current frame in each frame. Considering the different dimensions of the position covariance and attitude covariance, they are not directly summed; instead, the position sub-block of the posterior covariance is used. with posture sub-block Each variance is dimensionless using the reference variance of its corresponding dimension, and then compared with the normalized innovation square. Jointly constructed in exponential form

[0036] in Let be the trace of the position covariance sub-block, with the dimension of length squared. The positional reference variance is of the same dimension; Let be the trace of the covariance subblock of the attitude quaternion, and be a dimensionless quantity. The dimensionless attitude reference variance; It is a dimensionless Markovian-type innovation quantity. For the observation dimension; , , The adjustment coefficient is used. All three terms are dimensionless before being linearly combined, therefore the exponent is a dimensionless scalar. This applies when the sensor experiences instantaneous packet loss, enters a strong interference region, or experiences an abnormally large increase in information. The trace of the corresponding covariance sub-block rises. A decrease indicates that the frame is not entirely reliable; this scalar is written to shared memory along with the state and recorded sequentially for subsequent evaluation and verification of data quality. If no valid observation is received within the set timeout, the system skips the update step and only performs prediction, multiplying the prior covariance by an amplification factor. The confidence level is reduced accordingly, and an upper limit is set on the number of consecutive pure prediction frames. Once the limit is exceeded, the rendering pose is frozen and a data quality alarm is thrown. The filter converges again after the observation is restored.

[0037] Mechanical coupling unfolds in step S300, addressing the distortion caused by the molding and deformation of flexible instruments. The system discretizes the virtual catheter or guidewire into... A point mass with mass, the first The inverse mass of a particle is denoted as Establish distance constraints between adjacent particles To constrain the length from being stretched, its gradient , To prevent the stiffness of the apparatus from drifting with the number of solution iterations and the time step, a position dynamics solution with compliance is adopted, and a compliance parameter is introduced for each constraint. and according to Update the Lagrange multiplier increments, and then... Correct the position of the mass point, and take the corresponding values ​​for constraint flexibility and material stiffness. ,in For the Young's modulus of the corresponding tissue or instrument segment, For equivalent cross-sectional area, To constrain the feature length, thus ensuring that anatomical material parameters fall into the solver in a physically consistent manner, each rendering step is subdivided into several substeps, within which all constraints are projected in a finite number of Gaussian-Seidel iterations. The bending stiffness of the apparatus is expressed by bending constraints, and these constraints have explicit constraint equations: for every three adjacent mass points along the apparatus... , , Record the unit direction of two adjacent segments. and Constraint functions are taken

[0038] Its gradient with respect to the positions of the three particles is explicitly given by the composite derivative of the equation with respect to the coordinates of each particle, and iteratively projected together with the remaining constraints; reference bend angle The curvature is zeroed when the instrument is outside the body to constrain it to be straight. After entering the body, the measured curvature is reconstructed from the grating at that arc length. According to the relationship between discrete curvature and angle of refraction The conversion yields, where The dimension is the reciprocal of length. The average length of the segments on both sides of the vertex; this constraint is based on flexibility. Participating in the aforementioned iterations, The moment of inertia of the instrument rod section about the neutral axis, That is, the bending stiffness of this segment. The actual bending measured by the sensor is injected into the solution in the form of an explicit equation with soft constraints. It is projected sequentially in the same Gauss-Seidel loop as the distance constraint and the collision constraint. The collision is an inequality constraint, which is given priority at the point where the two conflict. The grating target approaches the solution in the way of minimizing the residual. In the free segment without collision information, the grating target dominates the shape, so that the simulated shape is consistent with the physical bending in each segment.

[0039] In this embodiment, the virtual tissue does not merely act as a rigid barrier, but rather as a deformable variant solved within the same frame as the instrument. The vertices of the tetrahedral mesh are also assigned inverse mass as points of mass in the solver. The boundary vertices that conform to near-rigid anatomical structures such as bone have their inverse mass set to zero to act as spatial anchors. The remaining vertices are assigned mass according to their neighborhood volume and tissue density. The tissue shape is maintained within the mesh by two types of constraints: one is a distance constraint along the tetrahedral edges, with the constraint edge length tending towards the resting length; the other is a tetrahedral volume constraint. The current volume of the cell is determined by the coordinates of its four vertices via a hybrid product. calculate, The resting volume of this unit is used to represent the near-incompressibility of soft tissue. The compliance of both types of constraints is tuned by the material parameters of the tissue as described above, so that different soft and hard tissues exhibit compliance consistent with their Young's modulus. Collision processing uses a hierarchical bounding box to perform continuous collision detection between instrument particles and triangular facets on the tissue surface. When the instrument particle... When falling into the mesh, along the outer normal of the surface patch it passes through. Find the minimum penetration depth Contact point Expressed by interpolation of the centroid coordinates of the three vertices of the facet, the inequality contact constraint is taken as... The constraint gradient acts simultaneously on the instrument mass and the three vertices of the surface via the centroid coordinates. Position correction is distributed bidirectionally according to their respective inverse mass weighting: soft tissue vertices are pressed in, forming indentations, while anchored or high-stiffness areas primarily push the instrument mass out. Because the forced movement of local masses disrupts the original distance, curvature, and volume constraints, the iterative process propagates the deviation backward along the conduit and diffuses itward to neighboring vertices within the tissue mesh. Ultimately, the continuous pressure-bending shape of the instrument conforming to the cavity curve and the pressure deformation of the tissue converge self-consistently within the same loop. Normal deformation displacement. The contact point's current position is obtained by projecting it relative to its position in the resting configuration along the outward normal. Equivalently, it is the weighted sum of the centroid coordinates of the displacements of the three vertices of the compressed surface. The projection on the surface; after the instrument is unloaded, the tissue apex returns to its resting shape under the traction of distance and volume constraints, and the deformation, springback and reaction force thus share the same set of material parameters without the need for a separate tissue mechanics solver.

[0040] The handling of tangential friction and the fact that hard switching between static and dynamic friction at discrete time steps can easily induce high-frequency chattering in the solver are considered. Therefore, regularized Coulomb friction with smooth transition is adopted: based on the Stribek type relationship. Make the coefficient of friction vary with the tangential relative velocity There is a continuous transition between static and kinetic friction, in which , These are the static and kinetic friction coefficients, respectively. The characteristic velocity of Stribek. For the transition steepness; calculate the tangential correction after completing the normal projection, when the relative tangential displacement is less than that calculated by... When the static friction upper limit is determined by the normal correction amount, a tangential correction constrained by a first-order relaxation factor is applied to gradually eliminate relative slip; otherwise, the tangential correction is applied according to... Scaling is used to reproduce the alternation of viscosity and slippage during the device's wall-hugging propulsion while suppressing numerical oscillations caused by state jumps. Macroscopic resistance is determined by the normal deformation displacement at the point of compression. With the stiffness coefficient of this part Inverse calculation, normal component taken The tangential component is taken in the opposite direction to the propulsion direction. The two are combined and packaged into a control command, which is then sent to the force feedback motor.

[0041] The reaction force is converted into physical damping by the holographic rendering and physical feedback driving module 103. This module will... and The target torque of the motor is mapped to generate a pulse width modulation duty cycle. The mapping process is further limited by upper limit saturation and rate of change, ensuring that the output is faithful to the mechanical calculation without being affected by sudden numerical changes. A watchdog timer monitors the timing and amplitude of downlink commands, and if the safe envelope is exceeded, the output falls back to the minimum damping. The rendering side reads the bending posture of the mass points and the deformation data of the tissue to update the vertex shader. Forward compensation is used to align the motion of the screen with the hand's movement in terms of time perception. When the mechanics thread cannot maintain the established frequency due to increased load, the rendering thread interpolates between the poses of the two most recent frames to maintain visual continuity. The mechanics thread, while maintaining stability, adaptively reduces the number of iterations to the lower limit, achieving a smooth degradation rather than a sudden stop when the load increases.

[0042] Trajectory evaluation is performed in step S400, provided that spatiotemporal consistency has been guaranteed. After each training session, the system retains a multi-dimensional temporal matrix of the student's actions. Frame-by-frame feature extraction And include the confidence of this frame. ,in This is the scalar propulsion velocity of the instrument in this frame, to distinguish it from the three-dimensional linear velocity vector in the state vector. , The force applied to the tissue in this frame. For attitude quaternions; the preloaded expert template is denoted as Trainees often pause, hesitate, or regress due to unfamiliarity with the process; therefore, the sequence length... Often significantly greater than This requires flexible alignment to absorb time axis scaling. Position is measured in millimeters, while scalar velocity is measured in millimeters per second; the two scales are vastly different. To make them comparable under the same distance measurement, position and scalar velocity are standardized with zero mean and unit variance. The mean and standard deviation used for standardization are taken from the statistics of the expert template in the corresponding dimensions, placing the trainee data within the coordinate system of the expert distribution and highlighting components deviating from the experts. The standardized position and scalar velocity are denoted as follows: and Attitude differences are measured by quaternion dot products, which are dimensionless and bounded, and therefore do not require standardization. The applied force is not used for frame-by-frame comparison with experts, but only enters the cost through a safety penalty; therefore, its Newtonian dimension is retained and relativized using the physical safety threshold of the anatomical region. Based on this, the local cost is...

[0043] in The term is a dot product of quaternions, and its absolute value is taken to eliminate the ambiguity of the quaternion sign; the last term is a non-linear safety penalty. Physical safety threshold of this part All terms are in Newtons, and their ratio is dimensionless. When the applied force does not exceed the threshold, this term is zero; after exceeding the threshold, it increases by the square of the relative excess, causing significant deductions for dangerous forces along regular paths. Thus, all four terms are dimensionless, and their dimensions are consistent. Safety Threshold Based on the spatial location analysis, the approach involves pre-labeling each tetrahedral element with its corresponding anatomical region and binding the region's physical safety propulsion speed, tangential physical force limit, and stiffness coefficient. During evaluation, the region is located using the centroid coordinates of the trainee's instrument within the tetrahedron, and the region is then selected based on the anatomical attributes matching those attributes. This allows for the application of different safety standards to different locations such as the glottis, trachea, esophageal inlet, and vascular bifurcation. The unstandardized physical locations used for zonal localization are derived from the standardized locations used in the tracking items, differing only in their measurement scale; the two are compatible. Weighting , , , The assessment criteria are dynamically updated based on the key assessment points. To avoid misjudging occasional upstream sensor degradation as student error, a reliability factor is constructed based on frame confidence. ,in To avoid completely ignoring the lower limit of a certain frame, we obtain... This reduces the contribution of low-confidence frames falling in packet loss or interference areas to the cumulative distance, and their abnormal force values ​​will not trigger false tissue damage judgments.

[0044] The optimal alignment is obtained by dynamic programming, and the cumulative distance satisfies

[0045] Diagonal transfers correspond to normal synchronous advancement, while longitudinal transfers incur a penalty for stagnation. To counter hesitation and unnecessary micro-movements, a lateral transfer with an additional jump penalty is applied. To address the issue of reckless advancement that skips necessary steps, two penalty factors are preset positive constants, their relative magnitudes adjusted according to the assessment strategy. At the algorithm level, this distinguishes between reasonable deceleration for safety and non-compliant prolonged pauses or dangerous overreach. To prevent a single frame of expert features from being mapped to a degenerate alignment of an extremely long segment of trainee frames and to constrain computational complexity, the path is restricted to a global band that adaptively widens with the length difference. , with half width ,in The baseline width is set to a positive integer to ensure that the alignment path still has minimal flexibility when the lengths of the two sequences are similar. This is a length difference widening factor, a positive number between zero and one, which widens the bandwidth proportionally to the difference in length between the trainee sequence and the expert sequence. Both are adjusted based on the expert template duration and the time distribution of the trainees. For example... Take 5% to 10% of the expert sequence length. Taking a value near 0.5 allows for significant time scaling while preventing ill-conditioned alignment. Matrix solution from... Starting from the beginning and setting the remaining boundaries to infinity, we recursively extend the process along rows and columns until... A backtracking pointer array is used to record the optimal alignment path for each cell in order to reconstruct the alignment mapping. Let the length of the optimal alignment path be... ,but The average local cost along the path is dimensionless, as all costs have been dimensionless. The student's overall score is calculated as follows:

[0046] Mapped to a percentage system, where A positive score sensitivity coefficient determines the steepness of score decay for each unit increase in average cost. Its tuning is based on a calibration sample, ensuring that the average cost of the benchmark operation sequence that achieves a passing teaching level is precisely mapped to a preset passing score, such as 80 points. Based on this, the following can be deduced: The system uses scores from all trainees in the same case, ensuring cross-personnel comparability. Beyond the overall score, the system segments the alignment path at several key anatomical nodes, normalizing each segment individually to provide a partial score. This score differentiates performance at different stages of the same operation. Furthermore, the raw scores can be quantile-calibrated using a monotonic mapping fitted to a reference population, ensuring comparability between different cases and batches. The evaluation process then traces back along the alignment path to identify local costs. For nodes exhibiting peak values, their timestamps and 3D coordinates are extracted and highlighted in a heatmap format on the virtual physiological model. The hue of the heatmap is determined by the peak cost through saturation mapping; the higher the cost, the more reddish it appears, while nodes falling within the low confidence interval... Nodes that do not participate in peak value determination are excluded to avoid generating false red zones.

[0047] Safety intervention takes effect immediately during operation in step S500. When the flexible body and biomechanical coupling module 102 detects an extremely dangerous penetrating trend, such as the instrument accidentally entering the esophagus during endotracheal intubation, or the pressure on the carotid sinus exceeding the limit during puncture, the system generates a flashing red zone alarm at the corresponding anatomical location on the 3D interface. On the other hand, it bypasses the conventional feedback path and directly sends an interruption and reverse-direction command to the lower-level server via a fast communication channel, instantly cutting off the motor's free sliding enable and locking the instrument's continued feed by an electromagnetic locking device. This forcibly establishes a safety boundary in the physical world, forcing the trainee to return to a safe position and readjust the angle. The trigger criterion combines the aforementioned spatial analysis safety threshold and penetration trend. Locking is only performed after maintaining the over-limit state for several consecutive frames to de-jitter the brief numerical jitter and avoid false locking. Unlocking is conditional on the instrument withdrawing in the reverse direction to a safe depth and the applied force falling back below the threshold, thus directly mapping the algorithm judgment on the virtual side to the physical blockage on the physical side.

[0048] See Figure 3The temporal coordination relationship shown indicates that the device operates asynchronously through multi-threaded collaboration in a complete training workflow, and its data flow can be divided into four step-by-step stages. In the preparation stage, the trainee selects a specific case on the host computer, and the main control process wakes up the lower-level microcontroller and the internal sensor array of the dummy. First, the reference point calibration in step S100 solves the affine transformation to make the two coordinate systems coincide. Then, the three-dimensional mesh, tissue biophysical parameters, and expert template of the case are loaded into high-speed memory to complete the warm-up. Once in runtime, the hardware acquisition thread continuously polls the six-axis torque sensor, the tip electromagnetic coil, and the grating demodulation channel at approximately 1000Hz. After analog-to-digital conversion and protocol unpacking, it pushes the timestamped coarse data into a pre-allocated unlocked ring buffer. The filtering and calculation thread self-buffers and pulls the latest data block at approximately 240Hz to perform state observation, hysteresis environment determination, online self-correction of observation noise, online delay estimation, and confidence extrapolation. It writes the rendering state and frame confidence into double-buffered shared memory. The biomechanics thread reads the predicted pose from the shared memory at approximately 120Hz, calls the position dynamics solver with flexibility to perform intersection tests on the duct multi-mass model and tissue mesh, iteratively calculates the bending morphology, and inversely calculates the directional and tangential reaction forces. This data is then rapidly sent to the lower-level machine via the serial bus along with the control frame to drive the motor torque. The rendering thread extracts bending posture and tissue deformation data at no less than 60Hz to update the shader and pushes the frame buffer. Forward compensation is used to align the visuals and the tactile feedback in time. The inherent time consumption of each stage accumulates segment by segment in the acquisition, bus transmission, filtering, coupling, and rendering processes. The system compensates for this end-to-end total time consumption by measuring the timestamp, thus reducing the asynchronous convergence of the equivalent residuals that provide a relatively tactile feel to the single-frame level. In parallel, a low-frequency observer process takes snapshots of the current state vector and its confidence at a fixed sampling frequency and pushes them onto the timing stack. When it detects events such as removing the device or clicking to confirm completion, it seals the multidimensional matrix and wakes up the evaluation process, formally initiating the elastic alignment solution in step S400, traversing the constrained cost matrix to deduce the alignment path with the minimum total penalty.

[0049] After the evaluation, the output phase begins. The front end enters a 3D debriefing mode, comparing the trainee's trajectory with the expert's trajectory side-by-side on the same timeline. The main viewport displays the expert's standard path as a semi-transparent solid line and the trainee's actual path as a dashed line, visually exposing spatial topological deviations. When playback reaches a high-risk error point, it automatically pauses and pulls out a data panel, providing micro-corrections based on the algorithm. For example, it suggests that the safe propulsion speed at the critical point of the throat turn into the subglottic cavity should be approximately [missing information]. The upper limit of tangential force is approximately When the frame snapshot shows the instantaneous propulsion speed reaches approximately Wrist rotation deviates from standard posture by approximately This causes the tangential force to increase sharply to approximately Based on this, it is determined that a mucosal scratch has occurred at this location, and corresponding points are deducted and a red zone warning is issued. At the same time, the contribution of this node to the sub-score and suggested improvement actions are given.

[0050] Therefore, this invention integrates the acquisition, time alignment, denoising, delay compensation, dynamic coupling, and morphological restoration of physical twin data into a unified intermediate solution layer. It also uses frame-level confidence to ensure synchronization quality throughout the evaluation process, and employs spatial resolution safety thresholds and flexible time warping to support micro-level corrections throughout the entire process. This results in high-fidelity virtual-real consistency and spatiotemporal trajectory-based behavioral evaluation that mutually support and close the loop, providing a reproducible and scalable system-level implementation for general practitioner education, complex surgical planning, and advanced interventional procedure training. All equivalent transformations made to the model order, update frequency, threshold and penalty factor values, weight configuration, constraint flexibility, noise gain construction, or formula expression under the concept of this invention do not exceed the essential scope of this invention.

[0051] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A medical education simulation method, characterized in that, Includes the following steps: Acquire multi-source sensor observation data of a simulated medical device during operation. The multi-source sensor observation data includes the pose observation data and shape observation data of the simulated medical device. Map the multi-source sensor observation data to a preset three-dimensional virtual world coordinate system. Adaptive filtering and delay compensation are performed on the mapped pose observation data to extract the state switching characteristics of the simulated medical device between the free state outside the body and the constrained state inside the body, dynamically adjust the process noise and observation noise, and combine the online estimated link delay and the fusion coefficient that shrinks adaptively with the motion stability to perform forward prediction, so as to obtain the digital twin pose state and frame-by-frame confidence scalar after time synchronization. Based on the digital twin pose state and the morphological observation data, the position dynamics of the multi-mass model of the simulated medical device and the preloaded anatomical and physiological model are solved with flexibility. The pressure bending morphology of the simulated medical device conforming to the anatomical cavity curve and the force applied to the virtual tissue are calculated, and the corresponding physical damping feedback command is generated. After the operation is completed, a temporal feature matrix containing the digital twin pose state, scalar velocity, force, and frame-by-frame confidence scalar is extracted. The local cost is weighted using the frame-by-frame confidence scalar, and the temporal feature matrix is ​​compared with the standard expert sequence by introducing an elastic alignment algorithm with stagnation penalty and jump penalty to generate a quantitative evaluation result.

2. The method according to claim 1, characterized in that, The adaptive filtering of the mapped pose observation data to extract the dynamic adjustment process noise and observation noise of the state switching characteristics of the simulated medical device between the in vitro free state and the in vivo constrained state includes: The linear acceleration is obtained by calculating the difference quotient based on the filtered posterior velocity estimate, and the jerk vector is derived from the difference quotient of the linear acceleration. The magnitude of the acceleration vector is averaged within a preset sliding window to obtain a scalar of acceleration magnitude used to characterize the smoothness of motion; The state of the simulated medical device is determined based on the accelerometer amplitude scalar and the preset dual-threshold hysteresis mechanism: when the trigger condition is met and the device is in an in vitro free state, the process noise matrix in the filter is increased to improve the sensitivity of the observation; when the trigger condition is met and the device is in an in vivo constrained state, the process noise matrix is ​​decreased and the observation noise matrix is ​​increased to filter out high-frequency jitter in complex media.

3. The method according to claim 1 or 2, characterized in that, The forward prediction, which combines the online estimated link delay with the fusion coefficient that adaptively shrinks with the motion smoothness, yields the time-synchronized digital twin pose state and frame-by-frame confidence scalar, including: The instantaneous end-to-end latency from data acquisition to rendering is statistically analyzed using data frame timestamps, and the online estimated link latency is obtained through recursive smoothing. Substitute the posterior estimated state into the state transition function and advance the duration corresponding to the link delay to obtain the advanced predicted state. The fusion coefficient is determined based on the scalar magnitude of the accelerometer, wherein the fusion coefficient shrinks exponentially with the increase of the scalar magnitude of the accelerometer and is limited by a preset lower limit of fusion. The fusion coefficients are used to perform a weighted summation of the posterior estimated state and the advanced predicted state to compensate for inherent delay and suppress visual overshoot, thereby obtaining the digital twin pose state for rendering. The position sub-block trace and attitude quaternion sub-block trace of the posterior covariance are respectively dimensionless according to the corresponding reference variance, and the frame-by-frame confidence scalar characterizing the observation reliability is jointly constructed by combining the dimensionless filtered innovation quantity in an exponential form.

4. The method according to claim 1, characterized in that, The step of performing a flexible positional dynamics solution on the multi-mass model and the pre-loaded anatomical and physiological model of the simulated medical device based on the digital twin pose state and the morphological observation data includes: In the multi-mass model, distance constraints are established between adjacent mass points, and bending constraints are established between three adjacent mass points. The reference bending angle of the bending constraint is obtained by converting the morphological observation data, and the compliance parameters of the distance constraint and the bending constraint are set according to the tensile stiffness and bending stiffness of the corresponding machine segment. Distance constraints and volume constraints are established within the deformable tissue mesh formed by the anatomical and physiological model. The constraint flexibility parameters of the deformable tissue mesh are tuned according to the Young's modulus of the target tissue. Continuous collision detection is performed on the surface of the multi-mass model and the deformable mesh. When collision penetration occurs, inequality contact constraints are constructed along the outer normal of the penetrated surface. The distance constraint, the bending constraint, the volume constraint, and the inequality contact constraint are sequentially projected in the same Gauss-Seidel iterative loop. The position correction amount is allocated according to the inverse mass weighting of each mass point, so that the pressure bending shape of the simulated medical device and the pressure deformation of the anatomical and physiological model converge self-consistently in the same loop.

5. The method according to claim 1, characterized in that, The step involves weighting the local cost using the frame-by-frame confidence scalar, and then comparing the temporal feature matrix with the standard expert sequence using an elastic alignment algorithm incorporating stagnation and jump penalties to generate a quantitative evaluation result, including: The temporal feature matrix and the standard expert sequence are standardized in terms of position and scalar velocity dimensions, and the standardized position difference, standardized scalar velocity difference, and dimensionless quaternion pose difference are calculated to construct the tracking cost. Based on the current physical position of the instrument, the anatomical region to which it belongs is located within the anatomical and physiological model, and the corresponding physical safety threshold is obtained. The portion of the force exceeding the physical safety threshold is constructed as a nonlinear safety penalty, and the tracking cost and the nonlinear safety penalty are combined to obtain the local cost. A reliability factor is constructed using the frame-by-frame confidence scalar. The local cost is then multiplied by the reliability factor to obtain a weighted local cost, thereby reducing the contribution of low-confidence frames to the cumulative distance. The cumulative distance matrix is ​​solved using a dynamic programming algorithm. The stagnation penalty is added during the vertical transition, and the jump penalty is added during the horizontal transition. The solution path is restricted to a global band that expands adaptively with the difference in length between the two sequences. The comprehensive evaluation score is calculated based on the average cost mapping on the optimal alignment path, and the three-dimensional coordinates corresponding to the time nodes when the weighted local cost reaches a peak and the confidence exceeds the limit are displayed as a warning on the interface.

6. A medical education simulation device, characterized in that, include: The multimodal fusion perception and analysis module is used to acquire multi-source sensor observation data of the simulated medical device during operation. The multi-source sensor observation data includes the pose observation data and shape observation data of the simulated medical device, and maps the multi-source sensor observation data to a preset three-dimensional virtual world coordinate system. The spatiotemporal synchronization and prediction solution module is used to perform adaptive filtering and delay compensation on the mapped pose observation data to extract the state switching characteristics of the simulated medical device between the free state outside the body and the constrained state inside the body, dynamically adjust the process noise and observation noise, and combine the online estimated link delay and the fusion coefficient that shrinks adaptively with the motion stability to perform forward prediction, so as to obtain the digital twin pose state and frame-by-frame confidence scalar after time synchronization. The flexible body and biomechanical coupling module is used to perform position dynamics solution with flexibility on the multi-mass model and preloaded anatomical and physiological model of the simulated medical device based on the digital twin pose state and the morphological observation data, and to back-calculate the pressure bending morphology of the simulated medical device conforming to the anatomical cavity curve and the force applied to the virtual tissue, and generate corresponding physical damping feedback commands. The multidimensional dynamic trajectory evaluation module is used to extract a temporal feature matrix containing the digital twin pose state, scalar velocity, force, and frame-by-frame confidence scalar after the operation is completed. The local cost is weighted using the frame-by-frame confidence scalar, and the temporal feature matrix is ​​compared with the standard expert sequence by introducing an elastic alignment algorithm with stagnation penalty and jump penalty to generate a quantitative evaluation result.

7. The apparatus according to claim 6, characterized in that, Before mapping the multi-source sensor observation data to a preset three-dimensional virtual world coordinate system, the multimodal fusion sensing and parsing module is configured to perform affine transformation relationship calibration: Obtain multiple sets of physical coordinate sampling points collected from fixed anatomical landmarks on the surface of a physical human body model in the physical coordinate system, and extract multiple sets of virtual coordinate known points in the corresponding virtual 3D scene; The physical coordinate sampling point set and the virtual coordinate known point set are de-centered and a cross-covariance matrix is ​​constructed. The rotation matrix and translation vector are obtained by performing singular value decomposition on the cross-covariance matrix. The rotation matrix and translation vector are used to establish a mapping relationship between the physical sensing coordinate system and the 3D engine world coordinate system. Recalibration is triggered when the cumulative offset of the fixed reference point exceeds the limit or the filtered information shows a one-sided systematic bias.

8. The apparatus according to claim 6 or 7, characterized in that, The spatiotemporal synchronization and prediction solution module includes an adaptive extended Kalman filter, which is configured as follows: A continuous state vector is constructed using position, linear velocity, attitude quaternions and angular velocity components. Linear acceleration and angular acceleration are introduced into the process noise in the form of zero-mean white noise, and the process noise matrix is ​​constructed by prior covariance recursion. The observation noise matrix is ​​then recursively matched with the sample covariance of the post-hoc residuals, including the forgetting factor, to achieve adaptive adjustment of the sensor signal-to-noise ratio as the medium changes. When a sensor experiences instantaneous packet loss or an abnormal increase in information, causing the frame-by-frame confidence scalar to decrease and no valid observation is received within a set timeout, the state update step is skipped and only the prediction step is executed. The prior covariance is reduced according to the amplification factor to decrease the prediction confidence. When the number of pure prediction frames exceeds the limit, the rendering pose is frozen and an alarm is output.

9. The apparatus according to claim 6, characterized in that, The device further includes a holographic rendering and physical feedback driving module, and the flexible body and biomechanical coupling module and the holographic rendering and physical feedback driving module are configured as follows: The flexible body and biomechanical coupling module calculates the normal resistance based on the normal deformation displacement of the collision contact point and the stiffness coefficient of the contact anatomical partition. At the same time, it uses the Stribek type relationship to make the friction coefficient continuously transition between static friction and dynamic friction with the tangential relative velocity to smoothly constrain the tangential correction, and calculates the tangential resistance in combination with the normal resistance. The holographic rendering and physical feedback driving module converts the synthesized normal drag and tangential drag into a target torque, generates a pulse width modulation signal, and after upper limit saturation and rate of change limiting processing, sends the physical damping feedback command to the force feedback servo component.

10. The apparatus according to claim 6, characterized in that, The device is also configured to perform a physical security response based on digital twin determination: The flexible body and biomechanical coupling module detects the penetration trend and local force application of the simulated medical device in the virtual tissue in real time. When it is determined that the simulated medical device has mistakenly entered the non-operational anatomical area or the physical force applied to the dangerous area exceeds the mapped safety threshold and the over-limit state is maintained for a preset number of frames, an over-limit interruption signal is generated. In response to the over-limit interruption signal, the device flashes a warning on the 3D rendering interface and sends an interruption and reverse-direction command directly to the servo side via the communication channel, bypassing the conventional feedback path. This cuts off the free sliding enable of the torque motor and triggers the electromagnetic locking device to lock the instrument feed until it is determined that the simulated medical device has retreated to a safe position in the reverse direction and the local force has fallen back below the safe threshold, at which point the locking is released.

Images