Real-time feedback interactive virtual diagnosis and treatment teaching and training system

Abstract

The invention relates to the technical field of virtual diagnosis and treatment, and discloses a real-time feedback interactive virtual diagnosis and treatment teaching and training system. Calculating an instant angular frequency, an external amplitude and a phase indication quantity in the sliding time window; intersecting a user sight ray with the pre-segmented liver segment model, determining a sight intersection point position and a segment tag, and extracting a liver three-dimensional amplitude parameter group from a segment parameter library; calculating a compensation displacement vector based on the system total time delay, the instant angular frequency, the phase indication quantity and the segment amplitude parameter group, and superposing the compensation displacement vector with the sight line intersection point position to obtain a high-detail loading area center; solving an amplitude scaling coefficient according to the external amplitude, and carrying out linear scaling on the three-dimensional amplitude to obtain a scaled segment amplitude; and calculating the three-way radius of the liver by using the total time delay of the system, the instant angular frequency and the scaled amplitude, and constructing an anisotropic high-detail loading region by taking the central position as the center and the radiuses in all directions as the half axes.

Description

Technical Field

[0001] This invention relates to the field of virtual medical technology, and more specifically, to an interactive virtual medical teaching and training system with real-time feedback. Background Technology

[0002] Virtual medical systems typically utilize VR technology to dynamically display the 3D human body structure and achieve interactive loading through gaze prediction and tracking. Specifically, the resolution and loading level of the model within the user's field of view are adjusted in real time based on changes in the user's gaze point to ensure an immersive experience with limited computing resources. This loading strategy performs well in ordinary virtual scenarios (such as games or architectural visualizations) because the observed objects are mostly static or rigid structures, and the system only needs to control the level of detail based on the viewing angle and distance. However, in medical virtual scenarios, the human body model is not a static geometry but involves complex movements and deformations caused by respiration, blood flow, and probe contact. The inherent difference between the spatial regularity of these internal changes and the temporal response of external gaze reduces the stability of traditional gaze-driven loading.

[0003] In abdominal diagnostic training (such as virtual ultrasound, laparoscopy, or radionavigation), the natural movement of organs and the mechanical deformation caused by manipulation together result in continuous geometric displacement. For example, different functional segments of the liver move with varying amplitudes and directions during a single respiratory cycle, with vertical displacements ranging from millimeters to centimeters, while anterior-posterior and lateral displacements are relatively smaller. Furthermore, contact with probes or instruments adds additional local compression, causing minute but significant shifts in the spatial position of anatomical structures over a short period. This displacement is anatomically zone-dependent: different segments exhibit different amplitudes and phases of movement, meaning that even if the user's line of sight prediction is perfectly accurate, the high-resolution area loaded by the system may spatially deviate from the actual diagnostic target.

[0004] The gaze prediction and tracking link suffers from a fixed system latency, including eye-tracking sampling, rendering calculations, and display refresh, typically in the tens of milliseconds. When a user gazes at a dynamic organ region, the high-detail portion actually loaded by the system is still based on the gaze data from the previous moment. If the organ is currently undergoing a vertical displacement crossing a zero point due to breathing or manipulation, the difference between the loaded position and the organ's actual position will peak instantaneously and appear periodically with the breathing rhythm. Due to the different motion characteristics of different segments within the organ, this misalignment exhibits directional differences in space and periodic changes in time. As a result, the system continuously allocates high-resolution rendering resources to slightly earlier or later spatial positions, causing lesions in virtual diagnosis to appear blurry, drifting, or jerky, affecting the accuracy of diagnosis and teaching. Summary of the Invention

[0005] This invention provides an interactive virtual diagnosis and treatment teaching and training system with real-time feedback, which solves the technical problems mentioned in the background art.

[0006] This invention provides a real-time feedback interactive virtual diagnosis and treatment teaching and training system, comprising: The latency module measures the latency of the virtual reality system's line-of-sight, rendering, and display processes to obtain the total system latency. The data acquisition and processing module acquires and processes the time series of external respiratory displacement of the human body within a sliding time window to obtain the instantaneous angular frequency, external amplitude and phase indication. The liver amplitude module intersects the user's line of sight in the virtual reality scene with the pre-segmented liver segment model to obtain the position of the line of sight intersection and the segment label; it then retrieves data from the preset segment parameter library according to the segment label to form a segment amplitude parameter set containing the three-dimensional amplitude of the liver. The center position calculation module calculates the compensation displacement vector based on the total system delay, instantaneous angular frequency, phase indication, and segment amplitude parameter set; the center position of the high detail loading area is obtained by summing the line-of-sight intersection position with the compensation displacement vector. The amplitude scaling module determines the amplitude scaling factor based on the external amplitude, and multiplies the amplitude of each direction in the segment amplitude parameter group with the amplitude scaling factor to obtain the scaled segment amplitude. The loading region determination module calculates the three-dimensional radius of the liver based on the total system delay, instantaneous angular frequency, and scaled segment amplitude; and forms an anisotropic high-detail loading region with the center position as the center and the anisotropic radius as the semi-axis.

[0007] The beneficial effects of this invention are as follows: By jointly modeling real-time gaze prediction data of virtual reality users with respiratory dynamic parameters, system latency, and liver segment displacement patterns, precise spatiotemporal synchronization and adaptive directional adjustment of the human body model loading are achieved. This invention can automatically predict and compensate for spatial misalignment when user gaze changes and non-rigid movements of internal organs coexist, ensuring that high-detail areas always align with the actual location of the lesion. This significantly improves spatial matching accuracy and visual stability in virtual diagnosis and treatment teaching, reduces model drift, jitter, and blurring caused by breathing and manipulation, and achieves a real-time responsive, physiologically consistent interactive loading mechanism, thereby providing operational feedback and visual experience closer to a real surgical environment in virtual diagnosis and treatment training. Attached Figure Description

[0008] Figure 1 This is a module diagram of an interactive virtual diagnosis and treatment teaching and training system with real-time feedback according to the present invention. Detailed Implementation

[0009] The subject matter described herein will now be discussed with reference to exemplary embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and implement the subject matter described herein, and changes may be made to the function and arrangement of the elements discussed without departing from the scope of this specification. Various processes or components may be omitted, substituted, or added as needed in the examples. Furthermore, features described in some examples may be combined in other examples.

[0010] like Figure 1 As shown, an interactive virtual medical teaching and training system with real-time feedback includes: The latency module measures the latency of the virtual reality system's line-of-sight, rendering, and display processes to obtain the total system latency. The data acquisition and processing module acquires and processes the time series of external respiratory displacement of the human body within a sliding time window to obtain the instantaneous angular frequency, external amplitude and phase indication. The liver amplitude module intersects the user's line of sight in the virtual reality scene with the pre-segmented liver segment model to obtain the position of the line of sight intersection and the segment label; it then retrieves data from the preset segment parameter library according to the segment label to form a segment amplitude parameter set containing the three-dimensional amplitude of the liver. The center position calculation module calculates the compensation displacement vector based on the total system delay, instantaneous angular frequency, phase indication, and segment amplitude parameter set; the center position of the high detail loading area is obtained by summing the line-of-sight intersection position with the compensation displacement vector. The amplitude scaling module determines the amplitude scaling factor based on the external amplitude, and multiplies the amplitude of each direction in the segment amplitude parameter group with the amplitude scaling factor to obtain the scaled segment amplitude. The loading region determination module calculates the three-dimensional radius of the liver based on the total system delay, instantaneous angular frequency, and scaled segment amplitude; and forms an anisotropic high-detail loading region with the center position as the center and the anisotropic radius as the semi-axis.

[0011] In one embodiment of the present invention, the total system latency is obtained by measuring the latency of the line-of-sight, rendering, and display stages of the virtual reality system, including: Time of completion of sampling by gaze sensor When the view data enters the rendering module ; Calculate the delay of the line of sight : ;in, This refers to the point in time when the gaze sensor in a virtual reality system completes a single user gaze data acquisition. This refers to the point in time when the collected gaze data is read by the rendering module of the virtual reality system and becomes usable. The time interval from when gaze data is sampled to when it is available to the rendering module; The frame submission time of the acquisition and rendering module ; Calculate rendering latency : ;in, This refers to the time point at which the rendering module uses the aforementioned gaze data to generate the corresponding frame image and submits it to the display link. This refers to the time interval between the entry of gaze data into the rendering module and the submission of the corresponding frame image for display; The actual time displayed on the acquisition display panel ; Calculate display latency : ;in, This refers to the actual point in time when the frame image submitted by the rendering module is displayed on the display panel of the virtual reality system. The time interval between submitting a frame image to the display link and its actual presentation on the display panel; Total system latency : ;in, The total time interval from the completion of gaze data sampling to the actual display of the corresponding frame image.

[0012] The moment when the gaze sensor completes sampling is the point in time in a virtual reality system when the sensor responsible for capturing the user's gaze completes the acquisition of a single user gaze data (such as gaze point coordinates and eye movement trajectory).

[0013] The gaze data enters the rendering module at the moment when the gaze data collected by the aforementioned gaze sensor is read by the rendering module responsible for image generation in the virtual reality system, and at the point in time when the data is ready to participate in the generation of frame images.

[0014] The gaze delay is the time interval from when gaze data is sampled to when it is usable by the rendering module, reflecting the time consumed in the transmission of gaze data. Specifically, the gaze delay is equal to the time when gaze data enters the rendering module minus the time when the gaze sensor completes sampling.

[0015] The frame submission moment of the rendering module is the point in time when the rendering module of the virtual reality system uses the read gaze data to generate the corresponding frame image (a virtual medical scene image adapted to the user's gaze) and submits the frame image to the display link (the transmission channel connecting the rendering module and the display panel).

[0016] Rendering latency is the time interval between the entry of gaze data into the rendering module and the submission of the frame image for display, reflecting the time consumed in frame image generation and submission; specifically, rendering latency equals the time when the rendering module submits the frame minus the time when gaze data enters the rendering module.

[0017] The actual time displayed on the display panel is the point in time when the frame image submitted by the rendering module to the display link is actually presented on the display panel (the display device directly observed by the user, such as a VR headset screen) of the virtual reality system.

[0018] Display latency is the time interval between the submission of a frame image to the display link and its actual presentation on the display panel, reflecting the time consumed in frame image transmission and display. Specifically, display latency equals the actual display time on the display panel minus the time when the rendering module submits the frame.

[0019] Total system latency is the complete time interval from when line-of-sight data is sampled to when the corresponding frame image is seen by the user.

[0020] In one embodiment of the present invention, the time series of external respiratory displacement of the human body is acquired and processed within a sliding time window to obtain instantaneous angular frequency, external amplitude, and phase indication quantities, including: Collect time series of external respiratory displacements of the human abdominal wall or thorax ; Set sliding time window ,time The corresponding sliding time window is ;in, The duration of the sliding time window. For time Centered on, duration is Time interval; Calculate the sliding time window Mean of internal and external respiratory displacement time series : ;in, For sliding time windows Integral variables within, For sliding time windows The average value of all displacement data within the range; The external respiratory displacement time series was detrended to obtain the detrended displacement series. ;in, To eliminate the mean trend in displacement data within the sliding time window; Determine the sliding time window displacement sequence after detrending The zero point; Sure Not greater than And satisfy The first most recent time point ; Sure Inner greater than And satisfy The second most recent time point ; Calculate instantaneous angular frequency ;in, For time The corresponding instantaneous angular frequency; Calculate external amplitude ;in, For time The corresponding external amplitude, For sliding time windows The maximum value of the internal and external respiratory displacement time series. For sliding time windows Minimum value of internal and external respiratory displacement time series; Calculate the time series of external respiratory displacement in time instantaneous derivative ; Calculate phase indication ;in, For time The corresponding phase indicator, For external respiratory displacement time series in time The rate of change at that point.

[0021] Human external respiratory displacement time series is a collection of displacement data of the human abdominal wall or thoracic cavity generated by respiratory movement, collected by displacement sensors and arranged in chronological order, reflecting the dynamic positional changes of the body surface during respiration.

[0022] A sliding time window is a time interval centered on the current time and containing a fixed duration. It is used to extract local data from the time series of respiratory displacement outside the human body, enabling real-time local analysis of respiratory dynamics.

[0023] The duration of the sliding time window is the time span of the sliding time window, which needs to cover a certain respiratory cycle to ensure the validity of local data. Specifically, the duration of the sliding time window is 1 to 3 seconds, and the specific value is set according to the adult resting respiratory rate (12-20 breaths / min) to ensure that the window contains 1 to 2 complete respiratory cycles.

[0024] The mean of the external respiratory displacement time series within and outside the sliding time window is the average level of all respiratory displacement data within the sliding time window, used to eliminate trend bias in the data; specifically, the mean of the external respiratory displacement time series within and outside the sliding time window is equal to the sum of all displacement data within the sliding time window divided by the duration of the sliding time window.

[0025] The integral variable is used to iterate through all time points within the sliding time window when calculating the average displacement within the sliding time window, corresponding to each instantaneous time within the time window.

[0026] The detrended displacement sequence is obtained by subtracting the mean displacement within the sliding time window from each data point in the human external respiratory displacement time series. It is used to remove static trends in the data and retain dynamic changes in respiration. Specifically, each data point in the detrended displacement sequence is equal to the human external respiratory displacement data at the corresponding time minus the mean of the external respiratory displacement time series within the sliding time window.

[0027] The zero point of the detrended displacement sequence within the sliding time window is the time point in the detrended displacement sequence where the value is equal to zero, reflecting the phase inflection point of respiratory motion. Specifically, when two adjacent data points in the detrended displacement sequence have one positive value and one negative value, the zero point time between the corresponding times of the two data points is calculated by linear interpolation. The interpolation formula is that the zero point time is equal to the time of the previous data point plus (the absolute value of the previous data point multiplied by the time interval between adjacent data points) divided by (the sum of the absolute values ​​of the previous data point and the absolute values ​​of the next data point).

[0028] The first most recent time point is the time point within the sliding time window that is no greater than the current time and equal to the zero point of the detrended displacement sequence. It is the most recent breathing phase inflection point before the current time.

[0029] The second most recent time point is the time point within the sliding time window that is greater than the current time and equal to the zero point of the detrended displacement sequence. It is the most recent breathing phase inflection point after the current time.

[0030] Instantaneous angular frequency is the respiratory angular frequency corresponding to the current time, reflecting the instantaneous speed of respiratory movements; specifically, instantaneous angular frequency equals 2 multiplied by pi, and then divided by the difference between the second most recent time point and the first most recent time point.

[0031] The external amplitude is the fluctuation range of respiratory displacement within the sliding time window, reflecting the strength of respiratory movement; specifically, the external amplitude is equal to the maximum value of the external respiratory displacement time series within the sliding time window minus the minimum value of the external respiratory displacement time series within the sliding time window, and then divided by 2.

[0032] The maximum value of the external respiratory displacement time series within the sliding time window is the maximum value among all external respiratory displacement data of the human body within the sliding time window.

[0033] The minimum value of the external respiratory displacement time series within the sliding time window is the minimum value among all external respiratory displacement data of the human body within the sliding time window.

[0034] The instantaneous derivative of the external respiratory displacement time series at time is the rate of change of respiratory displacement at the current time point, that is, the rate of change of the external respiratory displacement time series at time. Specifically, the instantaneous derivative of the external respiratory displacement time series at time is equal to the displacement data after the current time point minus the displacement data before the current time point, and then divided by the time interval between the two data points (i.e., the sampling period of the displacement sensor).

[0035] The phase indicator is a parameter that reflects the phase state of the current respiratory motion; specifically, the phase indicator is equal to the instantaneous derivative of the external respiratory displacement time series over time (i.e., the rate of change at time), divided by the product of the instantaneous angular frequency and the external amplitude.

[0036] The rate of change of the external respiratory displacement time series at time is the instantaneous derivative of the external respiratory displacement time series at time, which reflects how fast the respiratory displacement changes at the current time point.

[0037] In one embodiment of the present invention, the intersection of the user's line of sight in the virtual reality scene and the pre-segmented liver segment model is obtained to obtain the position of the line of sight intersection and the segment label; data is retrieved from a preset segment parameter library according to the segment label to form a segment amplitude parameter set containing the three-dimensional amplitude of the liver, including: In a virtual reality scene, the user's gaze ray is represented in parametric form as follows: : ;in, Let be the expression for the spatial trajectory of the line-of-sight ray in the virtual reality scene. This is the starting point of the line of sight ray. Let be the direction vector of the line of sight ray. For ray parameters; The pre-segmented liver segment model is based on The criteria for liver segmentation are constructed and represented as follows: : ;in, For a complete pre-segmented 3D liver model, For the first indivual The spatial region corresponding to the liver segment for The liver segments are numbered, with values ​​ranging from 1 to 8; Calculate the intersection parameters of the line-of-sight ray and the pre-segmented liver segment model. : ;in, The minimum non-negative ray parameter that satisfies the intersection condition between the line of sight ray and the pre-segmented liver segment model corresponds to the position where the line of sight ray first enters the pre-segmented liver segment model; Determine the location of the intersection of lines of sight ;in, The three-dimensional coordinates of the first intersection between the line of sight ray and the pre-segmented liver segment model; Set the rank tag determination function ; Calculate rank tags Determined; among them, The liver segment number to which the line of sight intersects is located and The liver segments are numbered identically. Used to determine the corresponding 3D coordinates based on the input. Liver segment numbering; based on The standard preset segment parameter library for liver segmentation stores the parameters for each segment. The amplitudes of respiratory movements in the vertical, anterior-posterior, and lateral directions corresponding to the liver segments, respectively; Set the preset rank parameter library retrieval function ; Calculate the segment amplitude parameter set : ;in, It is a set of parameters containing the three-dimensional amplitude of the liver. It is used to retrieve the triaxial respiratory motion amplitude of the corresponding liver segment from the preset segment parameter library based on the input segment label. This corresponds to the amplitude of respiratory movements in the vertical direction of the liver segment. This corresponds to the amplitude of respiratory movements in the anterior-posterior direction of the liver segment. This represents the amplitude of respiratory movements in the left and right directions corresponding to the liver segment.

[0038] The parametric form of the gaze ray is a mathematical expression describing the spatial trajectory of the user's gaze in a virtual reality scene. It is used to quantitatively calculate the intersection point between the gaze ray and the liver segment model. Specifically, the spatial trajectory expression of the gaze ray in a virtual reality scene is equal to the starting position of the gaze ray plus the ray parameter multiplied by the direction vector of the gaze ray.

[0039] The spatial trajectory expression of the gaze ray is a parametric description of the path that the user's gaze extends in the virtual reality scene, with each ray parameter corresponding to a spatial point on the trajectory.

[0040] The starting point of the gaze ray is the initial spatial coordinate of the user's gaze in the virtual reality scene, corresponding to the virtual position of the user's eyes in the scene; specifically, the starting point of the gaze ray is equal to the three-dimensional coordinates of the user's pupil center in the scene collected by the eye tracking module of the virtual reality device (such as a VR headset).

[0041] The direction vector of the gaze ray is a three-dimensional vector that describes the direction of the user's gaze in the virtual reality scene, reflecting the direction of the gaze. Specifically, the direction vector of the gaze ray is calculated by collecting the eye rotation angles (horizontal angle and vertical angle) from the eye tracking module of the VR headset. The horizontal angle corresponds to the left-right direction component, the vertical angle corresponds to the up-down direction component, and the front-back direction component is set by the scene coordinate system. Finally, the three components are normalized to obtain the unit direction vector.

[0042] The ray parameter is a variable that controls the extension length of the line-of-sight ray trajectory. It is a non-negative value, and the larger the value, the farther the point on the trajectory is from the starting point.

[0043] A pre-segmented liver model is a three-dimensional model of the liver divided according to specific medical standards, containing spatial regions of 8 independent liver segments. Specifically, the pre-segmented liver model reconstructs the three-dimensional structure of the liver through medical imaging (such as 4DCT or MRI), and then divides the three-dimensional structure into 8 spatial regions according to the Couinaud liver segmentation standard (with the hepatic vein and portal vein as the boundary), with each region corresponding to a liver segment.

[0044] A complete pre-segmented 3D liver model is a 3D structural model that covers the entire liver, formed by merging 8 spatial regions of the liver segment.

[0045] The spatial region corresponding to the kth liver segment is the three-dimensional spatial range of the kth liver segment (k ranges from 1 to 8) in the virtual reality scene, which is divided according to medical standards in the pre-segmented liver segment model.

[0046] The liver segments are numbered sequentially to identify the eight liver segments, with values ​​ranging from 1 to 8, consistent with the liver segment numbers in the Couinaud liver segmentation criteria.

[0047] The intersection parameter is the minimum non-negative ray parameter that ensures the line of sight ray falls within the pre-segmented liver segment model, guaranteeing that the corresponding intersection point is the position where the line of sight first contacts the liver. Specifically, the non-negative ray parameter values ​​are iterated, and each parameter value is substituted into the line of sight ray spatial trajectory expression. It is then determined whether the obtained spatial point is within the spatial region of the pre-segmented liver segment model. The minimum ray parameter value that meets the condition is selected as the intersection parameter.

[0048] The intersection point of the line of sight is the three-dimensional spatial coordinates of the line of sight when it first enters the pre-segmented liver segment model. Specifically, the spatial coordinates of the point obtained by substituting the intersection parameters of the line of sight and the pre-segmented liver segment model into the spatial trajectory expression of the line of sight are the intersection point of the line of sight.

[0049] The segment label determination function is used to determine the liver segment number to which a segment belongs based on its spatial coordinates. Specifically, the input of the segment label determination function is the three-dimensional coordinates of the intersection of the lines of sight. The function traverses the eight spatial regions of the pre-segmented liver segment model, determines which region the input coordinates belong to, and outputs the liver segment number (1 to 8) corresponding to that region.

[0050] The segment label is the number that identifies the liver segment to which the intersection of the lines of sight belongs, and it is consistent with the numbering of the liver segments (1 to 8). Specifically, the segment label is equal to the output result obtained after inputting the intersection of the lines of sight into the segment label determination function.

[0051] The liver segmentation standard is a medical standard used to divide the liver into 8 segments, specifically the Couinaud liver segmentation standard, which uses the hepatic vein and portal vein branches as anatomical boundaries.

[0052] The preset segment parameter library is a database that stores the three-dimensional respiratory motion amplitude of each liver segment in advance. Specifically, the data in the preset segment parameter library comes from the actual measurement of human liver respiratory motion by 4DCT or MRI. The respiratory motion amplitude of each Couinaud liver segment in the vertical, front-back, and left-right directions is extracted, classified and stored according to the liver segment number to form the preset segment parameter library.

[0053] The preset segment parameter library retrieval function is used to retrieve the corresponding three-dimensional amplitude from the preset segment parameter library based on the liver segment number. Specifically, the input of the preset segment parameter library retrieval function is the segment label (liver segment number). The function queries the preset segment parameter library based on the input number and outputs the vertical respiratory motion amplitude, the anterior-posterior respiratory motion amplitude, and the lateral respiratory motion amplitude corresponding to that liver segment.

[0054] The segment amplitude parameter set is a set of parameters that includes the three-dimensional respiratory motion amplitude of the liver segment to which the line of sight intersects; specifically, the segment amplitude parameter set is equal to the set of respiratory motion amplitudes in the vertical direction, the front-back direction, and the left-right direction after the segment label is input into the preset segment parameter library retrieval function.

[0055] The amplitude of respiratory motion in the vertical direction corresponding to the liver segment is the amplitude of the liver segment to which the line of sight intersects in the vertical direction as the respiratory motion occurs, which is stored in the preset segment parameter library.

[0056] The amplitude of respiratory motion in the anterior-posterior direction corresponding to the liver segment is the amplitude of respiratory motion in the anterior-posterior direction of the liver segment to which the line of sight intersects, which is stored in the preset segment parameter library.

[0057] The amplitude of respiratory movements in the left and right directions corresponding to the liver segment is the amplitude of the liver segment to which the line of sight intersects in the left and right directions as it moves with the breath, which is stored in the preset segment parameter library.

[0058] In one embodiment of the present invention, the compensation displacement vector is calculated based on the total system delay, instantaneous angular frequency, phase indication, and segment amplitude parameter set, including: Define the unit vector in the up and down directions. Unit vectors in the forward and backward directions and the unit vector in the left and right directions ; based on , and Forming an orthogonal basis in three-dimensional space; Calculate the vertical compensation displacement components : ;in, For time The corresponding vertical compensation displacement components; Calculate the forward and backward compensating displacement components : ;in, For time The corresponding forward and backward compensated displacement components; Calculate the compensation displacement components in the left and right directions : ;in, For time The corresponding left and right direction compensation displacement components; Calculate the compensation displacement vector ; ;in, For time The corresponding three-dimensional compensation displacement vector.

[0059] The unit vector in the vertical direction is a three-dimensional unit vector describing the vertical anatomical direction of the human liver in a virtual reality scene. It has a magnitude of 1 and is used to locate and compensate for the vertical component of displacement. Specifically, a right-handed anatomical coordinate system is used, and the vertical direction of the human model's torso is set as the vertical direction. The forward and backward component of this unit vector is 0, the vertical component is 1, and the left and right component is 0. That is, it only extends along the positive direction of the vertical axis and has a magnitude of 1.

[0060] The unit vector in the front-back direction is a three-dimensional unit vector describing the front-back anatomical direction of the human liver in a virtual reality scene. It has a magnitude of 1 and is used to locate and compensate for the displacement in the front-back direction. Specifically, based on the right-handed anatomical coordinate system mentioned above, the chest and back direction of the human model is set as the front-back direction. The front-back component of this unit vector is 1, the up-down component is 0, and the left-right component is 0. That is, it only extends along the positive front-back axis and has a magnitude of 1.

[0061] The unit vector in the left and right direction is a three-dimensional unit vector describing the left and right anatomical direction of the human liver in the virtual reality scene. It has a magnitude of 1 and is used to locate and compensate for the displacement in the left and right direction. Specifically, based on the right-handed anatomical coordinate system mentioned above, the horizontal lateral direction of the human body model's torso is set as the left and right direction. The front-back direction component of this unit vector is 0, the up-down direction component is 0, and the left-right direction component is 1. That is, it only extends along the positive left and right axis and has a magnitude of 1.

[0062] An orthogonal basis in three-dimensional space is a spatial reference frame composed of unit vectors in the up, down, front, back, and left directions, ensuring that compensation displacements can be accurately decomposed and synthesized in three-dimensional space. Specifically, an orthogonal basis in three-dimensional space satisfies three conditions: first, the dot product of the unit vectors in the up and down directions and the unit vectors in the front and back directions is 0 (pairwise perpendicular); second, the dot product of the unit vectors in the up and down directions and the unit vectors in the left and right directions is 0 (pairwise perpendicular); and third, the dot product of the unit vectors in the front and back directions and the unit vectors in the left and right directions is 0 (pairwise perpendicular), and the magnitude of all three unit vectors is 1.

[0063] The vertical compensation displacement component is the projection of the compensation displacement vector in the vertical direction, used to offset the loading deviation in this direction caused by system time delay and respiratory motion. Specifically, the vertical compensation displacement component is equal to the total system time delay multiplied by the instantaneous angular frequency, then multiplied by the phase indicator, and finally multiplied by the vertical respiratory motion amplitude of the corresponding liver segment in the segment amplitude parameter group.

[0064] The anterior-posterior direction compensation displacement component is the projection of the compensation displacement vector in the anterior-posterior direction, used to offset the loading deviation in this direction caused by system time delay and respiratory motion. Specifically, the anterior-posterior direction compensation displacement component is equal to the total system time delay multiplied by the instantaneous angular frequency, then multiplied by the phase indication, and finally multiplied by the anterior-posterior direction respiratory motion amplitude of the corresponding liver segment in the segment amplitude parameter group.

[0065] The left and right direction compensation displacement component is the projection of the compensation displacement vector in the left and right directions, used to offset the loading deviation in this direction caused by system time delay and respiratory motion. Specifically, the left and right direction compensation displacement component is equal to the total system time delay multiplied by the instantaneous angular frequency, then multiplied by the phase indication, and finally multiplied by the left and right direction respiratory motion amplitude of the corresponding liver segment in the segment amplitude parameter group.

[0066] The compensation displacement vector is the total vector in three-dimensional space used to correct the loading position deviation, integrating the compensation components in three directions; specifically, the compensation displacement vector is equal to the compensation displacement component in the vertical direction multiplied by the unit vector in the vertical direction, plus the compensation displacement component in the front-back direction multiplied by the unit vector in the front-back direction, plus the compensation displacement component in the left-right direction multiplied by the unit vector in the left-right direction.

[0067] In one embodiment of the present invention, the center position of the high-detail loading region is obtained by summing the position of the line-of-sight intersection with the compensation displacement vector, including: Determine the center position of the high detail loading region : ;in, For time The three-dimensional coordinates of the center position of the corresponding high-detail loading region.

[0068] In one embodiment of the present invention, an amplitude scaling factor is determined based on the external amplitude, and the amplitudes in each direction of the segment amplitude parameter group are multiplied by the amplitude scaling factor to obtain the scaled segment amplitude, including: Determine the initial calibration window length as The corresponding time period is ;in, The duration of the initial calibration window. This is the initial calibration time; Calculate the external amplitude reference within the initial calibration window. : ;in, This is a time series of external respiratory displacement in the human body. Calculate the amplitude scaling factor ;in, For time The corresponding amplitude scaling factor, For time Corresponding external amplitude; Calculate the amplitude after scaling in the vertical direction ;in, For time The amplitude after scaling in the vertical direction; Calculate the amplitude after scaling in the forward and backward directions ;in, For time The amplitude after scaling in the corresponding forward and backward directions; Calculate the amplitude after scaling in the left and right directions ;in, For time The amplitude after scaling in the corresponding left and right directions; based on , and This forms a scaled segment amplitude parameter set.

[0069] In one embodiment of the present invention, the three-dimensional radius of the liver is calculated based on the total system delay, instantaneous angular frequency, and scaled segment amplitude, including: Determine the base radius in the vertical direction : ;in, For time The corresponding base radius in the vertical direction; Determine the base radius in the front and back directions : ;in, For time The corresponding base radius in the front and rear directions; Determine the base radius in the left and right directions : ;in, For time The corresponding base radii in the left and right directions; Determine the radius in the vertical direction : ;in, For time The corresponding vertical radius of the liver; Determine the radius in the forward and backward directions : ;in, For time The corresponding anteroposterior radius of the liver; Determine the radius in the left and right directions : ;in, For time The corresponding left-right radius of the liver.

[0070] The intersection of the line of sight is the three-dimensional coordinate of the first intersection between the user's line of sight and the pre-segmented liver segment model in the virtual reality scene. It is the basic reference point for calculating the center position of the high-detail loading area.

[0071] The center position of the high-detail loading area is the reference 3D coordinate of the high-detail model loading range in the virtual reality scene, used to ensure that the high-detail area is aligned with the actual position of the liver. Specifically, the center position of the high-detail loading area is calculated by summing the 3D coordinate components. That is, the vertical coordinate of the center position is equal to the vertical coordinate of the line-of-sight intersection plus the vertical component of the compensation displacement vector; the horizontal coordinate of the center position is equal to the horizontal coordinate of the line-of-sight intersection plus the horizontal component of the compensation displacement vector; and the horizontal coordinate of the center position is equal to the horizontal coordinate of the line-of-sight intersection plus the horizontal component of the compensation displacement vector.

[0072] In one embodiment of the present invention, an anisotropic high-detail loading region is formed with the center position as the center and the anisotropic radius as the semi-axis, including: Determine the anisotropic high-detail loading region : ;in, For time The corresponding anisotropic high-detail loading region, The three-dimensional coordinates of any point in space within a virtual reality scene. Represents the dot product of vectors.

[0073] An anisotropic high-detail loading region is a high-detail model loading range in a virtual reality scene, divided by radii in different directions, with shapes adapted to the anisotropic motion differences of the liver. Specifically, the process involves the following steps: First, calculate the coordinate difference between any spatial point and the center position in the vertical direction. Squaring this difference and dividing it by the square of the vertical radius yields the vertical normalized distance term. Second, calculate the coordinate difference between any spatial point and the center position in the front-back direction. Squaring this difference and dividing it by the square of the front-back radius yields the front-back normalized distance term. Third, calculate the coordinate difference between any spatial point and the center position in the left-right direction. Squaring this difference and dividing it by the square of the left-right radius yields the left-right normalized distance term. Fourth, sum the three normalized distance terms. If the sum is less than or equal to 1, the spatial point belongs to the anisotropic high-detail loading region; if it is greater than 1, it does not belong to this region.

[0074] The three-dimensional coordinates in a virtual reality scene are the positional information of any spatial point in the virtual reality scene that is to be determined whether it belongs to a high-detail loading area, including coordinate components in three anatomical directions: up and down, front and back, and left and right.

[0075] The dot product of vectors is used to calculate the directional difference between any spatial point and the center position. Specifically, it is the product of the coordinate differences along the same anatomical direction (since the direction vector is a unit vector, the dot product result is equal to the coordinate difference in that direction). For example, in the up-down direction, the dot product of vectors is equal to (the up-down coordinates of any spatial point minus the up-down coordinates of the center position) multiplied by (the up-down components of the unit vector in the up-down direction). Since the up-down components of the unit vector in the up-down direction are 1, the dot product result is equal to the coordinate difference in that direction.

[0076] The embodiments of this example have been described above. However, this example is not limited to the specific implementation methods described above. The specific implementation methods described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms based on the guidance of this example, and all of them are within the protection scope of this example.

Claims

1. A real-time feedback interactive virtual diagnosis and treatment teaching and training system, characterized in that, include: The latency module measures the latency of the virtual reality system's line-of-sight, rendering, and display processes to obtain the total system latency. The data acquisition and processing module acquires and processes the time series of external respiratory displacement of the human body within a sliding time window to obtain the instantaneous angular frequency, external amplitude and phase indication. The liver amplitude module intersects the user's line of sight in the virtual reality scene with the pre-segmented liver segment model to obtain the position of the line of sight intersection and the segment label; it then retrieves data from the preset segment parameter library according to the segment label to form a segment amplitude parameter set containing the three-dimensional amplitude of the liver. The center position calculation module calculates the compensation displacement vector based on the total system delay, instantaneous angular frequency, phase indication, and segment amplitude parameter set; the center position of the high detail loading area is obtained by summing the line-of-sight intersection position with the compensation displacement vector. The amplitude scaling module determines the amplitude scaling factor based on the external amplitude, and multiplies the amplitude of each direction in the segment amplitude parameter group with the amplitude scaling factor to obtain the scaled segment amplitude. The loading region determination module calculates the three-dimensional radius of the liver based on the total system delay, instantaneous angular frequency, and scaled segment amplitude; and forms an anisotropic high-detail loading region with the center position as the center and the anisotropic radius as the semi-axis.

2. The interactive virtual diagnosis and treatment teaching and training system with real-time feedback according to claim 1, characterized in that, The total system latency is obtained by measuring the latency of the line-of-sight, rendering, and display stages of a virtual reality system, including: Time of completion of sampling by gaze sensor When the view data enters the rendering module ; Calculate the delay of the line of sight : ;in, This refers to the point in time when the gaze sensor in a virtual reality system completes a single user gaze data acquisition. This refers to the point in time when the collected gaze data is read by the rendering module of the virtual reality system and becomes usable. The time interval from when gaze data is sampled to when it is available to the rendering module; The frame submission time of the acquisition and rendering module ; Calculate rendering latency : ;in, This refers to the time point at which the rendering module uses the aforementioned gaze data to generate the corresponding frame image and submits it to the display link. This refers to the time interval between the entry of gaze data into the rendering module and the submission of the corresponding frame image for display; The actual time displayed on the acquisition display panel ; Calculate display latency : ;in, This refers to the actual point in time when the frame image submitted by the rendering module is displayed on the display panel of the virtual reality system. The time interval between submitting a frame image to the display link and its actual presentation on the display panel; Total system latency : ;in, The total time interval from the completion of gaze data sampling to the actual display of the corresponding frame image.

3. The interactive virtual diagnosis and treatment teaching and training system with real-time feedback according to claim 2, characterized in that, The time series of external respiratory displacement of the human body is acquired and processed within a sliding time window to obtain instantaneous angular frequency, external amplitude, and phase indication quantities, including: Collect time series of external respiratory displacements of the human abdominal wall or thorax ; Set sliding time window ,time The corresponding sliding time window is ;in, The duration of the sliding time window. For time Centered on, duration is Time interval; Calculate the sliding time window Mean of internal and external respiratory displacement time series : ;in, For sliding time windows Integral variables within, For sliding time windows The average value of all displacement data within the range; The external respiratory displacement time series was detrended to obtain the detrended displacement series. ;in, To eliminate the mean trend in displacement data within the sliding time window; Determine the sliding time window displacement sequence after detrending The zero point; Sure Not greater than And satisfy The first most recent time point ; Sure Inner greater than And satisfy The second most recent time point ; Calculate instantaneous angular frequency ;in, For time The corresponding instantaneous angular frequency; Calculate external amplitude ;in, For time The corresponding external amplitude, For sliding time windows The maximum value of the internal and external respiratory displacement time series. For sliding time windows Minimum value of internal and external respiratory displacement time series; Calculate the time series of external respiratory displacement in time instantaneous derivative ; Calculate phase indication ;in, For time The corresponding phase indicator, For external respiratory displacement time series in time The rate of change at that point.

4. The interactive virtual diagnosis and treatment teaching and training system with real-time feedback according to claim 3, characterized in that, The intersection of the user's gaze ray in the virtual reality scene and the pre-segmented liver segment model is obtained to get the location of the gaze intersection point and the segment label; data is retrieved from a preset segment parameter library according to the segment label to form a segment amplitude parameter set containing the three-dimensional amplitude of the liver, including: In a virtual reality scene, the user's gaze ray is represented in parametric form as follows: : ;in, Let be the expression for the spatial trajectory of the line-of-sight ray in the virtual reality scene. This is the starting point of the line of sight ray. Let be the direction vector of the line of sight ray. For ray parameters; The pre-segmented liver segment model is based on The criteria for liver segmentation are constructed and represented as follows: : ;in, For a complete pre-segmented 3D liver model, For the first indivual The spatial region corresponding to the liver segment for The liver segments are numbered, with values ​​ranging from 1 to 8; Calculate the intersection parameters of the line-of-sight ray and the pre-segmented liver segment model. : ;in, The minimum non-negative ray parameter that satisfies the intersection condition between the line of sight ray and the pre-segmented liver segment model corresponds to the position where the line of sight ray first enters the pre-segmented liver segment model; Determine the location of the intersection of lines of sight ;in, The three-dimensional coordinates of the first intersection between the line of sight ray and the pre-segmented liver segment model; Set the rank tag determination function ; Calculate rank tags Determined; among them, The liver segment number to which the line of sight intersects is located and The liver segments are numbered identically. Used to determine the corresponding 3D coordinates based on the input. Liver segment numbering; based on The standard preset segment parameter library for liver segmentation stores the parameters for each segment. The amplitudes of respiratory movements in the vertical, anterior-posterior, and lateral directions corresponding to the liver segments, respectively; Set the preset rank parameter library retrieval function ; Calculate the segment amplitude parameter set : ;in, It is a set of parameters containing the three-dimensional amplitude of the liver. It is used to retrieve the triaxial respiratory motion amplitude of the corresponding liver segment from the preset segment parameter library based on the input segment label. This corresponds to the amplitude of respiratory movements in the vertical direction of the liver segment. This corresponds to the amplitude of respiratory movements in the anterior-posterior direction of the liver segment. This represents the amplitude of respiratory movements in the left and right directions corresponding to the liver segment.

5. The interactive virtual diagnosis and treatment teaching and training system with real-time feedback according to claim 4, characterized in that, Based on the total system delay, instantaneous angular frequency, phase indication, and segment amplitude parameter set, the compensation displacement vector is calculated, including: Define the unit vector in the up and down directions. Unit vectors in the forward and backward directions and the unit vector in the left and right directions ; based on , and Forming an orthogonal basis in three-dimensional space; Calculate the vertical compensation displacement components : ;in, For time The corresponding vertical compensation displacement components; Calculate the forward and backward compensating displacement components : ;in, For time The corresponding forward and backward compensated displacement components; Calculate the compensation displacement components in the left and right directions : ;in, For time The corresponding left and right direction compensation displacement components; Calculate the compensation displacement vector ; ;in, For time The corresponding three-dimensional compensation displacement vector.

6. The interactive virtual diagnosis and treatment teaching and training system with real-time feedback according to claim 5, characterized in that, Summing the intersection point of the lines of sight with the compensation displacement vector yields the center position of the high-detail loading region, including: Determine the center position of the high detail loading region : ;in, For time The three-dimensional coordinates of the center position of the corresponding high-detail loading region.

7. The interactive virtual diagnosis and treatment teaching and training system with real-time feedback according to claim 6, characterized in that, The amplitude scaling factor is determined based on the external amplitude. The amplitudes in each direction of the segment amplitude parameter set are multiplied by the amplitude scaling factor to obtain the scaled segment amplitude, including: Determine the initial calibration window length as The corresponding time period is ;in, The duration of the initial calibration window. This is the initial calibration time; Calculate the external amplitude reference within the initial calibration window. : ;in, This is a time series of external respiratory displacement in the human body. Calculate the amplitude scaling factor ;in, For time The corresponding amplitude scaling factor, For time Corresponding external amplitude; Calculate the amplitude after scaling in the vertical direction ;in, For time The amplitude after scaling in the vertical direction; Calculate the amplitude after scaling in the forward and backward directions ;in, For time The amplitude after scaling in the corresponding forward and backward directions; Calculate the amplitude after scaling in the left and right directions ;in, For time The amplitude after scaling in the corresponding left and right directions; based on , and This forms a scaled segment amplitude parameter set.

8. The interactive virtual diagnosis and treatment teaching and training system with real-time feedback according to claim 7, characterized in that, Based on the total system delay, instantaneous angular frequency, and scaled segment amplitude, the three-dimensional radius of the liver is calculated, including: Determine the base radius in the vertical direction : ;in, For time The corresponding base radius in the vertical direction; Determine the base radius in the front and back directions : ;in, For time The corresponding base radius in the front and rear directions; Determine the base radius in the left and right directions : ;in, For time The corresponding base radii in the left and right directions; Determine the radius in the vertical direction : ;in, For time The corresponding vertical radius of the liver; Determine the radius in the forward and backward directions : ;in, For time The corresponding anteroposterior radius of the liver; Determine the radius in the left and right directions : ;in, For time The corresponding left-right radius of the liver.

9. The interactive virtual diagnosis and treatment teaching and training system with real-time feedback according to claim 8, characterized in that, An anisotropic high-detail loading region is formed with the center position as the center and the anisotropic radius as the semi-axis, including: Determine the anisotropic high-detail loading region : ;in, For time The corresponding anisotropic high-detail loading region, The three-dimensional coordinates of any point in space within a virtual reality scene. Represents the dot product of vectors.

Images