A visual tracking pose synchronization and real-time matching method and system for kidney puncture MR teaching

Abstract

This invention provides a method and system for visual tracking posture synchronization and real-time ultrasound image matching for MR teaching of renal biopsy. The method includes: acquiring the real-time spatial poses of a physical ultrasound probe and a physical puncture needle in a real-world operating space; mapping the real-time spatial poses of the physical ultrasound probe and the physical puncture needle to corresponding virtual models in a mixed reality scene; based on the real-time spatial pose of the virtual ultrasound probe, retrieving a matching target ultrasound image frame from a pre-constructed clinical ultrasound image database using a spatial distance metric; the spatial distance metric is as follows: calculating the difference in rotation matrix norm between the real-time pose of the probe and the corresponding pose of the image database frame as the rotational pose difference, and the difference in translation vector norm as the translational position difference; weighting and fusing the rotational pose difference and translational position difference using differentiated weights to obtain a comprehensive matching distance; and displaying the matched target ultrasound image frame in real-time to form an ultrasound cross-section updated synchronously with the movement of the virtual ultrasound probe.

Description

Technical Field

[0001] This invention belongs to the technical fields of mixed reality interaction, machine vision tracking, posture calculation algorithms and real-time matching of medical ultrasound images, specifically involving a method and system for visual tracking posture synchronization and real-time matching of ultrasound images for MR teaching of renal biopsy. Background Technology

[0002] Renal biopsy is currently the gold standard for pathological diagnosis, classification, staging, and prognostic assessment of kidney diseases, and it is also a core skill that nephrologists must master. This procedure must be performed under real-time ultrasound guidance, demanding extremely high levels of hand-eye coordination, spatial positioning ability, and ultrasound image interpretation skills from the operator. Improper operation can easily lead to complications such as perirenal hematoma and hematuria. Therefore, standardized preoperative teaching and training are crucial for ensuring operational safety, improving clinicians' practical skills, and are an important component of the standardized clinical training system.

[0003] Traditional kidney biopsy training relies heavily on specimens, animal models, and human simulators, all of which have inherent limitations. Specimens are scarce, training costs are high, and the anatomical structure and elasticity of ex vivo tissues differ significantly from those of living tissues, making it impossible to accurately reproduce the real tissue feedback in clinical practice. Animal models have fundamentally different kidney anatomy and ultrasound imaging characteristics compared to humans, making it difficult to create clinically relevant operational scenarios. Conventional human simulators are mostly static anatomical structures, unable to achieve dynamic linkage between operative movements and ultrasound images. Trainees struggle to establish the corresponding logic between operative adjustments and image changes, resulting in limited translatability of training into clinical practice and failing to meet the core needs of clinical skills training.

[0004] With the development of mixed reality and virtual reality technologies, digital interactive teaching systems have gradually become an important direction for renal biopsy training, and various virtual biopsy teaching systems have been put into application. However, existing systems still have multi-dimensional technical shortcomings and are difficult to adapt to the full-process requirements of standardized clinical training. In terms of interactive immersion, most existing systems still use general interactive handles to replace real clinical instruments, which cannot reproduce the grip feel, operating force, and operating stroke of clinical puncture needles and ultrasound probes. Trainees cannot obtain real operation feedback that is consistent with clinical practice, and there is a disconnect between training content and clinical practice scenarios, making it difficult to effectively cultivate operational muscle memory.

[0005] In terms of image fidelity, existing virtual teaching systems primarily generate ultrasound-guided images based on virtual anatomical models. These simulated images can only achieve simple grayscale mapping of anatomical structures and cannot reproduce the tissue echo characteristics, texture details, and physiological artifacts of real clinical ultrasound images. They also cannot replicate the individual image differences between patients, resulting in a significant gap between these images and those used in actual clinical practice. The image interpretation logic developed by trainees during training cannot be directly adapted to real clinical scenarios and may even lead to incorrect image perceptions, failing to meet the image fidelity requirements of clinical training.

[0006] In terms of instrument tracking and virtual-real interaction, existing teaching systems with physical instruments mostly adopt electromagnetic tracking or conventional visual marker tracking schemes, which have obvious shortcomings in robustness. Electromagnetic tracking schemes are susceptible to electromagnetic interference from metal equipment and electronic instruments in the teaching environment, and are prone to pose drift and accuracy degradation when multiple devices are used simultaneously. Conventional visual marker tracking schemes are prone to recognition failure and tracking loss in complex backgrounds, excessive viewing angle deflection, and slight marker occlusion scenarios, which cannot guarantee the stability of instrument pose recognition during long-term continuous training, thus leading to interruption of virtual-real interaction and affecting the continuity and smoothness of the teaching process.

[0007] In terms of real-time response and system stability, the matching logic between the ultrasound section and probe movement in existing systems mostly focuses only on the translational position of the probe, without taking into account the synchronous matching of the probe's rotational posture. This easily leads to problems such as ultrasound section jumps and matching lags when the probe is adjusted slightly, making it impossible to achieve real-time synchronous response between probe movement and image changes. At the same time, most existing systems use a single-threaded architecture to handle core tasks such as pose calculation, image rendering, and interactive response. When running continuously for a long time, they are prone to screen stuttering and response delays, making them unsuitable for large-scale, routine clinical teaching and training scenarios and failing to meet the core requirements of clinical training for system real-time performance and stability.

[0008] In summary, the existing teaching system for renal biopsy, whether the traditional offline training model or the existing digital virtual teaching system, cannot simultaneously meet the requirements of clinical operation fidelity, image fidelity, tracking stability, and real-time response. It is difficult to build a standardized, full-process practical training system that conforms to the actual clinical operation process, and it cannot meet the clinical-level practical training requirements for renal biopsy in the standardized training of clinicians. The relevant technical shortcomings still need to be further addressed and overcome. Summary of the Invention

[0009] To address the shortcomings and deficiencies of existing technologies, this invention provides a method and system for visual tracking posture synchronization and real-time ultrasound image matching in MR teaching for renal biopsy. This solution addresses the core pain point of existing renal biopsy teaching systems, which cannot simultaneously achieve clinical operation fidelity, image fidelity, tracking stability, and real-time response. It constructs a full-process clinical-grade training solution based on mixed reality interaction, machine vision tracking, and medical image matching technologies. By affixing high-contrast dot matrix identifiers to the physical puncture needle and ultrasound probe, combined with PnP pose calculation principles, stable and accurate acquisition of the spatial pose of physical instruments is achieved, supporting simultaneous tracking of multiple instruments. A quaternion pose synchronization algorithm with initial offset compensation and module length normalization constraints is employed to achieve real-time and accurate pose mapping between physical instruments and their corresponding virtual models. It effectively suppresses cumulative system errors and avoids gimbal lock-up issues; it pre-constructs a database of real human kidney ultrasound images with spatial pose indexes to replace traditional simulated rendering images, ensuring clinical-grade fidelity of ultrasound images; based on the real-time spatial pose of the probe, it completes ultrasound section matching by fusing differential weighted spatial distance measurements of rotational posture differences and translational position differences, and optimizes the retrieval range by combining X-ray detection technology, achieving low-latency, accurate matching and smooth switching of ultrasound sections and probe movements; at the same time, it adopts a multi-threaded architecture to separate core tasks, ensuring the stability and real-time performance of the system during long-term continuous operation, and can fully adapt to the entire process requirements of standardized clinical training for renal biopsy, helping trainees establish the corresponding logic between operational actions and changes in ultrasound images, improving the effectiveness of clinical training and operational conversion efficiency.

[0010] The specific technical solution adopted by this invention to solve its technical problem is as follows:

[0011] A method for visual tracking posture synchronization and real-time matching of ultrasound images for teaching renal biopsy using MR imaging includes:

[0012] Acquire the real-time spatial pose of the physical ultrasound probe and the physical puncture needle in the actual operating space;

[0013] The real-time spatial poses of the physical ultrasound probe and the physical puncture needle are mapped to the corresponding virtual models in the mixed reality scene to achieve real-time posture synchronization between the physical instruments and the virtual models.

[0014] Based on the real-time spatial pose of the virtual ultrasound probe, matching target ultrasound image frames are retrieved from a pre-constructed clinical ultrasound image library using a spatial distance measurement method.

[0015] The clinical ultrasound image library stores multiple frames of clinically acquired human kidney ultrasound images. Each frame of ultrasound image is associated with spatial pose information in the form of rotation matrix and translation vector of the ultrasound probe when the image was acquired.

[0016] The spatial distance measurement method is as follows: calculate the difference in the rotation matrix norm between the real-time pose of the probe and the corresponding pose of the image library frame as the rotation pose difference, and the difference in the translation vector norm as the translation position difference. The rotation pose difference and the translation position difference are weighted and fused using differentiated weights to obtain the comprehensive matching distance.

[0017] In the mixed reality scenario, the matched target ultrasound image frames are displayed in real time, forming an ultrasound cross-section that is updated synchronously with the movement of the virtual ultrasound probe.

[0018] Furthermore, the method for obtaining the real-time spatial pose of the physical ultrasound probe and the physical puncture needle is as follows: using a pre-calibrated vision camera, image information of high-contrast black and white coded dot matrix identifiers pasted on the surface of the physical ultrasound probe and the physical puncture needle is acquired, and the real-time spatial pose is calculated based on the PnP pose calculation principle; different physical instruments are configured with exclusive dot matrix identifiers, and the vision camera can simultaneously acquire image information of multiple sets of exclusive dot matrix identifiers and simultaneously calculate the spatial pose of the corresponding instruments.

[0019] Furthermore, the step of achieving real-time attitude synchronization between the physical instrument and the virtual model is implemented using a quaternion attitude synchronization algorithm. Specifically, the initial pose offset between the physical instrument and the corresponding virtual model is compensated by quaternion conjugation and quaternion multiplication operations to obtain a synchronized attitude quaternion. The synchronized attitude quaternion is then normalized to suppress the cumulative numerical error during system operation.

[0020] Furthermore, in the pre-constructed clinical ultrasound image library, all ultrasound images are rearranged according to the corresponding probe spatial pose at the time of acquisition, establishing a fast index structure with probe spatial pose data as the core.

[0021] Further, the difference in the rotation matrix norm is the Frobenius norm of the difference between the two rotation matrices, and the difference in the translation vector norm is the 2-norm of the difference between the two translation vectors; the specific steps for retrieving the matching target ultrasound image frame are as follows: a preset comprehensive matching distance threshold is set, all image frames in the clinical ultrasound image database with a comprehensive matching distance less than the threshold are retrieved, and the image frame with the smallest comprehensive matching distance is selected as the target ultrasound image frame.

[0022] Furthermore, when searching for matching target ultrasound image frames, the collision point coordinates between the virtual ultrasound probe and the virtual kidney anatomy model in the mixed reality scene are obtained through X-ray detection technology. The search range is narrowed by combining the hierarchical markers of the kidney anatomy region corresponding to the collision point coordinates, thereby optimizing the matching results.

[0023] Furthermore, when displaying the target ultrasound image frame in real time, smooth switching of ultrasound sections is achieved, and the corresponding position of the virtual puncture needle is displayed synchronously in the ultrasound section.

[0024] Furthermore, the core tasks of pose acquisition, pose synchronization, image retrieval and matching, and image rendering and display are allocated to independent threads using a multi-threaded architecture, and the real-time data interaction of each stage is ensured through a thread synchronization mechanism.

[0025] And, a visual tracking and real-time ultrasound image matching system for MR teaching of renal biopsy, comprising:

[0026] The pose acquisition module is used to acquire the real-time spatial pose of the physical ultrasound probe and the physical puncture needle in the actual operating space.

[0027] The attitude synchronization module is used to map the real-time spatial poses of the physical ultrasound probe and the physical puncture needle to the corresponding virtual models in the mixed reality scene, so as to realize the real-time attitude synchronization between the physical instruments and the virtual models.

[0028] The clinical ultrasound image library includes a storage module, which stores multiple frames of clinically acquired human kidney ultrasound images. Each frame of ultrasound image is associated with spatial pose information in the form of a rotation matrix and a translation vector corresponding to the ultrasound probe when the image was acquired.

[0029] The retrieval and matching module is used to retrieve matching target ultrasound image frames from the clinical ultrasound image database based on the real-time spatial pose of the virtual ultrasound probe and through a spatial distance measurement method. The spatial distance measurement method is as follows: the difference in the rotation matrix norm between the real-time pose of the probe and the corresponding pose of the image database frame is calculated as the rotational posture difference, and the difference in the translation vector norm is calculated as the translational position difference. The rotational posture difference and the translational position difference are weighted and fused using differential weights to obtain the comprehensive matching distance.

[0030] The real-time display module is used to display the matched target ultrasound image frames in real time in a mixed reality scene, forming an ultrasound cross-section that is updated synchronously with the movement of the virtual ultrasound probe.

[0031] Furthermore, it also includes a multi-threaded processing module, which is communicatively connected to the pose acquisition module, the pose synchronization module, the retrieval and matching module, and the real-time display module, respectively. This module is used to allocate the core tasks of each module to independent threads for execution and to realize real-time data interaction between modules through a thread synchronization mechanism. The pose acquisition module supports synchronous pose recognition and calculation of multiple physical instruments.

[0032] Compared to existing technologies, this invention and its preferred solutions achieve highly robust recognition of physical instrument poses. Even in complex teaching scenarios, it can stably acquire the spatial poses of physical instruments such as puncture needles and ultrasound probes, effectively reducing the impact of changes in viewing angle and slight occlusion on recognition results. It is less prone to tracking loss and pose drift, providing stable and reliable basic data support for virtual-real interaction in practical training. It achieves highly accurate and stable synchronization of virtual and real poses, effectively compensating for initial pose deviations between physical instruments and virtual models, suppressing cumulative errors caused by long-term system operation, avoiding pose jumps and gimbal lock-up problems, and enabling real-time and accurate linkage between physical instruments and virtual models. It reproduces operational feedback that closely matches clinical practice, helping trainees establish standardized operational muscle memory during training. It also achieves clinical-grade high-fidelity ultrasound image reproduction, using clinically acquired human kidney ultrasound images to construct a database, replacing traditional virtual model-based rendering. The simulated images reproduce the imaging characteristics of real clinical ultrasound, avoiding incorrect image perceptions among trainees and achieving deep adaptation between training content and clinical practice scenarios. This simultaneously improves trainees' ultrasound image interpretation and spatial positioning abilities during puncture procedures. It achieves high-precision, low-latency matching of ultrasound sections, taking into account both probe rotation and translational information to complete section matching. Combined with optimized retrieval range based on anatomical regions, it achieves real-time synchronization between ultrasound sections and probe movement, effectively avoiding section jumps and matching lags. This helps trainees quickly establish the corresponding logic between operational adjustments and image changes. Furthermore, it possesses excellent system stability and scenario adaptability. Utilizing a multi-threaded architecture to separate core processing tasks, it supports long-term continuous training and simultaneous operation of multiple instruments, adapting to large-scale, routine, and standardized clinical training scenarios. This effectively reduces the cost of renal biopsy training and improves the efficiency of translating training results into clinical practice. Attached Figure Description

[0033] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0034] Figure 1 This is a schematic diagram illustrating the construction of a virtual kidney model and a virtual instrument model based on CT scans with labeled renal medulla, according to an embodiment of the present invention.

[0035] Figure 2 This is a schematic diagram of a 3D-printed physical teaching device with a dot matrix identification code attached, according to an embodiment of the present invention.

[0036] Figure 3 This is a flowchart illustrating the functional implementation of the MR teaching system according to an embodiment of the present invention.

[0037] Figure 4 This is a schematic diagram of the deployment and operation interface of the renal biopsy MR teaching system according to an embodiment of the present invention;

[0038] Figure 5This is a schematic diagram illustrating a teaching and training scenario for renal biopsy using MR imaging, as described in an embodiment of the present invention.

[0039] Figure 6 This is a block diagram illustrating the overall architecture and working principle of a visual tracking posture synchronization and real-time ultrasound image matching system for MR teaching of renal biopsy, according to an embodiment of the present invention. Detailed Implementation

[0040] To make the features and advantages of the present invention more apparent and understandable, specific embodiments are described below in detail:

[0041] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0042] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0043] This embodiment addresses the engineering pain points of existing renal biopsy operation teaching systems, such as insufficient anatomical fidelity, poor spatial positioning accuracy, distortion of ultrasound image simulation, and high delay in section matching. It provides a method and system for visual tracking posture synchronization and real-time matching of ultrasound images for renal biopsy MR teaching. It can realize stable posture recognition of physical instruments, accurate posture synchronization of virtual and real models, and real-time matching and retrieval of clinical ultrasound images, effectively ensuring the engineering stability and real-time performance of the MR teaching system. It is suitable for the engineering deployment of renal biopsy MR teaching systems with high requirements for real-time performance and stability.

[0044] This solution is built upon mixed reality engineering algorithms, machine vision tracking, and real-time medical image matching technology. It addresses the core issues in existing renal biopsy MR teaching, such as low accuracy of virtual-real linkage, distortion of ultrasound image simulation, and high matching latency. It obtains the real-time spatial pose of physical instruments through dot matrix visual recognition combined with PnP pose calculation principles. A quaternion pose synchronization algorithm enables precise interactive linkage between virtual and real models. Simultaneously, a pre-built clinical ultrasound image library with spatial pose indexes is constructed. Based on spatial distance measurement formulas and X-ray detection technology, real-time matching of probe movement and ultrasound sections is achieved. This multi-dimensional improvement enhances the accuracy of virtual-real linkage and ultrasound image fidelity in the MR teaching system, while reducing system latency.

[0045] Specifically, the implementation of this solution mainly includes four core steps. The first is the pose recognition and calculation of the physical instrument. Dot matrix identification codes are affixed to the surface of the physical puncture instrument and the ultrasound probe. The image information of the identification codes is acquired in real time using a vision camera to complete feature recognition and spatial pose calculation, providing accurate basic data for subsequent virtual-real pose synchronization. The dot matrix identification codes used in this solution employ a high-contrast black and white encoding design, enabling stable recognition of the physical instrument's pose in complex teaching scenarios. This effectively reduces the impact of changes in viewing angle and slight occlusion on the recognition effect. 3D-printed physical teaching instruments with these identification codes can be found in [reference needed]. Figure 2 .

[0046] Secondly, the attitude synchronization between the virtual and real models is addressed. Based on the real-time pose of the physical instrument obtained from the solution, a quaternion attitude synchronization algorithm is used to compensate for the initial offset between the physical instrument and the virtual model, achieving real-time attitude mapping between the physical instrument and the virtual model. Simultaneously, this algorithm can effectively suppress the cumulative error generated by long-term system operation through attitude normalization constraints, avoiding attitude jumps and gimbal lock-up problems, thus ensuring the real-time performance and accuracy of the virtual-physical instrument attitude mapping.

[0047] The third step involves constructing a clinical ultrasound image database. Real clinical kidney ultrasound images are pre-collected, and corresponding probe spatial pose information is added to each frame to create an ultrasound image database with a pose index. Simultaneously, the ultrasound images within the database are rearranged according to spatial pose, establishing an index structure centered on probe pose data. This enables rapid nearest-neighbor retrieval based on probe position, providing data support for subsequent real-time ultrasound cross-section matching.

[0048] Fourthly, the real-time matching and display of ultrasound sections involves using a spatial neighborhood retrieval algorithm based on the real-time pose of the ultrasound probe. Simultaneously, X-ray detection is used to obtain the coordinates of the collision point between the virtual ultrasound probe and the virtual kidney model, further optimizing the retrieval range and matching results. This achieves precise matching and smooth switching of ultrasound sections, enabling real-time synchronous display of the ultrasound sections and probe movement. Figure 4 As shown. Furthermore, the core tasks of attitude synchronization and ultrasound matching in this solution are deployed using a multi-threaded architecture, which effectively ensures uninterrupted continuous training and stable image updates, meeting the real-time requirements of clinical teaching.

[0049] Corresponding to the above methods, this solution also provides a complete visual tracking posture synchronization and real-time ultrasound image matching system for MR teaching of renal biopsy. The core functional module architecture of this system can be found in [link to relevant documentation]. Figure 6Specifically, it includes a dot matrix code recognition module, a visual pose calculation module, a quaternion pose synchronization module, an ultrasound image library construction module, a spatial neighborhood retrieval module, and a real-time matching display module. The dot matrix code recognition module acquires images of the identification codes on physical instruments; the visual pose calculation module calculates the spatial pose of the physical instruments based on the acquired images; the quaternion pose synchronization module achieves accurate pose mapping between virtual and physical instruments; the ultrasound image library construction module creates a clinical ultrasound database with pose indexes; the spatial neighborhood retrieval module matches the real-time pose of the probe with the corresponding ultrasound sections; and the real-time matching display module enables dynamic and synchronized presentation of ultrasound images. These modules work together to complete the core functions of the entire process for teaching renal biopsy using MR technology.

[0050] The following sections will detail the complete technical implementation process of this solution, broken down into modules:

[0051] 1) Based on dot matrix code visual recognition and pose calculation

[0052] This module is the basic data input link of the entire system. Its core function is to acquire the real-time spatial pose of the physical puncture instrument and the ultrasound probe, so as to provide accurate pose data support for subsequent virtual and real pose synchronization and ultrasound section matching.

[0053] The specific implementation process is as follows: High-contrast black-and-white coded dot matrix identifiers are pasted onto the surface of the physical puncture needle and ultrasound probe, such as... Figure 2 As shown, the system acquires image information of the identification code in real time using a pre-calibrated vision camera. Based on the PnP (Perspective-n-Point) pose calculation principle, the system accurately calculates the spatial pose of the physical device by using the correspondence between the spatial coordinates of feature points and pixel coordinates. The core mathematical relationship is as follows:

[0054]

[0055] in, This is a scale factor used to unify the scale differences between the pixel coordinate system and the world coordinate system; R is the extrinsic parameter matrix of the physical instrument relative to the camera, where R is a 3×3 rotation matrix used to describe the spatial rotational attitude of the instrument, and t is a 3×1 translation vector used to describe the spatial position of the instrument; (X, Y, Z) are the 3D world coordinates of the feature points on the dot matrix identification code, which are pre-calibrated and obtained after the identification code is pasted; (u, v) are the 2D pixel coordinates of the above feature points in the image acquired by the camera, which are extracted from the real-time acquired image by the feature point detection algorithm.

[0056] This is the intrinsic parameter matrix of the visual camera, which is obtained in advance through camera calibration. The complete matrix form is as follows:

[0057] K=

[0058] Where fx and fy are the focal lengths of the camera in the x-axis and y-axis directions, respectively, and cx and cy are the coordinates of the principal point of the camera's imaging plane.

[0059] By solving the above formula, the initial pose data of the physical instrument can be obtained, providing accurate basic data for subsequent virtual-real pose synchronization.

[0060] 2) Quaternion attitude synchronization algorithm achieves precise mapping between virtual and real worlds.

[0061] This module receives the output data from the aforementioned pose calculation module. Its core function is to achieve precise pose synchronization between the physical instrument and the virtual model, solving the problems of low accuracy and easy pose jumps in existing systems. The virtual kidney anatomy model and virtual instrument model used in this solution can be found in [reference needed]. Figure 1 .

[0062] The specific implementation process is as follows: a quaternion attitude synchronization algorithm is adopted to compensate for the initial offset between the physical instrument and the virtual model. At the same time, the cumulative error is suppressed by normalization constraints to avoid the gimbal deadlock problem existing in the traditional Euler angle attitude representation and ensure the stability of attitude mapping.

[0063] First, the rotational attitude of the physical instrument is represented using a unit quaternion, which has the following form: Unit constraints must be met. ,in Let be the real part of the quaternion. This represents the imaginary part of the quaternion. The rotation matrix corresponding to this unit quaternion. The conversion formula is as follows:

[0064]

[0065] Based on this, attitude synchronization is achieved through the following two steps:

[0066] Initial offset compensation: Initial pose offset compensation between the physical instrument and the virtual model is achieved through quaternion multiplication. Let the initial offset quaternion be... (That is, the pose difference between the physical device and the virtual model in the initial state), the quaternion of the original pose of the physical device obtained by pose calculation is: The compensated synchronous attitude quaternion is:

[0067]

[0068] in, This is the quaternion multiplication operator. It is the conjugate of the initial offset quaternion, used to counteract the initial attitude deviation.

[0069] Attitude normalization constraint: The compensated synchronized attitude quaternions are normalized using the following formula:

[0070] q norm =q sync /

[0071] in, This represents the modulus of the quaternion. This normalization process effectively suppresses the cumulative numerical errors generated during long-term system operation, ensuring the long-term stability of the real-time attitude mapping between the virtual model and the physical instrument, and preventing attitude jumps.

[0072] Finally, the normalized attitude quaternion is converted into a rotation matrix, which can drive the virtual puncture needle, virtual ultrasound probe and physical instrument to achieve real-time synchronous movement and complete the precise mapping of virtual and real space.

[0073] 3) Construction of a clinical ultrasound image library

[0074] This module provides data support for subsequent real-time ultrasound cross-sectional matching. Its core function is to address the problem of significant differences and insufficient fidelity between simulated ultrasound images and real clinical images in existing systems. The specific implementation process is as follows:

[0075] Pre-acquire real clinical human kidney ultrasound images and attach the corresponding ultrasound probe spatial pose information at the time of acquisition to each ultrasound image. , where i is the index of the image frame. This is the rotation matrix of the probe during the acquisition of this frame of image. The translation vector of the probe during the acquisition of this frame of image is used to construct an ultrasound image database with pose index.

[0076] Meanwhile, all ultrasound images in the database are rearranged according to the corresponding probe spatial pose to establish a fast index structure with pose data as the core, which greatly improves the efficiency of subsequent real-time retrieval and provides a data foundation for low-latency matching of ultrasound sections.

[0077] 4) Spatial neighborhood retrieval and real-time matching

[0078] This module receives the real-time probe pose data from the aforementioned pose calculation module, as well as the pre-built data from the ultrasound image library. It is the core component of the system for achieving clinical-grade ultrasound-guided reconstruction.

[0079] The specific implementation process is as follows: based on the real-time pose of the ultrasonic probe. ,in For the real-time rotation matrix of the probe, The probe's real-time translation vector is used to quickly match the optimal ultrasonic section using a spatial neighborhood retrieval algorithm. The process is as follows:

[0080] First, a spatial distance metric formula is defined between the real-time pose of the probe and the corresponding pose of the i-th frame in the image library, which is used to quantify the degree of pose matching:

[0081]

[0082] in, is the Frobenius norm of the matrix, used to quantify the pose difference between two rotation matrices; It is the 2-norm of a vector, used to quantify the positional difference between two translation vectors; The weighting coefficients for the differences in rotational posture and translational position are respectively, and can be preset in advance according to the actual needs of clinical training; This is the combined distance between the real-time pose of the probe and the pose of the i-th frame image. The smaller the value, the higher the pose matching degree.

[0083] Simultaneously, the coordinates of the collision point between the virtual ultrasound probe and the virtual kidney model are obtained through a ray detection algorithm. The search range is further optimized based on the collision point location, improving the anatomical accuracy of the cross-sectional matching. The core formula for ray detection is as follows:

[0084]

[0085] in, O represents the coordinates of the collision point between the probe ray and the virtual kidney model; O represents the center coordinates of the virtual ultrasound probe. is the unit vector pointing towards the probe; k is the projection distance of the ray, which can be preset in advance according to the size of the virtual anatomical model.

[0086] Ultimately, the image database was searched to find images that met the requirements. ( All image frames (with a preset distance threshold used to limit the search range) are selected. The smallest frame is used as the optimal matching result. The system retrieves and renders the ultrasound image of that frame in real time, and combines the collision point coordinates to achieve smooth switching of the ultrasound section, ultimately achieving low-latency and accurate matching display of the ultrasound section and probe movement.

[0087] 5) Multi-threaded engineering deployment

[0088] This module provides engineering support for the stable operation of all the aforementioned core modules. Its core function is to ensure the stability and real-time performance of the system during long-term continuous operation, adapting to the large-scale and routine training needs of clinical teaching. For a complete deployment and implementation process of the MR teaching system, please refer to [link to relevant documentation]. Figure 3 .

[0089] The specific implementation process is as follows: A multi-threaded architecture is adopted, separating core tasks such as visual tracking, pose calculation, posture synchronization, ultrasound matching, and image rendering into independent threads. Different physical instruments use their own unique dot matrix identification codes for registration. The system can simultaneously and continuously identify and track multiple instruments, meeting the needs of teaching scenarios such as two-person collaborative operation. Simultaneously, a thread synchronization mechanism ensures real-time data interaction between modules, avoiding screen stuttering and response delays caused by excessive single-thread load, ensuring uninterrupted continuous training and stable screen updates, meeting the real-time requirements of clinical teaching.

[0090] Compared with the prior art, the advantages of the above solutions provided by the present invention include:

[0091] 1) Robust pose recognition: It adopts high-contrast black and white coded dot matrix identification code and combines the PnP pose calculation principle. It can still stably recognize the pose of physical instruments in complex backgrounds, changing viewpoints and slightly occluded scenes, providing an accurate data foundation for pose synchronization.

[0092] 2) Precise and stable attitude synchronization: Based on the quaternion attitude representation and transformation algorithm, attitude normalization constraints are realized, which effectively suppresses cumulative errors, avoids attitude jumps and gimbal deadlock, and ensures the real-time performance and accuracy of virtual and real instrument attitude mapping.

[0093] 3) Realistic ultrasound reproduction: Using clinical ultrasound images, a spatial neighborhood retrieval algorithm is used to achieve precise matching between probe movement and ultrasound cross-section, replacing simulated images and making it highly adaptable to clinical use.

[0094] 4) Matching real-time and low latency: The retrieval logic is optimized by using a spatial distance metric formula, and the core tasks are separated by a multi-threaded architecture. The ultrasound section response is fast, and the operation and image are synchronized without lag, meeting the needs of real-time teaching.

[0095] 5) Stable project deployment: The multi-threaded architecture supports long-term continuous training, adapts to large-scale applications, reduces teaching costs, and improves training effectiveness.

[0096] The following is a more specific application example to further demonstrate and introduce the above solution of the present invention.

[0097] This example is applied to the MR teaching and training scenario of renal biopsy in a standardized clinical training system. It fully covers the entire process from system deployment, pre-training calibration and initialization, real-time interactive guidance during training, to data acquisition and output. It completely reproduces the operational logic and scenario requirements of real clinical renal biopsy. The specific implementation process is as follows:

[0098] First, the basic system environment and virtual anatomical model were pre-built to provide the anatomical foundation for the entire training scenario. Real clinical abdominal kidney CT images were acquired and imported into the 3DSlicer platform. Three-dimensional layered reconstruction and structural boundary delineation of the kidney were completed. Clinicians meticulously annotated and verified key anatomical structures related to the puncture procedure, such as the renal cortex and medulla, ensuring complete consistency in morphology, size, and relative position between the virtual model and the real human anatomical structures. The annotated 3D anatomical model was then imported into the Unity development platform for rendering optimization, constructing a virtual anatomical scene that perfectly matches the physical teaching instruments and actual operating space. Independent collision bodies and hierarchical markers were configured for different anatomical regions of the kidney, providing underlying support for subsequent X-ray collision detection and precise ultrasound section matching. The virtual kidney anatomical model and virtual instrument model constructed in this example can be found in [reference needed]. Figure 1 .

[0099] After completing the basic scene setup, system calibration and virtual-real space alignment are carried out before practical training to ensure the accuracy of virtual-real interaction during subsequent training. The first step is to calibrate the visual camera of the MR teaching system, obtain the camera intrinsic parameter matrix K, and determine the camera's focal lengths fx and fy in the x and y axes, as well as the principal point coordinates cx and cy of the imaging plane. This completes the calibration of the camera imaging parameters and eliminates the influence of lens distortion on subsequent pose calculations. The second step is to affix high-contrast black-and-white coded dot matrix identifiers to designated positions on the physical puncture needle and ultrasound probe. The 3D world coordinates (X, Y, Z) of each feature point on the identifier are pre-recorded. The completed 3D-printed physical teaching equipment can be found in [reference needed]. Figure 2 The third step involves activating the vision camera to acquire images of the identification code in real time. A feature point detection algorithm is used to extract the 2D pixel coordinates (u, v) of the identification code's feature points in the image. These coordinates are then substituted into the pinhole camera model and the PnP pose calculation formula to calculate the initial spatial pose (R) of the physical puncture needle and ultrasonic probe. real , t real The fourth step involves collecting the initial offset data between the physical instrument and the virtual model, and constructing the initial offset quaternion q. off The initial offset compensation is achieved through quaternion multiplication to obtain the synchronous attitude quaternion. The compensated quaternion is then normalized to suppress cumulative errors that may occur during long-term system operation. The fifth step converts the normalized attitude quaternion into a corresponding rotation matrix, driving the virtual puncture needle and virtual ultrasound probe in the virtual scene to achieve real-time synchronous movement with the physical instruments, completing precise mapping and alignment between virtual and real spaces. For a complete MR teaching system deployment example, please refer to [link to example]. Figure 4 , Figure 5 .

[0100] Once the system calibration and alignment are complete, renal biopsy training can begin. During the training, the system simultaneously performs real-time interaction and full-process data acquisition. The trainee holds the calibrated physical ultrasound probe and puncture needle, and conducts simulated puncture training according to clinical operating procedures. The system continuously acquires identification code images through a vision camera, calculates instrument pose in real time, and synchronizes virtual and real postures, allowing the trainee to visually see the real-time position and posture of the virtual instrument corresponding to the physical instrument in the renal anatomical model through the MR device. Simultaneously, the system generates a unit vector n along the probe's orientation, using the center coordinate O of the virtual ultrasound probe as the origin, and performs ray collision detection using a formula... Accurately calculate the coordinates of the collision point between the probe rays and the virtual kidney model. Based on the hierarchical markers corresponding to the collision point coordinates, the system identifies the renal anatomical region corresponding to the current probe in real time, and simultaneously records all operational data during the trainee's operation, including puncture operation time, needle insertion angle, puncture depth, probe movement trajectory, and all changes in instrument position. The system automatically integrates the operational data with the corresponding position data and timestamps to form a raw operational dataset with time series, providing complete data support for subsequent training effect evaluation and operation error correction.

[0101] Throughout the entire process of the trainee operating the physical ultrasound probe, the system synchronously performs real-time ultrasound image matching and dynamic display functions, replicating the operating experience of a real clinical ultrasound-guided procedure. The system pre-acquires batches of real human kidney ultrasound images, adding probe spatial pose information (R0) corresponding to the acquisition time to each frame of ultrasound image. i ,t i The system constructs an ultrasound image database with a pose index and rearranges the images according to pose information, establishing a fast retrieval index centered on pose data. During the training, the system acquires the current pose (R) of the ultrasound probe in real time. curr ,t curr Substitute the values ​​into the spatial distance metric formula to calculate the comprehensive distance di between the current pose and the corresponding poses of all image frames in the image database; preset the distance threshold according to clinical training needs. Search the image database for images that satisfy d i < Select d from all image frames. i The smallest image frame is used as the optimal matching result. The system retrieves and renders the optimal matching clinical ultrasound image in the MR scene in real time. It combines the collision point coordinates obtained from X-ray detection to optimize the image display position and switching logic, so as to achieve a smooth transition of ultrasound sections. This ensures the clinical consistency of the three elements: "physical probe action - ultrasound section display - virtual anatomical position". It helps trainees establish the corresponding logic between operation actions and changes in ultrasound images during training, and simultaneously improves their ability to interpret ultrasound images and their spatial positioning ability for puncture operations.

[0102] This example employs a multi-threaded architecture throughout, assigning core tasks such as visual tracking, pose calculation, posture synchronization, ultrasound retrieval and matching, image rendering, and data acquisition to independent threads. The thread synchronization mechanism ensures real-time data interaction between modules, avoiding screen stuttering and response delays caused by excessive single-thread load. This ensures uninterrupted and stable screen updates during long-term continuous training, fully realizing the entire clinical-grade MR teaching and training process for renal biopsy.

[0103] It should be noted that, unless otherwise defined, the technical or scientific terms used in this invention should have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0104] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

[0105] This invention is not limited to the preferred embodiment described above. Anyone inspired by this invention can derive various other forms of visual tracking posture synchronization and real-time ultrasound image matching methods and systems for MR teaching of renal biopsy. All equivalent variations and modifications made within the scope of the claims of this invention shall fall within the scope of this invention.

Claims

1. A method for visual tracking posture synchronization and real-time matching of ultrasound images for teaching renal biopsy using MR imaging, characterized in that, include: Acquire the real-time spatial pose of the physical ultrasound probe and the physical puncture needle in the actual operating space; The real-time spatial poses of the physical ultrasound probe and the physical puncture needle are mapped to the corresponding virtual models in the mixed reality scene to achieve real-time posture synchronization between the physical instruments and the virtual models. Based on the real-time spatial pose of the virtual ultrasound probe, matching target ultrasound image frames are retrieved from a pre-constructed clinical ultrasound image library using a spatial distance measurement method. The clinical ultrasound image library stores multiple frames of clinically acquired human kidney ultrasound images. Each frame of ultrasound image is associated with spatial pose information in the form of rotation matrix and translation vector of the ultrasound probe when the image was acquired. The spatial distance measurement method is as follows: calculate the difference in the rotation matrix norm between the real-time pose of the probe and the corresponding pose of the image library frame as the rotation pose difference, and the difference in the translation vector norm as the translation position difference. The rotation pose difference and the translation position difference are weighted and fused using differentiated weights to obtain the comprehensive matching distance. In the mixed reality scenario, the matched target ultrasound image frames are displayed in real time, forming an ultrasound cross-section that is updated synchronously with the movement of the virtual ultrasound probe.

2. The method for visual tracking posture synchronization and real-time matching of ultrasound images for MR teaching of renal biopsy according to claim 1, characterized in that: The method for obtaining the real-time spatial pose of the physical ultrasound probe and the physical puncture needle is as follows: using a pre-calibrated vision camera, image information of high-contrast black and white coded dot matrix identifiers pasted on the surface of the physical ultrasound probe and the physical puncture needle is acquired, and the real-time spatial pose is calculated based on the PnP pose calculation principle; different physical instruments are equipped with exclusive dot matrix identifiers, and the vision camera can simultaneously acquire image information of multiple sets of exclusive dot matrix identifiers and simultaneously calculate the spatial pose of the corresponding instruments.

3. The method for visual tracking posture synchronization and real-time matching of ultrasound images for MR teaching of renal biopsy according to claim 1, characterized in that: The step of achieving real-time attitude synchronization between the physical instrument and the virtual model is implemented using a quaternion attitude synchronization algorithm. Specifically, the initial pose offset between the physical instrument and the corresponding virtual model is compensated by quaternion conjugation and quaternion multiplication operations to obtain a synchronized attitude quaternion. The synchronized attitude quaternion is then normalized to suppress the cumulative numerical error during system operation.

4. The method for visual tracking posture synchronization and real-time matching of ultrasound images for MR teaching of renal biopsy according to claim 1, characterized in that: In the pre-constructed clinical ultrasound image library, all ultrasound images are rearranged according to the corresponding probe spatial pose at the time of acquisition, establishing a fast index structure with probe spatial pose data as the core.

5. The method for visual tracking posture synchronization and real-time matching of ultrasound images for MR teaching of renal biopsy according to claim 1, characterized in that: The difference in the rotation matrix norm is the Frobenius norm of the difference between the two rotation matrices, and the difference in the translation vector norm is the 2-norm of the difference between the two translation vectors. The specific steps for retrieving the target ultrasound image frame are as follows: a preset comprehensive matching distance threshold is set, all image frames in the clinical ultrasound image database with a comprehensive matching distance less than the threshold are retrieved, and the image frame with the smallest comprehensive matching distance is selected as the target ultrasound image frame.

6. The method for visual tracking posture synchronization and real-time matching of ultrasound images for MR teaching of renal biopsy according to claim 1, characterized in that: When searching for matching target ultrasound image frames, the collision point coordinates between the virtual ultrasound probe and the virtual kidney anatomy model in the mixed reality scene are obtained through X-ray detection technology. The search range is narrowed by combining the hierarchical markers of the kidney anatomy region corresponding to the collision point coordinates, thereby optimizing the matching results.

7. The method for visual tracking posture synchronization and real-time matching of ultrasound images for MR teaching of renal biopsy according to claim 1, characterized in that: When displaying the target ultrasound image frame in real time, the ultrasound section can be smoothly switched, and the corresponding position of the virtual puncture needle can be displayed synchronously in the ultrasound section.

8. The method for visual tracking posture synchronization and real-time matching of ultrasound images for MR teaching of renal biopsy according to claim 1, characterized in that: The core tasks of pose acquisition, pose synchronization, image retrieval and matching, and image rendering and display are assigned to independent threads using a multi-threaded architecture, and the real-time data interaction of each stage is ensured through a thread synchronization mechanism.

9. A real-time visual tracking and ultrasound image matching system for MR teaching of renal biopsy, characterized in that, include: The pose acquisition module is used to acquire the real-time spatial pose of the physical ultrasound probe and the physical puncture needle in the actual operating space. The attitude synchronization module is used to map the real-time spatial poses of the physical ultrasound probe and the physical puncture needle to the corresponding virtual models in the mixed reality scene, so as to realize the real-time attitude synchronization between the physical instruments and the virtual models. The clinical ultrasound image library includes a storage module, which stores multiple frames of clinically acquired human kidney ultrasound images. Each frame of ultrasound image is associated with spatial pose information in the form of a rotation matrix and a translation vector corresponding to the ultrasound probe when the image was acquired. The retrieval and matching module is used to retrieve matching target ultrasound image frames from the clinical ultrasound image database based on the real-time spatial pose of the virtual ultrasound probe and through a spatial distance measurement method. The spatial distance measurement method is as follows: the difference in the rotation matrix norm between the real-time pose of the probe and the corresponding pose of the image database frame is calculated as the rotational posture difference, and the difference in the translation vector norm is calculated as the translational position difference. The rotational posture difference and the translational position difference are weighted and fused using differential weights to obtain the comprehensive matching distance. The real-time display module is used to display the matched target ultrasound image frames in real time in a mixed reality scene, forming an ultrasound cross-section that is updated synchronously with the movement of the virtual ultrasound probe.

10. A real-time visual tracking and ultrasound image matching system for MR teaching of renal biopsy according to claim 9, characterized in that, It also includes a multi-threaded processing module, which is communicatively connected to the pose acquisition module, the pose synchronization module, the retrieval and matching module, and the real-time display module. This module is used to allocate the core tasks of each module to independent threads for execution and to realize real-time data interaction between modules through a thread synchronization mechanism. The pose acquisition module supports synchronous pose recognition and calculation for multiple physical instruments.

Images