AI-based guidance system for operating intracardiac ultrasound examinations that continuously maintains visibility of the treatment device's tip.

Abstract

The present invention provides a system and method for tracking the tip of a treatment device during intracardiac ultrasound (ICE) procedures. [Solution] A system and method for estimating the incidence angle and trajectory of a treatment device in intracardiac ultrasound (ICE) images in real time, and providing guidance for ICE operation while maintaining visibility of the tip of the treatment device. An ICE imaging sequence is acquired. Spatial and temporal features are extracted from the ICE image sequence using a pre-trained ultrasound-based model. A transformer-based time-series deep learning model is used to integrate the extracted spatial information and incorporate information on previous trajectories and angles to track the catheter tip within the ICE imaging plane. The tip-tracking transformer-based time-series deep learning model improves the estimation of existing incidence angle and trajectory models during ICE procedures, thereby improving the accuracy and consistency of predictions.

Description

[Technical Field]

[0001] (Description of research and development funded by the federal government) This invention was made with government support under the grant number 1R01EB028278-01A1 from the NIH (National Institutes of Health) / NIBIB (National Institute of Medical Imaging and Biotechnology). The U.S. Government has certain rights to this invention.

[0002] (Cross-reference to related applications) This application claims the benefit of U.S. Provisional Patent Application No. 63 / 753, 524, filed on 4 February 2025 under 35 119(e) of the U.S. Patent Act.

[0003] This disclosure pertains to medical images. [Background technology]

[0004] Intracardiac echocardiography (ICE) is an important imaging modality in electrophysiological procedures (EP) and structural heart disease (SHD) interventions, enabling real-time, high-resolution visualization (or rendering) of structures within the cardiac chambers or heart. However, direct visualization has been limited during ICE catheter manipulation. In addition, frequent adjustments have presented several challenges. Consistent and accurate tracking of the device has been difficult. Nevertheless, in EP procedures, accurate tracking of the catheter tip is crucial for precise tissue preparation. Meanwhile, in SHD interventions, ICE assists in positioning structural treatment devices such as mitral valve clips (or MitraClips), occluders or occludas, and valve implants. [Overview of the Initiative] [Means for solving the problem]

[0005] The following describes, based on several preferred embodiments, an AI-based guidance system for the operation of intracardiac ultrasound examinations to continuously maintain the visibility of the tip of a treatment device. These embodiments include methods, systems, instructions, and / or computer-readable media.

[0006] In a first embodiment, a system is provided for tracking the tip of a treatment device during an intracardiac ultrasound (ICE) procedure. The system includes an ICE catheter configured to acquire an ICE imaging sequence of the patient during the ICE procedure, and a control device including at least a processor and memory. The control device described above is configured to track the tip of the treatment device in real time during the ICE imaging sequence using a tip-tracking model. The above-described tip tracking model includes a pre-trained ultrasound foundation model configured to extract features from multiple images of the ICE imaging sequence, and a transformer-based model configured to estimate the incidence angle and transit point of the tip of the treatment device in the ICE imaging sequence.

[0007] In a second aspect, a method is provided for constructing a transformer-based model for tracking the tip of a therapeutic device. This method provides a pre-trained ultrasound-based model; obtains a hybrid training dataset of multiple intracardiac ultrasound (ICE) imaging sequences, including clinical and synthetic sequence data; and trains the transformer-based model from multiple iterations. In each of the above iterations, the pre-trained ultrasound-based model extracts features from one of the multiple ICE imaging sequences; these features are input into the transformer-based model, which is configured using one or more encoders that model the temporal dependence by self-attention; the transformer-based model outputs the position and angle of the next frame; the position and angle of the next frame are compared with the position and angle of the ground truth using mean squared error (MSE) loss; and the attention weights and embeddings of the transformer-based model are updated to improve temporal consistency and spatial accuracy.

[0008] A third aspect provides a method for tracking the tip of a treatment device. This method involves acquiring an ICE imaging sequence during an ICE procedure using an intracardiac ultrasound (ICE) catheter; a control device estimating the position and orientation of the tip of the treatment device in real time during the ICE procedure, with the position and orientation being estimated using an ultrasound-based model and a transformer-based deep learning model; and a robotic catheter system controlling the ICE catheter during the ICE procedure based on at least a portion of the estimated position and orientation of the tip of the treatment device.

[0009] One or more of the above aspects may be used alone or in combination. These and other aspects, features and advantages will become apparent upon reading the preferred embodiments described below with reference to the accompanying drawings. The present invention is defined by the appended claims. The description herein should not be construed as limiting the individual claims. Further aspects and advantages of the present invention will be described in conjunction with the preferred embodiments below, and these may be set forth in the claims hereinafter, either alone or in combination.

[0010] The individual components and drawings are not necessarily drawn to scale; instead, emphasis is placed on illustrating the mechanism of each embodiment. Furthermore, in these drawings, like reference numerals indicate corresponding parts throughout the various drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] [Figure 1] Figure 1 illustrates a system for tracking the tip of a therapeutic device during an intracardiac echocardiography (ICE) procedure, according to an embodiment. [Figure 2] Figure 2 illustrates the architecture of a model for tracking the tip of a therapeutic device during an intracardiac echocardiography (ICE) procedure, according to an embodiment. [Figure 3] Figure 3 illustrates a method of training a model for tracking the tip of a therapeutic device during an intracardiac echocardiography (ICE) procedure, according to an embodiment. [Figure 4] Figure 4 illustrates the entry angle, rotation angle, and tip passage position. [Figure 5] Figure 5 illustrates a method of applying a model for tracking the tip of a therapeutic device during an intracardiac echocardiography (ICE) procedure, according to an embodiment. [Figure 6] Figure 6 illustrates the predicted angle and position of a therapeutic device, according to an embodiment. [Figure 7] Figure 7 illustrates a neural network, according to an embodiment. MODE FOR CARRYING OUT THE INVENTION

[0012] In each embodiment described in the present specification, guidance for ICE operation is provided so as to maintain continuous visibility of the tip of a treatment device by providing a model for estimating the incident angle and trajectory of the treatment device in real time in an ICE image. In this process, an ICE imaging sequence is acquired. A pre-trained ultrasound-based model is used to extract spatial and temporal features from the sequence of ICE images. A transformer-based time-series deep learning model is used to integrate the extracted spatial and temporal features, and the tip of the catheter is tracked in the ICE imaging plane by incorporating past or learned historical waypoints and angle information. The transformer-based time-series deep learning model improves the estimation of the incident angle and trajectory models during ICE procedures, and enhances the accuracy and consistency of prediction.

[0013] The proposed transformer-based time-series deep learning model differs from existing solutions in several aspects and is improved thereby. Unlike conventional manual adjustment and hardware-dependent EM tracking methods, the method and system described herein uses an innovative artificial intelligence model to predict the tip position and incident angle of a catheter in real time. Unlike conventional deep learning approaches, the system uses a transformer network to continuously perform ICE frame analysis, so as to improve accuracy and stability. This model is trained using a combination of actual clinical sequences and synthetic datasets, and corresponds to anatomical diversity to ensure robustness. This model provides automatic guidance for robotic ICE catheter adjustment, so as to maintain visibility of the device tip throughout the entire procedure. These advantages allow ICE-guided interventions to improve procedural accuracy, reduce the burden on the operator, and improve clinical outcomes.

[0014] Intracardiac echocardiography (ICE) is an important imaging technique used in electrophysiological (EP) procedures and interventions for structural heart disease (SHD). ICE is an imaging technique that provides real-time, high-resolution images of cardiac structures using an ultrasound catheter inserted into the heart. ICE can be performed under sedation without the need for tracheal intubation. Unlike previous transthoracic or transesophageal echocardiography (or cardiac ultrasound), which relied on external or semi-invasive techniques, ICE uses a catheter-based probe that is directly inserted into the heart or nearby blood vessels. This accessibility to the heart allows for real-time, high-resolution images of intracardiac structures, cardiac physiology, and therapeutic devices used in various procedures. The ICE catheter is a thin, flexible tube equipped with an ultrasound transducer at its tip, enabling the emission and reception of sound waves, allowing for real-time visualization of intracardiac structures and therapeutic devices. Steering control may allow a physician or robotic system to precisely guide and position the catheter within the heart.

[0015] ICE imaging can be used in a variety of procedures, including, for example, guiding catheters and / or devices for resection (or ablation) and mapping of arrhythmias. These devices include, for example, those for closing atrial septal defects (ASD) / patent foramen ovale (PFO), closing atrial appendage (AA) / left atrial appendage (LAA), and transcatheter aortic valve replacement (TAVR). While ICE resection is given as an example below, the model described herein can be applied to any procedure that uses tip position and orientation for control, without being limited to this example. ICE resection is a technique that uses ICE ultrasound imaging during cardiac resection to visualize and guide catheter placement, enabling accurate tissue contact, identification of anatomical structures, and early detection of complications such as pericardial effusion. Real-time imaging is useful for identifying abnormal cardiac tissue (matrix), improving the accuracy of scar tissue creation, blocking false signals, and potentially reducing fluoroscopy time, procedure time, and arrhythmia recurrence. In ICE resection, the left atrium is the most common anatomical structure of interest. Anatomical structures include the left atrial appendage (LAA), left inferior pulmonary vein (LIPV), left superior pulmonary vein (LSPV), right inferior pulmonary vein (RIPV), and right superior pulmonary vein (RSPV). Additional structures, different structures, or fewer structures or parts of interest may be included. Any structure can be imaged, for example, the roots or bases of veins. Pulmonary veins, particularly their junctions with the left atrium, around the pulmonary vein openings, or around other cardiac chambers may be anatomical sites of interest in other embodiments. Other imaging sites may be other anatomical structures of interest (e.g., arteries or veins).

[0016] Figure 1 illustrates an ICE system 100 used in electrophysiological procedures such as ICE resection. This system 100 includes an ICE catheter 140, a treatment device (or therapeutic apparatus) 160, a control unit (or control unit) 150, and optionally, a robotic catheter control system (or catheter system) 170. The ICE catheter 140 and / or treatment device 160 are controllable using the robotic catheter control system 170. Alternatively, the ICE catheter 140 and / or treatment device 160 may be controlled by a clinician during the procedure. The ICE catheter 140 is configured to acquire image data of the patient's tissues and / or organs. This image data may include the treatment device 160. The treatment device 160 (e.g., a resection catheter) is inserted into the patient 180 and configured to perform one or more therapeutic functions. The control unit 150 can process the image data in real time, in particular with respect to the angle of incidence and trajectory of one or more treatment devices 160, in order to estimate the position and orientation of the ICE catheter 140 or the treatment device 160. In this context, the improved models described herein are used, which incorporate both an ultrasound-based model and a time-series transformer-based deep learning model.

[0017] In one embodiment, the ICE catheter 140 included in the ICE system 100 is configured to image intracardiac and pericardial structures in high resolution and in real time. In one embodiment, the ICE catheter 140 integrates a phased array (or phased array) type ultrasound transducer capable of generating 2D, 3D, and / or 4D volumetric images. The 4D (3D + time) capability enables real-time visualization of dynamic cardiac structures (such as heart valves, septa or diaphragm, blood flow patterns, etc.) and real-time visualization of the treatment device 160, providing important spatial and temporal information for tracking and / or monitoring the treatment device 160. One of the main advantages of ICE is its ability to provide real-time guidance during minimally invasive cardiac procedures. For example, in catheter-based interventions / procedures, ICE facilitates precise catheter positioning and visualization of the target to be resected. Similarly, ICE helps to contour anatomical landmarks, aiding in confirmation of device placement. ICE provides direct visualization of the inside of the heart, potentially eliminating the need for external imaging modalities such as fluoroscopy and reducing radiation exposure to patients and medical teams.

[0018] Therefore, one or more treatment devices 160 may include any medical devices inserted into the patient, such as mitral valve clips, occluders or occludas, valve implants, etc. Mitral valve clips can bring the mitral valve leaflets closer together to reduce regurgitation (or valve insufficiency) without open-heart surgery. Occluders can close openings in atrial septal defects (ASDs), patent foramen ovale (PFOs), or left atrial appendages (LAAs) to prevent replacement or embolization. Valve implants can replace or reinforce bioprosthetic valves or bioprostheses (or bioprosthetic devices). Treatment devices 160 may include structural cardiac treatment devices used in transcatheter procedures, such as atrial septal defect (ASD) closure, patent foramen ovale (PFO) closure, left atrial appendage (LAA) closure, and transcatheter valve repair or replacement. Furthermore, the treatment device 160 may also be an intravascular intervention device, for example, a device used for thrombectomy, intravascular stent placement, or inferior vena cava filter placement. Alternatively, the treatment device 160 may be a catheter for drug or energy transfer, for example, for the transfer of local drugs, gene therapy vectors, or ultrasound-mediated energy for tissue modification. Other treatment devices 160 may also be used.

[0019] In one embodiment, a robotic catheter guidance system (or catheter system) 170 is used to control an ICE catheter 140 and / or a treatment device 160 (e.g., an excision catheter). For example, the robotic catheter guidance system 170 includes a catheter, a base, a catheter handle housing, an access point base, an access point guide, and an arm. In one embodiment, the robotic catheter guidance system 170 controls the ICE catheter 140 to acquire images of the treatment device 160. In another embodiment, for example, the robotic catheter guidance system 170 also controls the treatment device 160 in the case of an excision catheter. The robotic catheter guidance system 170 can also control other catheters. The base and catheter handle housing form a handle robot. The access point base and access point guide form an access point robot. The arm connects or links the handle robot to the access point robot. A cable is connected to an ultrasound device, for example, image processing, beamforming, and display of generated images. This robotic catheter guidance system 170 is just one example. Other configurations of the robotic catheter guidance system 170 are also available. In one embodiment, the robot controls the guidance (or navigation) and trajectory planning of the ICE catheter 140. As the ICE catheter 140 moves, closed-loop features of the robot control algorithm manage the robot's progress and continuously monitor the system for errors and deviations from the target.

[0020] In one embodiment, as the ICE catheter 140 moves within the patient 180, image data is acquired and processed, and the incidence angle and trajectory of the treatment device 160 are estimated in real time within the ICE image. Therefore, the ICE catheter 140 can continuously visualize the treatment device 160 throughout the procedure. Previous systems sometimes used electromagnetic (EM) sensor-based tracking, which incorporated EM sensors to track the position of the catheter and other devices. However, these systems required additional hardware and were susceptible to magnetic field interference. Therefore, robot-assisted control of the ICE catheter 140 has been considered to improve accuracy and reduce the workload on technicians. However, for effective robot assistance, it was crucial to accurately estimate the direction and angle of devices appearing in the ICE image in order to automatically guide the movement of the ICE catheter 140. Previous methods lacked robust, real-time guidance for tracking the tip of the device. Several reasons contributed to the difficulty in tracking the tip of the catheter. For example, one need is to keep the tip of the treatment device continuously visible within the imaging area (or image area) of ICE. Due to frequent adjustments and limited direct visualization, it was difficult to consistently track the tip of the device. For example, due to the dynamic nature of the procedure (or surgery), tracking of the catheter tip was sometimes lost during EP procedures. In SHD interventions, devices such as mitral valve clips, occlusion devices, and valve implants require careful positioning, but this positioning became complicated when these devices deviated from the ICE imaging plane.

[0021] In each embodiment described herein, an innovative AI-driven model is provided to ensure continuous visualization and a reduction in the technician's workload for estimating the incidence angle and trajectory of the treatment device 160 in ICE images. This innovative AI-driven model improves upon previous AI attempts by incorporating a transformer-based model 220 that estimates the incidence angle and trajectory of the treatment device 160, by incorporating an ultrasound-based model that has been pre-trained or pre-learned to extract features (or characteristics).

[0022] Figure 2 illustrates the architecture and pipeline processing of the AI-driven model of the tip-tracking model 200. As shown in Figure 2, the architecture of the AI-driven model of the tip-tracking model 200 includes an ultrasound-based model 210, a transformer-based model 220, and multiple linear layers 230, the linear layer 230 being provided for predicting tip passage points 252 and predicting tip incidence angles 254. The inputs are the ICE imaging sequence (or image sequence) 240 and, if available, previous passage points 242 and previous incidence angles 244. The outputs are the predicted tip passage points 252 and the predicted tip incidence angles 254. Different architectures can be employed, and for example, additional components or layers may be included. Image data from the ICE catheter 140 may be preprocessed to a predetermined resolution. Further processing may be performed on the input data and outputs as required for treatment or analysis. For example, additional data may be output for the robotic catheter control system 170, or the output data may be further processed for the automatic control of the catheter 140.

[0023] The ultrasound-based model 210 is configured to extract features 212 from input ICE images, for example, with respect to an ICE imaging sequence 240 acquired by an ICE catheter 140. In one embodiment, the ultrasound-based model 210 is a large neural network that has been trained on a wide range of diverse datasets using self-supervised or unsupervised learning to develop a widely transferable representation. Instead of being configured for a single task, the ultrasound-based model 210 is configured to learn general patterns, for example, multiple visual patterns that can be efficiently adapted to different downstream applications through additional adapters. To estimate the incidence angle and trajectory of the treatment device 160, the ultrasound-based model 210 processes the raw data into latent feature embeddings that capture structural information. The ultrasound-based model 210 may be pre-trained to learn invariance without explicit labeling in order to generalize the model across heterogeneous domains.

[0024] In one embodiment, the ultrasound-based model 210 is a vision-foundation model configured to provide a unified framework for cardiac ultrasound (or echocardiography) image analysis across multiple clinical applications. This ultrasound-based model 210 employs a transformer-based encoder architecture, pre-trained on a large, diverse, intradomain dataset of millions of cardiac ultrasound images via self-supervised learning through the DINOv2 framework. Through this extensive pre-training, the encoder learns a generalizable spatial representation of cardiac structure, motion, and image features, independent of transducer type, vendor, or patient population. The framework is configured to support both complete fine-tuning and parameter-efficient adaptation, enabling performance optimization with minimal computational resources. The pre-trained encoder ensures a common representational foundation for diverse clinical purposes, maintaining consistency across multiple tasks. Once pre-trained, the model 210 can extract features 212 from an input ICE imaging sequence 240. In other embodiments, a different model may be used for feature extraction.

[0025] As shown in Figure 2, features 212 extracted from the ICE imaging sequence 240 are input into a time-series transformer-based model 220, which estimates the incidence angle 254 and passpoint 252 of the treatment device 160 in real time. The transformer-based time-series model 220 is configured to take data with inherent temporal dependencies as input. Rather than relying on fixed receptor regions or repeating structures, the transformer-based time-series model 220 models the relationships between time steps using a self-attention mechanism. Each time token (or time-series token) represents a frame, time step, or feature embedding, and together with all others, allows the transformer-based time-series model 220 to capture short-term dynamics and long-term dependencies in a unified representation. Position or time encoding is added to maintain sequence order, enabling the network to infer temporal progression.

[0026] The tip-tracking model 200 is configured to capture the spatial and temporal dependencies of the tip of the treatment device 160 / catheter tip as it moves over time. In this configuration, each frame of the continuous (sequential) ICE image stream 240 is first processed by the ultrasound-based model 210 to extract spatial features describing the region around its tip, which include bounding boxes encoding the passing points and angles of incidence for characterizing the directional directivity of the motion.

[0027] A passage point refers to the spatial position at the moment when the distal tip or active segment of the treatment device 160 intersects or crosses the image plane of the ICE transducer. The passage point represents the projection of the device within a 2D (or two-dimensional) ultrasound view and corresponds to the location where the device's trajectory passes through the imaged plane. Because ICE captures thin slices of cardiac anatomy, the passage point identifies the moment and location when the device enters, exits, or crosses these 2D slices through 3D (or three-dimensional) motion. The angle of incidence refers to the angle between the trajectory of the distal axis or tip of the device and the normal vector to the image plane at the point of intersection. This angle quantifies how obliquely or perpendicularly the device crosses the ultrasound plane. A small angle of incidence indicates that the device passes approximately parallel to the plane, often appearing stretched or blurred, while a large angle indicates that the device intersects the plane approximately perpendicularly, in which case the tip appears as a compact or circular echo. The angle of incidence is crucial for understanding the true 3D orientation of the device from the image data. The angle of incidence affects how the projection and shape of the device appear in the ICE image, thus impacting the accuracy of localization.

[0028] Each frame's feature vector is incorporated into a time sequence (or time-series sequence), which serves as input to a transformer model containing multiple encoder layers and a multi-head self-attention module. The transformer encoder models the relationships between frames by allowing each feature vector to accompany all other features in the imaging sequence 240. The attention mechanism allows the network to learn long-term time dependencies, such as velocity changes, trajectory curvature, or transient occlusions. Temporal position embedding preserves the order of the frames, while the linear transformation layer integrates past tip coordinate and angular information into the same feature space to ensure consistent temporal coherence and robust motion continuity.

[0029] Model 200, described above, is trained to predict the coordinates and incidence angle of the next bounding box at the tip for each frame or short-term prediction range. This prediction process aggregates contextual information (or contextual data) across the entire sequence to enable the model to infer motion patterns even under noisy or partially visualized conditions. Optimization is performed using mean squared error (MSE) loss to minimize the difference between the predicted position and angle and the position and angle of the ground truth (or reliable ground truth data). This aims to facilitate smooth, temporally consistent trajectories and accurate spatial identification.

[0030] In one embodiment, multiple encoder layers of the transformer-based time-series model 220 form a hierarchical sequence processing structure that progressively refines the temporal and spatial feature representation. Each encoder layer includes multi-head self-attention and feedforward components, which, combined with normalization and residual connections, maintain the information flow. The self-attention component ensures that the features of each frame interact with all others to capture global temporal dependencies, while the feedforward component projects and non-linearly transforms the features of interest to enhance the abstraction. By stacking multiple encoder layers, the model progressively learns high levels of temporal correlation, from immediate motion continuity to complex orbital dynamics, ensuring stable, context-aware tracking performance.

[0031] The attention heads (or attention heads) of the transformer-based time series model 220 function as independent subspaces within the self-attention mechanism. Each attention head learns to focus on various aspects of temporal and spatial dependencies across the entire feature sequence. Each attention head computes weighted relationships between all pairs of time steps, enabling the model to capture diverse motion patterns (e.g., short-term displacements, long-term orbital trends, and changes in direction). By combining the outputs from multiple attention heads, the model can aggregate multiple complementary temporal cues into a unified representation. The multi-head structure improves comprehension and overall predictive stability in continuous tracking tasks.

[0032] The linear transformation layer of the transformer-based time series model 220 functions as a mechanism for integrating and aligning temporal and spatial information. The linear transformation layer projects heterogeneous inputs (e.g., past tip coordinates, bounding box features, and incidence angles) into a common embedding space compatible with the transformer encoder. By applying a learned affine mapping, the linear transformation layer ensures that positional and angular data consistently contribute to temporal feature representation. The linear transformation layer effectively encodes the continuity of motion and the geometry of space. Thus, the transformer is provided with context (or background) regarding directionality and past motion. This further facilitates improved prediction accuracy across continuous tracking frames.

[0033] The output of this model is a prediction of the tip passage point 252 and a prediction of the tip incidence angle 254. Unlike conventional CNN-based methods, this model leverages attention mechanisms to capture spatial and temporal dependencies. The system provides continuous probe tracking, reducing the need to manually reposition the ICE catheter 140 during the procedure. In one embodiment, this model 200 is trained using real-world and synthetic data. The combination of actual clinical ICE sequences and synthetic augmentation ensures the potential for generalization to various patient anatomy.

[0034] Figure 3 illustrates a workflow for training the tip-tracking model 200. In one embodiment, the base model is pre-trained to extract features (or characteristics) from an ultrasonic sequence. These operations (or actions) may be performed by the system shown in Figures 1, 2, and 7, or by other systems, workstations, computers, and / or servers. Additional operations, different operations, or fewer operations can be provided. These operations may be performed in the illustrated order (e.g., from top to bottom), or in other orders. Also, certain operations may be omitted or modified depending on the results of previous operations. Training the transformer-based model 220 involves sequentially optimizing the transformer-based model 220 so as to predict the exact tip-passing point 252 and incidence angle (or collision angle) 254 in time series, given the features 212 extracted from the ultrasonic base model 210. The input sequence includes spatial features, bounding box coordinates, and angular data, which are encoded and passed through a multi-layer transformer encoder that can model time-series dependencies through self-attention. Before making predictions, a linear transformation layer integrates past spatial and directional information. This model outputs the position and angle of the next frame, which are compared to ground truth using mean squared error (MSE) loss. Gradient is passed backward through these layers to update attention weights and embeddings, improving temporal consistency and spatial accuracy through iterative learning.

[0035] For training this model, operation A110 provides a pre-trained ultrasound-based model 210. The pre-trained ultrasound-based model 210 is configured to extract features 212 from input ICE image data. In one embodiment, the ultrasound-based model 210 functions as a domain-specific image encoder, which converts each 2D ICE frame into a compact embedding suitable for time-series modeling.

[0036] In operation A120, a training dataset (or learning dataset) is acquired. In one embodiment, the training dataset includes both synthetic and real data. Acquiring a large ICE dataset that includes accurate catheter tip position and ground truth angle of incidence can be difficult. To address this problem, a training dataset is acquired by simulating various device orientations, angles of incidence, and anatomical interactions using recorded clinical sequences and synthetic data generation, ensuring clinical relevance. Recorded clinical sequences can be obtained from actual scans of various patients by different demographics and technicians. Synthetic data can be generated using various different processes, including physically simulated and algorithmically simulated data.

[0037] In one embodiment, a water chamber is used to generate a composite sequence. Various device orientations and angles of incidence are simulated to reflect actual anatomical interactions, thus ensuring that the training dataset covers a wide range of scenarios. To capture the position and orientation of the catheter tip, both the ICE catheter 140 and the device catheter tip are equipped with EM sensors. The sensor frames are initialized so that the orientation of the ICE catheter 140 and its tip aligns with the z-axis, and the direction of the ICE's US fan aligns with the EM's x-axis. Synthesized and simulated ICE images are collected within the water chamber against a black background, enabling automated annotation (or labeling) using computer vision technology. For ground truth labeling, a rot However, this can sometimes be estimated from the direction of the diagonal of the bounding box, but this a rot This can be calculated from EM sensor data using the following formula.

[0038]

number

[0039] In the above formula, E world represents a global frame, and a entry is derived from the z-axis angle on the left side of the above formula (i.e., E tip ice ). In order to improve realism and prepare data for a base model (or fundamental model), the extracted tip images may be overlaid on actual ICE images from clinical resection procedures, which allows introducing intensity variations to maintain motion continuity and improve generalization. In each case, consecutive frames are included, ensuring continuous tip motion that is indispensable for training a real-time tracking model. To prevent overfitting, the synthetic tip sequences and actual ICE images may be strictly distinguished between the training dataset and the test dataset.

[0040] Note that FIG. 4 shows an example of a dataset configuration. In FIG. 4, (a) illustrates an entry angle, a entry . This angle is the angle at which the tip enters the US fan region. In (b), a rotation angle, a rot , is illustrated. This angle a rot is the rotation angle between the center line of a 2D US image and the tip. In (c), a tip passing position is illustrated, which represents the position of the instrument on the 2D US image.

[0041] In operation A130, features are extracted from a hybrid training dataset by a pre-trained ultrasound base model 210. In one embodiment, a sequence of ICE images, I1:N = [i1, i2, ..., iN], is processed, resized, and passed through the ultrasound base model 210 (M foundation ) to extract feature representations, FI = [f1, f2, ..., fN]. To maintain temporal consistency, a previous passing point B N-1 and an incident angle AN-1 These are projected into the same feature space. In one embodiment, the ultrasound frames of the ICE imaging sequence 240 are subjected to appropriate echo processing, which may include cone masking, intensity normalization, and resizing. A pre-trained ultrasound-based model 210 can tokenize each frame into patch tokens, add classification (CLS) tokens, and output feature embeddings that capture anatomical and other features. Regions of interest around the catheter tip or passage point may be cropped before encoding to highlight task-relevant signals. For each frame, the CLS embedding or pooled token representation is projected onto a fixed-dimensional vector. Bounding box parameters and incidence angles are linearly mapped into the same feature space and concatenated or fused with the visual embedding to form a frame descriptor. Sequential frame descriptors form a sequence that is sent to the transducer's temporal stack, which provides position encoding and time-dependent self-attention.

[0042] In operation A140, the extracted features are input into a transformer-based deep learning model 220 equipped with one or more encoders that model temporal dependence, by self-attention. In operation A150, the transformer-based deep learning model 220 outputs estimated waypoints and incidence angles. In one embodiment, the waypoints are represented as bounding boxes, B=[xmin, ymin, xmax, ymax], where (xmin, ymin) and (xmax, ymax) correspond to the top-left and bottom-right coordinates. The incidence angle is A=[a entry a rot ] is defined as. In this case, a entry This indicates the angle of incidence to the 2D ICE image plane, and a rotThis indicates the direction of rotation. To ensure temporal consistency, the previous passpoint BN-1 and the previous incidence angle AN-1 are projected into the same feature space via a linear transformation. The extracted features are concatenated with CLS tokens and passed through a transformer network (Mmain) consisting of eight encoder layers and six attention heads. This CLS output is processed via a linear layer 230 to predict the passpoint ^B (with a circumflex before B) and incidence angle ^A (with a circumflex before A).

[0043] In operation A160, the estimated position and angle of the next frame are compared to the position and angle of the ground truth using mean squared error (MSE) loss. As described above, the CLS output is processed through linear layer 230 to predict the passpoint^B and incidence angle^A, in which case the following equation can be referenced.

[0044]

number

[0045] In the above equation, BN and TN are ground truth values, and lmse is the MSE loss function. The MSE loss in the transformer-based time series model 220 quantifies the discrepancy between the predicted values ​​of tip position and incidence angle and the ground truth values. The MSE loss promotes accurate and stable tracking by calculating the average of the squared errors across all prediction parameters and imposing a heavier penalty for larger deviations. By minimizing the MSE, the model learns to generate continuous and temporally consistent predictions that approach and match the observed trajectory. The MSE loss function provides a smooth optimization outlook, facilitating convergence to accurate spatial localization and consistent angle estimation, which is crucial for reliable motion tracking in dynamic or clinical environments.

[0046] In operation A170, the attention weights (or attention weights) and embeddings of the transformer-based deep learning model 220 are updated or adjusted to improve temporal consistency and spatial accuracy. The attention weights and embeddings may be updated using back projection. In one embodiment, the weights of a pre-trained ultrasonic model are fixed (or frozen) when the transformer-based deep learning model 220 is configured. In other embodiments, the underlying model is pre-trained, but the entire model may be trained end-to-end as it is updated / fine-tuned during end-to-end training. This training process can be repeated multiple times to update the transformer-based deep learning model 220. In this case, the attention weights (or attention weights) and embeddings change with each iteration until the transformer-based deep learning model 220 can accurately estimate the position and angle of the next frame. For example, this model was trained on a single GPU (NVIDIA A100) using PyTorch over 117 epochs with a batch size of 6, achieving real-time performance at 25 Hz. It is possible to apply different GPUs (multiple) and the number of epochs (iterations). In operation A180, a pre-trained model is output because it is used in the imaging procedure.

[0047] Figure 5 illustrates a method for tracking the tip of the treatment device 160 during an ICE imaging procedure (or imaging procedure). These operations (or actions) can be performed by the systems shown in Figures 1, 2, and 7, or by other systems, workstations, computers, and / or servers. Additional operations, different operations, or fewer operations can be provided. These operations may be performed in the illustrated order (e.g., from top to bottom), or in other orders. Furthermore, certain operations may be omitted or modified depending on the results of previous operations.

[0048] In operation A210, an ICE imaging sequence 240 is acquired by an ICE catheter 140 during ICE treatment of patient 180. The transducer of the ICE catheter 140 scans a surface. The scan surface (or scanning plane) is oriented based on the position of the catheter. As the catheter moves (e.g., moves or rotates), different scan surfaces are scanned. Each scan generates a frame of data representing the scan surface at that time. The ultrasound data frames may be scalar values ​​or display values ​​(e.g., RGB) in polar or Cartesian coordinate form. The ultrasound data frames may be B-mode, color flow, or other ultrasound images. A sequence of data frames obtained from ICE imaging represents the ICE sequence. Each frame represents a 2D scan surface. Thus, a collection of frames representing different 2D scan surfaces of the volume of the heart and / or its surrounding volume is acquired. While the ICE imaging sequence 240 is being acquired, it is possible to display the ICE imaging sequence 240 using a display. In one embodiment, the acquisition of the ICE imaging sequence 240 is performed by a robotic control system.

[0049] In operation A220, the control unit 150 estimates the position and orientation of the catheter tip of the ICE catheter 140 during ICE treatment in real time as the ICE imaging sequence 240 is acquired. These positions and orientations are estimated by a tip-tracking model 200 applied by the control unit 150. These positions and orientations may be derived from tip passpoints and tip incidence angles provided by the model 200. Multiple sequences of ICE images are processed by the tip-tracking model 200, which includes an ultrasound-based model 210 and a time-series transformer-based model 220. The inputs to this model are the sequence, as well as previous passpoints and incidence angles. Features are extracted and combined by the base model and sent into the transformer-based model 220, where the final passpoints and incidence angles are predicted from a separate output layer.

[0050] In one embodiment, the ultrasound-based model 210 processes each frame of the ICE imaging sequence 240 to generate spatial feature embeddings. Furthermore, a temporal context stream incorporates previous passpoints and previous incidence angles from the most recent time step. A learned linear projection maps these past variables into the same embedding space used in the image features, encoding continuity of velocity, curvature, and direction. Feature fusion occurs through concatenation or cross-attention between the spatial embedding and the projected temporal embedding, followed by the addition of a learned temporal position encoding. The sequence enters the transformer's encoder stack, consisting of, for example, eight layers with six attention heads per layer. Causal masking enforces a temporal flow, in which multi-head attention allocates each time step to a global temporal context, effectively handling occlusion (or interference), speckle noise, and out-of-plane motion. Layer-norm and residual paths stabilize the optimization and maintain low latency information. Two task-specific output heads operate in the final hidden state. A regression head predicts how-to points as bounding box coordinates in image space, or as calibrated 3D coordinates when combined with probe geometry. A second regression head predicts the angle of incidence around the local tangent of the tip. Training is overseen by mean squared error loss on coordinates and angles.

[0051] Figure 6 illustrates the output of the tip-tracking model 200. This output includes angle estimation (prediction) and waypoint estimation (prediction). Figure 6 also shows the target (ground truth) for each frame.

[0052] In operation A230, the control device 150 controls the ICE catheter 140 during the ICE procedure. In this case, the orientation of the robotic catheter system 170 is determined, for example, at least in part, based on the estimated position and orientation of the catheter tip. The ICE catheter 140 may be controlled during the procedure so that the tip of the treatment device is continuously visible within the ICE imaging area. In one embodiment, the control device 150 inputs the updated position and orientation in real time (e.g., in video frame rate) and provides guidance to the robotic system or user interface (UI), which may include, for example, a confidence score of an assessment of the uncertainty of attention aggregation provided from a model.

[0053] Referring again to Figure 1, the control unit 150 includes a processor 110, memory 120, and interface 130. The processor 110 may include an image processor, which is configured to train a model and / or implement a model, as described herein. The image processor 110 may be a general-purpose processor, a digital signal processor, a three-dimensional data processor, a graphics processing unit, an application-specific integrated circuit, a field-programmable gate array, an artificial intelligence processor, a digital circuit, an analog circuit, or any combination thereof. Alternatively, the image processor 110 may be a currently known or futuristically developed device capable of training and implementing a model for tip-tracking of the treatment device 160 in the ICE imaging sequence 240. The image processor 110 may be a single device, multiple devices, or a network. If multiple devices are utilized, processing may be performed in parallel or sequentially. The image processor 110 may be configured with different devices to perform different functions. In one embodiment, the image processor 110 may be the control processor of the ICE imaging system, or it may be another processor. It is also possible to use another image processor of the ICE imaging system, or an image processor outside the ICE imaging system. The image processor 110 is configured using software, firmware, and / or hardware, and can process data acquired by the imaging device (or imaging device) to output one or more images.

[0054] Interface 130 includes input and output devices. The input may be an interface, for example, to a computer network, memory 120, a database, a medical image storage device, or other source of input data. The input may also be a user input device, for example, a mouse, trackpad, keyboard, rollerball, touchpad, touchscreen, or other device capable of receiving user input. The output is a display (or display device), but an interface is also applicable. The display may be a CRT, LCD, plasma, projector, printer, or other display device. The display is configured to load (read) an image onto the display surface or buffer. The display is configured to display a reconstructed image of a region of the patient 180. The interface may include a graphical user interface (GUI), which enables user operation of the medical imaging device, allowing for substantially real-time user modification and selection.

[0055] Instructions for implementing the processes, methods, and / or techniques described herein are provided on a non-temporary computer-readable storage medium or memory, for example, on a cache, buffer, RAM, removable media, hard drive, or other computer-readable storage medium, such as memory 120. These instructions may be executed by processor 110 or other processor 110. Computer-readable storage mediums include various types of volatile and non-volatile storage media. Functions, operations (or actions) or tasks described herein or illustrated in the drawings are executed in accordance with one or more sets of instructions stored in or on the computer-readable storage medium. Functions, operations or tasks are independent of the set of instructions, storage medium, processor or processing policy, and may be executed alone or in combination by software, hardware, integrated circuits, firmware, microcode, etc. In one embodiment, these instructions are stored on a removable media device (or detachable media device) so that they can be read by a local or remote system. In other embodiments, these instructions are stored remotely so that they can be transferred over a computer network. In further embodiments, these instructions are stored within a given computer, CPU, GPU, or system. Some of the components and steps of the system configuration illustrated in the attached drawings can be implemented by software. Therefore, depending on how this embodiment is programmed, the actual connections between system components (or process steps) may differ.

[0056] In one embodiment, to provide an end-tracking model 200, the processor 110 executes one or more machine learning networks stored in memory 120. In particular, the machine learning networks may include neural networks, such as deep neural networks, convolutional neural networks, or convolutional deep neural networks. The neural networks may also be adversarial networks, deep adversarial networks, generative networks, and / or adversarial generative networks. The network(s) may be provided or implemented by a neural network trained using deep learning. The network(s) may be defined as a series of consecutive feature units or layers. Continuity is used to illustrate a general flow of supplying output feature values ​​from one layer to the next. Information from the next layer is supplied to the next layer, and this continues until the final output. These layers may supply information only forward, or they may supply information bidirectionally, in which case some feedback to the previous layer may be included. The nodes of each layer or unit may be connected to all nodes or a subset (or part) of the nodes of the previous and / or next layer or unit. It is possible to apply skip connections; for example, one layer may output sequentially to the next layer, or to a different layer. Rather than pre-programming features and associating attributes with features, deep architectures are defined to learn features at different levels of different abstractions (actions) of the input data. These features are learned to reconstruct lower-level features (i.e., features at a more abstract or compressed level). Various units or layers can be used; for example, linear, convolution, pooling (e.g., max pooling), deconvolution (or transposed convolution), fully connected, or other types of layers can be used. Any number of nodes can be provided in a unit or layer. For example, 100 nodes may be provided. Subsequent units or the next unit may contain more nodes, fewer nodes, or the same number of nodes.In general, in the case of convolution, the following units may include more abstractions. Figure 7 illustrates one embodiment of an artificial neural network (ANN) 500 according to one or more embodiments. Alternative terms for "artificial neural network" include "neural network," "artificial neural network," and "neural network." The artificial neural network 500 can be used, for example, as part of one or more machine learning-based networks, these networks can be used in the base model or the transformer-based model 220, or as part thereof. Alternatively, the base model and the transformer-based model 220 may use other architectures, such as attention-based mechanisms.

[0057] The artificial neural network 500 includes nodes 502-522 and edges 532, 534, ..., 536. Each edge 532, 534, ..., 536 is a directed connection from the first node 502-522 to the second node 502-522. Generally, the first node 502-522 and the second node 502-522 are different nodes, but it is also possible for the first node 502-522 and the second node 502-522 to be the same. For example, in Figure 7, edge 532 is a directed connection from node 502 to node 506, and edge 534 is a directed connection from node 504 to node 506. Edges 532, 534, ..., 536 from the first node 502-522 to the second node 502-522 can be described as "incoming edges" going to the second node 502-522, and can also be described as "outgoing edges" from the first node 502-522.

[0058] In this embodiment, nodes 502-522 of the artificial neural network 500 can be placed in multiple layers 524-530. These layers may include a unique order introduced by edges 532, 534, ..., 536 between the multiple nodes 502-522. In particular, edges 532, 534, ..., 536 may only exist between adjacent layers of multiple nodes. In the embodiment illustrated in Figure 7, there is an input layer 524 containing only nodes 502 and 504, but with no incoming edges. There is also an output layer 530 containing only node 522, but with no outgoing edges. Furthermore, there are hidden layers 526 and 528 between the input layer 524 and the output layer 530. Generally, the number of hidden layers 526 and 528 can be arbitrarily selected. The number of nodes 502 and 504 in the input layer 524 is usually related to the number of input values ​​to the neural network 500. The number of nodes 522 in the output layer 530 is usually related to the number of output values ​​of the neural network 500.

[0059] In particular, for all nodes 502-522 of neural network 500, (real number / complex number) can be assigned as a value. In this case, x (n) i This represents the values ​​of the i-th nodes 502-522 in the n-th layer 524-530. The values ​​of nodes 502-522 in the input layer 524 correspond to the input values ​​of the neural network 500, and the value of node 522 in the output layer 530 corresponds to the output values ​​of the neural network 500. Furthermore, each edge 532, 534, ..., 536 may contain a weight that is a real number, in particular, a real number within the interval [-1,1] or a real number within the interval [0,1]. In this case, w (m、n) i、j This shows the edge weights between the i-th node 502-522 of the m-th layer 524-530 and the j-th node 502-522 of the n-th layer 524-530. Furthermore, the abbreviation w (n) i、j is weight w (n、n+1) i、j It is defined as follows.

[0060] In particular, input values ​​are transmitted through the neural network in order to calculate the output values ​​of the neural network 500. Specifically, the values ​​of several nodes 502-522 in the (n+1)th layer 524-530 can be calculated from the following equation based on the values ​​of nodes 502-522 in the nth layer 524-530.

[0061]

number

[0062] In the above equation, function f is a transfer function (also known as an "activation function"). Known transfer functions include the step function, sigmoid function (e.g., logistic function, generalized logistic function, hyperbolic tangent, arctangent, error function, smooth step function), or rectifier function. Transfer functions are primarily used for normalization purposes.

[0063] In particular, values ​​are transmitted layer by layer via the neural network. In this case, the value of the input layer 524 is given by the input of the neural network 500, the value of the first hidden layer 526 can be calculated based on the value of the input layer 524 of the neural network, and the value of the second hidden layer 528 can be calculated based on the value of the first hidden layer 526.

[0064] Edge's definition (m、n) i、j To set this up, we need to train the neural network 500 using the training data. In particular, this training data includes training input data and training output data (t i This includes (which may be shown as follows). For the training step, the neural network 500 is applied to the training input data to generate computed output data. In particular, the training data and computed output data contain a number of values ​​equal to the number of nodes in the output layer.

[0065] In particular, a comparison of the calculated output data with the training output data is used to recursively adapt the weights within the neural network 500 (reverse conduction algorithm). Specifically, the weights are changed according to the order.

[0066]

number

[0067] In the above equation, γ represents the learning rate, and the numerical value δ (n) j This can be calculated recursively according to the degree.

[0068]

number

[0069] In the above equation, δ (n+1) j This is based on the case where the (n+1)th layer is not the output layer, and furthermore, the following equation is provided.

[0070]

number

[0071] In the above equation, the (n+1)th layer is the output layer 530. In this case, f' is the first derivative of the activation function, and t (n+1) j This is the comparison training value for the j-th node of output layer 530.

[0072] In one embodiment, the attention layer of the transformer model comprehensively evaluates and utilizes the specific context of each part of the image data sequence. The model reads the input image data sequence and converts it into vector embeddings, where each element of the sequence is represented by its own feature vector(s), which numerically reflects the features. The model then determines the similarity, correlation, and other dependencies (or lack thereof) between each vector. By calculating the dot product between each vector, the importance of one vector to others can be determined. If the vectors are well-matched, their product is large. If the vectors are not well-matched, their dot product is small or negative. This matching score can be converted into attention weights. This may be done by using the matching score as input to a softmax activation function, but by normalizing all values ​​within the range of 0 to 1 so that the sum is 1. For example, assigning an attention weight of 0 between "vector A" and "vector B" means that when making predictions about vector A, vector B should be ignored. Conversely, assigning an attention weight of 1 to vector B means that when making decisions about vector A, the model should receive 100% of its attention. This attention weight is used to emphasize or weaken the influence of a particular input element at a given time point. Networks used for transformer-based time-series (or temporal) modeling for tip-tracking may be derived from Vision Transformer (ViT) networks. Both architectures share the transformer foundation and self-attention mechanisms, but differ in structural intent and data representation. ViT processes spatial tokens extracted from a single image to learn relationships between image patches to capture a global spatial context.In contrast, the time-series transformer networks described herein extend this concept to sequential data, where each token represents a feature vector from a frame or time step, rather than a spatial patch.

[0073] The present invention has been described above based on various embodiments; however, it should be understood that various changes and modifications can be made to these embodiments. Therefore, the above description is intended to be illustrative and not limiting. Furthermore, it should be understood that all equivalents and / or combinations of these embodiments are intended to be included in this specification. Whatever grammatical terms are used, those terms may include masculine, feminine, or any other gender-specific term.

[0074] The following are non-limiting examples of embodiments disclosed herein. Example of Embodiment 1 A system for tracking the tip of a treatment device during intracardiac ultrasound (ICE) procedures, During the above ICE procedure, an ICE catheter configured to acquire the patient's ICE imaging sequence, A control device including at least a processor and memory, The control device described above is configured to track the tip of the treatment device in real time during the ICE imaging sequence using a tip-tracking model. The above advanced tracking model, A pre-trained ultrasound-based model configured to extract features from multiple images in the above ICE imaging sequence, A transformer-based model configured to estimate the angle of incidence and the passage point of the tip of the treatment device in the above ICE imaging sequence, system.

[0075] Example of Embodiment 2 The system of Example 1 of the above embodiment may further include a robotic control system configured to control the ICE catheter at least partially based on the estimated angle of incidence and trajectory of the tip of the treatment device.

[0076] Example 3 of the Embodiment In the system of Example 2 of the above embodiment, the robot control system can be configured to control the ICE catheter so as to maintain continuous visibility of the tip of the treatment device within the ICE imaging area.

[0077] Example 4 of the Embodiment The system in the example embodiment described above may further include a display (or display device) configured to display the ICE imaging sequence.

[0078] Example 5 of the Embodiment In the system of the above embodiment, the pre-trained ultrasound-based model can be trained on millions of cardiac ultrasound images using a self-supervised learning method.

[0079] Example 6 of the Embodiment In the system of the above embodiment, the transformer-based model may include multiple encoder layers, multiple attention heads, and multiple linear layers.

[0080] Example 7 of the Embodiment In the system of Example 6 of the above embodiment, the transformer-based model is trained using an MSE loss function, in which extracted features provided by a pre-trained ultrasonic substrate are concatenated with CLS tokens and transmitted through the transformer-based model, which consists of multiple encoder layers and multiple attention heads, and the output can be processed by multiple linear layers to predict the waypoint and incidence angle.

[0081] Example of an Embodiment 8 In the system of Example 7 of the above embodiment, previous passing points and previous incidence angles can be projected into the feature space via a linear transformation and input into the transformer-based model.

[0082] Example 9 of the Embodiment In the system of the above embodiment, the transformer-based model can be trained using a hybrid set of datasets, including clinical sequencing and synthetic data generation.

[0083] Example 10 of the Embodiment In the system of Example 9 of the above embodiment, the synthesis data generation may include data simulated using a water chamber.

[0084] Example of an embodiment 11 A method for configuring a transformer-based model for tracking the tip of a treatment device, We provide a pre-trained ultrasonic-based model. We obtained a hybrid training dataset of multiple intracardiac ultrasound (ICE) imaging sequences, including clinical and synthetic sequence data. The above transformer-based model is trained through multiple iterations, and in each iteration, Using the above pre-trained ultrasound-based model, features are extracted from one (or several) of the above ICE imaging sequences. The above features are input into the transformer-based model, which is configured using one or more encoders that model temporal dependence through self-attention. The above transformer-based model outputs the position and angle of the next frame, Using mean squared error (MSE) loss, the position and angle of the next frame are compared with the position and angle of the ground truth. To improve temporal consistency and spatial accuracy, update the attention weights and embeddings of the above transformer-based model. method.

[0085] Example 12 of the Embodiment In the example method of the above embodiment, the synthesized data may include data simulated using a water chamber.

[0086] Example 13 of the Embodiment In the example method of the above embodiment, the weights of the pre-trained ultrasonic-based model can be fixed during the training of the transformer-based model.

[0087] Example 14 of the Embodiment In the example method of the above embodiment, the transformer-based model may include multiple encoder layers, multiple attention heads, and multiple linear layers.

[0088] Example 15 of the Embodiment In the method of Example 14 of the above embodiment, the features provided by the pre-trained ultrasonic base model are concatenated with CLS tokens and input into the transformer-based model, which consists of multiple encoder layers and multiple attention heads, and the output can be processed by multiple linear layers to predict the position and angle of the next frame.

[0089] Example 16 of the Embodiment In the method of Example 15 of the above embodiment, the previous passing point and the previous incidence angle can be projected into the feature space via a linear transformation and input into the transformer-based model along with the connected features.

[0090] Example 17 of the Embodiment In the example method of the above embodiment, the multiple iterations of learning may include 100 or more iterations.

[0091] Example of an Embodiment 18 A method for tracking the tip of a treatment device, During an intracardiac ultrasound (ICE) procedure, an ICE imaging sequence is acquired using an intracardiac ultrasound (ICE) catheter. The control device estimates the position and direction of the tip of the treatment device in real time during the ICE procedure, and in this process, the position and direction are estimated using an ultrasound-based model and a transformer-based deep learning model. The robotic catheter system controls the ICE catheter during the ICE procedure based on at least a portion of the estimated position and orientation of the tip of the treatment device. method.

[0092] Example 19 of the Embodiment In the method of Example 18 of the above embodiment, the ICE catheter can be controlled so that the tip of the treatment device is continuously visualized in the ICE imaging sequence during the ICE procedure.

[0093] Example of an embodiment 20 In the method of Example 18 of the above embodiment, the transformer-based deep learning model may include multiple encoder layers, multiple attention heads, and multiple linear layers.

Claims

1. A system for tracking the tip of a treatment device during intracardiac ultrasound (ICE) procedures, An ICE catheter configured to acquire the patient's ICE imaging sequence during the ICE procedure, A control device including at least a processor and memory, The control device is configured to track the tip of the treatment device in real time during the ICE imaging sequence using a tip-tracking model. The aforementioned advanced tracking model, A pre-trained ultrasound-based model configured to extract features from images of the aforementioned ICE imaging sequence, The system includes a transformer-based model configured to estimate the incident angle and transit point of the tip of the treatment device in the ICE imaging sequence, system.

2. The system according to claim 1, further comprising a robotic control system configured to control the ICE catheter at least partially based on the estimated angle of incidence and point of passage of the tip of the treatment device.

3. The system according to claim 2, wherein the robot control system is configured to control the ICE catheter so as to maintain continuous visibility of the tip of the treatment device within the ICE imaging area.

4. The system according to claim 1, further comprising a display configured to display the ICE imaging sequence.

5. The system according to claim 1, wherein the pre-trained ultrasound-based model is trained on millions of cardiac ultrasound images using a self-supervised learning method.

6. The system according to claim 1, wherein the transformer-based model includes a plurality of encoder layers, a plurality of attention heads, and a plurality of linear layers.

7. The system according to claim 6, wherein the transformer-based model is trained using an MSE loss function, in which extracted features provided by a pre-trained ultrasonic substrate are concatenated with CLS tokens and transmitted through the transformer-based model, the transformer-based model comprising a plurality of encoder layers and a plurality of attention heads, and the output is processed by a plurality of linear layers to predict the passage point and incidence angle of the tip of the treatment device.

8. The system according to claim 7, wherein the previous passing point and previous angle of incidence of the tip of the treatment device are projected into feature space via a linear transformation and input into a transformer-based model.

9. The system according to claim 1, wherein the transformer-based model is trained using a hybrid set of datasets including clinical sequences and synthetic data generation.

10. The system according to claim 9, wherein the synthesis data generation includes data simulated using a water chamber.

11. A method for constructing a transformer-based model for tracking the tip of a treatment device, We provide a pre-trained ultrasonic-based model. We obtained a hybrid training dataset of multiple intracardiac ultrasound (ICE) imaging sequences, including clinical and synthetic sequence data. The transformer-based model is trained through multiple iterations, and in each iteration, Using the aforementioned pre-trained ultrasound-based model, features are extracted from multiple ICE imaging sequences, The features are input into the transformer-based model, which is configured using one or more encoders that model temporal dependence through self-attention. The transformer-based model outputs the position and angle of the next frame at the tip of the treatment device. Using mean squared error (MSE) loss, the position and angle of the next frame are compared with the position and angle of the ground truth. The attention weights and embeddings of the transformer-based model are updated to improve temporal consistency and spatial accuracy. method.

12. The method according to claim 11, wherein the synthesized data includes data simulated using a water chamber.

13. The method according to claim 11, wherein the weights of the pre-trained ultrasonic-based model are fixed during the training of the transformer-based model.

14. The method according to claim 11, wherein the transformer-based model includes a plurality of encoder layers, a plurality of attention heads, and a plurality of linear layers.

15. The method according to claim 14, wherein the features provided by the pre-trained ultrasonic base model are linked with CLS tokens and input into the transformer base model comprising a plurality of encoder layers and a plurality of attention heads, and the output is processed by a plurality of linear layers to predict the position and angle of the next frame.

16. The method according to claim 15, wherein previous passing points and previous incidence angles are projected into feature space via a linear transformation and input into the transformer-based model together with the concatenated features.

17. The method according to claim 11, wherein the multiple iterations of the learning include 100 or more iterations.

18. A method for tracking the tip of a treatment device, During an intracardiac ultrasound (ICE) procedure, an ICE imaging sequence is acquired using an intracardiac ultrasound catheter. The control device estimates the position and direction of the tip of the treatment device in real time during the ICE procedure, and in this process, the position and direction are estimated using an ultrasound-based model and a transformer-based deep learning model. The robotic catheter system controls the ICE catheter during the ICE procedure based on at least a portion of the estimated position and orientation of the tip of the treatment device. method.

19. The method according to claim 18, wherein the ICE catheter is controlled such that the tip of the treatment device is continuously visualized in the ICE imaging sequence during the ICE procedure.

20. The method according to claim 18, wherein the transformer-based deep learning model includes a plurality of encoder layers, a plurality of attention heads, and a plurality of linear layers.

Images