Systems and methods for guiding ultrasound probes

Abstract

An ultrasound device (10) includes a probe (12) comprising a tube (14) sized for insertion into a patient and an ultrasound transducer (18) disposed at a distal end (16) of the tube. A camera (20) is mounted at the distal end of the tube in a fixed spatial relationship relative to the ultrasound transducer. At least one electronic processor (28) is programmed to: control the ultrasound transducer and the camera to acquire ultrasound images (19) and camera images (21) respectively when the ultrasound transducer is disposed in the patient; and to construct a keyframe (36) representing the intra-body location of the ultrasound transducer, the keyframe including at least one ultrasound image feature (38) extracted from an ultrasound image acquired at the intra-body location of the ultrasound transducer and a camera image feature (40) extracted from one of the camera images acquired at the intra-body location of the ultrasound transducer.

Description

Technical Field

[0001] The following content generally covers ultrasound technology, ultrasound imaging technology, ultrasound probe technology, ultrasound probe guidance technology, ultrasound catheter technology, transesophageal ultrasound (TEE) technology, and related technologies. Background Technology

[0002] Ultrasound imaging, particularly transesophageal echocardiography (TEE), using an array of ultrasound transducers mounted at the end of an insertion tube, is an existing imaging method with various applications, most commonly for diagnostic purposes in cardiac patients and for providing image guidance in catheter-based cardiac interventional procedures. TEE relates to a method for cardiac ultrasound imaging in which an ultrasound probe comprises a cable or tube, and an ultrasound transducer is positioned at the end of the cable or tube. The TEE probe is inserted into the esophagus to position the ultrasound transducer at its distal end, close to the heart.

[0003] In catheter-based structural cardiac interventions, transesophageal echocardiography (TEE) has been widely used as a reliable method for imaging interventional catheter devices used in the treatment of structural heart diseases. Three-dimensional (3D) transesophageal ultrasound (US) is used for interventional guidance in catheterization lab procedures because it provides real-time volumetric imaging compared to two-dimensional (2D) slice imaging using B-mode ultrasound. This enhances visualization of cardiac anatomy and provides particularly good soft tissue visualization, which is lacking in X-rays. TEE is commonly used for interventional treatments of many structural heart diseases (SHDs), such as mitral valve replacement.

[0004] Typically, the TEE probe is inserted into the esophagus by a trained ultrasound technician (or cardiologist) and manually adjusted towards some standard viewing positions so that specific anatomical structures and perspectives of the heart are within the field of view of the US device. Different measurements or examinations may require different fields of view / perspectives for the same anatomical structures, in which case the probe needs to be repositioned. During interventional procedures, the probe is frequently moved between viewing positions to accommodate X-ray imaging and to track the progress of the intervention as the device is manipulated. Sometimes, the probe may be accidentally moved due to physiological motion or unintentionally for other reasons and must be returned to the desired viewing position.

[0005] TEE probes typically include a cable-driven mechanical joint located at the distal end of the probe, which can be manually operated via a knob on the probe's handle. The distal flexibility provided by these joints, coupled with manually controlled rotation and insertion distance of the TEE probe, and electronic beam steering of the ultrasound imaging plane, offers considerable flexibility in positioning the ultrasound transducer and imaging plane to obtain the desired cardiac view. However, concerns include the risk of esophageal perforation, the difficulty in manipulating numerous controls to obtain the desired clinical view, and the harmful radiation exposure of the TEE operator to the X-ray source during interventional procedures.

[0006] Besides TEE, other types of ultrasound imaging use probes with tubes (i.e., catheters) that are sized for insertion into the patient's body, with an ultrasound transducer located at the distal end of the tube. Probes include: intracardiac echo (ICE) probes, which are typically thinner than TEE probes and are inserted into the blood vessel to allow the ultrasound transducer array to move within the heart; and intravascular ultrasound (IVUS) probes, which are also thin and inserted into the blood vessel to image various anatomical structures from an advantageous internal position.

[0007] US2012 / 302875A1 discloses a system and method for inserting an intracranial catheter. US2019 / 008490A1 discloses a lung biopsy device, system, and method for locating and biopsy objects. US2018 / 185008A1 discloses an imaging probe with a US transducer array and an integrated optical imaging subsystem. US2017 / 258440A1 discloses an apparatus, system, and method for ultrasound imaging that properly locates a transesophageal echocardiography probe using a camera. JP5118455B2 discloses an endoscopic device.

[0008] The following discloses some improvements to overcome these and other problems. Summary of the Invention

[0009] In one aspect, an ultrasound device includes a probe comprising a tube and an ultrasound transducer, the tube being sized for insertion into a patient, and the ultrasound transducer being positioned at a distal end of the tube. A camera is mounted at the distal end of the tube in a fixed spatial relationship relative to the ultrasound transducer. At least one electronic processor is programmed to: control the ultrasound transducer and the camera to acquire ultrasound images and camera images, respectively, while the ultrasound transducer is positioned within the patient; and to construct a keyframe representing the intra-body location of the ultrasound transducer, the keyframe including at least one ultrasound image feature extracted from at least one of the ultrasound images acquired at the intra-body location of the ultrasound transducer and a camera image feature extracted from one of the camera images acquired at the intra-body location of the ultrasound transducer. The keyframe also includes one or more settings of the ultrasound transducer at the time of acquisition of the ultrasound image acquired at the intra-body location of the ultrasound transducer.

[0010] One advantage is that it provides proper positioning of the ultrasound probe to acquire images of the heart in a specific view.

[0011] Another advantage is that it provides an ultrasound probe with multiple imaging devices to obtain cardiac images.

[0012] Another advantage is that it provides an ultrasonic probe that gives the user feedback on manipulating the ultrasonic probe.

[0013] Another advantage is that it provides an ultrasound probe with lower operational complexity, thereby reducing errors and costs.

[0014] Another advantage is that it provides an ultrasound probe with a servo motor that can automatically maneuver the ultrasound probe through the esophagus, blood vessels, or anatomical structures with lumens that can pass through.

[0015] Another advantage is that by actuating the ultrasound probe for remote and / or automatic control, the operator's X-ray exposure to ultrasound equipment is reduced.

[0016] The given embodiments may not provide, provide, provide, two, more or all of the above advantages, and / or may provide other advantages that will become apparent to those skilled in the art upon reading and understanding this disclosure. Attached Figure Description

[0017] This disclosure can take various forms, including the arrangement of components and components, and the arrangement of steps. The accompanying drawings are for illustrative purposes only and should not be construed as limiting this disclosure.

[0018] Figure 1 and Figure 2 An exemplary embodiment of an ultrasonic device according to one aspect is shown.

[0019] Figure 3 It shows Figure 1 and Figure 2 An exemplary flowchart of the operation of an ultrasound device.

[0020] Figure 4 It shows Figure 1 and Figure 2 The potential movable axis of the ultrasound equipment.

[0021] Figure 5 It shows Figure 1 and Figure 2 Another exemplary flowchart operation of an ultrasound device.

[0022] Figure 6 It shows Figure 1 and Figure 2 Exemplary use of ultrasound equipment. Detailed Implementation

[0023] The systems and methods disclosed herein provide keyframes. As used herein, a keyframe (and its variations) refers to an image feature representing a specific location of a TEE probe (or other catheter-based ultrasound probe). This paper recognizes that ultrasound images alone may not be sufficient to generate reliable keyframes because ultrasound imaging can be discontinuous and provides relatively low-resolution images. To provide stronger keyframes, a video camera is integrated into the probe tip, attached to the ultrasound transducer, or positioned close to the transducer on the probe for movement together.

[0024] In a typical workflow, the TEE probe acquires keyframes at various points along its traverse of the esophagus. For example, a new keyframe may be acquired each time image loss (due to movement and / or electron beam deflection) exceeds a threshold portion of image features (e.g., in ultrasound or camera images). Alternatively, keyframes may be manually acquired and labeled with the desired view as the physician reaches it. Alternatively, views may be automatically identified based on image analysis to define image features and label the corresponding keyframes. This can also be defined by a dwell time during which the probe undergoes minimal motion. If the physician then wishes to return to a previous view, the servo motors reverse to move the probe tip backward and compare the acquired images with keypoints along the way to automatically track and adjust (if necessary) the backtracking process. In other embodiments, the servo motors move via a sequence of servo motions corresponding to keyframe transitions that correlate the current view with the desired view. In some examples, visual servoing methods supplement keyframe-to-keyframe open-loop motion. Keyframe motion includes (but is not limited to) the physical movement of the probe and the electronic beam steering. This motion may also include imaging settings (e.g., ultrasound parameters, camera settings, etc.).

[0025] In some embodiments disclosed herein, a manual mode is implemented. In this case, the TEE probe is a manually operated probe with knobs for controlling the joints of the TEE probe, and the system provides control cues such as "advance insertion," "retraction," "rotation," "bend," "at view," etc., based on feedback obtained by comparing real-time images with previously acquired keyframes. In other embodiments, the TEE probe is partially or fully robotic, utilizing servo motors instead of mechanical manipulation to operate the TEE probe, such as distal joint control, insertion, rotation, etc. In this case, the system can directly control the servo motors to perform the desired TEE probe manipulation. The system can also control ultrasound imaging parameters (e.g., imaging depth, beam steering angle, gain, etc.) to recall previously set settings for a given view.

[0026] In some embodiments disclosed herein, the ultrasound transducer is side-firing, while the video camera is forward-looking. This is a convenient arrangement because a side-firing ultrasound transducer can be well positioned for imaging the heart, while a forward-looking video camera offers advantages that a side-firing transducer cannot provide. Particularly valuable is that the forward-looking camera can detect obstacles that would impede further insertion of the TEE probe and can visualize appropriate actions (e.g., rotating the probe's joints) to avoid collisions with obstacles.

[0027] Figure 1 and Figure 2 An exemplary embodiment of an ultrasound device 10 for medical procedures, particularly cardiac imaging procedures, is shown. Although referred to herein as a TEE ultrasound device, ultrasound device 10 can be any suitable catheter-based ultrasound device (e.g., an ultrasound device for intracardiac echo (ICE) procedures, intravascular ultrasound procedures, etc.). Figure 1 As shown, the ultrasound device 10 includes a probe 12 configured as, for example, a flexible cable or tube, for insertion into a patient's lumen (e.g., the lumen could be an esophageal lumen, a vascular lumen, etc.). The probe 12 can be any suitable, commercially available probe (e.g., the Philips x7-2 TEE probe, available from Koninklijke Philips NV in Eindhoven, Netherlands). As described herein, the probe 12 is described for use in TEE procedures involving inserting the probe into the patient's esophagus to obtain images of the patient's heart; however, it should be understood that the probe can be inserted into any part of the patient to obtain images of any target tissue.

[0028] The probe 12 includes a tube 14 sized for insertion into a portion of a patient's body (e.g., the esophagus). The tube 14 can be flexible or rigid. In some examples, the tube 14 has a handle 15 positioned externally to the patient and operated by a user. The tube 14 includes a distal end 16, at which an ultrasound transducer 18 is located. The ultrasound transducer 18 is configured to acquire ultrasound images 19 of a target tissue (e.g., the heart or surrounding blood vessels). A camera 20 (e.g., a video camera, such as an RGB or other color camera, a monochrome camera, an infrared (IR) camera, a stereo camera, a depth camera, a spectral camera, an optical coherence tomography (OCT) camera, etc.) is also located at the distal end 16 of the tube 14. The camera 20 is configured to acquire camera (e.g., still and / or video) images 21 of the target tissue. Camera 20 can be any suitable, commercially available camera (e.g., the camera described in Pattison et al., “Atrial pacing thresholds measured in anesthetized patients with the use of an esophagus stethoscope modified for pacing” (Journal of Clinical Anesthesia, Vol. 9, No. 6, 492).

[0029] The camera 20 is mounted in a fixed spatial relationship relative to the ultrasonic transducer 18. In one exemplary embodiment, the ultrasonic transducer 18 and the camera 20 are attached to each other, or as... Figure 1 and Figure 2 As shown, it is housed or otherwise secured to a common housing 22 located at the distal end 16 of the tube 14. In particular, as Figure 2 As shown, the ultrasonic transducer 18 is arranged to emit from the side, and the camera 20 is arranged to face forward. Advantageously, as Figure 1 This arrangement, as shown, provides that the side-firing ultrasound transducer 18 is well-positioned for imaging the heart, while the forward-looking camera 20 offers advantages that the side-firing transducer cannot provide (e.g., for the heart). In other embodiments, the camera 20 may be robotically actuated or steerable using cables or other means, such as a reflective lens (not shown). In another example, the ultrasound transducer 18 and camera 20 may be connected via some other mechanism (e.g., flexible cables) and tracked relative to each other.

[0030] The ultrasound device 10 also includes an electronic controller 24, which may include a workstation, such as an electronic processing device, a workstation computer, a smart tablet, or a more general computer. In a non-limiting illustrative example, the electronic controller 24 is a Philips EPIQ-class ultrasound workstation. (Note that the ultrasound workstation 24 and the TEE probe 12 are shown at different scales). The electronic controller 24 can control the operation of the ultrasound device 10, including, for example, controlling the ultrasound transducer 18 and / or the camera 20 to acquire images, and controlling the movement of the probe 12 through the esophagus by controlling one or more servo motors 26 of the ultrasound device 10, which are connected to drive its articulated joints (not shown) and / or rotate and / or extend and retract the tube 14. Alternatively, one or more knobs 27 may be provided, by which the user manually operates the drive joints to manipulate the probe through the esophagus.

[0031] Although Figure 1 The knobs and servo motor components 26, 27 are shown for illustrative purposes, but typically the ultrasound probe 12 will be manual (with only knobs) or robotic (with only servo motors), but a hybrid manual / robotic design is envisioned, such as where the user manually extends / retracts the tube 14, while the servo motor is configured to robotically operate the probe position and its joints.

[0032] Workstation 24 includes typical components such as at least one electronic processor 28 (e.g., a microprocessor) which includes a connector 29 for inserting an ultrasound probe. Figure 1 The dashed lines in the diagram schematically indicate the connection between the TEE probe 12 and the ultrasound workstation 24, at least one user input device (e.g., mouse, joystick, keyboard, trackball, and / or similar device) 30, and at least one display device 32 (e.g., LCD display, plasma display, head-up display, augmented reality display, cathode ray tube display, and / or etc.). The illustrative ultrasound workstation 24 includes two display devices 32: a larger upper display device that displays ultrasound images; and a smaller lower display device that displays a graphical user interface (GUI) 48 for controlling the workstation 24. In some embodiments, the display devices 32 may be components separate from the workstation 24. The display devices 32 may also include two or more display devices. In some embodiments, the user input device 30 may be a component separate from the workstation 24, and in some cases may be virtual, provided within an augmented reality system.

[0033] Electronic processor 28 is operatively connected to one or more non-transitory storage media 34. By way of non-limiting illustrative examples, non-transitory storage media 34 may include one or more of the following: disk, RAID, or other magnetic storage media; solid-state drives, flash drives, electronically erasable read-only memory (EEROM), or other electronic storage; optical disks or other optical storage devices; various combinations thereof; and may be, for example, network storage devices, internal hard drives of workstation 24, various combinations thereof. Although shown separately from controller 24, in some embodiments, a portion or all of one or more non-transitory storage media 34 may be integrated with ultrasonic workstation 24 (e.g., including internal hard drives or solid-state drives). It should be further understood that any non-transitory medium or media 34 mentioned herein should be broadly understood to include a single medium or multiple media of the same or different types. Similarly, electronic processor 28 may be implemented as a single electronic processor or two or more electronic processors. Non-transitory storage media 34 stores instructions executable by at least one electronic processor 28.

[0034] As described above, the ultrasound device 10 is configured to perform a control method or process 100 for controlling the movement of the probe 12. A non-transitory storage medium 32 stores instructions that can be read and executed by at least one electronic processor 28 of the workstation 24 to perform the disclosed operations, including executing the control method or process 100. In some examples, the control method 100 may be executed at least partially via cloud processing.

[0035] Now for reference Figure 3 And continue to refer to Figure 1 and Figure 2 An illustrative embodiment of the control method or process 100 is schematically shown as a flowchart. At operation 102, at least one electronic processor 28 is programmed to control an ultrasound transducer 18 and a camera 20 to acquire ultrasound images 19 and camera images 21, respectively, when the ultrasound transducer (and the camera 20 and the common rigid housing 22) are placed inside the patient's esophagus.

[0036] At operation 104, at least one electronic processor 28 is programmed to construct a keyframe 36 representing the in vivo location (e.g., within the esophagus) of the ultrasound transducer 18. To construct the keyframe 36, at least one electronic processor 28 is programmed to extract ultrasound image features 38 from at least one ultrasound image 19 and / or extract camera image features 40 from at least one camera image 21. In another example, the keyframe 36 may include the ultrasound image 19 and / or the camera image 21 itself. The ultrasound image 19 and the camera image 21 may be stored in one or more non-transitory computer media 34 and / or displayed on a display device 32. The extraction process may include algorithms for extracting a feature set between at least one ultrasound image feature 38 and at least one camera image feature 40. Such algorithms may include, for example, scale-invariant feature transformation (SIFT) algorithms, multi-scale oriented plaque (MOPS) algorithms, vascular tracking algorithms, or any other suitable matching algorithms known in the art. In a variant embodiment, operation 102 uses the ultrasonic transducer 18 to acquire only ultrasonic images (in which case, camera 20 may optionally be omitted), and operation 104 uses features 38 extracted only from the ultrasonic images to construct keyframes. However, it is desirable to use features extracted from both the ultrasonic image 19 and the camera image 21 to construct keyframe 36, which would provide a higher level of distinctiveness for uniquely identifying a given view. Furthermore, the camera image 21 is useful if the ultrasonic image has low contrast or otherwise information-deficient features (and vice versa, if the camera image is information-deficient, this would be compensated for by the features extracted from the ultrasonic image).

[0037] In one example, keyframe 36 may also include one or more settings of the ultrasound transducer 18 (e.g., beam steering, depth, resolution, width, etc.) at the time of acquisition of ultrasound image 19 at a transducer in-body location, from which image features 38 are extracted. In another example, keyframe 36 may include rotation settings and / or insertion settings and / or joint position settings of probe 12 at the time of acquisition of one or more of ultrasound image 19 and / or camera image 21. Joint position settings may include, for example, settings such as "insert," "rotate," "bend," etc. These settings may be determined from position feedback devices (not shown, such as encoders) and / or sensor feedback devices (not shown, such as force sensors or torque sensors) related to the shape and position of probe 12.

[0038] In some exemplary embodiments, operation 104 includes constructing a keyframe 36 in response to satisfying one or more keyframe acquisition criteria 42 (which may be stored in one or more non-transitory computer-readable media 34). In one example, keyframe acquisition criteria 42 may include a comparison between the “last acquired” keyframe 36 and the currently acquired ultrasound image 19 and / or the currently acquired camera image 21. In another example, keyframe acquisition criteria 42 may include a comparison between a keyframe 36 acquired at a preset previous time interval (e.g., a keyframe acquired 30 seconds prior) and the currently acquired ultrasound image 19 and / or the currently acquired camera image 21. Keyframe 36 may be stored in one or more non-transitory computer media 34 and / or displayed on a display device 32. Once stored, keyframe 36 can be accessed by a user at any time via workstation 24. The comparison may include a comparison of changes in the number of features between the last acquired keyframe 36 and the ultrasound image 19 / camera image 21, spatial movement of one of the ultrasound images 19 or one of the camera images relative to the last acquired keyframe, etc. In another example, keyframe acquisition criterion 42 may include the identification of defined image features of the target tissue (e.g., left or right ventricle, left or right aorta, specific blood vessels of the patient's heart, such as the aorta or vena cava, etc.) imaged in the current ultrasound image 19. The comparison process may include a matching algorithm for matching feature sets 38 and 40 of at least one ultrasound image 19 and at least one camera image 21, respectively. Such an algorithm may include, for example, using the sum of squared differences (SSD) algorithm. In some examples, deformable registration algorithms known in the art are used, which use feature sets 38 and 40 to improve the matching between multiple keyframes 36. To improve the robustness of keyframe matching, a sequence of recently generated keyframes 36 is used during the matching process.

[0039] In optional operation 106, at least one electronic processor 28 is programmed to mark keyframes 36 representing the in vivo location of the ultrasonic transducer 18 with a tag 44 when receiving user input from a user via at least one user input device 30 of the workstation 24. In one approach, GUI 48 may provide a drop-down list-style GUI dialog box of standard anatomical views (mid-esophageal (ME) four-chamber view, ME (long axis (LAX) view, transgastric (TG) mid-papillary muscle short axis (SAX) view, etc.), through which the user can select one of the listed items as label 44. Alternatively, a free-form text-input GUI dialog box may be provided, through which the user enters label 44, or further annotates the label selected from the drop-down list. Furthermore, keyframe 36 may also be marked to indicate or represent the intermediate position of ultrasound transducer 18 (e.g., the position of the ultrasound transducer between the positions shown in "adjacent" ultrasound images 19 and / or camera image 21). In another example, a 2D or 3D visual representation of a canonical view of probe 12 may be displayed, showing the current state of the probe, while views of other previously acquired probes may be selected by the user for display. Label 44 and marked keyframe 36 may be stored in one or more non-transitory computer-readable media 34.

[0040] In some examples, instead of (or in addition to) manual labeling, at least one electronic processor 28 may be programmed to label or otherwise classify ultrasound images 19 and / or photographic images 21 based on the specific anatomical view displayed in the images (e.g., ME four-chamber view, ME LAX view, TG mid-papillary muscle SAX view, etc.). Images 19 and 21 may be manually labeled by a user via at least one user input device 30, or automatically labeled using ultrasound image matching algorithms known in the art.

[0041] Now, a brief reference. Figure 4 And continue to refer to Figures 1 to 3 The probe 12 can be manipulated in various ways (manually using the knob 27 or other manual operation, and / or robotically using the servo motor 26, depending on the embodiment). The probe 12 is capable of advancing in the "insertion" direction (i.e., into the esophagus). Figure 3The probe 12 is configured to move along axis 1(a) (marked); and along axis 1(b) in the "retracted" direction; rotate / turn along the front angle direction of axis 2(a); and rotate / turn along the rear angle direction of axis 2(b). The distal end 16 of the probe 12 is configured to move along axes 3(a) and 3(b) in the forward / backward bending direction (via user operation of knob 27); and along axes 4(a) and 4(b) in the right / left direction. The probe 12 can be moved by direct user manipulation along axes 1(a), 1(b), 2(a), 2(b), and by using knob 27 along axes 3(a), 3(b), 4(a), 4(b). These are illustrative degrees of freedom; specific ultrasound probe implementations may provide more, fewer, and / or different degrees of freedom for manipulating the probe position within the body, and subsets of these degrees of freedom may be manually actuated while others may be passively implemented.

[0042] Return to Figures 1 to 3 In another optional operation 108, at least one electronic processor 28 is programmed to guide (and, in the case of the robotic embodiment, control) the movement of probe 12 through the esophagus by constructing a plurality of keyframes 36. For this purpose, at least one electronic processor 28 is programmed to construct keyframes 36 representing a first view including a first in vivo location of the ultrasonic transducer 18. During the traversal of the ultrasonic transducer 18 from the first view to a second view including a second in vivo location of the ultrasonic transducer, at least one electronic processor 28 is programmed to construct keyframes 36 representing “intermediate” locations of the ultrasonic transducer (e.g., locations between the first and second views). In one example, these intermediate locations are implicitly included in the ultrasound image 19 and the camera image 21, in which case probe 12 can be moved to match a target keyframe 36. In another example, the intermediate location can be an estimate of the motion from one keyframe 36 to the next, relative to the ultrasound volume, the camera image space, or a kinematic transformation in Cartesian or joint space, or any combination thereof. At the end of the passage of the ultrasonic transducer 18, at least one electronic processor 28 is programmed to construct a second keyframe 36 representing the second view.

[0043] Operation 108 may include an operation in which at least one electronic processor 28 is programmed to detect when a new keyframe 36 representing an “intermediate position” should be acquired and saved (i.e., during a transition from a first view to a second view). For this purpose, the recently constructed keyframe 36 is compared with the recently acquired ultrasound image 19 and the recently acquired camera image 21. In one example, a new keyframe is generated if the number of features (e.g., anatomical features, etc.) in images 19, 21 changes relative to the number of features in keyframe 36 in a manner exceeding a predetermined comparison threshold (25% of the features). In another example, a new keyframe is generated if the average pixel displacement in the acquired images 19, 21 changes relative to the pixel displacement of keyframe 36 by a predetermined comparison threshold (e.g., x% of the image size). Other examples may include deformable matching algorithms known in the art to improve image tracking of images 19, 21. These thresholds may be adjusted empirically.

[0044] In one exemplary embodiment, operation 108 is implemented in a manual mode. For this purpose, at least one electronic processor 28 is programmed to provide human-perceptible guidance 46 during a retrograde movement (i.e., “reverse” movement) of the ultrasonic transducer 18 from a second view back to a first view (or an intermediate view between the two) performed manually (e.g., via knob 27). Guidance 46 is based on a comparison of the ultrasonic image 19 and camera image 21 (acquired during the retrograde movement) with a keyframe 36 representing the intermediate position and a keyframe representing the first view. Guidance 46 may include commands including one or more of the following: the ultrasonic device 10 advances through the esophagus (e.g., “forward” or variations thereof); the ultrasonic device retracts through the esophagus (e.g., “reverse” or variations thereof), “rotate,” “capture keyframe,” etc. Guidance 46 may be output visually on a display device 32, audibly via a speaker (not shown), etc. Furthermore, guidance 46 may be displayed as an overlap with images 19 and 21 displayed on the display device 32.

[0045] In another exemplary embodiment, operation 108 is implemented in an automatic mode, wherein the probe 12 moves automatically through the esophagus by the action of servo motors 26. For this purpose, at least one electronic processor 28 is programmed to control one or more servo motors 26 of the probe 12 to perform a traverse of the ultrasonic transducer 18 from a first view to a second view. Then, at least one electronic processor 28 is programmed to control the servo motors 26 of the probe 12 to perform a traverse of the ultrasonic transducer 18 from the second view back to the first view based on a comparison of the ultrasonic image 19 and the camera image 21 (acquired during the traverse) with a keyframe 36 representing an intermediate position and a keyframe representing the first view.

[0046] In both manual and automatic modes, at least one electronic processor 28 is programmed to guide the user regarding the movement of the probe 12 through the esophagus by generating a GUI 48 for display on the display device 32. The user can use the GUI 48, employing at least one user input device 30, to select a desired view of keyframe 36. The desired view of keyframe 36 may include keyframes previously acquired and stored in a non-transitory computer-readable medium 34, keyframes acquired during the current operation, or predefined keyframes stored in a non-transitory computer-readable medium. Matching algorithms for image feature sets 38, 40 can be used to find a set of keyframes 36 that most closely approximates the currently acquired keyframes as shown on the display device 30. For example, keyframes 36 from “View A” to “View N” are created by the user at the start of the operation and stored in the non-transitory computer-readable medium 34. Views between adjacent views (e.g., “View A” to “View B”, “View B” to “View C”, etc.) are linked using “intermediate” keyframes 36. To this end, incremental motion between the current keyframe (e.g., "View B") and the next keyframe (e.g., "View C") uses, for example, motion estimation methods (such as the basic optical flow of features) to estimate how probe 12 should move. The incremental motion direction required to move probe 12 to the next keyframe to the desired view is implemented on GUI 48. The incremental motion can be given relative to, for example, the view of camera 20, the view of ultrasound transducer 18, a model of probe 12, a model of the heart, a model of the patient, etc. The incremental motion can be displayed, for example, as a three-dimensional area indicating the direction of movement.

[0047] In some embodiments, once the desired final view is near, a beamforming-based image steering process can be used to obtain the desired final view.

[0048] In other embodiments, knob 27 can be used to change the probe's bending and extension settings to a relevant dataset of image features 38, 40 to improve the matching of keyframe 36 and generated guide 46. Knob 27 can be manipulated (i.e., rotated or toggled), and the bending and extension settings can be included in GUI 48.

[0049] In a further embodiment, the electronic processor 28 can use the servo motor 26 to perform robotic control of the ultrasound probe 12. By adding keyframe 36 for tracking and guidance 46 as feedback to the robot control, a Cartesian velocity control loop can be used to smoothly and reliably move the probe 12 to the desired view. This allows for efficient, accurate, and safe automatic positioning of the probe 12 between different views. In some embodiments, the robot's forward kinematic posture can be used as another feature in the generation of the keyframe 36. In cases where the ultrasound image or camera image does not match the keyframe 36 well, pre-stored kinematic posture information can be used to transition to the next keyframe.

[0050] In some embodiments, a data-driven approach can be used to estimate the current keyframe 36 relative to a keyframe atlas (not shown) from a number of patients stored in a non-transitory computer-readable medium 34. This would enable guidance within the patient without first seeing the desired view and establishing the keyframe 36.

[0051] In other embodiments, for example, when the sequence of images 19, 21 displays images that are close to the desired camera images in a known sequence, but contain new information, such as blood, or decolorization, or new anatomical topographic features (such as tears), the detection of adverse events can be displayed on GUI 48.

[0052] In some embodiments, the estimated motion of the probe 12 can be used for specific volume stitching of multiple ultrasound images 38. In the case of 3D data, this can provide excellent initialization information.

[0053] In a further embodiment, a 3D model (not shown) can be created based on the orientation of the probe 12 from the keyframe 36 position and the superimposed ultrasound image 19. The user can select a specific keyframe 36 on the 3D model as the desired view, or view the stored image 19 associated with that area for more information.

[0054] In some embodiments, probe 12 may include an integrated force sensing mechanism (not shown) that can be used for a variety of operations, including: retrieving a desired view involving pressure on the esophagus; maintaining contact and drift to limit deformation on the esophagus to manage the risk of perforation; and elastography to create a 3D thermal map of the force-loaded region of the esophagus or stomach.

[0055] Figure 5An exemplary flowchart of movement operation 108 is shown. At operation 50, a view labeled "View A" is acquired for probe 12. At operation 52, the positions of any joint positions (not shown) of the ultrasonic transducer 18, camera 20, and probe 12 at view A are saved. At operation 54, probe 12 is moved from view A toward a new desired view labeled "View B". At operation 56, the positions of the ultrasonic transducer 18, camera 20, and any joint positions (not shown) of probe 12 as it moves from view A to view B are recorded. At operation 58, an intermediate keyframe 36 with a 50% change between view A and view B is created. At operation 60, probe 12 is moved until it reaches view B. At operation 62, using visual servos on GUI 48, probe 12 is moved back from view B toward view A in reverse sequence of intermediate keyframes 36. At operation 64, probe 12 is moved to pass through a predefined intermediate view shown in intermediate keyframe 36. At operation 66, as probe 12 approaches view A, probe 12 is switched to an ultrasound-based servo process. At operation 68, probe 12 is moved until it reaches view A. In addition to the traversal from view B to view A (e.g., operations 64, 66, and 68), keyframe 36 can be updated or added to the set describing the traversal from A to B, and vice versa. These can be used to enrich the resolution and richness of keyframe 36.

[0056] Figure 6 An exemplary use of inserting an ultrasound device 10 into a patient's esophagus is shown. Figure 6 As shown, probe 12 is inserted downwards into the patient's esophagus, allowing ultrasound transducer 18 and camera 20 to acquire corresponding ultrasound images 19 and 21 of the patient's heart. It should be understood that this is merely one specific application of the disclosed method for guiding a catheter-based ultrasound probe. For example, intracardiac echocardiography (ICE) or intravascular ultrasound (IVUS) probes can be similarly guided through the patient's major blood vessels to reach the desired anatomical view and back to the previous anatomical view.

[0057] This disclosure has been described with reference to preferred embodiments. Modifications and variations may be made by others after reading and understanding the foregoing detailed description. The exemplary embodiments are intended to be interpreted as including all such modifications and variations, provided they are within the scope of the appended claims or their equivalents.

Claims

1. An ultrasonic device (10), comprising: The probe (12) includes a tube (14) and an ultrasonic transducer (18), the tube being sized for insertion into a patient, and the ultrasonic transducer being positioned at the distal end (16) of the tube. A camera (20) is mounted at the distal end of the tube in a fixed spatial relationship relative to the ultrasonic transducer; and At least one electronic processor (28), said electronic processor being programmed to: Control the ultrasound transducer and the camera to acquire ultrasound images (19) and camera images (21) respectively when the ultrasound transducer is placed inside the patient; and Construct a keyframe (36) representing the in vivo location of the ultrasonic transducer, the keyframe including at least one ultrasonic image feature (38) extracted from at least one of the ultrasonic images acquired at the in vivo location of the ultrasonic transducer and a camera image feature (40) extracted from one of the camera images acquired at the in vivo location of the ultrasonic transducer, characterized in that the keyframe (36) also includes one or more settings of the ultrasonic transducer (18) at the time of acquisition of the ultrasonic image (19) acquired at the in vivo location of the ultrasonic transducer. The keyframe is configured to guide the movement of the probe by comparing the ultrasound image and the camera image with the keyframe.

2. The ultrasonic device (10) according to claim 1, wherein, The keyframe (36) also includes the rotation setting, insertion setting, and joint position setting of the probe (12) at the acquisition time of the ultrasound image (19) acquired at the in vivo location of the ultrasound transducer (18).

3. The ultrasonic device (10) according to any one of claims 1-2, wherein, The ultrasonic transducer (18) and the camera (20) are attached to each other or housed in or fixed to a common rigid housing (22) located at the distal end (16) of the tube (14).

4. The ultrasonic device (10) according to any one of claims 1-2, wherein, The ultrasonic transducer (18) is positioned at the distal end (16) of the tube (14) for lateral emission, while the camera (20) is positioned at the distal end of the tube for forward orientation.

5. The ultrasonic device (10) according to any one of claims 1-2, wherein, The at least one electronic processor (28) is programmed to construct the keyframe (36) in response to meeting the keyframe acquisition criteria (42).

6. The ultrasonic device (10) according to claim 5, wherein, The keyframe acquisition criteria (42) include a comparison between the last keyframe (36) and the currently acquired ultrasound image (19) and camera image (21).

7. The ultrasonic device (10) according to claim 5, wherein, The keyframe acquisition criteria (42) include the identification of defined image features of the target tissue imaged in the current ultrasound image (19).

8. The ultrasonic device (10) according to any one of claims 1-2, wherein, The ultrasound device further includes at least one user input device (30); and the at least one electronic processor (28) is programmed to: When user input is received via the at least one user input device, the keyframe (36) representing the in vivo location of the ultrasonic transducer (18) is marked.

9. The ultrasonic device (10) according to any one of claims 1-2, wherein, The at least one electronic processor (28) is also programmed to: Construct a keyframe (36) representing a first view, which includes the first in-body location of the ultrasonic transducer (18); During the traversal of the ultrasonic transducer from the first view to a second view including the second in-body position of the ultrasonic transducer, keyframes representing intermediate positions of the ultrasonic transducer are constructed; and At the end of the traversal, a keyframe representing the second view is constructed.

10. The ultrasonic device (10) according to claim 9, wherein, The at least one electronic processor (28) is also programmed to: During the backtracking journey of the ultrasonic transducer (18) from the second view back to the first view or an intermediate view between the two, human-perceptible guidance (46) is provided for the manual control of the probe (12) based on the comparison of the ultrasonic images (19) and camera images (21) acquired during the backtracking journey with keyframes (36) representing the intermediate position and keyframes representing the first view.

11. The ultrasonic device (10) according to claim 10, wherein, The human-perceptible guidance (46) includes commands including one or more advances of the ultrasound device through the esophagus, retraction of the ultrasound device through the esophagus, and capture of keyframes (36).

12. The ultrasonic device (10) according to claim 9, wherein, The at least one electronic processor (28) is also programmed to: The servo motor (26) of the probe (12) is controlled to perform the passage of the ultrasonic transducer (18) from the first view to the second view; and The servo motor of the probe is controlled to perform the backtracking of the ultrasonic transducer from the second view to the first view based on a comparison of the ultrasonic images (19) and camera images (21) acquired during the backtracking process with keyframes (36) representing the intermediate position and keyframes representing the first view.

13. The ultrasonic device (10) according to any one of claims 1-2, wherein, The probe (12) includes a transesophageal echocardiography probe.

Images