Systems and methods utilizing machine learning for in vivo navigation

Abstract

A method for providing in-vivo navigation of a medical device includes receiving input medical imaging data of a patient's anatomical structure, receiving input non-optical in-vivo image data from a sensor on a distal end of the device within the anatomical structure, and identifying a position of the distal end in the input image data using a trained model, the model being trained based on (i) training medical imaging data and training non-optical in-vivo image data of the anatomical structure of one or more individuals, and (ii) registration data that associates the training image data with positions in the training imaging data as ground truth to learn an association between the training image data and the training imaging data, the method including determining an output position of the medical device using the learned association and the input data, modifying the input imaging data to depict the determined position, and causing a display to output the modified input imaging data.

Description

Technical Field

[0001] Various embodiments of the present disclosure generally relate to machine learning-based techniques for in vivo navigation, and more particularly, to systems and methods for determining registration between non-optical image data, such as ultrasound image data, and medical imaging data.

Background Art

[0002] In certain medical procedures, a medical device is advanced at least partially into a patient's body. For example, during a lung ablation procedure to remove unwanted tissue from within a patient's lung, an ablation device is advanced into the peripheral portion of the lung that has the unwanted tissue. Techniques that use direct insertion through a needle, for example, are used, but such techniques generally have a high risk of complications. Less invasive techniques, such as techniques that utilize a bronchoscope, have been developed. However, such techniques also have drawbacks.

[0003] The present disclosure is directed to addressing the above-described problems. The description of the background art provided herein is for the purpose of generally presenting the background of the present disclosure. Unless otherwise indicated herein, the materials described in this section are not prior art to the claims of the present application and are not admitted to be prior art or suggestions of prior art by including them in this section.

Summary of the Invention

[0004] According to certain aspects of the present disclosure, methods and systems for providing in vivo navigation of a medical device are disclosed. In one embodiment, an exemplary embodiment of a system for providing in vivo navigation for a medical device may include memory, a display, and a processor operably connected to the display and memory. The memory may store instructions and a trained machine learning model. The machine learning model may be trained on (i) training medical imaging data and training non-optical in vivo image data of at least a portion of the anatomical structures of one or more individuals, and (ii) registration data relating the training non-optical in vivo image data to locations in the training medical imaging data as ground truth. The training may be configured to cause the trained machine learning model to learn associations between the training non-optical in vivo image data and the training medical imaging data. The processor may be configured to execute instructions in memory and perform operations. The operation may include receiving input medical imaging data associated with at least a portion of the patient's anatomical structure; receiving input non-optical in vivo imaging data from a sensor positioned on the distal end of a medical device that is advanced into the portion of the patient's anatomical structure; determining the position of the distal end of the medical device in the input medical imaging data using the learned associations; modifying the input medical imaging data to include a position indicator showing the determined position of the distal end of the medical device; and causing a display to output the modified input medical imaging data including the position indicator.

[0005] In some embodiments, the operation may further include receiving additional non-optical in vivo image data from sensors as the medical device moves within a portion of the patient's anatomical structure, using learned associations to determine the updated position of the distal end of the medical device based on the additional non-optical in vivo image data, updating the input medical imaging data to adjust the position indicator based on the updated position of the distal end of the medical device, and causing a display to output the updated input medical imaging data.

[0006] In some embodiments, the determination of the updated position, updating of the input medical imaging data, and output of the updated input medical imaging data via the display may occur in real time or near real time, such that the display is configured to output the live position of the distal end of the medical device.

[0007] In some embodiments, the trained machine learning model may be configured to learn associations between a series of non-optical in vivo images of training non-optical in vivo image data and movement paths in training medical imaging data.

[0008] In some embodiments, the trained machine learning model may be configured to determine the position of the distal end of a medical device in the input medical imaging data by using input non-optical in vivo imaging data to predict the migration path of the distal end of the medical device from a previous position in the input medical imaging data.

[0009] In some embodiments, the operation may further include extracting at least one three-dimensional structure from input non-optical in vivo image data and registering at least one three-dimensional structure to the geometric shape of at least a portion of an anatomical structure from input medical imaging data. In some embodiments, the positioning of the distal end of a medical device may further depend on the registration to the geometric shape of at least one three-dimensional structure.

[0010] In some embodiments, the trained machine learning model may include one or more of the following: a long-term short-term memory network or a sequence-to-sequence model. In some embodiments, the operation may further include receiving a position signal from a position sensor positioned close to the distal end of the medical device. In some embodiments, the determination of the position of the distal end of the medical device may further depend on the position signal.

[0011] In some embodiments, the operation may further include using position signals to restrict the location of the distal end of the medical device to a region within a portion of the patient's anatomical structure. In some embodiments, using learned associations to determine the location of the distal end of the medical device in input medical imaging data may include using learned associations to identify the location of the distal end within a restricted region.

[0012] In some embodiments, the input non-optical in vivo image data may include 360-degree image data from a phased transducer array. In some embodiments, training may be configured to associate training non-optical in vivo image data with the diameter of the internal portion of an anatomical structure. In some embodiments, determining the position of the distal end of a medical device using the learned associations may include using the learned associations to determine the diameter of the internal portion of the patient's anatomical structure at the current position of the distal end of the medical device, comparing the current diameter with the geometric shape of the input medical imaging data to identify the position in the input medical imaging data that matches the determined diameter.

[0013] In some embodiments, the trained machine learning model may be configured to learn associations between a set of diameters determined based on training non-optical in vivo image data and movement paths in the training medical imaging data. In some embodiments, the trained machine learning model may be configured to determine the position of the distal end of a medical device in the input medical imaging data by using input non-optical in vivo image data to predict the movement path of the distal end of the medical device from a previous position in the input medical imaging data.

[0014] In some embodiments, the portion of the patient's anatomical structure may include the peripheral portion of the patient's lungs. In some embodiments, the input non-optical in vivo image data may include ultrasound data.

[0015] In some embodiments, the trained machine learning model may be configured to determine the position of the distal end of the medical device in input medical imaging data based on shape information associated with the medical device received from further sensors of the medical device.

[0016] In another embodiment, exemplary embodiments of a method for providing in vivo navigation of a medical device may include receiving input medical imaging data associated with at least a portion of a patient's anatomical structure, receiving input non-optical in vivo imaging data from a sensor positioned on the distal end of a medical device to be advanced into the portion of the patient's anatomical structure, and using a trained machine learning model to determine the position of the distal end of the medical device in the input medical imaging data, wherein the trained machine learning model uses (i) training medical imaging data and training non-optical in vivo imaging data of at least a portion of the anatomical structure of one or more individuals, and (ii) training medical imaging data as ground truth. The method is trained on registration data that associates training non-optical in vivo image data with positions, the training is configured to cause the trained machine learning model to learn associations between training non-optical in vivo image data and training medical imaging data, the trained machine learning model is configured to use the learned associations to determine the position of the distal end of a medical device in the input medical imaging data based on the input non-optical in vivo image data, the method may include modifying the input medical imaging data to include a position indicator showing the determined position of the distal end of the medical device, and causing a display to output the modified input medical imaging data including the position indicator.

[0017] In some embodiments, the input non-optical intravitreal imaging data may include ultrasound data. In some embodiments, the portion of the patient's anatomical structure may include the peripheral portion of the patient's lungs.

[0018] In a further embodiment, an exemplary embodiment of a method for training a machine learning model to determine the output position of the distal end of a medical device in the patient's anatomical structure in the input medical imaging data, in response to receiving input medical imaging data and input non-optical in vivo imaging data from a sensor located at the distal end of the medical device, may include inputting training data into the machine learning model, the training data including training medical imaging data and training non-optical in vivo imaging data of at least a portion of the anatomical structure of one or more individuals; inputting ground truth into the machine learning model, which includes registration data relating the training non-optical in vivo imaging data to a position in the training medical imaging data; and using the training data and ground truth together with the machine learning model to learn associations between the training non-optical in vivo imaging data and the training medical imaging data that can be used by the machine learning model to determine the output position of the distal end of the medical device.

[0019] In some embodiments, the method may further include using training data and ground truth together with a machine learning model to learn associations between a series of training non-optical in vivo images and movement paths in training medical imaging data, thereby configuring the machine learning model to determine the position of the distal end of a medical device in the input medical imaging data by using input non-optical in vivo image data to predict the movement path of the distal end of the medical device from a previous position in the input medical imaging data.

[0020] In some embodiments, the training non-optical in vivo imaging data is ultrasound data. Please understand that both the above overview and the following detailed description are illustrative and descriptive only and do not limit the disclosed embodiments described in the claims. [Brief explanation of the drawing]

[0021] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate various exemplary embodiments and, together with the description, serve to explain the principles of the disclosed embodiments. [Figure 1] FIG. 1 shows an exemplary environment for training and / or using a machine learning model to provide in-vivo navigation of a medical device according to one or more embodiments. [Figure 2A] FIG. 2A shows an exemplary embodiment of a medical device that can be used in the environment of FIG. 1 according to one or more embodiments. [Figure 2B] FIG. 2B shows an exemplary ultrasonic image generated by a transducer operating in air. [Figure 2C] FIG. 2C shows another exemplary ultrasonic image generated by a transducer operating in vivo with a gap between the transducer and the surrounding tissue. [Figure 3] FIG. 3 shows a flowchart of an exemplary method for training a machine learning model to provide in-vivo navigation of a medical device according to one or more embodiments. [Figure 4A-1] FIG. 4A (FIGS. 4A-1 and 4A-2) shows a flowchart of an exemplary method for using a machine learning model trained to provide in-vivo navigation of a medical device according to one or more embodiments. [Figure 4A-2] FIG. 4A (FIGS. 4A-1 and 4A-2) shows a flowchart of an exemplary method for using a machine learning model trained to provide in-vivo navigation of a medical device according to one or more embodiments. [Figure 4B] FIGS. 4B and 4C show exemplary embodiments of navigation outputs generated by a navigation system according to one or more embodiments. [Figure 4C] FIGS. 4B and 4C show exemplary embodiments of navigation outputs generated by a navigation system according to one or more embodiments. [Figure 5]FIG. 5 shows an example of a computing device according to one or more embodiments. DETAILED DESCRIPTION

[0022] According to certain aspects of the present disclosure, methods and systems are disclosed for providing in vivo navigation of a medical device, such as an ablation device that is navigated within the periphery of a patient's lung. In certain medical procedures, it may be desirable to navigate a medical device to a location within the body. However, conventional navigation techniques may not be suitable. For example, conventional techniques may not be accurate enough in navigating to a target site and / or confirming that the target site has been reached. Also, conventional navigation techniques may rely on including a light source, camera, and / or lens, and as a result, the size of the medical device may become too large to be navigable in some target sites.

[0023] As described in more detail below, in various embodiments, a machine learning model is trained, such as through supervised or semi-supervised learning, to learn an association between non-optical in vivo image data received from sensors disposed at the distal end of a medical device and medical imaging data, such as preoperative CT scan data, and based on this, systems and methods for identifying the position of the distal end of the medical device are described. The trained machine learning model can be used to provide navigation information regarding the medical device, such as a position indicator in medical imaging data indicating the live position of the distal end of the medical device.

[0024] Any reference to any specific procedure is provided for convenience in this disclosure and is not intended to limit the disclosure. Those skilled in the art will recognize that the concepts underlying the disclosed devices and methods may be used in any appropriate procedure. This disclosure can be understood by referring to the following description and accompanying drawings, where similar elements are referred to by the same reference numerals.

[0025] The terms used herein, even when used in conjunction with the detailed descriptions of certain examples in this disclosure, should be interpreted in their broadest and most reasonable manner. Indeed, any term that is intended to be interpreted restrictively, although certain terms may be emphasized below, is so explicitly and specifically defined in this detailed description section. Both the general descriptions above and the detailed descriptions below are illustrative and descriptive only and do not limit the features claimed.

[0026] For ease of explanation, parts of a device and / or its components are referred to as the proximal and distal portions. Note that the term “proximal” is intended to refer to the part of the device closer to the user, while the term “distal” is used herein to refer to the part further away from the user. Similarly, “extending distally” indicates that a component extends in the distal direction, and “extending proximally” indicates that a component extends in the proximal direction.

[0027] In this disclosure, the term “based on” means “at least partially based on.” The singular forms “a,” “an,” and “the” refer to multiple objects unless the context indicates otherwise. The term “exemplary” is used to mean “example” rather than “ideal.” The terms “equip,” “equip,” “include,” “contain,” or other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, or product comprising a list of elements does not necessarily contain only those elements, but may include other elements not explicitly enumerated, or other elements specific to such a process, method, article, or apparatus. The term “or” is used disjunctively, such that “at least one of A or B” includes (A), (B), (A and A), (A and B), etc. Relative terms such as “substantially” and “generally” are used to indicate a possible variation of ±10% of the stated or understood values.

[0028] Where used herein, terms such as “medical imaging data” generally encompass data associated with and / or illustrating the geometric shape and / or physiological function of a patient, which may be generated, for example, through medical imaging, and / or represented as images of a patient’s anatomical structure, e.g., two-dimensional images, three-dimensional images or models, videos, time-varying images, etc. Medical imaging generally encompasses techniques in which signals (light, electromagnetic energy, radiation, etc.) are generated and measurements are taken that show how those signals interact with and / or are affected by the patient, how they penetrate the patient, etc. Examples of medical imaging techniques include CT scans, MRI scans, X-ray scans, or any other appropriate modality that may be used, for example, to visualize the interior of at least a portion of a patient’s anatomical structure. Medical imaging data may include, for example, two-dimensional data and / or images, three-dimensional data and / or images, voxel data, geometric models of at least a portion of a patient's anatomical structure, solid models of portions of a patient's anatomical structure, meshes of nodes or points representing the properties of portions of anatomical structure and / or portions of anatomical structure, and / or any other appropriate data associated with the patient and / or medical imaging.

[0029] As used herein, “non-optical image data” generally includes data that represents, is associated with, and / or can be used to generate an image, and includes data generated using non-optical signals, for example, via signals generated by an ultrasonic transducer.

[0030] As used herein, “machine learning model” generally encompasses instructions, data, and / or models configured to take input and apply one or more of the following to the input: weights, biases, classifications, or analyses, to produce an output. The output may include, for example, a classification of the input, an analysis based on the input, a design, process, prediction, or recommendation associated with the input, or any other appropriate type of output. Machine learning models are generally trained using training data, e.g., empirical data and / or samples of input data, which are fed to the model to establish, adjust, or modify one or more aspects of the model, e.g., weights, biases, criteria for forming classifications or clusters, etc. Aspects of a machine learning model may operate linearly and in parallel with respect to the input, via a network (e.g., a neural network) or via any appropriate configuration.

[0031] Running a machine learning model may involve deploying one or more machine learning techniques, such as linear regression, logistic regression, random forest, gradient boosted machine (GBM), deep learning, and / or deep neural networks. Supervised and / or unsupervised training may be used. For example, supervised learning may involve providing training data and labels corresponding to the training data. Unsupervised approaches may include clustering, classification, etc. K-means clustering or K-Nearest Neighbors may also be used, which may be supervised or unsupervised. A combination of K-Nearest Neighbors and unsupervised clustering techniques may also be used. For example, any appropriate type of training can be used, such as stochastic, gradient boosted, random seeded, recursive, epoch-based, or batch-based.

[0032] In certain medical procedures, such as the ablation of unwanted tissue, it may be desirable to navigate a medical device to a target site within the patient's body, for example, the periphery of the patient's lungs. However, conventional in vivo navigation techniques, such as conventional bronchoscopy, may not be sufficiently accurate in navigating to the target site or verifying that the target site has been reached.

[0033] Concerns regarding bronchoscopy techniques include not only the ability to accurately navigate to the target for ablation, but also the ability to confirm that the target has been reached. Generally, during such procedures, medical imaging, such as CT images of the patient acquired before or during the procedure, can be used as a passive map in conjunction with active navigation via the bronchoscope. Bronchoscopy techniques for such purposes include navigation (electromagnetic and / or video) bronchoscopy, radial probe intrabronchial ultrasound, and robotic bronchoscopy.

[0034] However, conventional navigation technologies, including those mentioned above, cannot adequately address the problems of accurate navigation to the target and target confirmation. Generally, electromagnetic navigation alone is insufficient to provide fine detail and / or confirm that the target site has been reached. Video navigation can be used to confirm arrival at the target site, but the information provided is limited, and furthermore, because bronchoscopes require a light source, camera, and lens, the device diameter can be increased to a size that is too large to reach the desired portion around the lung. Radial probes require the device to rotate for imaging, which may pose a risk to the patient and / or may not be possible depending on the patient's physiological function and / or the position of the device. Also, robotic bronchoscopy generally involves large and / or complex machines located outside the patient that operate in conjunction with the procedure, such as CT scanners, which can be cumbersome and / or expensive. Conventional technologies may be too large for medical devices to navigate to the target site, potentially posing a risk to the patient, and / or may require large, complex, and / or expensive external machines to operate in conjunction with the procedure. Therefore, improvements in in-vivo navigation technology for medical devices are needed.

[0035] The following description will illustrate embodiments with reference to the accompanying drawings. Systems and methods for providing in vivo navigation of medical devices will be described in various embodiments, as will be discussed in more detail below.

[0036] In exemplary use cases, a medical procedure involves introducing and / or advancing a medical device to a target site within a patient's body. Medical imaging, such as a CT scan, can be acquired for at least a portion of the patient's body, including the target site, for example, before and / or during the procedure. The location of the target site in the patient's body can be identified in the medical imaging. The medical device can be introduced into the patient's body so as to be advanced toward the target site. For example, the medical device may be introduced through the patient's airway so as to be advanced toward a target site in the patient's lungs. The medical device may include an end effector, such as an ablation device, positioned on the distal end of the medical device to perform a therapeutic procedure. The medical device may further include sensors positioned on the distal end of the medical device, such as a transducer configured to generate signals indicating an ultrasound medical image. In some examples, the medical device may not include one or more of a camera, a light source, or a lens. A navigation system may be configured to receive signals generated by the sensors, for example, by receiving non-optical in vivo image data from the sensors. The navigation system may include a trained machine learning model configured to determine the position of the distal end of a medical device within a patient's body in medical imaging data, based on non-optical in vivo image data received from sensors. The navigation system can modify the medical imaging data to include position indicators showing the position of the distal end of the medical device within the patient's anatomical structures indicated by the medical imaging data, and can output the modified medical imaging data to a display. For example, the display can depict the live position of the distal end of the medical device in the medical imaging data as the medical device moves within the patient's anatomical structures.

[0037] In another exemplary use case, a machine learning model may be trained to determine the position of the distal end of a medical device being advanced into a patient's anatomical structure within medical imaging data. Training data, including medical imaging data and non-optical in vivo imaging data of at least a portion of the anatomical structures of one or more individuals, may be input to the machine learning model. Ground truth, including registration data relating non-optical in vivo imaging data to positions within the medical imaging data, may also be input to the machine learning model. The training data and ground truth can be used with the machine learning model to develop associations between non-optical in vivo imaging data and medical imaging data that can be used by the machine learning model to determine the output position of the distal end of the medical device.

[0038] In some cases, ground truth can be developed at least partially using additional navigation techniques. For example, training data and ground truth can be acquired using medical devices that include sensors such as ultrasound transducers, and also optical sensors such as cameras. Video bronchoscopy may be used to determine and / or validate the position of medical devices in order to generate ground truth associations between signals from sensors and the position of medical devices in medical imaging data. Furthermore, training of machine learning models can be validated by comparing the positions determined through the trained models with the positions determined through video bronchoscopy.

[0039] While some of the above examples involve ultrasound, it should be understood that the techniques described herein can be adapted to any suitable type of non-optical imaging. For example, pressure sensors, medical imaging techniques, etc., can be used to determine pressure, temperature, or other biological or physiological characteristics within a patient's body. The medical device may include, in place of or in addition to, the aforementioned sensors, further sensors configured to sense one or more of such biological or physiological characteristics. One or more determined characteristics within the body, and one or more determined characteristics sensed by the further sensors, can be used as input for a machine learning model. Furthermore, while some of the examples above involve bronchoscopy and / or navigation and / or ablation of tissues within the periphery of the lungs, it should be understood that the techniques described herein may be adapted to any appropriate procedure involving in vivo navigation of a medical device, including, for example, procedures on the heart or heart valves, any procedure in the lungs, gastrointestinal tract, urinary tract, or other body tracts, any procedure using an endoscope, bronchoscope, colonoscope, ureteroscope, or other similar device, and / or any therapeutic or diagnostic procedure, including, for example, biopsy, ablation, resection, incision, injection, application of drugs or therapeutic agents, or a combination thereof. It should also be understood that the examples above are illustrative only. The techniques and technologies of this disclosure may be adapted to any appropriate activity.

[0040] Various embodiments of machine learning techniques that can be adapted for in vivo navigation of medical devices are presented below. As will be discussed in more detail below, machine learning techniques adapted to determine the position and / or movement path of a medical device within the anatomical structure of a patient, with reference to medical imaging data, may include one or more embodiments of the present disclosure, such as a specific selection of training data, a specific training process for a machine learning model, the operation of a specific device suitable for use with the trained machine learning model, the operation of a machine learning model in combination with specific data such as medical imaging data, the modification of such specific data by the machine learning model, and / or other embodiments that may be apparent to those skilled in the art based on the present disclosure.

[0041] Figure 1 shows an exemplary environment 100 that may be used with the technology presented herein. One or more user devices 105, one or more medical devices 110, one or more displays 115, one or more medical providers 120, and one or more data storage systems 125 may communicate via an electronic network 130. One or more navigation systems 135 may communicate via the electronic network 130 with one or more of the other components of the environment 100, as will be described in more detail below. One or more user devices 105 may be associated with a user 140, for example, a user associated with one or more of the following: generating, training, or tuning machine learning models for providing in-vivo navigation of medical devices, generating, acquiring, or analyzing medical imaging data, and / or performing medical procedures.

[0042] In some embodiments, the components of environment 100 are associated with a common entity, such as a hospital or facility. In some embodiments, one or more of the components of the environment are associated with a different entity from another entity. The systems and devices of environment 100 can communicate in any configuration. As discussed herein, the systems and / or devices of environment 100 can communicate for one or more of the following activities, among others: generating, training, or using machine learning models to provide in vivo navigation for a medical device 110.

[0043] The user device 105 may be configured to allow the user 140 to access and / or interact with other systems within the environment 100. For example, the user device 105 may be a computer system such as a desktop computer, a mobile device, or a tablet. In some embodiments, the user device 105 may include one or more electronic applications installed on the user device 105's memory, such as programs, plugins, or browser extensions. In some embodiments, the electronic applications may be associated with one or more other components within the environment 100. For example, the electronic applications may include one or more system control software, system monitoring software, or software development tools.

[0044] Figure 2A shows an exemplary embodiment of the medical device 110. However, it should be understood that the embodiment in Figure 2 is illustrative only, and any suitable medical device for in vivo navigation to a target site may be used. The medical device 110 may include a distal end 205 connected to a proximal end 210 via a tube 215.

[0045] The distal end 205 may include one or more sections 220 configured to receive components or communicate with a lumen located in the tube 215. For example, at least one sensor 225 may be located in one of the sections 220. In another example, a tool having an end effector 230, such as an ablation device, forceps, net, or orifice for taking in or outputting fluid and / or material, may be located in another of the sections 220. The sensor 225 may include, for example, a transducer, an electromagnetic position sensor, an optical fiber position sensor, etc. In the embodiment shown in Figure 2, the sensor 225 includes a transducer array, but it should be understood that any suitable type of non-optical sensor may be used.

[0046] In some embodiments, the tube 215 may be formed from a flexible material. The tube 215 may include one or more lumens (not shown) communicating between the distal end 205 and the proximal end 210. In some embodiments, the tube 215 may further include and / or house components at the distal end 205, such as a sensor 225 and other elements such as a wire connector configured to communicate data between it and the proximal end 210.

[0047] The proximal end 210 may include, for example, a handle portion 245 that allows an operator to manipulate, advance, retract, and / or direct the distal end 205. The proximal end 210 may further include one or more interfaces 250, such as an umbilicus, for outputting data, transmitting or receiving electrical signals, and / or allowing fluid or material to enter or leave the medical device 110. The interfaces for data may include one or more wired or wireless connections. The interfaces 250 may be configured to receive power to operate the sensor 225 or the end effector 230.

[0048] In this embodiment, the medical device 110 does not include optical fiber lines and visual navigation elements such as lenses and cameras. As a result, the distal end 205, and in some embodiments, the tube 215, can have a smaller outer diameter compared to conventional medical devices such as bronchoscopes. For example, the medical device 110 can have an outer diameter suitable for navigation to the periphery of the lung, for example, a diameter of 3 millimeters or less.

[0049] In some embodiments, the medical device 110, or at least a portion thereof, is configured to be disposable, for example, a single-use device. By not including a visual navigation element, the costs associated with the disposal of the medical device 110 can be reduced compared to conventional medical devices.

[0050] Referring again to Figure 1, the display 115 may be configured to output information received from other systems in the environment 100. For example, the display 115 may be a monitor, tablet, television, mobile device, etc. In some embodiments, the display 115 may be integrated into another component of the environment, such as a user device 105.

[0051] The medical provider 120 may include and / or represent a person using a computer system, a computer system, and / or an entity using a computer system. For example, the medical provider 120 may include a medical imaging device such as a CT scanner, an entity such as a hospital or outpatient facility using the medical imaging device, and a medical data exchange system. The medical provider 120 may generate or otherwise acquire medical imaging data, for example by performing medical imaging on a patient, and / or perform analysis of the acquired medical imaging data. For example, the medical provider 120 may perform a CT scan on a patient to generate a three-dimensional model and / or two-dimensional images of at least some of the patient's anatomical structures. The medical provider 120 may also acquire any appropriate patient-specific information, such as age and medical history. The medical provider 120 may provide and / or provide access to the medical imaging data and / or any other data to one or more other components of the environment 100, for example, a navigation system 135, which will be discussed in more detail below.

[0052] The data storage system 125 may include a server system, an electronic medical data system, and computer-readable memory such as hard drives, flash drives, and disks. In some embodiments, the data storage system 125 includes and / or interacts with an application programming interface for exchanging data with other systems, e.g., one or more other components of the environment. The data storage system 125 includes and / or can function as a repository or source for medical imaging data. For example, medical imaging data obtained from a CT scan may be stored by the data storage system 125 and / or provided by the data storage system 125 to the navigation system 135, as will be described in more detail below.

[0053] In various embodiments, the electronic network 130 may be a wide area network ("WAN"), a local area network ("LAN"), a personal area network ("PAN"), and the like. In some embodiments, the electronic network 130 includes the Internet, and information and data provided between various systems occur online. "Online" may mean connecting to or accessing source data or information from a location remote from other devices or networks connected to the Internet. Alternatively, "online" may mean connecting to or accessing an electronic network (wired or wireless) via a mobile communication network or device. The Internet is a global system of computer networks, a network of networks in which parties in one computer or other device connected to the network can obtain information from any other computer and communicate with parties in other computers or devices. The most widely used part of the Internet is the World Wide Web (often abbreviated as "WWW" or simply "Web"). A “website page” generally encompasses location, data stores, etc., and may include data that is hosted and / or operated by a computer system to be accessible online, and configured to cause programs such as web browsers to perform actions such as sending, receiving, or processing data, and to generate visual displays and / or interactive interfaces, etc.

[0054] As will be described in more detail below, the navigation system 135 may, among other activities, (i) generate, store, train, or use a machine learning model configured to determine the position of the distal end 205 of the medical device 110; adjust the patient's medical imaging data based on the determined position of the distal end 205 to include a visual indicator of that position; or operate the display 115 to display the adjusted medical imaging data. The navigation system 135 may include the machine learning model and / or instructions associated with the machine learning model, such as instructions for generating the machine learning model, instructions for training the machine learning model, and instructions for using the machine learning model. The navigation system 135 may include instructions for retrieving medical imaging data, adjusting the medical imaging data based on the output of the machine learning model, and / or operating the display 115 to output the medical imaging data adjusted based on the machine learning model, for example. The navigation system 135 may include training data, such as medical imaging data and non-optical in vivo image data from one or more individuals, and may also include ground truth, such as registration data that associates non-optical in vivo image data with the location in the medical imaging data.

[0055] In some embodiments, non-optical image data includes ultrasound data. Ultrasound data generally includes data associated with the internal structure of a portion of the patient's anatomical structure, generated by the application of ultrasound to the patient's anatomical structure, thereby transmitting pulses of high-frequency vibrations into the tissue using a probe, e.g., an ultrasound transducer. The vibrations are reflected, at least partially, from surfaces, e.g., the geometric shape of structures or tissues, representing changes in acoustic impedance within the body. The reflected vibrations returning to the transducer can be transmitted, for example, via a wire in a tube 215 to a connector on the proximal end 210 and / or to a medical provider system 120 for processing into image data. The generation of image data is based on the time it takes for the reflections to return to the transducer after the application of vibrations and the intensity of the returned reflections. Conventional transducers are generally configured to receive variations in signal response across only one dimension. In other words, for a static position of the transducer, only one column of pixel data for an ultrasound image can be received. Therefore, to generate an image, the transducer is generally swept across the field of view and rotated, for example, back and forth, to continuously add and / or refresh columns of values ​​to the data.

[0056] Since data is collected based on received reflections, generally, in order to receive a signal at a given location, the transducer must be in contact with the surrounding tissue. However, this is not always the case, especially if the medical device has a diameter smaller than the size of the anatomical structure it is navigating. Gaps in air or gas between the transducer and the surrounding tissue generally cause signals from the transducer to be reflected back. Figure 2B shows an exemplary ultrasound image of a transducer operating in air, resulting in an essentially blank ultrasound image. Figure 2C shows an exemplary ultrasound image of a transducer operating in the airway, where a gap exists between the transducer and the surrounding tissue for at least part of the transducer's sweep. This gap results in an artifact known as a “ringdown” artifact in the image data, which is generally considered to reduce the diagnostic usability of the image data. However, the image data can still be used for navigation purposes even if such artifacts are present. For example, the presence of ring-down artifacts may indicate that the diameter of the lumen through which the medical device is moving is too large relative to the device, which could serve as an indication that the device has not advanced sufficiently to the periphery of the lung where the lumen diameter decreases. Further use of such imaging data will be discussed in more detail below.

[0057] As described above, in some embodiments, the medical device 110 includes a transducer array. The transducer array may include multiple transducers, for example, arranged parallel to each other and distributed over at least a portion of the outer circumference of the distal end. As a result, multiple rows of data can be sensed at once without rotating the sensor 225. In various embodiments, any suitable transducers in parallel may be used. Additional transducers effectively expand the static field of view of the sensor 225. In various embodiments, the sensor 225, the medical device 110, and / or another system are configured to control data acquisition using the sensor with parallel transducers and / or to control the coupling of signals from the parallel transducers into combined medical imaging data.

[0058] In some embodiments, a system or device other than the navigation system 135 is used to generate and / or train a machine learning model. For example, such a system may include instructions for generating a machine learning model, training data and ground truth, and / or instructions for training the machine learning model. The resulting trained machine learning model may then be provided to the navigation system 135.

[0059] Generally, a machine learning model includes a set of variables, such as nodes, neurons, and filters, that are adjusted to different values, for example, weighted or biased, through the application of training data. In supervised learning, for example, if the ground truth is known about the provided training data, training can proceed by feeding samples of the training data to a model that has variables set to initialized values, for example, based on Gaussian noise or a pre-trained model. The output can be compared to the ground truth to determine the error, and the error can then be backpropagated through the model to adjust the values ​​of the variables.

[0060] Training may be performed in any suitable way, e.g., in batches, and may include any suitable training method, e.g., stochastic or non-stochastic gradient descent, gradient boosting, random forest, etc. In some embodiments, a portion of the training data may not be provided during training and / or may be used to validate the trained machine learning model, for example, by comparing the output of the trained model to the ground truth for that portion of the training data in order to evaluate the accuracy of the trained model. Training of a machine learning model may be configured to cause the machine learning model to learn associations between non-optical in vivo image data and medical imaging data, so that the trained machine learning model is configured to determine the output location in the input medical imaging data in response to input medical imaging data and input non-optical in vivo image data, based on the learned associations.

[0061] As described above, machine learning models may be configured to receive medical imaging data and non-optical in vivo imaging data as input. Such data can generally be represented as arrays of pixels or voxels. For example, a monochrome two-dimensional image may be represented as a two-dimensional array of values ​​corresponding to the intensity of pixels in the image. Three-dimensional imaging data obtained from, for example, a CT scan, may be represented as a three-dimensional array. The variables of the machine learning model perform operations on the input data to produce an output. The output may be a location within the medical imaging data, e.g., a three-dimensional coordinate or data indicating it, as described above. It should be understood that the number of dimensions of the data described above are illustrative, and any appropriate type of data, e.g., data with a time component, can be used.

[0062] In various embodiments, the variables of a machine learning model may be related to each other in any suitable array to produce an output. For example, in some embodiments, the machine learning model may include an image processing architecture configured to identify, separate, and / or extract features, geometric shapes, and / or structures in one or more medical imaging data and / or non-optical in vivo imaging data. For example, the machine learning model may include one or more convolutional neural networks ("CNNs") configured to identify features in medical imaging data and / or non-optical in vivo imaging data, and may include further architectures, such as connection layers and neural networks, configured to determine the relationships between the identified features in order to determine their location in the medical imaging data.

[0063] In some examples, different samples of training data and / or input data may not be independent. For example, as the distal end 205 of the medical device 110 moves within the patient's anatomical structure, non-optical in vivo image data sensed by sensor 225 at the current position may be related to non-optical in vivo image data sensed by sensor 225 at the previous position. In other words, one or more geometric features of the patient's anatomical structure and factors such as the continuous movement of the distal end 225 may result in non-optical in vivo image data sensed by sensor 225 being related to the continuous instants. Therefore, in some embodiments, the machine learning model may be configured to explain and / or determine the relationships between multiple samples.

[0064] For example, in some embodiments, the machine learning model of the navigation system 135 may include a recurrent neural network ("RNN"). Generally, an RNN is a type of feedforward neural network suitable for processing a sequence of inputs. In some embodiments, the machine learning model may include a long short-term memory ("LSTM") model and / or a sequence-to-sequence ("Seq2Seq") model. The LSTM model may be configured to produce an output from samples that take into account at least some previous samples and / or outputs. The Seq2Seq model may, for example, receive a series of non-optical in vivo images as input and produce a series of locations in medical imaging data, e.g., a path, as output.

[0065] Although shown as separate components in Figure 1, it should be understood that, in some embodiments, components or parts of components of environment 100 may be integrated with or incorporated into one or more other components. For example, part of the display 115 may be integrated with an entity user device 105 or a computer system associated with a medical provider 120. In another example, the navigation system 135 may be integrated with a medical provider system 120 and / or a data storage system 125. In some embodiments, the operation or manner of one or more of the components described above may be distributed among one or more other components. Any suitable arrangement and / or integration of the various systems and devices of environment 100 may be used.

[0066] Further aspects of the machine learning model, and / or how the machine learning model may be used in conjunction with the medical device 110 and / or medical procedure to navigate the medical device to a target site within the patient's anatomical structure, will be described in more detail in the following manner. In the following manner, various operations may be described as being performed or carried out by components from Figure 1, such as the navigation system 135, the user device 105, the medical device 110, the display 115, the medical provider system 120, or their components. However, it should be understood that in various embodiments, various components of the environment 100 described above may execute instructions or perform operations including those described below. Operations performed by a device may be considered to be performed by a processor, actuator, etc. associated with that device. Furthermore, it should be understood that in various embodiments, various steps may be added, omitted, and / or rearranged in any appropriate manner.

[0067] Figure 3 illustrates an exemplary process for training a machine learning model to determine the output position of the distal end 205 of the medical device 110 in the patient's anatomical structure within the first medical imaging data, in response to the input of first medical imaging data and first non-optical in vivo imaging data received from a sensor located at the distal end of the medical device, as in the various examples described above. In step 305, the medical provider 120 can acquire medical imaging data for one or more individuals. For example, the medical provider 120 may perform a CT scan of a portion of the anatomical structure of one or more individuals, e.g., the peripheral portion of an individual's lung, and / or read such medical imaging data from another source, e.g., a data storage system 115, or another entity such as a hospital or outpatient facility, e.g., via an electronic medical database. In some embodiments, individuals may be classified based on one or more criteria, e.g., age, sex, height, weight, and / or any other appropriate demographic data. In some embodiments, the individual may not be a human. For example, training data can be generated from animal studies using species that have at least some anatomical similarity to humans, such as pigs. In general, acquired medical imaging data can be used to depict visual representations of parts of each individual's anatomical structure.

[0068] In step 310, the medical provider 120 can acquire non-optical in vivo imaging data of at least a portion of the anatomical structures of one or more individuals. For example, the medical provider 120, such as a physician or operator, can introduce a medical device, such as a medical device 110, into the body of one or more individuals and acquire non-optical in vivo imaging data as the distal end of the medical device is navigated to a target site within each individual's body.

[0069] In some embodiments, non-optical in vivo imaging data is ultrasound data. In some embodiments, the medical device includes a transducer array so that the ultrasound data is received without requiring a distal sweep or rotation. In some embodiments, the non-optical in vivo imaging data includes image data associated with at least a portion of the surrounding area inside an anatomical structure. For example, in some embodiments, the field of view of the non-optical in vivo imaging data may be 30 degrees, 90 degrees, 180 degrees, 360 degrees, etc. In some embodiments, the transducer array is configured to continuously acquire data so that, for example, a continuous sequence of data values ​​is acquired for each segment of the transducer array as the transducer array moves within the anatomical structure of the individual.

[0070] In step 315, the medical provider 120 can acquire positional information associated with the position of the distal end of the medical device when non-optical image data is acquired. Any suitable type of positional information can be used. In some embodiments, the distal end of the medical device 110 may include an electromagnetic position sensor, for example, which uses one or more electromagnetic signals to determine the three-dimensional position of the position sensor. In some embodiments, the distal end of the medical device may include an optical navigation element, such as a camera, optical fiber, or lens, which allows the medical provider 120 to visually inspect the position of the distal end of the medical device within the patient's anatomical structure and input such data, for example, via a user device 105. In some embodiments, the medical device 110 may include an optical fiber shape sensing mechanism. In some embodiments, the positional information includes shape information associated with the shape of the medical device 110. In some embodiments, an external scanner, such as a CT scanner or X-ray scanner, may be operated in conjunction with the movement of the medical device within the individual and may be used to determine the position of the distal end within the individual. While the medical device in this method may utilize some of the aforementioned optical navigation elements and techniques for generating training data, it should be understood that, as will be discussed in more detail below, even if the training data used to train the model is collected using a medical device employing optical navigation, such elements or techniques do not need to be used during the procedure using the trained machine learning model.

[0071] In some embodiments, the medical provider 120 may acquire additional data in addition to and / or based on medical imaging data and / or non-optical in vivo image data. For example, in some embodiments, the medical provider 120 may extract at least one three-dimensional structure from the medical imaging data and / or non-optical in vivo image data. For example, the medical provider 120 may generate a three-dimensional model based on the medical imaging data.

[0072] In step 320, the navigation system 135 can receive acquired medical imaging data, non-optical in vivo imaging data, location information, and optionally additional data, and can generate registration data that associates the locations where the non-optical in vivo imaging data was acquired with locations in the medical imaging data. In some embodiments, generating registration data may include registering an individual's anatomical structures with the medical imaging data and / or a generated three-dimensional model, and then associating the locations where the non-optical in vivo imaging data was acquired with corresponding locations in the registered medical imaging data. In some embodiments, the medical provider may register the locations of structures extracted from the medical imaging data with similar structures extracted from the non-optical in vivo imaging data. Any appropriate measure of structural similarity may be used. In some embodiments, the medical provider 120 may be configured to receive user input, for example, to set, adjust, or fine-tune the location information for the medical imaging data. For example, in some embodiments, the display 115 may output medical imaging data in conjunction with the output of an optical navigation element, allowing the user to set, select, adjust, or synchronize the current position of the distal end of the medical device in the medical imaging data. In some embodiments, the shape of the medical device 110 may be registered to the geometric shape of the medical imaging data. The above are merely examples, and any suitable technique for registering medical imaging data and non-optical in vivo imaging data using positional information may be used.

[0073] In step 325, the navigation system 135 may input medical imaging data and non-optical in vivo imaging data of at least a portion of the anatomical structures of one or more individuals as training data into the machine learning model. In some embodiments, the training data is input in batches. In some embodiments, at least a portion of the training data is not provided to the machine learning model to be used as validation data. In some embodiments, the training data is input as separate sequences corresponding to each of one or more individuals.

[0074] In step 330, the navigation system 135 can input registration data into the machine learning model as ground truth. In some embodiments, step 330 is performed simultaneously with, in parallel with, or sequentially with step 325, for example, alternately.

[0075] In step 335, the navigation system 135 can use the training data and ground truth together with the machine learning model to develop associations between non-optical in vivo image data and medical imaging data that can be used by the machine learning model to determine the output position of the distal end of a medical device. For example, the navigation system 135 can use the machine learning model to determine the error between the training data and ground truth for each sample of the training data, a batch of the training data, etc., and backpropagate the error to tune one aspect of the machine learning model. By tuning aspects of the machine learning model, such as variables, weights, biases, nodes, neurons, etc., the machine learning model is trained to learn associations between non-optical in vivo image data and medical imaging data that can be used by the machine learning model to determine the output position of the distal end of a medical device.

[0076] In some embodiments, by learning associations, the machine learning model may be configured to determine the position of the distal end 205 of the medical device 110 in the first medical imaging data by using the first non-optical in vivo imaging data to predict the movement path of the distal end 205 of the medical device 110 from a previous position in the first medical imaging data. For example, in some embodiments, the machine learning model and / or navigation system 135 may be configured to track and / or store the position of the distal end 205 over time and / or determine the current position of the distal end 205 based on its previous position. In exemplary embodiments, the machine learning model may include one or more long-term short-term memory networks or sequence-to-sequence models, as discussed in one or more of the examples above.

[0077] In some embodiments, the machine learning model is configured to learn an association between the shape of the medical device 110, for example, the shape that changes over time as the medical device 110 is moved, and the position of the distal end 205 in the medical imaging data.

[0078] In some embodiments, a machine learning model is configured to learn an association between a set of dimensions or measurements determined based on non-optical in vivo image data, such as diameters (e.g., cross-sectional diameters of body lumens), and a migration path within the medical imaging data. For example, in some embodiments, a medical provider 120 and / or a navigation system 135 can determine the diameter of a location in the medical imaging data and / or the diameter for non-optical in vivo image data, and such determined diameters can be used as further input to a machine learning model. In some embodiments, the diameter of a location in the medical imaging data may be determined based on the geometric shape of the part of an anatomical structure associated with the medical imaging data. In some embodiments, the diameters of locations in the medical imaging data and non-optical in vivo image data are used as training data, and the determined diameters of locations are used as a ground truth for a machine learning model and / or another machine learning model configured to output diameters in response to inputs of non-optical in vivo image data. Any suitable technique for determining the diameter of internal parts of anatomical structures based on non-optical in vivo image data may be used. While some of the embodiments described above relate to diameter, it should be understood that in at least some embodiments, the dimensions or measurements are not limited to circles or approximations of circles, and any suitable dimensions, measurements, and / or geometric shapes may be used.

[0079] Optionally, in step 340, the navigation system 135 may validate the trained machine learning model. For example, the navigation system 135 may input training data, e.g., a portion of the training data not provided to the machine learning model during training, as validation data, and use the trained machine learning model to generate output locations in the medical imaging data. The navigation system 135 can then generate the accuracy of the trained machine learning model by comparing the generated output locations with the locations from the ground truth registration data corresponding to the input validation data. For example, the navigation system 135 may determine the accuracy based on the average distance between each location in the output and the corresponding location in the registration data. Any appropriate accuracy metric can be used. The navigation system 135 may validate or reject the training of the machine learning model based on whether the accuracy is above or below a predetermined threshold.

[0080] Figure 4 illustrates an exemplary process for providing in vivo navigation of a medical device by utilizing a trained machine learning model, such as a machine learning model trained according to one or more embodiments described above. In step 405, the navigation system 135 can receive first medical imaging data associated with at least a portion of the patient's anatomical structure. The first medical imaging data may be associated with the patient's CT scan, for example. The first medical imaging data may be received from the data storage system 125. For example, the first medical imaging data may have been acquired at a previous time, for example, before surgery, via the patient's medical imaging. The first medical imaging data may be received from the medical provider 120, for example, from a medical imaging scanning device such as a CT scanner operating in conjunction with the method. At least a portion of the anatomical structure may be the periphery of the patient's lungs. The first medical imaging data can identify target sites within the patient's anatomical structure, such as the location of undesirable tissue to be ablated, the location of disease or illness such as a lesion, a foreign body, or any other appropriate medically relevant location.

[0081] In step 410, the medical provider 120 can insert the distal end 205 of the medical device 110 into the patient's body and advance the distal end 205 toward the target site. For example, the medical provider 120 can insert the distal end 205 into the patient's body bronchially, endoscopically, laparoscopically, or via any other suitable technique.

[0082] In step 415, the navigation system 135 can receive first non-optical in vivo image data from a sensor located at the distal end 205 of the medical device 110. The first non-optical in vivo image data may include ultrasound data. The sensor may include an ultrasound transducer. The ultrasound transducer may be a transducer array. The first non-optical in vivo image data may include non-optical image data extending over a circumferential sweep with a field of view such as 30 degrees, 90 degrees, 180 degrees, or 360 degrees, thereby enabling the acquisition of a field of view without sweeping or rotating the sensor. The first non-optical image data may be received, for example, via an interface on the proximal end 210 of the medical device 110.

[0083] Optionally, in step 420, the navigation system 135 may receive a position signal from a position sensor positioned close to the distal end 205 of the medical device 110, for example, via an interface. The position signal may contain information that can be used to localize the position of the position sensor to a given area. For example, the position signal may contain three-dimensional position information with an accuracy of about 6 inches, 3 inches, 1 inch, etc. In various embodiments, the position sensor may include, for example, an electromagnetic position sensor, an optical fiber shape sensing mechanism, or a combination thereof. In some embodiments, the position signal may contain information related to the shape of the medical device 110, etc.

[0084] Optionally, in step 425, the navigation system 135 may extract one or more three-dimensional structures from one or more of the first medical imaging data or the first non-optical in vivo imaging data. In some embodiments, the data received in step 405 may include one or more extracted structures, such as a geometric three-dimensional model of the patient's anatomical structure. In some embodiments, the extracted structures include the diameter of the internal portion of the patient's anatomical structure.

[0085] In step 430, the navigation system 135 can determine the position of the distal end 205 of the medical device 110 in the first medical imaging data using a trained machine learning model, for example, a model trained according to the method of Figure 3 and / or other embodiments described above. For example, the trained machine learning model may be trained on (i) second medical imaging data and second non-optical in vivo imaging data of at least a portion of the anatomical structures of one or more individuals as training data, and (ii) registration data as ground truth relating the second non-optical in vivo imaging data to a position in the second medical imaging data. The training may be configured to cause the trained machine learning model to learn associations between non-optical in vivo imaging data and medical imaging data, thereby configuring the trained machine learning model to determine an output position in the input medical imaging data in response to the input medical imaging data and input non-optical in vivo imaging data, based on the learned associations. In some embodiments, the trained machine learning model includes one or more of a long short-term memory network or a sequence-to-sequence model.

[0086] In some embodiments, the navigation system 135 can use position signals to restrict the location of the distal end of a medical device to a region within a portion of the patient's anatomical structure. In some embodiments, the navigation system 135 restricts the first medical imaging data input to a trained machine learning model to only the restricted region. In some embodiments, the navigation system 135 inputs the restricted region as further input to the trained machine learning model. In some embodiments, the trained machine learning model is further configured to receive position signals as input. In some embodiments, the position signals include one or more of three-dimensional coordinates, three-dimensional regions or volumes, such as the shape of the medical device 110 associated with an optical fiber shape sensor.

[0087] In some embodiments, the navigation system 135 is configured to register at least one structure extracted from first non-optical in vivo imaging data to the geometric shape of at least a portion of the patient's anatomical structure in the first medical imaging data, e.g., at least one structure extracted from the first medical imaging data. In some embodiments, the positioning of the distal end 205 of the medical device 110 is further based on the registration to the geometric shape of at least one three-dimensional structure. For example, the registration and / or one or more extracted structures may be used as further input to a trained machine learning model. In another example, the registration can be used to identify a limited region of an anatomical structure for positioning.

[0088] In some embodiments, a trained machine learning model is trained to learn associations between non-optical in vivo image data and dimensions or measurements, such as the diameter of an internal part of an anatomical structure. In some embodiments, the navigation system 135 can use the trained machine learning model to determine the diameter of an internal part of a patient's anatomical structure at the current position of the distal end 205 of the medical device 110. In some embodiments, the navigation system 135 can, for example, compare the current diameter to the geometric shape of medical imaging data to identify a position in the medical imaging data that matches the determined diameter, in order to determine the position of the distal end 205 of the medical device 110.

[0089] In some embodiments, a trained machine learning model is trained to learn associations between a series of non-optical in vivo images of non-optical in vivo image data and movement paths in medical imaging data. In some embodiments, the trained machine learning model is configured to determine the position of the distal end 205 of a medical device 110 in first medical imaging data by using first non-optical in vivo image data to predict the movement path of the distal end 205 of the medical device from a previous position in first medical imaging data. For example, in some embodiments, the trained machine learning model may be configured to accept a series of non-optical in vivo images of first medical imaging data and first non-optical in vivo image data as input and to generate a series of positions, e.g., paths, in the first medical imaging data as output.

[0090] In some embodiments, the trained machine learning model is trained to learn associations between a set of dimensions or measurements, such as diameter determined based on non-optical in vivo image data, and movement paths in medical imaging data. The trained machine learning model may be configured to determine the position of the distal end 205 of the medical device 110 in the first medical imaging data by using the first non-optical in vivo image data to predict a sequence showing the movement path of the distal end 205 of the medical device 110 from a previous position in the first medical imaging data.

[0091] In step 435, the navigation system 135 may modify the first medical imaging data to include a position indicator showing the determined position of the distal end of the medical device. In various embodiments, the position indicator may include one or more graphics or objects indicating the position of the distal end 205, e.g., geometric shapes such as arrows and circles, or graphics or objects indicating the path of the distal end 205, e.g., solid lines, dashed lines, coloring of the portion(s) of the first medical imaging data traversed. In some embodiments, the position indicator includes a representation of the medical device 110 and / or the distal end 205 in the medical imaging data. In some embodiments, the first medical imaging data may be further modified to include a depiction of at least one structure extracted from one or more of the first medical imaging data or the first non-optical in vivo imaging data. In some embodiments, the first medical imaging data may be further modified to include a rendering or three-dimensional model of the patient's anatomical structure at the current position of the distal end 205. In some embodiments, the first medical imaging data may be further modified to include images generated based on the first non-optical in vivo imaging data. In some embodiments, the first medical imaging data may be further modified to include a visual depiction of additional data, such as the distance between the distal end 205 and the target site, or confirmation that the target site has been reached by the distal end 205, as determined, for example, by the navigation system 135.

[0092] In step 440, the navigation system 135 can cause the display 115 to output modified first medical imaging data, including a position indicator. For example, the display 115 can display the first medical imaging data as a map of the patient's anatomical structure, along with a position indicator that identifies the current position of the distal end 205 of the medical imaging device.

[0093] Figures 4B and 4C show different exemplary embodiments of the output 475 that may be generated by the navigation system 135. As shown in Figures 4B and 4C, the output 475 generated by the navigation system 135 may include one or more of the following: medical imaging data 477 showing at least a portion of the patient's anatomical structure; location information 479 and / or route information 481 superimposed on the medical imaging data 477 and showing the current and / or past location of the medical device 110; and ultrasound image data 483 superimposed on the corresponding location in the medical imaging data.

[0094] In some embodiments, the navigation system 135, user device 105, touchscreen input of display 115, etc., are configured to receive input from user 140 and, for example, manipulate the viewpoint of first medical imaging data and include, move, adjust, and / or remove further information in the output, such as first non-optical in vivo image data or images generated based on the aforementioned additional data.

[0095] Returning to Figure 4A (Figures 4A-1 and 4A-2), optionally, in step 445, the medical provider 120 may move the medical device 110, thereby changing, for example, the position of the distal end 205 within the patient's anatomical structure.

[0096] Optionally, in step 450, one or more of steps 415 to 440 may be repeated to take into account, for example, the new position of the distal end 205 of the medical device 110. For example, the navigation system 135 may receive additional non-optical in vivo image data from the sensor and use a trained machine learning model to determine the updated position of the distal end 205 of the medical device 110 based on the additional non-optical in vivo image data. Furthermore, the navigation system 135 may update the first medical imaging data to adjust the position indicator based on the updated position of the distal end 205 of the medical device 110 and update the display 115 to output the updated first medical imaging data. In some embodiments, such iterations may be performed in real time or near real time, thereby configuring the display 115 to output the live position of the distal end 205 of the medical device 110.

[0097] Optionally, in step 455, the navigation system may be configured to output a target site confirmation message to the display 115 when the navigation system 135 determines that the distal end 205 has reached the target site.

[0098] Optionally, in step 460, the medical provider 120 may perform a procedure at the target site using the medical device 110. For example, the medical provider may activate an end effector 230, such as an ablation device, to excise tissue at the target site.

[0099] Optionally, in step 465, the navigation system 135 and / or the medical provider may confirm the completion of the procedure based on further non-optical in vivo imaging data received from the sensor located at the distal end 205. For example, the procedure may involve modifications to the geometric shape of the patient's anatomical structure. The navigation system 135 may be configured to extract one or more modified structures of the patient's anatomical structure and compare the modified(s) structures with previously extracted(s). For example, the first medical imaging data may identify tissue to be excised, and the navigation system 135 may be configured to identify whether the tissue has been excised or remains in the patient.

[0100] In step 470, the medical provider 120 can withdraw the medical device 110 from the patient's body. In some embodiments, the medical provider 120 may dispose of the medical device 110.

[0101] It should be understood that the embodiments in this disclosure are illustrative only, and other embodiments may include various combinations of features from other embodiments, as well as additional or fewer features. For example, some of the embodiments described above relate to ablation of tissue within the periphery of the lung. However, any suitable procedure may be used. Furthermore, while some of the embodiments described above relate to ultrasound, any suitable non-optical imaging modality or technique may be used. In the exemplary embodiments, instead of, or in addition to, a sensor, the medical device 110 includes an optical fiber light and a receiver optical fiber that can be used for position sensing of the distal end 205.

[0102] In general, any process or operation described in this disclosure that is understood to be computer implementable, such as the processes shown in Figures 3 and 4, can be executed by one or more processors in a computer system, such as any of the systems or devices in Environment 100 of Figure 1 as described above. A process or process step executed by one or more processors may also be called an operation. One or more processors may be configured to execute such a process by having access to instructions (e.g., software or computer-readable code) that cause one or more processors to execute the process, when executed by one or more processors. The instructions may be stored in the memory of the computer system. The processors may be a central processing unit (CPU), a graphics processing unit (GPU), or any suitable type of processing unit.

[0103] A computer system, such as a system or device that implements a process or operation in the above example, may include one or more computing devices, such as one or more of the systems or devices in Figure 1. One or more processors in a computer system may be contained in a single computing device or distributed among multiple computing devices. The memory of a computer system may include the memory of each of the multiple computing devices.

[0104] Figure 5 is a simplified functional block diagram of a computer 500 that may be configured as a device for performing the methods of Figures 3 and 4 according to exemplary embodiments of the present disclosure. For example, the computer 500 may be configured as a navigation system 135 and / or another system according to exemplary embodiments of the present disclosure. In various embodiments, any of the systems described herein may be a computer 500 including, for example, a data communication interface 520 for packet data communication. The computer 500 may also include a central processing unit ("CPU") 502 in the form of one or more processors for executing program instructions. The computer 500 may include an internal communication bus 508 and a storage unit 506 (such as ROM, HDD, SSD) on which data can be stored on a computer-readable medium 522, but the computer 500 may receive programming and data via network communication. The computer 500 may also have memory 504 (such as RAM) for storing instructions 524 for performing the techniques presented herein, although the instructions 524 may be temporarily or permanently stored in other modules of the computer 500 (e.g., a processor 502 and / or computer-readable media 522). The computer 500 may also include input / output ports 512 and / or a display 510 for connecting to input / output devices such as a keyboard, mouse, touchscreen, monitor, and display. Various system functions may be implemented in a distributed manner on multiple similar platforms to distribute the processing load. Alternatively, the system may be implemented by appropriate programming on a single computer hardware platform.

[0105] The programmatic aspects of this technology can typically be considered as “products” or “manufactured goods” in the form of executable code and / or associated data carried or embodied on a certain type of machine-readable medium. “Storage” type media include any or all of tangible memory such as computers, processors, or their associated modules, such as various semiconductor memories, tape drives, and disk drives, which can provide non-temporary storage for software programming at any time. All or part of the software may be communicated over the Internet or various other telecommunication networks. Such communication can enable, for example, the loading of software from one computer or processor to another, for example, from a management server or host computer on a mobile communication network to a server computer platform, and / or from a server to a mobile device. Therefore, other types of media that can carry software elements include light, electricity, and electromagnetic waves, such as those used across physical interfaces between local devices, through wired and optical terrestrial communication line networks, and through various air links. Physical elements that carry such waves, such as wired or wireless links and optical links, can also be considered media that carry software. As used herein, unless limited to non-temporary tangible “storage” media, terms such as computer or machine “readable media” refer to any medium involved in providing instructions to a processor for execution.

[0106] While the disclosed methods, devices, and systems are described with illustrative reference to data transmission, it should be understood that the disclosed embodiments are applicable to any environment, such as desktop or laptop computers, automotive entertainment systems, and home entertainment systems. Furthermore, the disclosed embodiments may be applicable to any type of Internet protocol.

[0107] In the above description of exemplary embodiments of the present invention, it should be understood that various features of the invention may be summarized in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various embodiments of the invention. However, this method of disclosure should not be interpreted as reflecting an intention that the claimed invention requires more features than expressly described in each claim. Rather, as reflected in the following claims, the embodiments of the invention have fewer features than all of the features of the single embodiment disclosed above. Accordingly, the claims following the detailed description are explicitly incorporated into this detailed description by this specification, and each claim stands independently as a separate embodiment of the invention.

[0108] Furthermore, some embodiments described herein include some features included in other embodiments, and do not include other features, but as will be understood by those skilled in the art, combinations of features of different embodiments are within the scope of the invention and are intended to form different embodiments. For example, in the following claims, any of the claimed embodiments may be used in any combination.

[0109] Therefore, although specific embodiments have been described, those skilled in the art will recognize that other modifications and further modifications can be made thereto without departing from the technical spirit of the invention, and that all such changes and modifications falling within the scope of the invention are intended to be claimed. For example, functions may be added to or removed from the block diagram, and operations may be exchanged between function blocks. Steps may be added to or removed from the methods described within the scope of the invention.

[0110] The subject matter disclosed above should be considered illustrative rather than restrictive, and the attached claims are intended to encompass all such modifications, enhancements, and other implementations that fall within the true technical spirit and scope of this disclosure. Therefore, to the maximum extent permitted by law, the scope of this disclosure should be determined by the broadest permissible interpretation of the following claims and their equivalents, and should not be limited or restricted by the detailed description above. While various implementations of this disclosure have been described, it will be apparent to those skilled in the art that many more implementations are possible within the scope of this disclosure. Therefore, this disclosure should not be limited except by consideration of the attached claims and their equivalents.

Claims

1. A system for providing in-vivo navigation for medical devices, It has memory to store instructions and trained machine learning models, The trained machine learning model is trained on (i) training medical imaging data and training non-optical in vivo imaging data of at least a portion of the anatomical structures of one or more individuals, and (ii) registration data relating the training non-optical in vivo imaging data to a location in the training medical imaging data as ground truth. The training is configured to cause the trained machine learning model to learn the association between the training non-optical in vivo image data and the training medical imaging data. The aforementioned system, The display and A processor configured to be operably connected to the display and the memory and to execute the instructions and perform operations. The operation is provided, Receiving input medical imaging data associated with at least a portion of the patient's anatomical structure, Receiving input non-optical in vivo image data from a sensor positioned on the distal end of a medical device that is advanced into the portion of the anatomical structure of the patient, Using learned associations, determine the position of the distal end of the medical device in the input medical imaging data based on the input medical imaging data and the input non-optical in vivo imaging data. Modifying the input medical imaging data to include a position indicator showing the determined position of the distal end of the medical device. The display will output the modified input medical imaging data, including the position indicator. A system that includes this.

2. The aforementioned operation is, As the medical device moves within the portion of the patient's anatomical structure, it receives further non-optical in vivo image data from the sensor. Using the learned associations, determine the updated position of the distal end of the medical device based on the further non-optical in vivo image data. Updating the input medical imaging data and adjusting the position indicator based on the updated position of the distal end of the medical device, To output updated input medical imaging data to the aforementioned display. The system according to claim 1, including the following:

3. The system according to claim 2, wherein the determination of the updated position, the updating of the input medical imaging data, and the output of the updated input medical imaging data via the display are performed in real time or near real time, and the display is configured to output the live position of the distal end of the medical device.

4. The system according to any one of claims 1 to 3, wherein the trained machine learning model is configured to learn associations between a series of non-optical in vivo images of the training non-optical in vivo image data and movement paths in the training medical imaging data.

5. The system according to claim 4, wherein the trained machine learning model is configured to determine the position of the distal end of the medical device in the input medical imaging data by using the input non-optical in vivo image data to predict the migration path of the distal end of the medical device from a previous position in the input medical imaging data.

6. The aforementioned operation is, Extracting at least one three-dimensional structure from the aforementioned non-optical in vivo image data input, Registering the at least one three-dimensional structure to the geometric shape of at least a portion of the anatomical structure from the input medical imaging data. It further includes, The system according to any one of claims 1 to 3, wherein the determination of the position of the distal end of the medical device is further based on the registration with the geometric shape of the at least one three-dimensional structure.

7. The system according to any one of claims 1 to 3, wherein the trained machine learning model includes one or more long-term short-term memory networks or sequence-to-sequence models.

8. The operation further includes receiving a position signal from a position sensor positioned in close proximity to the distal end of the medical device, The system according to any one of claims 1 to 3, wherein the determination of the position of the distal end of the medical device is further based on the position signal.

9. The operation further includes using the position signal to restrict the position of the distal end of the medical device to a region within the portion of the patient's anatomical structure, The system according to claim 8, wherein determining the position of the distal end of the medical device in the input medical imaging data using the learned associations includes identifying the position of the distal end within a limited region using the learned associations.

10. The system according to any one of claims 1 to 3, wherein the input non-optical in vivo image data includes 360-degree image data from a phased transducer array.

11. The learned association between the training non-optical in vivo image data and the training medical imaging data includes an association between the diameter of the internal portion of the anatomical structure contained in the training non-optical in vivo image data and the geometric shape of the anatomical structure contained in the training medical imaging data, Using the learned associations, determining the position of the distal end of the medical device is: Based on the input non-optical in vivo image data, determine the diameter of the internal portion of the patient's anatomical structure at the current position of the distal end of the medical device. The current diameter is compared with the geometric shape of the input medical imaging data to identify the position in the input medical imaging data that matches the determined diameter. The system according to any one of claims 1 to 3, including the system described in any one of claims 1 to 3.

12. The trained machine learning model is configured to learn associations between a set of diameters determined based on the training non-optical in vivo image data and the movement paths in the training medical imaging data. The system according to any one of claims 1 to 3, wherein the trained machine learning model is configured to determine the position of the distal end of the medical device in the input medical imaging data by using the input non-optical in vivo image data to predict the migration path of the distal end of the medical device from a previous position in the input medical imaging data.

13. The system according to any one of claims 1 to 3, wherein the portion of the anatomical structure of the patient includes the peripheral portion of the patient's lungs.

14. The system according to any one of claims 1 to 3, wherein the input non-optical intra vivo image data is ultrasound data.

15. The system according to any one of claims 1 to 3, wherein the trained machine learning model is configured to determine the position of the distal end of the medical device in the input medical imaging data based on shape information associated with the medical device received from further sensors of the medical device.

Images