Systems and methods for guiding medical devices using artificial intelligence

Abstract

Systems and methods for generating navigational guidance for a medical device in vivo are disclosed. A computer-implemented method may include receiving, at a computer server, image data associated with at least one anatomical object; determining, using a processor associated with the computer server and via applying a trained predictive navigational guidance model to the image data, navigational guidance for the medical device associated with the at least one anatomical object; generating at least one visual representation associated with the navigation guidance based on the determination; and transmitting instructions to a user device in network communication with the computer server to display the at least one visual representation associated with the navigation guidance over the image data on a display screen of the user device.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of priority under 35 U.S.C. §119 to U.S. Provisional Application No. 63 / 368,529, filed on July 15, 2022, which is incorporated herein by reference in its entirety. Technical Field

[0002] Aspects of the present invention generally relate to systems and methods for providing navigation guidance for medical devices that perform actions in vivo using artificial intelligence. More specifically, in an embodiment, the present invention relates to applying a trained machine learning model to data associated with providing predictive navigation guidance for physicians operating medical devices. Background Art

[0003] Certain medical procedures may be performed to examine and treat problems within the body. For example, during endoscopic surgery, a long, thin tube is inserted directly into the body to provide a detailed view of internal organs or tissues. This procedure may also be used to perform other tasks, including imaging and minor surgeries. In some endoscopic procedures, cannulation of various anatomical objects (e.g., one or more catheters, etc.) may need to be accomplished via insertion of endoscopic components (e.g., guidewires). This manipulation may be very challenging and may come with a steep learning curve. Therefore, it may take a very long time for a novice physician to become proficient in this procedure.

[0004] The present invention is directed to solving the challenges mentioned above. The background description provided herein is intended to generally present the background of the present invention. Unless otherwise indicated herein, by inclusion in this section, the materials described in this section are not prior art for the claims in this application and are not admitted to be prior art, or suggestions of prior art. Summary of the invention

[0005] Each of the aspects disclosed herein may include one or more of the features described in conjunction with any of the other disclosed aspects.

[0006] Aspects of the present invention relate to systems and methods for generating navigation guidance for medical devices operating in vivo, among other things. According to one example, a computer-implemented method for generating navigation guidance for a medical device in vivo is provided. The computer-implemented method includes: receiving image data associated with at least one anatomical object at a computer server; determining navigation guidance for the medical device related to at least one anatomical object using a processor associated with the computer server and applying a trained predictive navigation guidance model to the image data; generating at least one visual representation associated with the navigation guidance based on the determination; and transmitting instructions to a user device in network communication with the computer server to display at least one visual representation associated with the navigation guidance on a display screen of the user device above the image data.

[0007] Any of the computer-implemented methods for generating navigation guidance may include any of the following features and / or processes. The medical device may be an endoscope with an extendable guidewire. At least one anatomical object may correspond to one or more of the following: a nipple, an orifice, and / or an internal catheter. The navigation guidance may include a path of the medical device for cannulating the anatomical object. The image data may be captured by at least one sensor associated with the medical device and / or at least one other imaging device. At least one sensor may include a camera sensor, and the image data captured by the camera sensor may include at least one of the following: shape data, orientation data, and / or appearance data of at least one anatomical object. At least one other imaging device may include an X-ray device and / or an ultrasound device, and the image data captured by at least one other imaging device may include anatomical structure data. One or more other sensors may be utilized, including at least one of the following: an electromagnetic sensor, an accelerometer, a gyroscope, a fiber optic sensor, an ultrasound transducer, a capacitive position sensor, and / or an inductive position sensor. One or more other sensors may capture position data associated with the medical device. Determining the navigation guidance may include identifying anatomical feature data from the image data using a predictive navigation guidance model. Identifying anatomical feature data may include: identifying a first classification associated with a first anatomical object within a first target region of the image data; identifying a second classification associated with a second anatomical object from within a second target region bounded by the first target region; detecting the location of one or more third anatomical objects from within the second target region; and detecting one or more other anatomical objects associated with the first anatomical object. Determining navigation guidance for a medical device may include: identifying a confidence weight held by a predictive navigation guidance model for at least one anatomical object; and determining whether the confidence weight is greater than a predetermined confidence threshold; wherein the generation of navigation guidance is performed only in response to determining that the confidence weight is greater than the predetermined confidence threshold. The at least one visual representation may include one or more of: at least one trajectory overlay, at least one annotation, and / or at least one feedback notification. The at least one trajectory overlay may include a visual indication of a planned path to an entry point to at least one anatomical object superimposed on an image of the at least one anatomical object, and a component of the medical device may follow the planned path to cannulate the at least one anatomical object. The computer-implemented method may also receive position data for the medical device and identify a deviation of the medical device from the planned path based on an analysis of the position data. Generating the feedback notification in this case may be in response to detecting that the deviation of the medical device from the planned path is greater than a predetermined amount. At least one annotation may include one or more visual indications superimposed on the image of at least one anatomical object, which indicate predetermined features associated with the at least one anatomical object. The one or more visual indications may include one or more of the following: a color indication, an outline indication, and / or a text-based indication.

[0008] According to another example, a computer-implemented method for training a predictive navigation guidance model is provided. The computer-implemented method includes: receiving a training data set from a database, the training data set including historical medical surgery data associated with a plurality of completed medical surgeries; extracting anatomical feature data from image data in the training data set; extracting medical device positioning data from sensor data in the training data set; extracting surgical outcome data from the training data set; and training the predictive navigation guidance model using the extracted anatomical feature data, the extracted medical device positioning data, and the extracted surgical outcome data.

[0009] Any of the computer-implemented methods for training a predictive navigation guidance model may include any of the following features and / or processes. The training data set may be annotated with identification data. Extracting anatomical feature data may include: identifying a classification associated with a first anatomical object; determining an identity of a second anatomical object within a second target region bounded by a first target region in the image data; detecting at least one location associated with one or more third anatomical objects; and detecting one or more other anatomical objects associated with the first anatomical object. The computer-implemented method may also identify new surgical results and update the database with data associated with the new surgical results.

[0010] According to another example, a computer system for generating navigation guidance for a medical device in the body includes: at least one memory storing instructions; at least one processor configured to execute the instructions to perform operations, the operations including: receiving image data associated with at least one anatomical object; using the at least one processor and determining navigation guidance for the medical device related to the at least one anatomical object by applying a trained predictive navigation guidance model to the image data; generating at least one visual representation associated with the navigation guidance based on the determination; and transmitting instructions to a user device to display at least one visual representation associated with the navigation guidance on a display screen of the user device above the image data.

[0011] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention, as claimed. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate various aspects of the invention and, together with the description, serve to explain the principles of the invention.

[0013] Figure 1 depicts an exemplary environment for training and / or utilizing a machine learning model to apply to medical surgical data to generate predictive navigation guidance in accordance with one or more embodiments;

[0014] Figure 2 A flowchart depicting an exemplary method of training a machine learning model to provide predictive navigation guidance during a medical procedure in accordance with one or more embodiments;

[0015] Figure 3 A flowchart depicting an exemplary method of training a machine learning model to extract anatomical feature data in accordance with one or more embodiments;

[0016] Figure 4 depicts a plurality of views of different types of anatomical objects according to one or more embodiments;

[0017] Figure 5 depicting a plurality of views of different characteristic patterns of an anatomical object according to one or more embodiments;

[0018] Figure 6 depicts a plurality of views of differentiation of anatomical objects based on characteristic patterns of the anatomical objects in accordance with one or more embodiments;

[0019] Figure 7 depicts a flow chart of an exemplary method for providing predictive navigation guidance during a medical procedure using a trained machine learning model in accordance with one or more embodiments;

[0020] Figure 8 depicts an illustration of annotation-based prediction guidance in accordance with one or more embodiments;

[0021] Fig. 9 depicts an illustration of trajectory-based predictive guidance in accordance with one or more embodiments;

[0022] Fig.10 An example of a computing device is depicted in accordance with one or more embodiments.

[0023] The foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosed embodiments, as claimed. DETAILED DESCRIPTION

[0024] The terminology used below is to be interpreted in the broadest reasonable manner, even though it is used in conjunction with the detailed description of certain specific examples of the invention. Indeed, certain terms may even be emphasized below; however, any term intended to be interpreted in any limiting manner will be openly and specifically defined in the detailed description. The foregoing general description and the following detailed description are merely exemplary and illustrative of the features as claimed, and are not limiting descriptions.

[0025] In the present invention, the term "based on" means "based at least in part on". Unless the context dictates otherwise, the singular forms "a", "an" and "the" include plural referents. The term "exemplary" is used in the sense of "example" rather than "exemplary". The terms "include", "including", "contain", "include" or other variations thereof are intended to cover non-exclusive inclusions, so that a process, method or product including a list of elements does not necessarily include only those elements, but may include other elements that are not explicitly listed or inherent to such a process, method, article or device. In the case where an element is not circular, the term "diameter" may refer to the width. The term "top" refers to the direction or side of the device relative to its orientation during use, and the term "bottom" refers to the direction or side of the device relative to its orientation during use, which is opposite to the "top". The term "exemplary" is used in the sense of "example" rather than "exemplary". Relative terms, such as "substantially" and "approximately" are used to indicate possible variations of ±10% of the values ​​described or understood.

[0026] Reference to any particular surgery is provided in the present invention only for convenience and is not intended to limit the present invention. One of ordinary skill in the art will recognize that the underlying concepts of the disclosed devices and methods can be used in any suitable surgery. For ease of description, portions of the device and / or its components are referred to as proximal and distal portions. It should be noted that the term "proximal" is intended to refer to the portion closer to the user of the device, and the term "distal" is used herein to refer to the portion further away from the user. Similarly, "distal" extension indicates that the component extends in a distal direction, and "proximally" extension indicates that the component extends in a proximal direction.

[0027] In the following description, embodiments will be described with reference to the accompanying drawings. As will be discussed in more detail below, according to certain aspects of the present invention, methods and systems are disclosed for capturing information associated with one or more biological components using a medical device (e.g., during a medical procedure), comparing the captured information with a database of historical surgical data or applying a model trained on historical surgical data to the captured information, and then providing various types of guidance based on the results of the comparison and / or analysis.

[0028] Endoscopic retrograde cholangiopancreatography (ERCP) is a procedure routinely used to examine the bile duct. During the operation, an endoscope is inserted through the oral cavity and leads to the duodenum. The duodenum is then insufflated, and the entry point of the common duct of the bile duct and the pancreatic duct is identified. A sphincterotomy can be performed using a tracheotomy to widen the opening, making intubation easier to perform. A guide wire can then be used to enter the common duct, and the guide wire is manipulated to the bile duct. Once intubation of the duct has been achieved, a biliary endoscope can be inserted over the guide wire and inserted into the duct. Contrast agents can then be injected and used in conjunction with X-rays to identify areas of interest. Thereafter, the physician can perform various operations, such as stone management or the treatment of biliary malignancies.

[0029] Traditionally, cannulation of an appropriate duct may be very challenging for a variety of reasons. For example, the ergonomics of manipulating an 8-degree-of-freedom endoscope to enter a precise location may be difficult, even for experienced and skilled physicians. Additionally, as another example, the lack of visualization of the duct path outside the common entry point may exacerbate the difficulty of the task. More specifically, although physicians can obtain various types of visualization of the target area during the ERCP procedure (e.g., preoperative magnetic resonance cholangiopancreatography (MRCP), high-resolution imaging after cannulation, X-rays, preoperative CT scans, etc.), only direct visualization (i.e., as provided by an endoscope) is used specifically for the cannulation process. This limited visualization does not provide information about the anatomical structure of the duct outside the common entry point, which is particularly problematic because the anatomical architecture of the duct is patient-specific (i.e., the characteristics of each individual papilla that the guidewire needs to enter are different). Therefore, in view of the above challenges, a common surgical outcome is pancreatic duct disorder (e.g., caused by misplaced guidewires, etc., by physicians). In more severe cases, this disorder may lead to pancreatitis.

[0030] The inherent difficulty of endoscopic manipulation, coupled with the lack of proper visualization during intubation, results in a steep learning curve for physicians trying to become proficient at performing ERCP procedures. Furthermore, even after becoming proficient, physicians are required to continually perform these types of procedures to maintain their skill level (e.g., at least one ERCP procedure per week), which may be very demanding, burdensome, and / or infeasible (e.g., the physician may not be in an area where a large number of ERCP procedures are performed, etc.). Therefore, there is a need to simplify or modify the ERCP procedure so that more physicians can master the procedure in a shorter period of time, which may result in better patient care.

[0031] As will be discussed in more detail below, the present invention provides a platform that can provide dynamic guidance to a physician during a procedure, such as an ERCP procedure, by applying a predictive navigation guidance model (i.e., trained based on data associated with historical procedures stored in an accessible ERCP database) to data obtained and associated with a real-time medical procedure. More specifically, anatomical feature data (e.g., characteristics of a target nipple and / or the anatomical structure of an associated duct) can be extracted from image data associated with an endoscope and / or one or more other imaging modes (e.g., fluoroscopy, ultrasound, etc.) initially captured using one or more sensors (e.g., camera / video sensors, etc.). Additionally, in some embodiments, position data of a medical device (e.g., the position, angle, and / or movement of a medical device relative to a target anatomical object) can be captured using one or more other sensors (e.g., electromagnetic sensors, etc.). The accumulated real-time surgical data can then be submitted as input to a predictive navigation guidance model, which can then analyze the data to determine navigation guidance to manipulate an endoscope's guidewire through the appropriate orifice of the nipple. The guidance can be transmitted to a user device (e.g., a computing device integrally or operably coupled to the endoscope, etc.) and can be presented as one or more visual indications (e.g., recommended intubation trajectories, annotations, notifications, etc.) that can be superimposed on the real-time surgical image data to assist the physician in completing the procedure.

[0032] It is important to note that while the techniques utilized herein are described with explicit reference to ERCP procedures, such designation is not limiting. More particularly, the machine learning models described herein can be trained to recognize characteristic features associated with other anatomical objects / structures and can be correspondingly used to provide guidance for other types of medical procedures.

[0033] Figure 1 An exemplary environment 100 is depicted that can be used with the techniques described herein. One or more user devices 105 can communicate with one or more medical devices 110 and / or one or more server systems 115 across a network 101. One or more user devices 105 can be associated with a user, for example, a user associated with generating, training, using, or adjusting one or more of a machine learning model for providing predictive navigation guidance during a medical procedure. For example, one or more user devices 105 can be associated with a physician performing a medical procedure (e.g., an ERCP procedure) and seeking to gain benefits from the capabilities of the server system 115.

[0034] In some embodiments, the components of environment 100 may be associated with a common entity, such as a single enterprise or organization, or alternatively, one or more of the components may be associated with an entity different from another. The systems and devices of environment 100 may communicate in any arrangement. For example, one or more user devices 105 and / or medical devices 110 may be associated with one or more customers or service subscribers, and server system 115 may be associated with a service provider that is responsible for receiving surgical data from one or more customers or service subscribers and then utilizing the capabilities of server system 115 to return output to one or more customers or service subscribers. As will be discussed further herein, the systems and / or devices of environment 100 may communicate to generate, train, and / or utilize machine learning models to characterize various aspects of medical surgery and dynamically provide activities such as predictive navigation guidance.

[0035] The user device 105 may be configured to enable a user to access and / or interact with other systems in the environment 100. For example, the user device 105 may be a computer system, such as, for example, a desktop computer, a mobile device, a tablet computer, etc. In some embodiments, the user device 105 may include one or more electronic applications, such as programs, plug-ins, browser extensions, etc., installed on a memory of the user device 105.

[0036] The user device 105 may include a display user interface (UI) 105A, a processor 105B, a memory 105C, and a network interface 105D. The user device 105 may execute an operating system (O / S) and at least one electronic application (each stored in the memory 105C) through the processor 105B. The electronic application may be a desktop program, a browser program, a web client or a mobile application (which may also be a browser program in a mobile O / S), an applicant-specific program, a system control software, a system monitoring software, a software development tool, etc. For example, the environment 100 may extend information on a web client that may be accessed through a web browser. In some embodiments, the electronic application may be associated with one or more of the other components in the environment 100. The application may manage the memory 105C, such as a database, to transmit the medical operation data to the network 101. The display / UI 105A may be a touch screen or a display with other input systems (e.g., a mouse, a keyboard, etc.) so that the user can interact with the application and / or the O / S. The network interface 105D may be a TCP / IP network interface, which is used, for example, for Ethernet or wireless communication with the network 110. Processor 105B may generate data while executing applications and / or receive user input from display / UI 105A and / or receive / transmit messages to server system 115 , and may also perform one or more operations before providing output to network 110 .

[0037] The medical devices 110 in the environment 100 may include one or more medical devices (e.g., endoscopes, other internal imaging devices, etc.) that are integrally (e.g., via a wired connection, etc.) or operatively (e.g., via a wireless connection, etc.) coupled to the user device 105 and / or the server system 115. Data obtained by sensors of the medical devices 110 (e.g., image / video data, location data, etc.) may be transmitted to one or both of the user device 105 and / or the server system 115.

[0038] In various embodiments, network 101 may be a wide area network ("WAN"), a local area network ("LAN"), a personal area network ("PAN"), etc. In some embodiments, network 101 includes the Internet, and information and data provided between various systems occurs online. "Online" may mean connecting to or accessing source data or information from a location away from other devices or networks connected to the Internet. Alternatively, "online" may refer to connecting to or accessing a network (wired or wireless) via a mobile communication network or device. The Internet is a global system of computer networks - a network of networks, in which a party at a computer or other device connected to the network can obtain information from any other computer and communicate with parties at other computers or devices. The most widely used part of the Internet is the World Wide Web (often abbreviated as "WWW" or referred to as "Web"). "Website pages" generally cover locations, data storage devices, etc., which, for example, are hosted and / or operated by a computer system so as to be accessible online, and which may include data configured to enable a program, such as a web browser, to perform operations, such as sending, receiving or processing data, generating visual displays and / or interactive interfaces, etc.

[0039] Server system 115 may include an electronic data system, computer readable memory such as a hard drive, flash drive, disk, etc. In some embodiments, server system 115 includes and / or interacts with an application programming interface for exchanging data with other systems, e.g., one or more of the other components of environment 100. Server system 115 may include and / or serve as a repository or source of extracted raw data set information.

[0040] The server system 115 may include a database 115A and at least one server 115B. The server system 115 may be a computer, a system of computers (e.g., a rack server), and / or a cloud service computer system. The server system may store or access the database 115A (e.g., hosted on a third-party server or in a memory 115E). The server may include a display / UI 115C, a processor 115D, a memory 115E, and / or a network interface 115F. The display / UI 115C may be a touch screen or a display with other input systems (e.g., a mouse, a keyboard, etc.) for an operator of the server 115B to control the functions of the server 115B. The server system 115 may execute at least one instance of an operating system (O / S) and a servlet program (each stored in the memory 115E) through the processor 115D. When the user device 105 or the medical device 110 sends medical surgery data to the server system 115, the received data set and / or data set information may be stored in the memory 115E or the database 115A. The network interface 115F may be a TCP / IP network interface for, for example, Ethernet or wireless communication with the network 101 .

[0041] The processor 115D may include and / or execute instructions to implement a predictive navigation guidance platform 120, which may include a medical surgery database 120A (e.g., containing data associated with historical ERCP surgeries, etc.) and / or a navigation guidance model 120B. The medical surgery database 120A may be continuously updated (e.g., with new medical surgery data). Additionally, the medical surgery database 120A may also be used to train the navigation guidance model 120B to dynamically identify correlations between characteristics associated with certain anatomical objects, positioning of one or more components of a medical device, and / or corresponding outcomes of the surgery from data associated with the immediate medical surgery. The process by which these correlations may be identified will be described later herein in conjunction with Figure 3 The associated public content is described.

[0042] In one embodiment, both the medical procedure database 120A and the navigation guidance model 120B may be included within the predictive navigation guidance platform 120. Alternatively, one or both of these components may be subcomponents of the other component within each other, or may reside on other components of the environment 100. For example, the medical procedure database 120A may be incorporated into an application platform on the user device 205, while the navigation guidance model 120B may reside on the server 115B of the server system 115.

[0043] As discussed in further detail below, the server system 115 can generate, store, train, or use one or more machine learning models configured to analyze medical procedure data and provide predictive navigation guidance based on the analysis. The server system 115 can include one or more machine learning models and / or instructions associated with each of the one or more machine learning models, e.g., instructions for generating a machine learning model, training a machine learning model, using a machine learning model, etc. The server system 115 can include instructions for retrieving output features, e.g., based on the output of a machine learning model, and / or operating the displays 105A and / or 115C to generate one or more output features, e.g., adjusted based on a machine learning model.

[0044] The server system 115 may include one or more sets of training data. The training data may include various types of historical data related to a particular medical procedure, such as ERCP. For example, the training data may include characteristic information associated with various types of detected papillae (e.g., shape data, size data, orientation data, appearance data, etc.), orifice characteristic information associated with each of the detected papillae (e.g., number, size, location, duct association, etc.), anatomical information associated with the location and / or structure of the bile duct and / or pancreatic duct, additional anatomical feature information associated with the detected papillae (e.g., the presence or absence of intramural folds, oral protrusions, frenulums and / or grooves, etc.), the position of the endoscope relative to the papillae before and / or during intubation, historical ERCP procedure results, etc.

[0045] In some embodiments, a machine learning model may be generated and / or trained using a system or device other than server system 115. For example, such a system may include instructions for generating a machine learning model, training data, and ground truth, and / or instructions for training a machine learning model. The resulting trained machine learning model may then be provided to server system 115.

[0046] In some embodiments, a neural network-based machine learning model includes a set of variables, such as nodes, neurons, filters, etc., which are adjusted, for example, weighted or biased to different values, via application of training data. In other embodiments, the machine learning model can be based on an architecture such as a support vector machine, a decision tree, a random forest, or a gradient boosting machine (GBM). Alternative embodiments include the use of techniques such as transfer learning, in which one or more pre-trained machine learning models on large general or domain-specific data sets can be utilized to analyze training data.

[0047] In supervised learning, for example, where the ground truth for the provided training data is known, training can be done by feeding samples of the training data into the model with the variables set at initialization values, e.g., randomly, based on Gaussian noise, a pre-trained model, etc. The output can be compared to the ground truth to determine the error, which can then be back-propagated through the model to adjust the values ​​of the variables.

[0048] Training can be performed in any suitable manner, for example, in batches, and can include any suitable training method, for example, random or non-stochastic gradient descent, gradient boosting, random forest, etc. In some embodiments, a portion of the training data can be retained during training and / or used to validate the trained machine learning model, for example, comparing the output of the trained model with the ground truth for the portion of the training data to evaluate the accuracy of the trained model. The training of the machine learning model can be configured to enable the machine learning model to learn the contextual association between the original surgical data and the context associated with it (for example, which anatomical features and / or the actions of the medical device affect the success rate of the ERCP procedure, etc.), so that the trained machine learning model is configured to provide predictive guidance that can increase the success rate of the ERCP procedure.

[0049] In various embodiments, the variables of the machine learning model can be interrelated in any suitable arrangement to generate an output. For example, in some embodiments, the machine learning model can include a signal processing architecture that is configured to identify, isolate and / or extract features, patterns and / or structures in an image or video. For example, the machine learning model can include one or more convolutional neural networks ("CNNs") that are configured to identify anatomical features associated with the nipple and related anatomical structures, and can include further architectures, e.g., connection layers, neural networks, etc., that are configured to determine relationships between the identified features and structures in order to determine an optimal cannulation path.

[0050] For example, in some embodiments, the machine learning model of the server system 115 may include a recurrent neural network ("RNN"). Typically, an RNN is a type of feedforward neural network that is well adapted to process a sequence of inputs. In some embodiments, the machine learning model may include a long short-term memory ("LSTM") model and / or a sequence-to-sequence ("Seq2Seq") model. The LSTM model may be configured to generate outputs from samples that take into account at least some previous samples and / or outputs. The Seq2Seq model may be configured, for example, to receive a sequence of images as input and subsequently generate a sequence of annotations and / or predicted medical device movement trajectories as output.

[0051] Despite Figure 1Although depicted as separate components in environment 100, in some embodiments, components or portions of components in environment 100 may be integrated with or incorporated into one or more other components. For example, a portion of display 115C may be integrated into user device 105, etc. In some embodiments, operations or aspects of one or more of the components discussed above may be distributed among one or more other components. Any suitable arrangement and / or integration of the various systems and devices of environment 100 may be used. Train machine learning models to provide predictive navigation guidance

[0052] Figure 2 An exemplary process for training a machine learning model, such as the navigation guidance model 120B, to identify key anatomical features and provide dynamic guidance to assist the surgeon during an ERCP procedure is shown.

[0053] At step 205 of the training process, the method may include receiving a training data set, such as a compilation of data associated with previously completed ERCP procedures. More particularly, for each completed ERCP procedure, the training data may include images, videos, medical reports, etc. associated with one or more anatomical objects of interest (e.g., nipples, one or more orifices on nipples, bile ducts, pancreatic ducts, etc.) detected during the previously completed ERCP procedure. The data may have been captured using one or more sensors (e.g., optical cameras) and / or other imaging modalities (X-ray imaging, fluoroscopy, etc.) associated with the medical device. In one embodiment, the training data may also include position and / or movement data of the medical device (e.g., endoscope) and / or its components (e.g., guidewires) during the procedure relative to one or more anatomical objects. The position and / or motion data may have been captured using one or more other sensors (e.g., electromagnetic (EM) sensors, accelerometers, gyroscopes, optical fibers, ultrasonic transducers, capacitive or inductive position sensors, etc.), and / or may have been obtained via any other suitable means, such as, via observation by a person and / or an automated system, via feedback from a controller for the medical device, etc. In one embodiment, the training data may also include an indication of the outcome of each of the completed ERCP procedures (eg, positive outcome, negative outcome, severity of negative outcome, etc.).

[0054] In one embodiment, each piece of training data may be pre-annotated with relevant anatomical feature information. For example, each image of the nipple may identify one or more of the following: a classification associated with the nipple based on shape, orientation, and / or appearance data; an orifice on the nipple and its corresponding association (e.g., an orifice associated with the pancreatic duct, an orifice associated with the bile duct, etc.); an orientation of each orifice (e.g., facing downward, facing sideways, etc.); intramural ligament features (e.g., intramural folds), diverticula, ostial protrusions; frenulum and / or grooves, etc. In one embodiment, each piece of training data may identify a path followed by a medical device to cannulate one or both of the orifices.

[0055] In one embodiment, a server system (e.g., server system 115) may receive a training data set and may store the training data set in a database (e.g., ERCP database 120A, database 115A, etc. on the predictive navigation guidance platform 120) and / or a memory (e.g., memory 115E). In one embodiment, a user may upload the training data set to a user device (e.g., user device 105) to manually annotate each piece of training data. The user device 105 may or may not store the training data set in a memory (e.g., 105C). Once annotated, the user device 105 may transmit the annotated training data set to the server system 115 via the network 101.

[0056] At step 210, the method may include, for each training data set associated with an ERCP procedure, extracting anatomical feature data from the annotated training data. The extracted anatomical feature data may be used to train a machine learning model to correctly identify and distinguish important anatomical objects associated with the ERCP procedure during real-time surgery. Figure 3 Additional disclosure regarding how to train a machine learning model based on extracted anatomical feature data is further provided in the discussion of .

[0057] Back to Figure 2At step 215, position data associated with the recorded position, angle, and / or movement of the endoscope (and its components) relative to the nipple during intubation in the ERCP procedure in the training data set can be extracted. The position data can initially be obtained using one or more sensors integrally or operably connected to the endoscope and / or guidewire, including but not limited to electromagnetic (EM) sensors, accelerometers, gyroscopes, optical fibers, ultrasonic transducers, capacitive or inductive position sensors, etc. In one embodiment, the position data can be annotated with positive annotations and / or negative annotations. For example, position data that resulted in a successful procedure (e.g., resulting from an optimal position and / or angle of the endoscope relative to the nipple, etc.) can be classified as positive, while position data that resulted in an unsuccessful procedure (e.g., due to approach angles, etc.) can be classified as negative. Therefore, the predictive navigation guidance model can be trained on the accumulation of position data to identify the ideal position, angle, and / or movement of the endoscope and guidewire to successfully intubate the nipple.

[0058] At step 220, the outcome data associated with the ERCP procedures in the training data set may be extracted and used to train the predictive navigation guidance model. More particularly, each of the ERCP procedures in the training data set may contain one or more indications as to whether the ERCP procedure was successful or unsuccessful. For the training data set, the outcome data for each ERCP procedure may be explicitly annotated so that the predictive navigation guidance model may learn to dynamically distinguish between successful and / or unsuccessful procedures and their sub-steps.

[0059] In one embodiment, the result data can provide a binary indication of the success status of the ERCP operation (e.g., the ERCP operation is generally successful or unsuccessful, etc.). In this regard, the success status of the ERCP operation can be based on whether the bile duct has been successfully cannulated. Additionally or alternatively, in another embodiment, the result data can provide a more refined indication of the surgical results that occurred during the ERCP surgical procedure. More particularly, the result data can depict the parts of the ERCP operation that are successful (e.g., the bile duct orifice and the pancreatic duct orifice are successfully distinguished from each other) and unsuccessful (e.g., the cannulation of the bile duct is unsuccessful due to the approach angle of the guide wire, etc.).

[0060] At step 225, the accumulation of all extracted data from steps 210 to 220 may be utilized to train a predictive navigation guidance model. In this regard, the trained predictive navigation guidance model may subsequently be able to receive data associated with a real-time ERCP procedure and apply the knowledge gained from the training program to identify correlations between aspects of the real-time ERCP procedure and aspects associated with previously completed ERCP procedures (e.g., those embodied in the trained data set, etc.). Thereafter, the predictive navigation guidance model may be able to provide dynamic guidance to an operator of a medical device (e.g., a physician, etc.), as further described herein.

[0061] Figure 3 An exemplary process of extracting anatomical feature data from an annotated training data set is shown. More particularly, a combination of various visual AI neural network frameworks may be utilized to isolate and evaluate progressively smaller and / or more specific regions of a target image based on ROIs and specific targets. Once trained, the predictive navigation guidance model 120B may be able to dynamically and accurately identify specific objects and ROIs from received data associated with a real-time ERCP procedure.

[0062] At step 305, the method may first include training a predictive navigation guidance model to classify nipple types. This classification is important because the type of nipple present may determine the location, orientation, and / or structure of other anatomical objects (e.g., orifices, etc.). Possible nipple types include: regular, small, protruding or pendulous, and wrinkled or ridged. For example, referring to Figure 4 , showing a plurality of possible nipple types. For example, with respect to nipple 405, ROI 405A includes a "regular" nipple 405B, i.e., a nipple that contains no distinctive features and has a "classic" appearance. With respect to nipple 410, ROI 410A includes a "small" nipple 410B, i.e., a nipple that is generally flat and has a diameter less than or equal to 3 mm. With respect to nipple 415, ROI 415A may include a protruding or pendulous nipple 415B, i.e., a nipple that may protrude, protrude, or bulge into the duodenal lumen or that sometimes sags and may have a pendulous appearance and have an orifice toward the tail. With respect to nipple 420, ROI 420A may include a wrinkled or ridged nipple 420B, i.e., a nipple in which the ductal mucosa appears to extend distally from the nipple orifice on a ridge or in a wrinkle.

[0063] In one embodiment, each set of annotated training data may contain an explicit designation of the ROI and an explicit indication of the nipple type. In one embodiment, each image in the training set may be captured typically with a "frontal" alignment to the target nipple. In one embodiment, for step 305, a regional convolutional neural network (R-CNN) framework, such as RESNET-18, may be employed and trained on a large set of annotated images of nipple types to facilitate proper nipple type classification during real-time ERCP procedures.

[0064] At step 310, the method may include training a predictive navigation guidance model to identify orifice types. More specifically, in one embodiment, the predictive navigation guidance model may be trained to identify a second ROI (i.e., ROI-2) bounded by a first ROI (i.e., ROI-1) that is associated with a particular nipple pattern. The type of nipple pattern identified may correspondingly determine the characteristics of one or more orifices residing on the nipple. Turning now to Figure 5 , showing multiple known nipple patterns. For example, papilla-A 505 may correspond to a typical nipple pattern having an annular shape, some of which have a tubercular variation on the oral side of the center. Papilla-U 510 may correspond to a nipple pattern that is generally unstructured and has no clear orifice. Papilla-LO 515 may correspond to a nipple pattern having a longitudinal groove that is continuous with the orifice, wherein the length of the groove is greater than the transverse diameter of the bile duct axis of the papilla. Papilla-I 520 may correspond to a nipple pattern having two separate, isolated orifices for the bile duct and the pancreatic duct, wherein the opening on the oral side is the opening of the bile duct, and the opening on the anal side is the opening of the pancreatic duct. Papilla-G 525 may correspond to a nipple pattern having a convoluted structure.

[0065] In one embodiment, each set of annotated training data may contain one or more explicit designations that identify: the type of nipple pattern expressed by the nipple, the region on the nipple where one or more orifices are located based on the nipple pattern, and the type of approach that the orifice may present. In one embodiment, since the focal region of ROI-2 is smaller than that of ROI-1, fewer convolutions may be required to accurately classify the orifice type. Therefore, for step 310, a FastR-CNN framework, such as RESTNET-9 or other faster traditional R-CNNs, may be employed and trained on a large batch of annotated nipple patterns and corresponding orifice features.

[0066] At step 315, the method can include training a predictive navigation guidance model to detect the location of the bile duct and / or the pancreatic duct. More particularly, in one embodiment, the bile duct and / or the pancreatic duct can be located on the image and, if possible, can be distinguished from each other and / or from other anatomical objects (e.g., based on annotations in the training data). Turning now to Figure 6 , provides a non-limiting example, where the bile duct and / or pancreatic duct may be located at a site originating from Figure 5 For example, papilla 605 shows the location of both bile duct 605A and pancreatic duct 605B, papilla 610 shows the location of bile duct 610A, papilla 615 shows the location of bile duct 615A, papilla 620 shows the location of both bile duct 620A and pancreatic duct 620B, and papilla 625 shows the location of bile duct 625A.

[0067] In one embodiment, duct differentiation can be achieved using a semantic segmentation model (e.g., SegNet) that can employ a fully convolutional network only on the region bounded by ROI-2. Such a model can utilize a two-stage approach to first distinguish ducts from surrounding anatomical features found on the nipple, and then regression can be performed to distinguish ducts from each other. In one embodiment, a fully convolutional network can be trained using a large batch of annotated images that depict the identity of the bile duct and / or pancreatic duct.

[0068] At steps 320 to 330, one or more detection algorithms may be utilized to identify specific anatomical features on or associated with the nipple. For example, at step 320, the detection algorithm may be trained to determine if any intramural folds are present. At step 325, either after or independently of the detection training process performed at step 320, the same or a different detection algorithm may be trained to determine if an oral protuberance is present. At step 330, either after or independently of the detection processes performed at steps 320 and 325, the same or a different detection algorithm may be trained to determine if a frenulum and / or sulcus are present. In order to train the detection algorithm at each of the steps in steps 320 to 330 to be ready to detect the specific anatomical features associated with each step, the training data set may be annotated with the relevant anatomical objects. Applying trained machine learning models to real-time surgical data to generate predictive navigation guidance

[0069] Figure 7 An exemplary process for determining predictive navigation guidance for an ERCP procedure and subsequently providing guidance to the surgeon is shown.

[0070] At step 705, an embodiment of the trained predictive navigation model may receive image data associated with one or more anatomical objects associated with a real-time ERCP procedure. For example, the one or more anatomical objects may correspond to a nipple, one or more orifices residing on the nipple, a bile duct and / or a pancreatic duct, other anatomical features or structures associated with any of the foregoing, and the like. In one embodiment, the image data may be captured by one or more optical sensors of a medical device used in an ERCP procedure. For example, the image data may be captured by one or more optical sensors positioned on a distal end of an endoscope.

[0071] At step 710, an embodiment of the trained predictive navigation guidance model can determine predictive navigation guidance of a medical device used in a medical procedure relative to a target anatomical object. In this regard, an embodiment can apply image data as input to the trained predictive navigation guidance model. The trained predictive navigation guidance model can be configured to analyze various aspects of the image data to determine relevant correlations between historical ERCP procedures and real-time ERCP procedures. Additionally, in some embodiments, available position data associated with the medical device can also be provided as input to the trained model.

[0072] In response to determining at step 710 that one or more types of predictive navigation guidance cannot be determined (e.g., due to lack of necessary information, inability of the machine learning model to identify correlations between real-time surgical data and historical surgical data, etc.), an embodiment may transmit an alert notification at step 715 (e.g., to the user device 105). The alert notification may be an audio notification, a visual notification, or a combination thereof, and may include an explanation indicating why dynamic guidance cannot be provided. Alternatively, in another embodiment, no additional action may be taken.

[0073] In contrast to the foregoing, in response to determining one or more types of predicted navigation guidance at step 710, an embodiment may generate one or more visual representations associated with the determined predicted navigation guidance at step 720. In one embodiment, the one or more visual representations may correspond to one or more of: annotations identifying relevant anatomical objects, trajectory recommendations for manipulating a medical device and / or components thereof, and / or feedback notifications alerting a medical device operator of updates occurring during a medical procedure.

[0074] At step 725, an embodiment may transmit instructions to a user device to display / overlay a visual representation of the prediction guidance over some or all portions of the image data. For example, in one embodiment, the server system 115 may be configured to transmit instructions to a user device to annotate one or more relevant anatomical objects during a medical procedure. In one embodiment, possible annotations may include anatomical object coloring / highlighting (e.g., wherein each detected relevant anatomical object is colored with a different specific color, etc.), ROI designations (e.g., wherein relevant regions in the image data are depicted via target boxes or outlines, etc.), text identifiers (e.g., wherein each detected relevant anatomical object is identified by text, etc.), combinations thereof, etc. Turning now to Figure 8 , provides a non-limiting example of annotations superimposed on image data associated with a target nipple. Figure 8 Annotations present in the include visually distinguishable anatomical objects (e.g., the intramural segment / diverticulum 82 may be colored blue, the intramural bile duct 84 may be colored gray, the bile duct ostium 86 may be colored green, and the pancreatic duct ostium 88 may be colored red) as well as ROI designations (e.g., the papilla identifier box 90 and the ostium region outline 92).

[0075] In another embodiment, the server system 115 can be configured to transmit instructions to the user device to provide a recommended trajectory to assist in positioning, aligning, and / or advancing the ERCP scope and guidewire. In one embodiment, the recommended trajectory can be provided as an overlay on top of the image data of the anatomical object. For example, referring to Fig. 9 , provides a non-limiting example implementation of how trajectory overlay can be achieved in two different surgical situations. Fig. 9, a conventional endoscopic view of two papillae, 905A and 910A, is provided. Each papilla contains a different papilla pattern that affects the location of the orifices of the bile duct and the pancreatic duct. More particularly, the positioning of the ducts and the corresponding orifices can vary between the two papillae shown in 905A and 910A. These differences can be represented in Figures 905B and 910B. Specifically, it can be seen that both the bile duct and the pancreatic duct can enter via a single orifice in the papilla, as shown in 905A, while the bile duct and the pancreatic duct can enter via dedicated orifices in the papilla, as shown in 910A. Based on the learned knowledge, the predictive navigation guidance model may be able to identify the appropriate duct location for each papilla type, and then provide a recommended trajectory overlay for entering the target duct, as shown in 905C and 910C. More particularly, referring to 905C, although the bile duct and pancreatic duct may be accessed through a single orifice, the trajectory overlay may still provide an approach trajectory for cannulation of the bile duct at 905C-1 and / or an approach trajectory for cannulation of the pancreatic duct at 905C-2. Additionally, referring to 910C, the predictive navigation guidance model may be able to distinguish between the two identified orifices and then provide an approach trajectory for cannulation of the bile duct at 910C-1 and / or an approach trajectory for cannulation of the pancreatic duct at 910C-2.

[0076] In another embodiment, the server system 115 may be configured to transmit instructions to the user device to provide feedback to the physician when the movement of the endoscope and / or guidewire deviates from the recommended trajectory. In one embodiment, the server system 115 may be configured to provide feedback immediately (e.g., when initially detecting a deviation from the recommended trajectory), or alternatively, when the degree of deviation from the recommended trajectory exceeds a predetermined threshold. In one embodiment, feedback can be presented in one or more different forms. For example, feedback can be presented as a visual alarm (e.g., a text alarm, an icon alarm, an animation alarm, etc., which is presented on the display screen of the user device, etc.), an auditory alarm (e.g., provided via one or more speakers associated with the user device, etc.), a tactile alarm (e.g., vibrating the medical device via one or more actuators, etc.), or a combination thereof. In one embodiment, feedback can be presented only once, presented at predetermined intervals (e.g., every 5 seconds, 10 seconds, etc.) or presented continuously. In one embodiment, feedback can be guiding and can suggest adjustments that the operator of the medical device can make to align the planned approach path with the recommended trajectory.

[0077] In one embodiment, the server system 115 can be configured to not transmit any predictive navigation guidance unless the confidence weight of the predictive navigation guidance model with respect to the target anatomical object is greater than a predetermined threshold. More specifically, the confidence weight held by the predictive navigation guidance model for a particular anatomical object (e.g., a nipple) can first be identified. The confidence weight can be based on or reflected by the training of the predictive navigation guidance model on a specific anatomical object (e.g., a specific type of nipple or nipple pattern, etc.), where more training can correspond to a higher confidence. Then, an embodiment can determine whether the confidence weight is greater than a predetermined confidence threshold, and, in response to determining that it is not greater than, the transmission of the predictive guidance to the operator of the medical device can be stopped.

[0078] Back to Figure 7 At 730, one embodiment may optionally update the ERCP database with data associated with the real-time medical procedure. This updated data may then be used to further train the predictive navigation guidance model (e.g., using data previously acquired in Figures 2 to 3 ). In one embodiment, the type of data obtained from the real-time medical procedure that can be used to update the ERCP database may include one or more of the following: captured anatomical feature data, detected medical device position / movement data, outcome data of the medical procedure (e.g., whether the overall cannulation was successful or unsuccessful, which parts of the ERCP procedure were successful or unsuccessful, whether there were postoperative complications such as pancreatitis, etc.), navigation guidance accuracy data (e.g., how accurate are the annotations used to identify anatomical objects, how accurate are the recommended trajectories, etc.), etc. In one embodiment, the update to the ERCP database may be a single batch update. For example, one embodiment may retain all captured data associated with the medical procedure until the medical procedure is completed. Thereafter, one embodiment may transfer all data associated with the medical procedure to the ERCP database. Alternatively, in another embodiment, the ERCP database may be continuously updated with medical procedure data (e.g., as data accumulates). For example, one embodiment may transfer image data associated with the target anatomical object to the ERCP database substantially immediately upon capture, one embodiment may continuously transfer movement data associated with the medical device during the medical procedure to the ERCP database substantially once detected, etc.

[0079] Fig.10 is an exemplary embodiment of the present invention may be configured to perform Figures 2 to 3 and Figure 7A simplified functional block diagram of a computer 1100 of an apparatus for performing the method of the present invention is shown. For example, the apparatus 1000 may include a central processing unit (CPU) 1020. The CPU 1020 may be any type of processor device, including, for example, any type of special or general purpose microprocessor device. As will be appreciated by those skilled in the relevant art, the CPU 1020 may also be a single processor in a multi-core / multi-processor system that operates alone or in a cluster of computing devices that operate in a cluster or server farm. The CPU 1020 may be connected to a data communications infrastructure 1010, such as a bus, a message queue, a network, or a multi-core messaging scheme.

[0080] The device 1000 may also include a main memory 1040, such as a random access memory (RAM), and may also include an auxiliary memory 1030. The auxiliary memory 1030, such as a read-only memory (ROM), may be, for example, a hard disk drive or a removable storage drive. Such a removable storage drive may include, for example, a floppy disk drive, a tape drive, an optical disk drive, a flash memory, etc. The removable storage drive in this example reads from and / or writes to a removable storage unit in a well-known manner. The removable storage unit may include a floppy disk, a tape, an optical disk, etc., which can be read by and written to the removable storage drive. As will be appreciated by those skilled in the relevant art, such a removable storage unit typically includes a computer-usable storage medium in which computer software and / or data are stored.

[0081] In alternative embodiments, secondary memory 1030 may include other similar devices that allow computer programs or other instructions to be loaded into device 1000. Examples of such devices may include program cartridges and cartridge interfaces (such as those found in video game devices), removable memory chips (such as EPROM or PROM) and associated slots and other removable storage units and interfaces that allow software and data to be transferred from removable storage units to device 1000.

[0082] The device 1000 may also include a communication interface ("COM") 1060. The communication interface 1060 allows software and data to be transferred between the device 1000 and external devices. The communication interface 1060 may include a modem, a network interface (such as an Ethernet card), a communication port, a PCMCIA slot and card, etc. The software and data transferred via the communication interface 1060 may be in the form of signals, which may be electronic signals, electromagnetic signals, optical signals, or other signals capable of being received by the communication interface 1060. These signals may be provided to the communication interface 1060 via a communication path of the device 1000, which may be implemented using, for example, wire or cable, optical fiber, a telephone line, a cellular phone link, an RF link, or other communication channel.

[0083] The hardware elements, operating systems, and programming languages ​​of such devices are conventional in nature, and it is assumed that those skilled in the art are sufficiently familiar with them. The device 1000 may also include input and output ports 1050 to connect to input and output devices such as keyboards, mice, touch screens, monitors, displays, etc. Of course, the various server functions can be implemented in a distributed manner on multiple similar platforms to distribute the processing load. Alternatively, the server can be implemented by appropriately programming a computer hardware platform.

[0084] The systems, devices, apparatuses and methods disclosed herein are described in detail by way of example and with reference to the accompanying drawings. The examples discussed herein are merely examples and are provided to help explain the devices, apparatuses, systems and methods described herein. Unless specifically designated as mandatory, any features or components shown in the figures or discussed below should not be considered mandatory for any particular embodiment of these devices, apparatuses, systems or methods. For ease of reading and clarity, certain components, modules or methods may be described only in conjunction with specific drawings. In the present invention, any identification of a particular technology, arrangement, etc. is either related to the specific examples presented, or is merely a general description of such technology, arrangement, etc. Therefore, unless specifically designated, identification of specific details or examples is not intended to be and should not be interpreted as mandatory or restrictive. Any combination or sub-combination of components that fails to be specifically described should not be understood to indicate that any combination or sub-combination is impossible. It should be understood that the disclosed and described examples, arrangements, configurations, components, elements, equipment, devices, systems, methods, etc. may be modified, and these modifications may be required for specific applications. In addition, for any method described, whether or not the method is described in conjunction with a flowchart, it should be understood that, unless the context dictates otherwise, any explicit or implicit ordering of the steps performed during the method execution does not mean that those steps must be performed in the order presented, but may be performed in a different order or in parallel.

[0085] Throughout the present invention, references to components or modules generally refer to items that can be logically combined together to perform a function or a group of related functions. Similar reference numerals are generally intended to refer to the same or similar components. Components and modules can be implemented with software, hardware, or a combination of software and hardware. The term "software" is used broadly to include not only executable code, such as machine executable or machine interpretable instructions, but also data structures, data storage devices, and computing instructions stored in any suitable electronic format, including firmware and embedded software. The terms "information" and "data" are used broadly and include various electronic information, including executable code; content, such as text, video data, and audio data; and various codes or signs. The terms "information", "data", and "content" are sometimes used interchangeably where the context permits.

[0086] The program aspects of the present technology can be considered as "products" or "manufactures", which typically use executable code and / or associated data carried or embodied on or in a type of machine-readable medium. "Storage" type media include any or all of computers, processors, etc. or their associated modules, such as various semiconductor memories, tape drives, disk drives, etc. (which can provide non-temporary storage for software programming at any time). All or part of the software can sometimes communicate through the Internet or various other telecommunications networks. For example, such communication can enable software to be loaded from one computer or processor to another, for example, from a management server or host of a mobile communication network to a computer platform of a server and / or from a server to a mobile device. Therefore, another type of medium that can carry software elements includes light waves, radio waves, and electromagnetic waves, such as across physical interfaces between local devices, through wired and optical fixed telephone line networks and through various air links. Physical elements that carry such waves, such as wired or wireless links, optical links, etc., can also be regarded as media that carry software. As used herein, unless restricted to non-transitory, tangible "storage" media, terms such as computer or machine "readable media" refer to any medium that participates in providing instructions to a processor for execution.

[0087] Although the disclosed methods, apparatuses, and systems are described with exemplary reference to transmitting data, it should be understood that the disclosed embodiments may be applicable to any environment, such as a desktop or laptop computer, a car entertainment system, a home entertainment system, etc. In addition, the disclosed embodiments may be applicable to any type of Internet protocol.

[0088] It should be understood that in the above description of exemplary embodiments of the present invention, various features of the present invention are sometimes combined together in a single embodiment, figure, or description thereof to simplify the present invention and to aid in understanding one or more of the various inventive aspects. However, this approach to the present invention should not be interpreted as reflecting an intention that the claimed invention requires more features than those explicitly recited in each claim. On the contrary, as reflected in the following claims, inventive aspects exist in features that are less than all of the features in a single, previously disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into the detailed description, with each claim existing independently as a separate embodiment of the present invention.

[0089] In addition, although some embodiments described herein include some features included in other embodiments, but not other features, the combination of features of different embodiments should be within the scope of the present invention and form different embodiments, as will be understood by those skilled in the art. For example, in the following claims, any of the claimed embodiments can be used in any combination.

[0090] Therefore, although certain embodiments have been described, those skilled in the art will recognize that other and further modifications may be made thereto without departing from the spirit of the invention, and it is intended that all such changes and modifications fall within the scope of the invention. For example, functions may be added or deleted from the block diagrams, and operations may be interchanged between functional blocks. Steps may be added or deleted from the method within the scope of the invention.

[0091] The above disclosed subject matter should be considered illustrative, not restrictive, and the appended claims are intended to cover all such modifications, enhancements, and other embodiments, which fall within the true spirit and scope of the present invention. Therefore, to the maximum extent permitted by law, the scope of the present invention should be determined by the broadest permissible interpretation of the following claims and their equivalents, and should not be restricted or limited by the preceding detailed description. Although various embodiments of the present invention have been described, it is apparent to those of ordinary skill in the art that more embodiments may be within the scope of the present invention. Therefore, the present invention is not limited except according to the appended claims and their equivalents.

Claims

1. A computer-implemented method for generating navigation guidance for a medical device in a body, the computer-implemented method comprising: receiving, at a computer server, image data associated with at least one anatomical object; determining, using a processor associated with the computer server and via applying a trained predictive navigation guidance model to the image data, navigation guidance for the medical device associated with the at least one anatomical object; generating at least one visual representation associated with the navigation guidance based on the determination; as well as Instructions are transmitted to a user device in network communication with the computer server to display the at least one visual representation associated with the navigation guidance over the image data on a display screen of the user device.

2. The computer-implemented method of claim 1 , wherein: The medical device is an endoscope including an extendable guidewire.

3. A computer-implemented method according to any one of claims 1 or 2, wherein: The at least one anatomical object corresponds to one or more of: a nipple, an orifice, or an internal duct, and the navigation guidance includes a path of the medical device for cannulating the anatomical object.

4. A computer-implemented method according to any one of claims 1 to 3, wherein: The image data is captured by at least one sensor and / or at least one other imaging device associated with the medical device.

5. The computer-implemented method of claim 4, wherein: The at least one sensor associated with the medical device comprises a camera sensor, and wherein the image data captured by the camera sensor comprises at least one of: shape data, orientation data and / or appearance data of the at least one anatomical object.

6. A computer-implemented method according to any one of claims 4 or 5, wherein: The at least one other imaging device comprises an X-ray device and / or an ultrasound device, and wherein the image data captured by the at least one other imaging device comprises anatomical data.

7. The computer-implemented method of any one of claims 1 to 6, further comprising receiving position data captured by one or more other sensors associated with the medical device, wherein: The one or more other sensors include at least one of the following: an electromagnetic sensor, an accelerometer, a gyroscope, a fiber optic sensor, an ultrasonic transducer, a capacitive position sensor, or an inductive position sensor.

8. A computer-implemented method according to any one of claims 1 to 7, wherein: The determining the navigation guidance includes identifying anatomical feature data from the image data using the predictive navigation guidance model.

9. The computer-implemented method of claim 8, wherein: The identifying the anatomical feature data comprises: identifying, within a first target region of the image data, a first classification associated with a first anatomical object of the at least one anatomical object; identifying a second classification associated with a second anatomical object of the at least one anatomical object within a second target region bounded by the first target region; detecting the location of one or more third anatomical objects within the second target region; and One or more other anatomical objects associated with the first anatomical object are detected.

10. The computer-implemented method of any one of claims 1 to 9, wherein: The determination includes: identifying a confidence weight held by the predictive navigation guidance model for the at least one anatomical object; and determining whether the confidence weight is greater than a predetermined confidence threshold; Wherein, the generating of the navigation instruction is performed only in response to determining that the confidence weight is greater than the predetermined confidence threshold.

11. A computer-implemented method according to any one of claims 1 to 10, wherein: The at least one visual representation includes one or more of: at least one trajectory overlay, at least one annotation, and / or at least one feedback notification.

12. The computer-implemented method of claim 11, wherein: The at least one trajectory overlay includes a visual indication, superimposed on an image of the at least one anatomical object, of a planned path to an entry point to the at least one anatomical object that a component of the medical device may follow to cannulate the at least one anatomical object.

13. The computer-implemented method of claim 12, further comprising: receiving, at a computer server, location data for the medical device; as well as identifying a deviation of the medical device from the planned path based on an analysis of the received position data; Wherein, generating the feedback notification includes generating in response to detecting that the deviation of the medical device from the planned path is greater than a predetermined amount.

14. A computer-implemented method according to any one of claims 11 to 13, wherein: The at least one annotation includes one or more visual indications superimposed on the image of the at least one anatomical object that indicate predetermined features associated with the at least one anatomical object.

15. The computer-implemented method of claim 14, wherein: The one or more visual indications include one or more of: a color indication, an outline indication, and / or a text-based indication.

Images