Systems and methods for using artificial intelligence to guide medical devices

Abstract

A system and method for generating navigation guidance for an in-vivo medical device are disclosed. One method implemented by a computer includes receiving, at a computer server, image data related to at least one anatomical object, using a processor associated with the computer server to determine navigation guidance for a medical device related to the at least one anatomical object by applying a trained predictive navigation guidance model to the image data, generating at least one visual representation associated with the navigation guidance based on the determination, and sending, to a user device in network communication with the computer server, instructions for displaying at least one visual representation associated with the navigation guidance over image data on a display screen of the user device.

Description

Technical Field

[0001] Various aspects of the present disclosure generally relate to systems and methods that utilize artificial intelligence to provide navigation guidance for medical devices that act within the body. More specifically, in some embodiments, the present disclosure relates to the application of trained machine learning models to data related to providing predictive navigation guidance for physicians operating medical devices.

Background Art

[0002] To examine and treat problems inside the body, certain medical procedures may be performed. For example, in an endoscopic procedure, a long, thin tube is inserted directly into the body to observe organs and tissues in detail. Such procedures may also be used to perform other tasks, including imaging and minor surgery. In some endoscopic procedures, cannulation of various anatomical objects (e.g., one or more ducts, etc.) may need to be achieved via the insertion of endoscopic components (e.g., guidewires). Such operations are very difficult and may involve a steep learning curve. As a result, the time it takes for inexperienced physicians to become proficient in such procedures can be very long.

[0003] The present disclosure is directed to addressing the above-described problems. The description of the background art provided herein is intended to generally present the background of the present disclosure. Unless otherwise stated herein, the materials described in this section are not prior art to the claims of this application and are not admitted to be prior art or suggestions of prior art by including them in this section.

Summary of the Invention

[0004] Each of the aspects disclosed herein may include one or more of the features described in relation to any of the other disclosed aspects. Aspects of the present disclosure relate, inter alia, to systems and methods for generating navigation guidance for medical devices operating within the body. According to one example, a method implemented by a computer is provided for generating navigation guidance for a medical device with respect to at least one anatomical object. The method implemented by the computer includes receiving, at a computer server, image data associated with at least one anatomical object, using a processor associated with the computer server to apply a trained predictive navigation guidance model to the image data to determine navigation guidance for a medical device with respect to at least one anatomical object, generating at least one visual representation associated with the navigation guidance based on the determination, and sending instructions to a user device in network communication with the computer server to display at least one visual representation associated with the navigation guidance on top of the image data on a display screen of the user device.

[0005] Any of the methods implemented by a computer to generate navigation guidance may include any of the following features and / or processes. The medical device may be an endoscope having an extendable guide wire. At least one anatomical object may correspond to one or more of the papilla, the opening, and / or the internal duct. The navigation guidance may include a path of the medical device for cannula insertion of the anatomical object. The image data may be captured by at least one sensor associated with the medical device and / or by at least one other imaging device. The at least one sensor may include a camera sensor, and the image data captured by the camera sensor may include at least one of the shape data, the orientation data, and / or the appearance data of at least one anatomical object. The at least one other imaging device may include an X-ray device and / or an ultrasonic device, and the image data captured by the at least one other imaging device may include anatomical structure data. One or more other sensors, including at least one of an electromagnetic sensor, an accelerometer, a gyroscope, an optical fiber sensor, an ultrasonic transducer, a capacitive position sensor, and / or an inductive position sensor, may be utilized. The one or more other sensors may capture position data related to the medical device. The determination of the navigation guidance may include identifying anatomical feature data from the image data using a predictive navigation guidance model. The identification of the anatomical feature data may include identifying a first classification associated with a first anatomical object within a first target region of the image data, identifying a second classification associated with a second anatomical object from within a second target region surrounded by the first target region, detecting the positions of one or more third anatomical objects from within the second target region, and detecting one or more other anatomical objects associated with the first anatomical object.Determining navigation guidance for a medical device may include identifying a confidence weight held by a predictive navigation guidance model for at least one anatomical object, determining whether the confidence weight is greater than a predetermined confidence threshold, and generating navigation guidance is only performed in response to determining that the confidence weight is greater than a predetermined confidence threshold. At least one visual representation may include one or more of at least one trajectory overlay, at least one annotation, and / or at least one feedback notification. At least one trajectory overlay may include a visual indication of a predicted path to an access point of at least one anatomical object that can be followed by a component of the medical device to cannulate the at least one anatomical object, overlaid on an image of the at least one anatomical object. A computer-implemented method may also receive position data of the medical device and identify a deviation of the medical device from the predicted path based on an analysis of the position data. Generation of a feedback notification in this context may also be in response to detecting that a deviation of the medical device from the predicted path is greater than a predetermined amount. At least one annotation may include one or more visual displays overlaid on an image of the at least one anatomical object that indicate a predetermined feature associated with the at least one anatomical object. The one or more visual displays may include one or more of a color display, a contour display, and / or a text-based display.

[0006] According to another example, a computer-implemented method for training a predictive navigation guidance model is provided. The computer-implemented method includes receiving, from a database, a training data set including past medical treatment data associated with a plurality of completed medical treatments; extracting anatomical feature data from the image data of the training data set; extracting medical device placement data from the sensor data of the training data set; extracting treatment result data from the training data set; and training a predictive navigation guidance model using the extracted anatomical feature data, the extracted medical device placement data, and the extracted treatment result data.

[0007] Any of the methods implemented by a computer for training a predictive navigation guidance model may include any of the following features and / or processes. The training data set may be annotated with identification data. The extraction of anatomical feature data may include identifying a classification associated with a first anatomical object; determining identification information of a second anatomical object from within a second target region surrounded by a first target region of the image data; detecting at least one position associated with one or more third anatomical objects; and detecting one or more other anatomical objects associated with the first anatomical object. The computer-implemented method may also identify new treatment results and update the database with data associated with the new treatment results.

[0008] According to another example, a computer system for generating navigation guidance for a medical device within a body includes at least one memory storing instructions and at least one processor executing the instructions to perform operations, the operations including receiving image data associated with at least one anatomical object, using the at least one processor to apply a trained predictive navigation guidance model to the image data to determine navigation guidance for the medical device with respect to the at least one anatomical object, generating at least one visual representation associated with the navigation guidance based on the determination, and sending instructions to a user device to display at least one visual representation associated with the navigation guidance on top of the image data on a display screen of the user device.

[0009] It should be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention described in the claims.

Brief Description of the Drawings

[0010] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

DETAILED DESCRIPTION OF THE INVENTION

[0011] Both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosed embodiments recited in the claims. The terms used hereinafter, even when used in conjunction with the detailed description of a particular example of the present disclosure, may be construed in the broadest reasonable manner. Indeed, although certain terms may be emphasized hereinafter, any term that is intended to be construed restrictively is clearly and specifically defined as such in this section of the detailed description. Both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the features as claimed.

[0012] In the present disclosure, the term "based on" means "at least partially based on". The singular forms "a", "an", and "the" include plural referents unless the context indicates otherwise. The term "exemplary" is used in the sense of "an example" rather than "ideal". The terms "comprise", "comprising", "include", "including", or other variations thereof are intended to cover non-exclusive inclusion such that a process, method, or product that comprises a list of elements does not necessarily include only those elements, but may include other elements not expressly listed or other elements inherent to such a process, method, article, or apparatus. The term "diameter" may refer to the width when the element is not circular. The term "upper" refers to the direction or side of the device with respect to the orientation during use, and the term "bottom" refers to the opposite of "upper", the direction or side of the device with respect to the orientation during use. The term "exemplary" is used in the sense of "an example" rather than "ideal". Relative terms such as "substantially" and "generally" are used to indicate a possible variation of ±10% of the recited or understood value.

[0013] Any reference to any specific procedure is provided for convenience only in the present disclosure and is not intended to limit the present disclosure. Those skilled in the art will recognize that the concepts underlying the disclosed devices and methods can be utilized in any suitable procedure. For ease of explanation, parts of the device and / or its components are referred to as proximal and distal parts. It should be noted that the term "proximal" is intended to refer to the part closer to the user of the device, and the term "distal" is used herein to refer to the part further away from the user. Similarly, "extending distally" indicates that the component extends in the distal direction, and "extending proximally" indicates that the component extends in the proximal direction.

[0014] In the following description, embodiments are described with reference to the accompanying drawings. According to certain aspects of the present disclosure, as will be discussed in more detail below, a medical device is used to capture information related to one or more biological components (e.g., during a medical procedure), and the captured information is compared to a database of past treatment data or a model trained on past treatment data is applied to the captured information, and then various types of guidance are provided based on the results of the comparison and / or analysis.

[0015] Endoscopic Retrograde Choloangio-Panceatography (ERCP) is a conventionally used procedure for examining the bile ducts. In this procedure, an endoscope is inserted through the mouth and passed into the duodenum. Next, gas is injected into the duodenum, and the entrance point of the common duct for the bile and pancreatic ducts is identified. Sphincterotomy is performed using a tome to widen the opening, thereby making cannula insertion easier. Next, a guidewire is used to enter the common duct and is manipulated up to the bile duct. Once cannulation of the duct is achieved, a cholangioscope can be inserted into the duct via the guidewire. Then, a contrast agent is injected and used in combination with X-rays to identify the area of interest. The physician can then perform various procedures such as stone management or treatment of biliary malignancies.

[0016] Conventionally, cannula insertion into the appropriate duct can be very difficult for various reasons. For example, operating an endoscope with eight degrees of freedom to reach the correct position can be ergonomically difficult even for an experienced and skilled physician. Additionally, as another example, the difficulty of the task can be exacerbated if the duct path beyond the common entry point is not visualized. More specifically, during an ERCP procedure, various types of visualization of the target area are available to the physician (e.g., preoperative magnetic resonance cholangiopancreatography (MRCP), high-resolution imaging after cannula insertion, X-ray, preoperative CT scan, etc.), but during the cannula insertion process, only direct visualization (i.e., provided by the endoscope) is utilized. This limited visualization does not provide information about the anatomical structure of the duct beyond the common entry point, which is particularly problematic because the anatomical structure of the duct is patient-specific (i.e., the characteristics of the individual single papilla into which the guidewire needs to be inserted vary). As a result, considering the aforementioned issues, the outcome of a typical procedure is damage to the pancreatic duct (e.g., due to misplacement of the guidewire by the physician, etc.). In more severe cases, this damage can potentially cause pancreatitis.

[0017] The operation of the endoscope is inherently very difficult, and combined with the lack of appropriate visualization during cannula insertion, the learning curve is steep for physicians attempting to master the ERCP procedure. Furthermore, even after proficiency, physicians need to continuously perform these types of procedures (e.g., at least one ERCP procedure per week) to maintain their skill level, which can be very demanding, burdensome, and / or not feasible (e.g., if the physician cannot be in an area where ERCP procedures are frequently performed). Therefore, there is a need to simplify or modify the ERCP procedure to enable more physicians to acquire the procedure in a shorter period, which could potentially lead to better patient care.

[0018] As discussed in more detail below, the present disclosure provides a platform that can provide dynamic guidance to a physician during a procedure, such as an ERCP procedure, by applying a predictive navigation guidance model (i.e., trained from past procedure-related data stored in an accessible ERCP database) to data acquired and associated with a live medical procedure. More specifically, anatomical feature data (e.g., characteristics of the target papilla and / or the anatomical structure of the associated ducts) can be extracted from image data initially captured using one or more sensors (e.g., camera / video sensors, etc.) associated with an endoscope and / or one or more other imaging modalities (e.g., fluoroscopy, ultrasound, etc.). Additionally, in some embodiments, medical device position data (e.g., the position, angle, and / or movement of a medical device relative to a target anatomical object) may be captured using one or more other sensors (e.g., electromagnetic sensors, etc.). The accumulated live procedure data is then submitted as an input to the predictive navigation guidance model, which can then analyze the data to determine navigation guidance for manipulating the guidewire of the endoscope through the appropriate opening of the papilla. This guidance may be transmitted to a user device (e.g., a computing device integrally or operably coupled to the endoscope, etc.) and may appear as one or more visual displays (e.g., recommended cannula insertion trajectories, annotations, notifications, etc.) that can be overlaid on the live procedure image data to assist the physician in completing the procedure.

[0019] It is important to note that the techniques utilized herein are described with explicit reference to ERCP procedures, but such designation is not limiting. More specifically, the machine learning models described herein may be trained to identify features associated with other anatomical objects / structures and, correspondingly, may be utilized to provide guidance for other types of medical procedures.

[0020] Figure 1 shows an exemplary environment 100 that can be utilized with the techniques presented herein. One or more user devices 105 can communicate via network 101 with one or more medical devices 110 and / or one or more server systems 115. One or more user devices 105 can be associated with a user, e.g., a user related to generating, training, using, or tuning a machine learning model to provide predictive navigation guidance during a medical procedure. For example, one or more user devices 105 can be associated with a physician performing a medical procedure, e.g., an ERCP procedure, seeking to obtain benefits derived from the capabilities of the (one or more) server systems 115.

[0021] In some embodiments, the components of environment 100 may be associated with a common entity, e.g., a single business or organization, or one or more of the components may be associated with an entity different from other entities. The systems and devices of environment 100 can communicate in any configuration. For example, one or more user devices 105 and / or medical devices 110 may be associated with one or more clients or service subscribers, and the (one or more) server systems 115 may be associated with a service provider responsible for receiving treatment data from one or more clients or service subscribers and then returning an output to one or more clients or service subscribers using the capabilities of the server systems 115. As further described herein, the systems and / or devices of environment 100 can communicate to generate, train, and / or utilize a machine learning model to characterize aspects of a medical procedure and dynamically provide predictive navigation guidance, among other activities.

[0022] The user device 105 may be configured to enable a user to access and / or interact with other systems of the environment 100. For example, the user device 105 may be a computer system such as, for example, a desktop computer, a mobile device, a tablet, etc. In some embodiments, the user device 105 may include one or more electronic applications installed on the memory of the user device 105, such as, for example, programs, plugins, browser extensions, etc.

[0023] User device 105 may include a display user interface (UI) 105A, a processor 105B, a memory 105C, and a network interface 105D. The user device 105 can execute an operating system (O / S) and at least one electronic application (each stored in the memory 105C) by the processor 105B. The electronic application can be a desktop program, a browser program, a web client, or a mobile application program (which may be a browser program of a mobile O / S), a program exclusive to the applicant, system control software, system monitoring software, software development tools, etc. For example, the environment 100 can expand information on a web client that can be accessed via a web browser. In some embodiments, the (one or more) electronic applications may be associated with one or more of the other components within the environment 100. The application can manage a memory 105C such as a database and transmit medical treatment data to the network 101. The display / UI 105A may be a touch screen or a display with other input systems (such as a mouse, keyboard, etc.) so that one or more users can interact with the application and / or the O / S. The network interface 105D may be, for example, an Ethernet (registered trademark) or a TCP / IP network interface for wireless communication with the network 110. The processor 105B can generate data, receive user input from the display / UI 105A, and / or receive / send messages to / from the server system 115 while executing an application, and can further execute one or more operations before providing an output to the network 110.

[0024] One or more medical devices 110 of environment 100 may be integrally (e.g., via a wired connection or the like) or operably (e.g., via a wireless connection or the like) coupled to one or more user devices 105 and / or server system 115, and may include one or more medical devices (e.g., endoscopes, other internal imaging devices, etc.). Data (e.g., image / video data, position data, etc.) obtained by sensors of medical device 110 may be transmitted to one or both of user device 105 and / or server system 115.

[0025] In various embodiments, network 101 may be a wide area network ("WAN"), a local area network ("LAN"), a personal area network ("PAN"), etc. In some embodiments, network 101 includes the Internet, and information and data provided between various systems are generated online. "Online" may mean connecting or accessing source data or information from a location remote from other devices or networks connected to the Internet. Alternatively, "online" may refer to connecting or accessing a network (wired or wireless) via a mobile communication network or device. The Internet is a worldwide system of computer networks, a network of networks in which parties at one computer or other device connected to the network can obtain information from any other computer and communicate with parties at other computers or devices. The most widely used part of the Internet is the World Wide Web (often abbreviated as "WWW" or called "the Web"). A "web site page" generally includes a location, a data store, etc., which is hosted and / or operated by a computer system so as to be accessible online, and includes data configured to cause a program such as a web browser to perform operations such as transmitting, receiving, or processing data, and generating a visual display and / or an interactive interface, etc.

[0026] The server system 115 can include an electronic data system and a computer-readable memory such as a hard drive, a flash drive, a disk, etc. In some embodiments, the server system 115 includes and / or interacts with an application programming interface for exchanging data with other systems, e.g., one or more of the other components of the environment 100. The server system 115 can include a repository or source for the extracted raw dataset information and / or can function as a repository or source for the extracted raw dataset information.

[0027] The server system 115 may include a database 115A and at least one server 115B. The server system 115 may be a computer, a computer system (e.g., a (one or more) rack server), and / or a cloud service computer system. The server system may store or have access to a database 115A (e.g., hosted on a third-party server or in memory 115E). The server may include a display / UI 115C, a processor 115D, a memory 115E, and / or a network interface 115F. The display / UI 115C may be a touch screen or a display with other input systems (e.g., a mouse, a keyboard, etc.) for an operator of the server 115B to control the functions of the server 115B. The server system 115 may execute, by the processor 115D, at least one instance of an operating system (O / S) and a servlet program (each stored in the memory 115E). When the user device 105 or the medical device 110 transmits medical treatment data to the server system 115, the received data set and / or data set information may be stored in the memory 115E or the database 115A. The network interface 115F may be, for example, an Ethernet (registered trademark) or a TCP / IP network interface for wireless communication with the network 101.

[0028] Processor 115D may include instructions for implementing and / or executing a predictive navigation guidance platform 120 that may include a medical treatment database 120A (e.g., containing data associated with past ERCP procedures, etc.) and / or a navigation guidance model 120B. The medical treatment database 120A may be continuously updated (e.g., with new medical treatment data). Additionally, the medical treatment database 120A may be used to train the navigation guidance model 120B to dynamically identify correlations between characteristics associated with specific anatomical objects, the placement of one or more components of a medical device, and / or the corresponding results of a procedure from data associated with an immediate medical treatment. The process by which these correlations may be identified will be described later herein by the disclosure related to FIG. 3.

[0029] In one embodiment, both the medical treatment database 120A and the navigation guidance model 120B may be included within the predictive navigation guidance platform 120. Alternatively, one or both of these components may be sub-components of other components within each other, or may exist on other components of the environment 100. For example, the medical treatment database 120A may be incorporated into an application platform on the user device 205, and the navigation guidance model 120B may reside on the server 115B of the server system 115.

[0030] As will be described in further detail below, the server system 115 can generate, store, train, or use one or more machine learning models configured to analyze medical treatment data and provide predictive navigation guidance based on that analysis. The server system 115 can include one or more machine learning models and / or instructions associated with each of the one or more machine learning models, such as instructions for generating a machine learning model, instructions for training a machine learning model, instructions for using a machine learning model, and the like. The server system 115 can include, for example, instructions for reading output features based on the output of a machine learning model and / or instructions for operating the displays 105A and / or 115C to generate one or more output features adjusted based on, for example, a machine learning model.

[0031] The server system 115 can include one or more sets of training data. The training data can include various types of past data related to a specific medical treatment such as ERCP. For example, the training data can include characteristic information (e.g., shape data, size data, orientation data, appearance data, etc.) associated with various types of detected papillae, aperture characteristic information (e.g., number, size, position, association with a duct, etc.) associated with each of the detected papillae, anatomical information associated with the position and / or structure of the bile duct and / or pancreatic duct, additional anatomical feature information (e.g., presence or absence of intramural folds, oral protrusions, ligaments and / or grooves, etc.) associated with the detected papillae, the position of the endoscope with respect to the papilla before and / or during cannula insertion, past ERCP treatment results, and the like.

[0032] In some embodiments, systems or devices other than server system 115 can be used to generate and / or train a machine learning model. For example, such a system may include instructions for generating a machine learning model, training data and ground truth, and / or instructions for training a machine learning model. The resulting trained machine learning model may then be provided to server system 115.

[0033] In some embodiments, a neural network-based machine learning model includes a set of variables, such as nodes, neurons, filters, etc., that are adjusted to different values, e.g., weighted or biased, through the application of training data. In other embodiments, the machine learning model may be based on an architecture such as a support vector machine, decision tree, random forest, or gradient boosting machine (GBM). Alternative embodiments include using techniques such as transfer learning, where one or more machine learning models pre-trained on a large common dataset or a domain-specific dataset can be utilized to analyze the training data.

[0034] In supervised learning, for example, when the ground truth is known for the provided training data, training can proceed by feeding samples of the training data to a model with variables set to initialized values, randomly, e.g., based on Gaussian noise or a pre-trained model. The output can be compared to the ground truth to determine an error, which can then be backpropagated through the model to adjust the values of the variables.

[0035] Training may be performed in any suitable manner, such as in batches, and may include any suitable training method, such as stochastic or non-stochastic gradient descent, gradient boosting, random forest, etc. In some embodiments, a portion of the training data may not be provided during training and / or may be used to validate the trained machine learning model. For example, to evaluate the accuracy of the trained model, the output of the trained model may be compared to the ground truth for that portion of the training data. Training of the machine learning model may be configured to cause the machine learning model to learn the contextual relevance between raw treatment data and the context with which it is associated (e.g., which anatomical features and / or operation of a medical device affected the success rate of an ERCP procedure, etc.). As a result, the trained machine learning model is configured to provide predictive guidance that can increase the success rate of an ERCP procedure.

[0036] In various embodiments, the variables of the machine learning model may be associated with each other in any suitable arrangement to generate an output. For example, in some embodiments, the machine learning model may include a signal processing architecture configured to identify, separate, and / or extract features, patterns, and / or structures of an image or video. For example, the machine learning model may include one or more convolutional neural networks ("CNNs") configured to identify anatomical features associated with a papilla and related anatomical structures, and may include additional architectures, such as connection layers, neural networks, etc., configured to determine the relationships between the identified features and structures to determine an optimal cannula insertion path.

[0037] For example, in some embodiments, the machine learning model of the server system 115 may include a recurrent neural network (RNN). Generally, an RNN is a type of feedforward neural network suitable for processing a series of inputs. In some embodiments, the machine learning model may include a long short-term memory (LSTM) model and / or a sequence-to-sequence (Seq2Seq) model. The LSTM model may be configured to generate an output from samples that take into account at least some previous samples and / or outputs. The Seq2Seq model may be configured to receive, for example, a series of images as input and then generate, as output, a series of annotations and / or predicted medical device movement trajectories.

[0038] Although shown as separate components in FIG. 1, in some embodiments, a component or a part of a component of the environment 100 may be integrated with or incorporated into one or more other components. For example, a part of the display 115C may be integrated into the user device 105 or the like. In some embodiments, one or more operations or aspects of the above-described components may be distributed among one or more other components. Any suitable arrangement and / or integration of the various systems and devices of the environment 100 may be used.

[0039] Training of a Machine Learning Model for Providing Predictive Navigation Guidance FIG. 2 shows an exemplary process for training a machine learning model, such as the navigation guidance model 120B, to identify important anatomical features during an ERCP procedure and provide dynamic guidance to assist the surgeon during the operation.

[0040] In step 205 of the training process, the method may include receiving a compilation of training data sets, for example, data associated with previously completed ERCP procedures. More specifically, for each completed ERCP procedure, the training data may include images, videos, medical reports, etc. associated with one or more anatomical objects of interest (e.g., the papilla, one or more openings on the papilla, bile ducts, pancreatic ducts, etc.) detected during the previously completed ERCP procedure. This data may be captured using one or more sensors associated with a medical device (e.g., an optical camera) and / or other imaging modalities (e.g., X-ray imaging, fluoroscopy, etc.). In one embodiment, the training data may include position and / or movement data of a medical device (e.g., an endoscope) and / or its components (e.g., a guidewire) with respect to one or more anatomical objects during the procedure. The position and / or movement data may be captured using one or more other sensors (e.g., electromagnetic (EM) sensors, accelerometers, gyroscopes, optical fibers, ultrasonic transducers, capacitive or inductive position sensors, etc.) and / or via any other suitable means, such as via observation by a person and / or an automated system, via feedback from a controller for the medical device, etc. In one embodiment, the training data may include an indication of the result of each completed ERCP procedure (e.g., positive result, negative result, severity of the negative result, etc.).

[0041] In one embodiment, each article of training data may be pre-annotated with relevant anatomical feature information. For example, each image of a papilla may be associated with a classification related to the papilla based on shape, orientation, and / or appearance data, openings on the papilla and their corresponding associations (e.g., openings associated with the pancreatic duct, openings associated with the bile duct, etc.), the orientation of each opening (e.g., downward, lateral, etc.), features of the intramural ligament (e.g., intramural folds), diverticula, oral protrusions, zonula and / or grooves, etc. One or more of these can be identified. In one embodiment, each article of training data can identify the path that a medical device follows to insert a cannula into one or both of the openings.

[0042] In one embodiment, a server system (e.g., server system 115) can receive a training data set and store the training data set in a database (e.g., ERCP database 120A on prediction navigation guidance platform 120, database 115A, etc.) and / or memory (e.g., memory 115E). In one embodiment, a user can upload a training data set to a user device (e.g., user device 105) and manually annotate each article of the training data. The user device 105 may or may not store the training data set in memory (e.g., 105C). Once annotated, the user device 105 can send the annotated training data set to the server system 115 via the network 101.

[0043] In step 210, the method may include extracting anatomical feature data from the annotated training data for each training data set related to an ERCP procedure. The extracted anatomical feature data can be used to train a machine learning model to correctly identify and distinguish important anatomical objects related to the ERCP procedure during a live procedure. Additional disclosure regarding how the machine learning model is trained from the extracted anatomical feature data is provided further in the description of FIG. 3 below.

[0044] Returning to FIG. 2, at step 215, position data associated with the recorded positions, angles, and / or movements of the endoscope (and its components) relative to the papilla during cannula insertion in an ERCP procedure within the training data set can be extracted. This position data can be initially obtained using one or more sensors integrally or operably coupled to the endoscope and / or guide wire, including but not limited to electromagnetic (EM) sensors, accelerometers, gyroscopes, fiber optics, ultrasonic transducers, capacitive or inductive position sensors, etc. In one embodiment, the position data may be annotated with positive and / or negative annotations. For example, position data resulting in a successful procedure (e.g., due to an optimal position and / or angle of the endoscope relative to the papilla) may be classified as positive, and position data resulting in an unsuccessful procedure (e.g., due to the angle of approach) may be classified as negative. Thus, the predictive navigation guidance model can be trained based on the accumulation of position data to identify the ideal positions, angles, and / or movements of the endoscope and guide wire for successful cannula insertion into the papilla.

[0045] At step 220, result data associated with the ERCP procedures within the training data set is extracted and can be utilized to train the predictive navigation guidance model. More specifically, each of the ERCP procedures within the training data set can include one or more indicia of how successful or unsuccessful the ERCP procedure was. For the training data set, the result data for each ERCP procedure may be explicitly annotated such that the predictive navigation guidance model can learn to dynamically distinguish between successful and / or unsuccessful procedures and their sub-steps.

[0046] In one embodiment, the result data can provide a binary indication of the success status of an ERCP procedure (e.g., whether the ERCP procedure was overall successful or unsuccessful). In this regard, the success status of the ERCP procedure may be based on whether the cannula insertion into the bile duct was successful. Additionally or alternatively, in another embodiment, the result data may provide a more granular indication of the treatment results occurring during the ERCP procedure. More specifically, the result data can indicate the successful parts of the ERCP procedure (e.g., where the bile duct opening and the pancreatic duct opening were successfully differentiated) and the unsuccessful parts (e.g., where the cannula insertion into the bile duct was unsuccessful due to the approach angle of the guide wire).

[0047] In step 225, the accumulated data extracted from steps 210 - 220 can be used to train a predictive navigation guidance model. In this regard, the trained predictive navigation guidance model can then receive data associated with a live ERCP procedure and apply the knowledge obtained from the training procedure to identify the correlation between the manner of the live ERCP procedure and the manner associated with previously completed ERCP procedures (e.g., as embodied in the trained dataset). Thereafter, the predictive navigation guidance model may be able to provide dynamic guidance to the operator of the medical device (e.g., a physician, etc.), as further described herein.

[0048] Figure 3 shows an exemplary process for extracting anatomical feature data from an annotated training dataset. More specifically, by using a combination of various visual AI neural network frameworks, regions of the target image that gradually become smaller and / or more specific regions can be separated and evaluated based on the ROI and specific targets. Once trained, the predictive navigation guidance model 120B may be able to dynamically and accurately identify specific objects and ROIs from the received data associated with a live ERCP procedure.

[0049] In step 305, the method may first include training a predictive navigation guidance model to classify the nipple type. Such classification is important because the type of nipple present may determine the position, orientation, and / or structure of other anatomical objects (e.g., openings, etc.). Nipple types include regular, small, protruding or pendulous, and wrinkled or raised. For example, referring to FIG. 4, a plurality of potential nipple types are shown. For example, with respect to nipple 405, ROI 405A includes "normal" nipple 405B, i.e., a nipple without specific features and having a "typical" appearance. With respect to nipple 410, ROI 410A includes a "small" nipple 410B having a diameter of 3 millimeters or less, i.e., a nipple that is often flat. With respect to nipple 415, ROI 415A may include a protruding or pendulous nipple 415B, i.e., a nipple that may protrude, project, or bulge into the duodenal lumen, or sometimes hang down and have an appearance with an opening oriented caudally. With respect to nipple 420, ROI 420A may include a wrinkled or raised nipple 420B, i.e., a nipple where the tubular mucosa appears to extend distally from the nipple opening either on a bulge or in a wrinkle.

[0050] In one embodiment, each set of annotated training data can include an explicit designation of the ROI and an explicit labeling of the nipple type. In one embodiment, each image in the training set can generally be captured using "en face" alignment with respect to the target nipple. In one embodiment, for step 305, a Region Convolutional Neural Network (R-CNN) framework, e.g., RESNET-18, is employed and can be trained on a high batch volume of annotated nipple type images to facilitate proper nipple type classification during a live ERCP procedure.

[0051] In step 310, the method may include training a predictive navigation guidance model to identify the opening type. More specifically, in one embodiment, the predictive navigation guidance model may be trained to identify a second ROI (i.e., ROI-2) associated with a specific nipple pattern surrounded by a first ROI (i.e., ROI-1). The type of nipple pattern identified can, correspondingly, determine the characteristics of one or more openings present on the nipple. Referring now to FIG. 5, a plurality of known nipple patterns are shown. For example, nipple-A 505 can correspond to a typical nipple pattern having an annular shape, and some have nodular changes on the central oral side. Nipple-U 510 can correspond to a generally unstructured nipple pattern without a distinct opening. Nipple-LO 515 can correspond to a nipple pattern having a longitudinal groove continuous with the opening, and the length of the groove is longer than the transverse diameter of the bile duct axis of the nipple. Nipple-I 520 can correspond to a nipple pattern having two separate openings for the bile duct and the pancreatic duct, where the oral-side opening is the opening of the bile duct and the anal-side opening is the opening of the pancreatic duct. Nipple-G 525 can correspond to a nipple pattern having a swirling structure.

[0052] In one embodiment, each set of annotated training data can include one or more explicit designations that identify the type of nipple pattern represented by the nipple, identify the region on the nipple where one or more openings are located based on the nipple pattern, and identify the type of access that the opening can present. In one embodiment, as a result of the smaller focus area of ROI-2 compared to ROI-1, fewer convolutions may be required to accurately classify the opening type. Thus, for step 310, a fast R-CNN framework, such as RESTNET-9 or other faster conventional R-CNNs, can be employed and trained on a large amount of annotated nipple patterns and corresponding opening characteristics.

[0053] In step 315, the method may include training a predictive navigation guidance model to detect the position of the bile duct and / or pancreatic duct. More specifically, in one embodiment, the bile duct and / or pancreatic duct are located on the image and can be distinguished from each other and / or from other anatomical objects, if possible (e.g., based on annotations within the training data). Referring now to FIG. 6, a non-limiting example of a situation is provided where the bile duct and / or pancreatic duct can be explicitly depicted in each of the images from FIG. 5. For example, papilla 605 indicates the position of both the bile duct 605A and the pancreatic duct 605B, papilla 610 indicates the position of the bile duct 610A, papilla 615 indicates the position of the bile duct 615A, papilla 620 indicates the position of both the bile duct 620A and the pancreatic duct 620B, and papilla 625 indicates the position of the bile duct 625A.

[0054] In one embodiment, a semantic segmentation model (e.g., SegNet) that can employ a full convolutional network only for the region enclosed by ROI-2 can be used to achieve tube differentiation. Such a model can utilize a two-stage approach that first differentiates the tubes from the surrounding anatomical features found on the papilla and then performs regression to differentiate the tubes from each other. In one embodiment, the full convolutional network can be trained using a large number of annotated images depicting the identification of the bile duct and / or pancreatic duct.

[0055] In steps 320 - 330, one or more detection algorithms can be utilized to identify specific anatomical features on or related to the papilla. For example, in step 320, a detection algorithm can be trained to determine whether there are folds within the wall. In step 325, following or independent of the detection training process performed in step 320, the same or a different detection algorithm can be trained to determine whether there is an oral protrusion. In step 330, following or independent of the detection processes performed in steps 320 and 325, the same or a different detection algorithm can be trained to determine whether there are ligaments and / or grooves. To train the detection algorithms in each of steps 320 - 330 to prepare for detecting the (one or more) specific anatomical features associated with each step, the training dataset may be annotated with the relevant anatomical objects.

[0056] Generation of predictive navigation guidance by applying a trained machine learning model to live procedure data FIG. 7 shows an exemplary process for determining predictive navigation guidance for an ERCP procedure and then providing the guidance to the physician performing the procedure.

[0057] In step 705, an embodiment of the trained predictive navigation model can receive image data related to one or more anatomical objects associated with a live ERCP procedure. For example, the one or more anatomical objects can correspond to the papilla, one or more openings present on the papilla, the bile duct and / or pancreatic duct, other anatomical features or structures related to any of the foregoing, etc. In one embodiment, the image data may be captured by one or more optical sensors of a medical device utilized in the ERCP procedure. For example, the image data may be captured by one or more optical sensors disposed at the distal end of an endoscope.

[0058] In step 710, one embodiment of the trained predictive navigation guidance model can determine the predictive navigation guidance of a medical device used in a medical procedure in relation to a target anatomical object. In this regard, one embodiment can apply image data as an input to the trained predictive navigation guidance model. The trained predictive navigation guidance model may be configured to analyze aspects of the image data to determine relevant correlations between past ERCP procedures and live ERCP procedures. Additionally, in some embodiments, available position data associated with the medical device may also be provided as an input to the trained model.

[0059] In step 710, in response to determining that one or more types of predictive navigation guidance cannot be determined (e.g., due to lack of necessary information, the machine learning model being unable to identify correlations between live procedure data and past procedure data, etc.), one embodiment can send an alert notification (e.g., to the user device 105) in step 715. The alert notification may be an audio notification, a visual notification, or a combination thereof, and may include an explanation indicating the reason why dynamic guidance could not be provided. Alternatively, in another embodiment, no additional action may be taken.

[0060] Conversely, in step 710, in response to determining one or more types of predictive navigation guidance, one embodiment can generate one or more visual representations related to the determined predictive navigation guidance in step 720. In one embodiment, the one or more visual representations may correspond to one or more of an annotation identifying the relevant anatomical object, a recommended trajectory for operating the medical device and / or its components, and / or a feedback notification warning the operator of the medical device of updates occurring during the medical procedure.

[0061] In step 725, one embodiment can send instructions to the user device to display / overlay a visual representation of the prediction guidance over part or all of the image data. For example, in one embodiment, the server system 115 may be configured to send instructions to the user device to annotate one or more relevant anatomical objects during a medical procedure. In one embodiment, potential annotations can include anatomical object coloring / highlighting (such as when each detected relevant anatomical object is colored with a different specific color), ROI designation (such as when the relevant zone within the image data is shown via a target box or contour), text identifiers (such as when each detected relevant anatomical object is identified with text), combinations thereof, and the like. Referring now to FIG. 8, a non-limiting example of an annotation overlaid on image data related to the target papilla is provided. The annotations present in FIG. 8 can include visually distinct anatomical objects (e.g., the intramural segment / diverticulum 82 can be colored blue, the intramural bile duct 84 can be colored gray, the bile duct opening 86 can be colored green, and the pancreatic duct opening 88 can be colored red), as well as ROI designations (e.g., the papilla identification box 90 and the opening area contour 92).

[0062] In another embodiment, the server system 115 may be configured to send instructions to the user device to provide a recommended trajectory to assist in placing, aligning, and / or advancing the ERCP scope and guidewire. In one embodiment, the recommended trajectory may be provided as an overlay on the image data of the anatomical object(s). For example, referring to FIG. 9, a non-limiting exemplary implementation of how the trajectory overlay may be implemented in two different treatment situations is provided. In FIG. 9, conventional endoscopic views of two papillae, namely 905A and 910A, are provided. Each papilla includes a different papilla pattern that affects the location of the openings of the bile duct and pancreatic duct. More specifically, the arrangement of the ducts and corresponding openings may vary between the two papillae shown in 905A and 910A. These differences may be represented in diagrams 905B and 910B. Specifically, it can be seen that while access to both the bile duct and pancreatic duct is possible through a single opening of the papilla shown in 905A, access to the bile duct and pancreatic duct may be possible through dedicated openings of the papilla shown in 910A. Based on the learned knowledge, the predictive navigation guidance model may be able to identify the appropriate duct position for each papilla type and then provide a recommended trajectory overlay for access to the target duct, as shown in 905C and 910C. More specifically, referring to 905C, even though the bile duct and pancreatic duct are accessible through a single opening, the trajectory overlay can still provide an approach trajectory for inserting a cannula into the bile duct at 905C-1 and / or an approach trajectory for inserting a cannula into the pancreatic duct at 905C-2. Additionally, referring to 910C, the predictive navigation guidance model may be able to distinguish between the two identified openings and then provide an approach trajectory for inserting a cannula into the bile duct at 910C-1 and / or an approach trajectory for inserting a cannula into the pancreatic duct at 910C-2.

[0063] In another embodiment, the server system 115 may be configured to send instructions to the user device to provide feedback to the physician when the movement of the endoscope and / or guide wire deviates from the recommended trajectory. In one embodiment, the server system 115 may be configured to provide feedback immediately (e.g., when the deviation from the recommended trajectory is first detected), or alternatively, when the degree of deviation from the recommended trajectory exceeds a predetermined threshold. In one embodiment, the feedback may appear in one or more different forms. For example, the feedback may be a visual alert (e.g., a text alert, an icon alert, an animation alert, etc. presented on the display screen of the user device), an auditory alert (e.g., provided via one or more speakers associated with the user device), a tactile alert (e.g., vibration of the medical device via one or more actuators), or a combination thereof. In one embodiment, the feedback may be presented only once, at predetermined intervals (e.g., every 5 seconds, every 10 seconds, etc.), or continuously. In one embodiment, the feedback may be directive and may suggest adjustments that the medical device operator can make to align the predicted approach path with the recommended trajectory.

[0064] In one embodiment, the server system 115 may be configured not to send predictive navigation guidance unless a confidence weight of a predictive navigation guidance model for a target anatomical object is greater than a predetermined threshold. More specifically, a confidence weight held by a predictive navigation guidance model for a particular anatomical object (e.g., a nipple) may first be identified. The confidence weight may be based on or reflected by training that the predictive navigation guidance model has performed on a particular anatomical object (e.g., a particular type of nipple or nipple pattern, etc.), and more training may correspond to a higher confidence. Next, one embodiment can determine whether this confidence weight is greater than a predetermined confidence threshold, and in response to determining otherwise, can refrain from sending the predictive guidance to the medical device operator.

[0065] Returning to FIG. 7, at 730, one embodiment can optionally update the ERCP database with data associated with a live medical procedure. This updated data can then be used to further train the prediction navigation guidance model (e.g., using the processes described above in FIGS. 2-3). In one embodiment, the types of data obtained from the live medical procedure that can be used to update the ERCP database can include one or more of captured anatomical feature data, detected medical device position / movement data, medical procedure outcome data (e.g., whether the cannula insertion was overall successful or unsuccessful, which parts of the ERCP procedure were successful or unsuccessful, whether there were post-procedure complications such as pancreatitis, etc.), navigation guidance accuracy data (e.g., how accurate the annotations were for identifying anatomical objects, how accurate the recommended trajectories were, etc.). In one embodiment, the update to the ERCP database can be a single batch update. For example, one embodiment can hold all of the captured data related to the medical procedure until the medical procedure is complete. Then, one embodiment can send all of the data related to the medical procedure to the ERCP database. Alternatively, in another embodiment, the ERCP database can be updated continuously (e.g., as the data accumulates) with medical procedure data. For example, one embodiment can send the image data associated with the target anatomical object to the ERCP database substantially immediately upon its capture, and one embodiment can continuously send the movement data associated with the medical device during the medical procedure to the ERCP database substantially immediately upon its detection, etc.

[0066] FIG. 10 is a simplified functional block diagram of a computer 1100 that may be configured as a device for performing the methods of FIGS. 2-3 and 7, according to an exemplary embodiment of the present disclosure. For example, device 1000 may include a central processing unit (CPU) 1020. CPU 1020 may be any type of processor device, including, for example, any type of dedicated or general-purpose microprocessor device. As will be understood by those skilled in the art, CPU 1020 may be a single processor within a multi-core / multi-processor system, and such a system may operate alone or as part of a cluster or a cluster of computing devices operating within a server farm. CPU 1020 may be connected to a data communication infrastructure 1010, such as a bus, message queue, network, or multi-core message passing scheme.

[0067] Device 1000 may also include a main memory 1040, such as random access memory (RAM), and may also include a secondary memory 1030. Secondary memory 1030, such as read-only memory (ROM), may be, for example, a hard disk drive or a removable storage drive. Such a removable storage drive may include, for example, a floppy (registered trademark) disk drive, magnetic tape drive, optical disk drive, flash memory, and the like. The removable storage drive in this example reads from and / or writes to a removable storage unit in a well-known manner. The removable storage unit may include, for example, a floppy (registered trademark) disk, magnetic tape, optical disk, etc., that is read from and written to by the removable storage drive. As will be understood by those skilled in the art, such a removable storage unit generally includes a computer-usable storage medium that stores computer software and / or data.

[0068] In an alternative implementation, the secondary memory 1030 may include other similar means for enabling a computer program or other instructions to be loaded into the device 1000. Examples of such means include program cartridges and cartridge interfaces (such as those found in video game devices), removable memory chips (such as EPROMs or PROMs) and associated sockets, and other removable storage units and interfaces that enable software and data to be transferred from a removable storage unit to the device 1000.

[0069] The device 1000 may also include a communication interface ("COM") 1060. The communication interface 1060 enables software and data to be transferred between the device 1000 and an external device. The communication interface 1060 may include a modem, a network interface (such as an Ethernet (registered trademark) card), a communication port, a PCMCIA slot and card, etc. The software and data transferred via the communication interface 1060 may be in the form of signals that can be electronic signals, electromagnetic signals, optical signals, or other signals that can be received by the communication interface 1060. These signals can be provided to the communication interface 1060 via the communication path of the device 1000, which can be implemented using, for example, wires or cables, optical fibers, telephone lines, cellular phone links, RF links, or other communication channels.

[0070] The hardware elements, operating systems, and programming languages of such devices are essentially conventional, and those skilled in the art are presumed to be sufficiently proficient in them. Device 1000 may also include input and output ports 1050 for connecting to input and output devices such as keyboards, mice, touchscreens, monitors, displays, and the like. Of course, various server functions can be implemented in a distributed manner on multiple similar platforms to distribute the processing load. Alternatively, the server may be implemented by appropriate programming of a single computer hardware platform.

[0071] The systems, apparatuses, devices, and methods disclosed herein are described in detail by way of example, with reference to the figures. The examples described herein are merely illustrative and are provided to assist in the description of the apparatuses, devices, systems, and methods described herein. None of the features or components shown in the drawings or described below are to be considered essential for any particular implementation of these apparatuses, devices, systems, or methods, unless specifically designated as such. For ease of reading and clarity, some components, modules, or methods may be described only with respect to a particular figure. In this disclosure, the identification of a particular technology, arrangement, etc. is either in relation to the particular examples presented or is merely a general description of such technology, arrangement, etc. The identification of specific details or examples is not intended, and should not be construed, as essential or limiting, unless specifically so designated. Even if a combination or partial combination of components is not specifically described, it should not be understood as indicating that the combination or partial combination is not possible. It will be understood that modifications may be made to the examples, arrangements, configurations, components, elements, apparatuses, devices, systems, methods, etc. disclosed and described, and may be desirable for a particular application. Also, with respect to any method described, regardless of whether the method is described in conjunction with a flowchart, unless otherwise specified by the context, any explicit or implicit ordering of the steps performed in the execution of the method does not mean that those steps must be performed in the order in which they are presented. Instead, it should be understood that they may be performed in a different order or in parallel.

[0072] Throughout this disclosure, references to components or modules generally refer to items that can be logically grouped together to perform a function or a group of related functions. Like reference numerals are generally intended to refer to the same or similar components. Components and modules can be implemented in software, hardware, or a combination of software and hardware. The term "software" is used in a broad sense to include not only executable code, such as machine-executable instructions or machine-interpretable instructions, but also firmware and embedded software, data structures, data stores, and computing instructions stored in any suitable electronic format. The terms "information" and "data" are used broadly and include a variety of electronic information, including executable code, in particular content such as text, video data, and audio data, as well as various codes or flags. The terms "information," "data," and "content" may be used interchangeably where context permits.

[0073] The program aspect of the present technology can typically be considered a "product" or "manufactured article" in the form of executable code and / or associated data carried or embodied on a type of machine-readable medium. "Storage" type media include various semiconductor memories, tape drives, disk drives, etc. that can provide non-transitory storage for software programming at any time, tangible memories such as computers, processors, or any or all of their associated modules. All or part of the software may be communicated via the Internet or various other telecommunications networks. Such communication can enable, for example, the loading of software from one computer or processor to another, such as from a management server or host computer of a mobile communication network to a server's computer platform, and / or from a server to a mobile device. Thus, another type of media that can carry software elements includes light, electrical, and electromagnetic waves such as those used over physical interfaces between local devices, through wired and optical terrestrial communication line networks, and via various air links. Physical elements that carry such waves, such as wired or wireless links, optical links, etc., can also be considered media that carry software. As used herein, unless limited to non-transitory tangible "storage" media, terms such as computer or machine "readable media" refer to any media involved in providing instructions to a processor for execution.

[0074] The disclosed methods, devices, and systems are described by way of example with reference to transmitting data, but it should be understood that the disclosed embodiments are applicable to any environment, such as desktop or laptop computers, automotive entertainment systems, home entertainment systems, etc. Also, the disclosed embodiments may be applicable to any type of Internet protocol.

[0075] In the foregoing description of exemplary embodiments of the present invention, it should be understood that various features of the present invention may be grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and facilitating understanding of one or more of the various aspects of the invention. However, this method of disclosure should not be construed as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as reflected by the following claims, aspects of the invention lie in less than all of the features of the single foregoing disclosed embodiment. Accordingly, the claims following the detailed description are hereby expressly incorporated herein by reference in this detailed description, and each claim stands on its own as a separate embodiment of the present invention.

[0076] Furthermore, some of the embodiments described herein include some features included in other embodiments and do not include other features. However, as will be understood by those skilled in the art, combinations of features of different embodiments are within the scope of the present invention and are intended to form different embodiments. For example, in the following claims, any of the claimed embodiments may be used in any combination.

[0077] Accordingly, while particular embodiments have been described, those skilled in the art will recognize that other modifications and further modifications can be made thereto without departing from the technical idea of the present invention, and it is intended to claim all such changes and modifications that fall within the scope of the present invention. For example, functions may be added or removed from a block diagram, and operations may be exchanged between functional blocks. Steps may be added or removed from the methods described within the scope of the present invention.

[0078] The subject matter disclosed above should be regarded as illustrative, not limiting, and the appended claims are intended to cover all such modifications, enhancements, and other implementations that fall within the true spirit and scope of this disclosure. Accordingly, to the fullest extent permitted by law, the scope of this disclosure should be determined by the broadest permissible interpretation of the following claims and their equivalents, and should not be limited or restricted by the foregoing detailed description. Although various implementations of this disclosure have been described, it will be apparent to those skilled in the art that many more implementations are possible within the scope of this disclosure. Accordingly, this disclosure should not be limited except in consideration of the appended claims and their equivalents.

Claims

1. A method performed by a computer to generate navigation guidance for medical devices in the body, In a computer server, receive image data related to at least one anatomical object. Using a processor associated with the computer server, a trained predictive navigation guidance model is applied to the image data to determine navigation guidance for the medical device regarding the at least one anatomical object. Based on the above determination, generate at least one visual representation associated with the navigation guidance. To transmit a command to a user device that communicates with the computer server over a network, to display the at least one visual representation associated with the navigation guidance on top of the image data on the display screen of the user device. A method performed by a computer, including the above.

2. The computer-based method according to claim 1, wherein the medical device is an endoscope equipped with an extendable guidewire.

3. The computer-based method according to claim 1 or 2, wherein the at least one anatomical object corresponds to one or more of a papilla, an opening, or an internal conduit, and the navigation guidance includes a route for the medical device for cannula insertion of the anatomical object.

4. The computer-based method according to claim 1 or 2, wherein the image data is captured by at least one sensor associated with the medical device and / or at least one other imaging device.

5. The computer-based method according to claim 4, wherein the at least one sensor associated with the medical device includes a camera sensor, and the image data captured by the camera sensor includes at least one of shape data, orientation data, and / or appearance data of the at least one anatomical object.

6. The computer-based method according to claim 4, wherein the at least one other imaging device includes an X-ray device and / or an ultrasound device, and the image data captured by the at least one other imaging device includes anatomical structure data.

7. The method performed by a computer according to claim 1 or 2, further comprising receiving position data captured by one or more other sensors associated with the medical device, wherein the one or more other sensors include at least one of an electromagnetic sensor, an accelerometer, a gyroscope, an optical fiber sensor, an ultrasonic transducer, a capacitive position sensor, or an inductive position sensor.

8. A computer-based method according to claim 1 or 2, wherein determining the navigation guidance includes identifying anatomical feature data from the image data using the predictive navigation guidance model.

9. Identifying the aforementioned anatomical feature data is Within the first target region of the image data, identify a first classification associated with a first anatomical object among the at least one anatomical object. Identifying a second classification associated with a second anatomical object among the at least one anatomical object from within a second target region enclosed by the first target region, To detect the location of one or more third anatomical objects within the second target region, To detect one or more other anatomical objects related to the first anatomical object. A computer-based method according to claim 8, including the method described in claim 8.

10. The above decision is, Identifying the confidence weights held by the predictive navigation guidance model for at least one anatomical object, Determining whether the aforementioned confidence weight is greater than a predetermined confidence threshold. Includes, The computer-based method according to claim 1 or 2, wherein generating the navigation guidance is performed only in response to the determination that the confidence weight is greater than a predetermined confidence threshold.

11. The computer-based method according to claim 1 or 2, wherein the at least one visual representation includes one or more of at least one trajectory overlay, at least one annotation, and / or at least one feedback notification.

12. The computer-based method according to claim 11, wherein the at least one trajectory overlay includes a visual representation of a predicted path to an access point of the at least one anatomical object, which a component of the medical device can follow to insert a cannula into the at least one anatomical object, overlaid on an image of the at least one anatomical object.

13. The computer server receives location data of the medical device. Based on the analysis of the received position data, the deviation of the medical device from the predicted path is identified. It further includes, The method performed by a computer according to claim 12, comprising generating the feedback notification in response to the detection that the deviation of the medical device from the predicted path is greater than a predetermined amount.

14. The computer-based method according to claim 11, wherein the at least one annotation includes one or more visual representations overlaid on an image of the at least one anatomical object, each representing a predetermined feature related to the at least one anatomical object.

15. The computer-based method according to claim 14, wherein the one or more visual displays include one or more of a color display, an outline display, and / or a text-based display.

Images