Registration geometry reconstruction based on constant fluoroscope snapshots

Abstract

In one embodiment, a method for generating a three-dimensional (3D) anatomic map is provided, the method comprising applying a trained artificial neural network to (a) a set of two-dimensional (2D) fluoroscopy images of a body part of a living subject, and (b) corresponding first 3D coordinates of the set of 2D fluoroscopy images, thereby producing second 3D coordinates of the 3D anatomic map, and rendering the 3D anatomic map to a display in response to the second 3D coordinates.

Description

[0001] Relevant application information This application claims the benefit of U.S. Provisional Patent Application Serial No. 63 / 118,047, filed November 25, 2020, the disclosure of which is incorporated herein by reference. Technical Field

[0002] The present invention relates to medical devices, and specifically, but not exclusively, to the generation of anatomical mapping. Background Technology

[0003] Many medical procedures involve placing probes, such as catheters, inside a patient's body. Position sensing systems have been developed to track these probes. Magnetic position sensing is one method known in the art. In magnetic position sensing, a magnetic field generator is typically placed at a known location outside the patient's body. A magnetic field sensor within the distal end of the probe generates electrical signals in response to these magnetic fields, and these signals are processed to determine the coordinate position of the distal end of the probe. These methods and systems are described in U.S. Patents 5,391,199, 6,690,963, 6,484,118, 6,239,724, 6,618,612, and 6,332,089, in PCT International Patent Publication WO 1996 / 005768, and in U.S. Patent Application Publications 2002 / 0065455, 2003 / 0120150, and 2004 / 0068178. Impedance- or current-based systems can also be used to track position.

[0004] Treatment of arrhythmias involves a medical procedure in which these types of probes or catheters have proven extremely useful. Arrhythmias, and specifically atrial fibrillation, have always been a common and dangerous medical condition, especially among the elderly.

[0005] The diagnosis and treatment of cardiac arrhythmias involve mapping the electrical properties of cardiac tissue, particularly the endocardium, and selectively ablating cardiac tissue by applying energy. Such ablation can stop or alter unwanted electrical signals propagating from one part of the heart to another. Ablation methods disrupt unwanted electrical pathways by creating a non-conductive ablation focus. Various forms of energy delivery for creating ablation focuses have been disclosed, including the use of microwaves, lasers, and more commonly, radiofrequency energy to create conduction blocks along the cardiac tissue walls. In a two-step procedure (mapping followed by ablation), electrical activity at various points within the heart is typically sensed and measured by advancing a catheter containing one or more electrical sensors into the heart and acquiring data at multiple points. This data is then used to select the target endocardial region for ablation.

[0006] Electrode catheters have been widely used in medical practice for many years. They are used to stimulate and map electrical activity in the heart, as well as to ablate sites of abnormal electrical activity. In use, the electrode catheter is inserted into a major vein or artery, such as the femoral vein, and then guided to the cardiac chamber of interest. A typical ablation procedure involves inserting a catheter with one or more electrodes at its distal end into the cardiac chamber. A reference electrode can be provided, typically taped to the patient's skin, or a second catheter positioned in or near the heart can be used to provide the reference electrode. RF (radio frequency) current is applied between the catheter electrodes of the ablation catheter and an unrelated electrode (which may be one of the catheter electrodes), and the current flows through the medium between these electrodes (i.e., blood and tissue). The current distribution can depend on the amount of contact between the electrode surface and the tissue compared to blood, which has a higher conductivity than tissue. Heating of the tissue occurs due to its resistance. The tissue is sufficiently heated to destroy cells in the cardiac tissue, resulting in the formation of a non-conductive ablation focus within the cardiac tissue. In some applications, irreversible electroporation can be performed to ablate the tissue.

[0007] Sensors within the cardiac chambers can detect far-field electrical activity, i.e., peripheral electrical activity originating far from the sensor, which can distort or obscure local electrical activity, i.e., signals originating at or near the sensor. U.S. Patent Application Publication 2014 / 0005664, jointly assigned to Govari et al., discloses the differentiation between local components of intracardiac electrode signals arising from tissue in contact with the electrodes and their far-field contributions to the signal, and explains how therapeutic processes applied to the tissue can be controlled in response to the differentiated local components. Summary of the Invention

[0008] According to embodiments of the present disclosure, a method for generating a three-dimensional (3D) anatomical mapping is provided, the method comprising applying a trained artificial neural network to (a) a set of two-dimensional (2D) fluorescence microscopy images of a body part of a living subject, and (b) corresponding first 3D coordinates of the set of 2D fluorescence microscopy images, thereby generating second 3D coordinates defining the 3D anatomical mapping, and rendering the 3D anatomical mapping to a display in response to the second 3D coordinates.

[0009] Furthermore, according to an embodiment of this disclosure, the set of 2D fluorescence images includes only two 2D fluorescence images.

[0010] Further according to the embodiments of this disclosure, the set of 2D fluorescent mirror images includes the anterior and posterior projections of the body parts and the left anterior oblique projection of the body parts.

[0011] Additionally, according to embodiments of this disclosure, the second 3D coordinates include one or more of the following: mesh vertices of a 3D mesh, and a 3D point cloud.

[0012] Furthermore, according to embodiments of this disclosure, the method includes improving 3D anatomical mapping in response to signals received from electrodes of a catheter inserted into a body part of a living subject.

[0013] Furthermore, according to the embodiments of this disclosure, the first 3D coordinate and the second 3D coordinate are in the same coordinate space.

[0014] Further according to an embodiment of the present disclosure, the method includes training an artificial neural network to generate 3D anatomical maps in response to training data, the training data including multiple sets of 2D fluorescence microscopy images of a corresponding body part of a corresponding live subject, the corresponding 3D coordinates of the multiple sets of 2D fluorescence microscopy images, and the corresponding 3D coordinates of multiple 3D anatomical maps of the corresponding body part of the corresponding live subject.

[0015] In addition, according to the embodiments of this disclosure, the method includes inputting multiple sets of 2D fluorescence microscopy images of corresponding body parts of a corresponding live subject and the corresponding 3D coordinates of the multiple sets of 2D fluorescence microscopy images into an artificial neural network, and iteratively adjusting the parameters of the artificial neural network to reduce the difference between the output of the artificial neural network and the desired output, the desired output including the corresponding 3D coordinates of the multiple 3D anatomical maps.

[0016] Furthermore, according to embodiments of this disclosure, the method includes generating multiple 3D anatomical maps of training data in response to signals received from electrodes of at least one catheter inserted into a body part of a corresponding living subject.

[0017] Further according to an embodiment of this disclosure, each set of 2D fluorescence images in the plurality of sets of 2D fluorescence images includes only two 2D fluorescence images.

[0018] Further according to the embodiments of this disclosure, the multiple sets of 2D fluorescent mirror images include corresponding anterior and posterior projections and corresponding left anterior oblique projections of the corresponding body parts.

[0019] According to another embodiment of this disclosure, a medical system is also provided, comprising: a fluorescence imaging device configured to capture a set of two-dimensional (2D) fluorescence images of a body part of a living subject; a display; and processing circuitry configured to apply a trained artificial neural network to (a) the set of two-dimensional (2D) fluorescence images of a body part of a living subject, and (b) corresponding first 3D coordinates of the set of 2D fluorescence images, thereby generating second 3D coordinates of a 3D anatomical mapping, and rendering the 3D anatomical mapping to the display in response to the second 3D coordinates.

[0020] Furthermore, according to the embodiments of this disclosure, the set of 2D fluorescence images includes only two 2D fluorescence images.

[0021] Furthermore, according to embodiments of this disclosure, the set of 2D fluorescent mirror images includes front and rear projections of body parts and left anterior oblique projections of body parts.

[0022] Further according to an embodiment of this disclosure, the second 3D coordinate includes one or more of the following: the grid vertices of the 3D mesh, and the 3D point cloud.

[0023] Further according to embodiments of the present disclosure, the system includes a catheter that includes electrodes and is configured to be inserted into a body part of a living subject, and the processing circuitry is configured to improve 3D anatomical mapping in response to signals received from the electrodes of the catheter.

[0024] Furthermore, according to the embodiments of this disclosure, the first 3D coordinates and the second 3D coordinates are in the same coordinate space.

[0025] Furthermore, according to embodiments of this disclosure, the fluorescence imaging device is configured to capture multiple sets of two-dimensional (2D) fluorescence images of a corresponding body part of a corresponding live subject, and the processing circuit is configured to train an artificial neural network to generate 3D anatomical maps in response to training data, the training data including multiple sets of 2D fluorescence images of a corresponding body part of a corresponding live subject, the corresponding 3D coordinates of the multiple sets of 2D fluorescence images, and the corresponding 3D coordinates of multiple 3D anatomical maps of the corresponding body part of the corresponding live subject.

[0026] According to a further embodiment of the present disclosure, the processing circuit is configured to input multiple sets of 2D fluorescence microscopy images of corresponding body parts of a corresponding live subject and the corresponding 3D coordinates of the multiple sets of 2D fluorescence microscopy images into an artificial neural network, and to iteratively adjust the parameters of the artificial neural network to reduce the difference between the output of the artificial neural network and the desired output, the desired output including the corresponding 3D coordinates of the multiple 3D anatomical maps.

[0027] Further according to an embodiment of the present disclosure, the system includes at least one catheter including electrodes and configured to be inserted into a body part of a corresponding live subject, and the processing circuitry is configured to generate a plurality of 3D anatomical maps of training data in response to signals received from the electrodes of the at least one catheter inserted into the body part of the corresponding live subject.

[0028] Furthermore, according to the embodiments of this disclosure, each set of 2D fluorescein images in the plurality of sets of 2D fluorescein images includes only two 2D fluorescein images.

[0029] Furthermore, according to the embodiments of this disclosure, the multiple sets of 2D fluorescent mirror images include corresponding anterior and posterior projections and corresponding left anterior oblique projections of the corresponding body parts.

[0030] According to another embodiment of this disclosure, a software product is also provided, comprising a non-transitory computer-readable medium therein storing program instructions that, when read by a central processing unit (CPU), cause the CPU to apply a trained artificial neural network to (a) a set of two-dimensional (2D) fluorescence microscopy images of a body part of a living subject, and (b) corresponding first 3D coordinates of the set of 2D fluorescence microscopy images, thereby generating second 3D coordinates of a 3D anatomical mapping, and rendering the 3D anatomical mapping to a display in response to the second 3D coordinates. Attached Figure Description

[0031] The invention will be understood from the following detailed description taken in conjunction with the accompanying drawings, wherein: Figure 1 A schematic diagram of a medical system constructed and operated according to an exemplary embodiment of the present invention; Figure 2 For including training used Figure 1 A flowchart of the steps in the artificial neural network method of the system; Figure 3 In order to be in Figure 1 A schematic diagram of an artificial neural network trained in the system; Figure 4 For including training used Figure 1 A flowchart detailing the steps in the artificial neural network method for the system; Figure 5 To include Figure 1 A flowchart of the steps in a method that applies a trained artificial neural network to a system; and Figure 6 In order to be in Figure 1 A schematic diagram of the trained artificial network used in the system. Detailed Implementation

[0032] Overview Before catheter-based cardiac chamber mapping, physicians only used fluorescein microscopy to guide the catheter during surgery. The main drawback of fluorescein microscopy was the risk of radiation. However, it did have some advantages, as it provided views of various veins and heartbeats.

[0033] Therefore, catheter-based mapping is commonly used to map the heart chambers based on the movement of the catheter around them to generate a three-dimensional (3D) anatomical mapping of the heart. Catheter tracking can be used using magnetic and / or impedance-based position tracking without fluorescence microscopy. However, fluorescence microscopy can be used, for example, to correctly introduce the sheath. Additionally, fluorescence microscopic images can be registered with magnetic and / or impedance-based position tracking and with the generated 3D mapping. Figure 1 It is rendered to the display.

[0034] While catheter-based mapping offers many advantages over fluorescence microscopy, it can miss certain features, such as veins, which are observed under fluorescence microscopy. For example, some people may have three pulmonary veins, while others may have four. A physician might overlook the fourth vein while moving the catheter and create an anatomical mapping that does not show the fourth vein.

[0035] Embodiments of the present invention address the aforementioned problems by generating an initial 3D anatomical mapping based on two two-dimensional (2D) fluorescence microscopy images of a body part (e.g., anterior-posterior (AP) and left anterior oblique (LAO) projections or any suitable fluorescence microscopy image projection pair) using a trained artificial neural network (ANN). The initial 3D anatomical mapping can then be refined based on this catheter-based mapping by moving a catheter around the body part (e.g., a heart chamber).

[0036] In some implementations, an ANN is trained to generate 3D anatomical mappings based on two 2D fluorescence microscopy images (e.g., anterior-posterior (AP) and left anterior oblique (LAO) projections or any suitable pair of fluorescence microscopy image projections) and corresponding 3D coordinates of the 2D fluorescence microscopy images based on the training data. The training data includes: (a) multiple sets of 2D fluorescence microscopy images from a corresponding electrophysiological (EP) procedure as input to the ANN and the corresponding 3D coordinates of these 2D fluorescence microscopy images; and (b) a carefully acquired catheter-based 3D anatomical mapping from the corresponding EP procedure as the desired output of the ANN. During ANN training, the parameters of the ANN (e.g., weights) are adjusted such that the output of the ANN approximates the desired output within given constraints.

[0037] The coordinates of the fluorescence microscope image (used during the training and application of the ANN) are typically registered with a magnetic and / or impedance-based position tracking system, allowing the 3D anatomical mapping generated by the ANN to be rendered to a display based on the known registration. Therefore, registration (i.e., the coordinates of the fluorescence microscope image to the corresponding coordinates of the 3D anatomical mapping) becomes part of the training process.

[0038] Once trained, the ANN takes two 2D fluorescein images (e.g., anterior-posterior (AP), left anterior oblique (LAO), or any suitable fluorescein image projection pair) and the corresponding 3D coordinates of these two 2D fluorescein images as input, and outputs a 3D anatomical mapping with coordinates in a coordinate system based on a magnetic and / or impedance-based position tracking system. The 3D anatomical mapping can be represented by the vertices of a 3D mesh or a 3D point cloud. For example, the 3D point cloud can be used to generate encapsulated meshes, for instance, using suitable algorithms such as “Marching cubes,” a computer graphics algorithm published by Lorensen and Cline in the 1987 SIGGRAPH conference proceedings for extracting polygonal meshes of isosurfaces from a three-dimensional discrete scalar field.

[0039] System Description Now for reference Figure 1 The figure is a schematic diagram of a medical system 10 constructed and operated according to an exemplary embodiment of the present invention. The medical system 10 is configured to perform catheter insertion surgery on the heart 12 of a living subject, constructed and operated according to the disclosed embodiment of the present invention. The system includes a catheter 14, which is inserted by an operator 16 through the skin into a chamber or vascular structure of the heart 12 into the patient's vascular system. The operator 16 (typically a physician) contacts the distal end 18 of the catheter against the heart wall at the ablation target site. The processing circuitry 22 located in the console 24 can then be used to prepare an electroactivation mapping, anatomical location (i.e., the location of the distal end 18 of the catheter 14), and other functional images according to any suitable method (e.g., the method disclosed in U.S. Patents 6,226,542, 6,301,496, and 6,892,091). An article including elements of the system 10 may be CARTO ® The system, purchased from Biosense Webster, Inc. (Irvine, CA, USA), is capable of generating electroanatomical mappings of the heart as needed for ablation. This system can be modified by those skilled in the art to implement the principles of the embodiments of the invention described herein.

[0040] For example, areas identified as abnormal by evaluation of electrical activation mapping can be ablated by applying thermal energy, for instance, by conducting radiofrequency current through a wire in catheter 14 (or another catheter) to one or more electrodes 21 at the distal end 18 (only some electrodes are labeled for simplicity), which apply radiofrequency energy to the myocardium of heart 12. The energy is absorbed into the tissue, thereby heating (or cooling) the tissue to a point where the tissue permanently loses its electrical excitability (typically about 60°C). If this procedure is successful, non-conductive ablation foci are formed in the cardiac tissue, which can interrupt the abnormal electrical pathways leading to arrhythmias. The principles of this invention can be applied to different ventricles to treat a variety of different arrhythmias.

[0041] The catheter 14 typically includes a handle 20 with suitable controls to allow the operator 16 to manipulate, position, and orient the distal end 18 of the catheter 14 as needed for mapping and ablation. To assist the operator 16, the distal portion of the catheter 14 includes a position sensor (not shown) that provides a signal to processing circuitry 22, which calculates the position of the distal end 18.

[0042] Ablation energy and electrical signals can be transmitted back and forth between the heart 12 and the control console 24 via cable 34. Pacing signals and other control signals can be transmitted from the control console 24 to the heart 12 via cable 34 and electrodes 21.

[0043] The wire connector 35 connects the console 24 to the surface electrode 30 and other components of the positioning subsystem. Electrode 21 and surface electrode 30 can be used to measure tissue impedance at the ablation site, as taught in U.S. Patent 7,536,218.

[0044] The console 24 typically includes one or more ablation power generators 25. The catheter 14 may be adapted to deliver ablation energy (e.g., radiofrequency energy, irreversible electroporation, ultrasound energy, cryotherapy, and laser-generated light energy) to the heart using any known ablation technique. Such methods are disclosed in U.S. Patents 6,814,733, 6,997,924, and 7,156,816.

[0045] Processing circuitry 22 may be an element of the positioning subsystem in system 10 that measures the position and orientation coordinates of catheter 14. In one embodiment, the positioning subsystem includes a magnetic positioning tracking arrangement that determines the position and orientation of catheter 14 by generating a magnetic field in a predetermined workspace using magnetic field generating coil 28 and sensing these magnetic fields at catheter 14. The positioning subsystem may employ impedance measurement, such as that taught in U.S. Patents 7,756,576 and 7,536,218.

[0046] The fluoroscope imaging device 37 includes a C-arm 39, an X-ray source 41, an image intensifier module 43, and an adjustable collimator 45. A control processor (not shown), which may be located in the console 24, allows the operator to control the operation of the fluoroscope imaging device 37, for example, by setting imaging parameters and controlling the collimator 45 to adjust the size and position of the field of view. The control processor can communicate with the fluoroscope imaging device 37 via cable 51 to enable and disable the X-ray source 41, or to limit the emission of the X-ray source to a desired region of interest by controlling the collimator 45, and to acquire image data from the image intensifier module 43. An optional display monitor 49 connected to the control processor allows the operator 16 to view the images produced by the fluoroscope imaging device 37. When the display monitor 49 is not included, the fluoroscope images can be viewed on the display 29 in split-screen mode or alternately with other non-fluoroscope images.

[0047] As described above, catheter 14 is coupled to console 24, allowing operator 16 to observe and control the function of catheter 14. Processing circuitry 22 is typically a computer with appropriate signal processing circuitry. Processing circuitry 22 is coupled to drive display 29. The signal processing circuitry typically receives, amplifies, filters, and digitizes signals from catheter 14, including signals generated by the aforementioned sensors and electrodes 21 located distal to catheter 14. Console 24 and the positioning subsystem receive and use the digitized signals to calculate the position and orientation of catheter 14, analyze the electrical signals from electrodes 21, and generate the desired electroanatomical mapping.

[0048] Typically, system 10 includes other components, but these are not shown in the figures for simplicity. For example, system 10 may include an electrocardiogram (ECG) monitor coupled to receive signals from one or more body surface electrodes to provide ECG synchronization signals to console 24. As described above, system 10 typically also includes a reference position sensor, either located on an externally applied reference patch attached to the outside of the subject's body or on an internal catheter inserted into and held in a fixed position relative to the heart 12. Conventional pumps and tubing are provided for circulating fluid through catheter 14 to cool the ablation site.

[0049] The fluorescence imaging device 37 is typically registered with the coordinate space 31 of the positioning subsystem, which is associated with the field generating coil 28 and the distal end 18 of the catheter 14. Therefore, the image captured by the fluorescence imaging device 37 can be used with the positioning subsystem. For example, a representation of the distal end 18 of the catheter 14 can be rendered onto the display 29 and superimposed on the X-ray image captured by the fluorescence imaging device 37.

[0050] Now for reference Figure 2 and Figure 3 . Figure 2For including training used Figure 1 Flowchart 100 of the steps in the method of artificial neural network 52 of system 10. Figure 3 In order to be in Figure 1 A schematic diagram of the artificial neural network 52 trained in System 10.

[0051] Fluorescent mirror imaging device 37 ( Figure 1 The device is configured to capture (box 102) multiple sets of 2D fluorescein images 54 of a corresponding body part 56 of a live subject 58. Each set of 2D fluorescein images 54 is associated with 3D coordinates 60 of the image. For example, each 2D fluorescein image may contain coordinates in coordinate space identifying at least two given points in the image (e.g., two corners of the image, or a corner and center of the image, or any other suitable point). Alternatively, each 2D fluorescein image may contain coordinates identifying a given point in the image and the orientation of the image in coordinate space. The coordinates may be referenced by the controller of the fluorescein imaging device 37 to any other coordinate space registered with coordinate space 31. Figure 1 If the fluorescence imaging device 37 is fixed relative to the coordinate space 31, then it is not necessary for the controller of the fluorescence imaging device 37 to supply the 3D coordinates of the multiple sets of 2D fluorescence images 54 for each of the multiple sets of 2D perspective images 54, because the 3D coordinates of the multiple sets of 2D fluorescence images 54 are known to the medical system 10.

[0052] In some embodiments, each set of 2D fluorescein images 54 comprises only two 2D fluorescein images. These two 2D fluorescein images are typically orthogonal projections of body part 56. In some embodiments, the multiple sets of 2D fluorescein images 54 include a corresponding anterior-posterior projection and a corresponding left anterior oblique projection of the corresponding body part 56, such as... Figure 3 As shown.

[0053] For each set of 2D fluorescence microscopy images 54, a corresponding anatomical mapping map 62 is generated using a catheter-based method now described in more detail. At least one catheter (e.g., catheter 14) includes an electrode (e.g., electrode 21) and is configured to be inserted (box 104) into a body part 56 of the corresponding living subject 58. Processing circuitry 22 is configured to receive (box 106) signals from the electrodes (e.g., electrode 21) of the catheter (e.g., catheter 14). The catheter is carefully moved around the body part 56 to ensure that the mapping map generated based on the catheter movement is accurate. Processing circuitry 22 ( Figure 1The system is configured to generate (box 108) multiple 3D anatomical maps 62 as training data for the artificial neural network 52 in response to signals received from electrodes of catheters inserted into body parts 56 of the corresponding living subject 58. These multiple 3D anatomical maps 62 can be defined with reference to the corresponding 3D coordinates 64 of mesh vertices and / or 3D point clouds, which may include a 3D mesh.

[0054] Processing circuitry 22 is configured to train (box 110) artificial neural network 52 to generate 3D anatomical maps in response to training data, which includes: (a) multiple sets of 2D fluorescence microscopy images 54 (captured by fluorescence microscopy imaging device 37) of a corresponding body part 56 of a corresponding live subject 58; (b) corresponding 3D coordinates 60 of the multiple sets of 2D fluorescence microscopy images 54; and (c) corresponding 3D coordinates 64 of multiple 3D anatomical maps 62 of a corresponding body part 56 of a corresponding live subject 58. Each set of 2D fluorescence microscopy images 54 and its corresponding 3D coordinates 60 have an associated 3D anatomical map 62 (with corresponding 3D coordinates 64) captured for the corresponding body part 56 of the corresponding live subject 58. In other words, the training data includes a set of 2D fluorescence microscopy images 54, the corresponding 3D coordinates 60 of the set of 2D fluorescence microscopy images 54, and the 3D coordinates 64 of one of a plurality of 3D anatomical mappings 62 of body parts for each of the living subjects 58. Body parts 56 can include any suitable body part, such as a heart chamber. The body parts 56 used to train the artificial neural network 52 are of the same type, such as heart chambers.

[0055] Now for reference Figure 4 It includes Figure 2 The flowchart details the sub-steps within the steps of box 110. See also: Figure 3 .

[0056] A neural network is a network or circuit of neurons, or, in the modern sense, an artificial neural network composed of artificial neurons or nodes. The connections of biological neurons are modeled as weights. Positive weights reflect excitatory connections, while negative values ​​represent inhibitory connections. The input is modified by the weights and summed using a linear combination. Activation functions control the amplitude of the output. For example, an acceptable output range is typically between 0 and 1, or the range can be between -1 and 1.

[0057] These artificial networks can be used for predictive modeling, adaptive control, and applications, and can be trained on datasets. Experience-based self-learning can occur within the network, drawing conclusions from complex and seemingly unrelated groups of information.

[0058] For completeness, a biological neural network consists of one or more groups of chemically connected or functionally related neurons. A single neuron can connect to many other neurons, and the total number of neurons and connections in a network can be extensive. Connections (called synapses) typically form from an axon to a dendrite, but dendritic synapses and other connections are also possible. In addition to electrical signals, other forms of signaling, caused by the diffusion of neurotransmitters, exist.

[0059] Artificial intelligence, cognitive modeling, and neural networks are information processing paradigms inspired by how biological nervous systems process data. Artificial intelligence and cognitive modeling attempt to simulate some characteristics of biological neural networks. In the field of artificial intelligence, artificial neural networks have been successfully applied to speech recognition, image analysis, and adaptive control to build software agents or autonomous robots (in computers and video games).

[0060] In terms of artificial neurons, often referred to as artificial neural networks (ANNs) or simulated neural networks (SNNs), a neural network (NN) is a set of interconnected natural or artificial neurons that process information using mathematical or computational models based on computational connection methods. In most cases, an ANN is an adaptive system that changes its structure based on the flow of external or internal information through the network. More practically, the term neural network is used for modeling nonlinear statistical data or as a decision-making tool. These terms can be used to model complex relationships between inputs and outputs or to find patterns in data.

[0061] In some embodiments, the artificial neural network 52 includes a fully connected neural network (e.g., a convolutional neural network). In other embodiments, the artificial neural network 52 may include any suitable ANN. The artificial neural network 52 may include processing circuitry 22 ( Figure 1 The software that executes and / or the hardware modules that are configured to perform the functions of the artificial neural network 52.

[0062] The artificial neural network 52 includes an input layer 80 from which input is received, and one or more hidden layers 82 that progressively process the input to an output layer 84 from which the output of the artificial neural network 52 is provided. The artificial neural network 52 may include layer weights between layers 80, 82, and 84. The artificial neural network 52 manipulates the data received at the input layer 80 based on the values ​​of the respective layer weights between layers 80, 82, and 84.

[0063] During the training of the artificial neural network 52, the layer weights of the artificial neural network 52 are updated, so that the artificial neural network 52 can perform the data manipulation tasks that the artificial neural network 52 was trained to perform.

[0064] The number and width of layers in the artificial neural network 52 can be configured. As the number and width of layers increase, the accuracy with which the artificial neural network 52 can manipulate data according to the task at hand also increases. However, a larger number of layers and wider layers typically require more training data, more training time, and the training may not converge. For example, the input layer 80 may include 400 neurons (e.g., to compress a batch of 400 samples), and the output layer may also include 400 neurons.

[0065] Training an artificial neural network 52 is typically an iterative process. One method for training the artificial neural network 52 will now be described below. Processing circuit 22 ( Figure 1 The artificial neural network 52 is configured to iteratively adjust (box 112) the parameters (e.g., layer weights) of the artificial neural network 52 to reduce the difference between the output of the artificial neural network 52 and the desired output of the artificial neural network 52. The desired output includes the corresponding 3D coordinates 64 of a plurality of 3D anatomical maps 62.

[0066] The sub-steps of the steps in box 112 are now described below.

[0067] Processing circuit 22 ( Figure 1 The processing circuit 22 is configured to input (box 114, arrow 70) multiple sets of 2D fluorescence microscopy images 54 of corresponding body parts 56 of a corresponding live subject 58 and the corresponding 3D coordinates 60 of the multiple sets of 2D fluorescence microscopy images 54 into the input layer 80 of the artificial neural network 52. The processing circuit 22 is configured to compare the output of the artificial neural network 52 with the desired output (i.e., the corresponding 3D coordinates 64 of the multiple 3D anatomical maps 62) (box 116, arrow 72). This comparison is typically performed using a suitable loss function that calculates the overall difference between all outputs of the artificial neural network 52 and all desired outputs (e.g., the 3D coordinates 64 of all corresponding multiple 3D anatomical maps 62).

[0068] At decision box 118, processing circuit 22 ( Figure 1 The processing circuit 22 is configured to determine whether the difference between the output of the artificial neural network 52 and the desired output is sufficiently small. If the difference between the output of the artificial neural network 52 and the desired output is sufficiently small (branch 120), then the processing circuit 22 is configured to store (box 122) the parameters (e.g., weights) of the trained artificial neural network 52 for application to the trained artificial neural network 52, as referenced. Figure 5 and Figure 6 A more detailed description.

[0069] If the difference is not small enough (branch 124), processing circuit 22 is configured to correct (box 126) the parameters (e.g., weights) of artificial neural network 52 to reduce the difference between the output of artificial neural network 52 and the desired output of artificial neural network 52. In the example above, the difference minimized is the overall difference between all outputs of artificial neural network 52 and all desired outputs (e.g., the 3D coordinates 64 of all corresponding plurality of 3D anatomical maps 62). Processing circuit 22 is configured to correct the parameters using any suitable optimization algorithm (e.g., gradient descent algorithms such as the Adam optimization algorithm). Steps in boxes 114-118 are then repeated.

[0070] Now for reference Figure 5 and Figure 6 . Figure 5 To include Figure 1 Flowchart 200 of the steps in the method of applying the trained artificial neural network 52 in System 10. Figure 6 In order to be in Figure 1 A schematic diagram of the trained artificial network 52 used in system 10.

[0071] Fluorescent mirror imaging device 37 ( Figure 1 The device is configured to capture (box 202) a set of 2D fluorescence microscopic images 86 of a body part 88 of a living subject 90. In some embodiments, the set of 2D fluorescence microscopic images 86 includes only two 2D fluorescence microscopic images. In some embodiments, the set of 2D fluorescence microscopic images 86 includes an anteroposterior (AP) projection and a left anterior oblique (LAO) projection of the body part 88, or any other suitable projection pair. The body part 88 can be any suitable body part, such as a heart ventricle.

[0072] Processing circuit 22 ( Figure 1 The device is configured to apply a trained artificial neural network 52 to (box 204): (a) a set of 2D fluorescence microscopy images 86 of a body part 88 of a live subject 90; and (b) the corresponding 3D coordinates 96 of the set of 2D fluorescence microscopy images 86, thereby generating 3D coordinates 92 of a 3D anatomical mapping 94. In some embodiments, the 3D coordinates 96 of the set of 2D fluorescence microscopy images 86 and the 3D coordinates 92 of the 3D anatomical mapping 94 are in the same coordinate space and are consistent with the above reference. Figure 1 The coordinate space registration of the described positioning subsystem (e.g., Figure 1 The coordinate space 31). The 3D coordinates 92 of the 3D anatomical mapping map 94 may include the mesh vertices of the 3D mesh and / or the 3D point cloud.

[0073] Processing circuitry 22 is configured to render (frame 206) 3D anatomical mapping 94 to display 29 in response to 3D coordinates 92. Figure 1 ).

[0074] Catheter 14 ( Figure 1 The electrode 21 is configured to be inserted (box 208) into and around the body part 88 of the living subject 90 in order to acquire signals via the electrode 21. Figure 1 The processing circuit 22 is configured to receive (box 210) signals from the electrode 21 of the catheter 14 to correct and improve the 3D anatomical mapping 94. The processing circuit 22 is configured to improve (box 212) the 3D anatomical mapping 94 in response to the signals received from the electrode 21 of the catheter 14.

[0075] In implementation, some or all of the functions of processing circuitry 22 may be combined in a single physical component, or alternatively, implemented using multiple physical components. These physical components may include hardwired or programmable devices, or a combination of both. In some embodiments, at least some of the functions of processing circuitry 22 may be implemented by a programmable processor under the control of suitable software. This software may be downloaded to the device electronically via, for example, a network. Alternatively or otherwise, the software may be stored in a tangible, non-transitory computer-readable storage medium, such as optical, magnetic, or electronic memory.

[0076] As used herein, the term “about” or “approximately” for any numerical value or range indicates appropriate dimensional tolerances that allow a collection of parts or components to achieve the intended purpose as described herein. More specifically, “about” or “approximately” may refer to a range of ±20% of the enumerated value; for example, “about 90%” may refer to a range of values ​​from 72% to 108%.

[0077] For clarity, the various features of the invention described in the context of individual embodiments may also be provided in combination in a single embodiment. Conversely, for brevity, the various features of the invention described in the context of individual embodiments may also be provided individually or in any suitable sub-combination.

[0078] The above embodiments are cited by way of example, and the invention is not limited to the specific examples shown and described above. Rather, the scope of the invention includes combinations and sub-combinations of the various features described above, as well as variations and modifications thereof, which will occur to those skilled in the art upon reading the above description and are not disclosed in the prior art.

Claims

1. A method for generating three-dimensional anatomical mapping maps, comprising: The trained artificial neural network was applied to: (a) a set of two-dimensional 2D fluorescence microscopy images of body parts of a live subject; (b) The corresponding first 3D coordinates in coordinate space for each 2D fluorescence microscopy image in the set of 2D fluorescence microscopy images, thereby generating second 3D coordinates that define the 3D anatomical mapping; and The 3D anatomical mapping is rendered to the display in response to the second 3D coordinates.

2. The method according to claim 1, wherein the set of 2D fluorescence images comprises only two 2D fluorescence images.

3. The method according to claim 2, wherein the set of 2D fluorescent images includes the front and rear projections of the body part and the left front oblique projection of the body part.

4. The method of claim 1, wherein the second 3D coordinate comprises one or more of the following: grid vertices of a 3D mesh; and a 3D point cloud.

5. The method of claim 1, further comprising improving the 3D anatomical mapping in response to signals received from electrodes of a catheter inserted into the body part of the living subject.

6. The method according to claim 1, wherein the first 3D coordinate and the second 3D coordinate are in the same coordinate space.

7. The method of claim 1, further comprising training the artificial neural network to generate 3D anatomical mappings in response to training data, the training data comprising: Multiple sets of 2D fluorescence microscopy images of corresponding body parts of the corresponding live subjects; The corresponding 3D coordinates of the multiple sets of 2D fluorescent mirror images; and The corresponding 3D coordinates of multiple 3D anatomical mappings of the corresponding body parts of the corresponding live subject.

8. The method according to claim 7, further comprising: The multiple sets of 2D fluorescence microscopy images of corresponding body parts of the corresponding live subjects and the corresponding 3D coordinates of the multiple sets of 2D fluorescence microscopy images are input into the artificial neural network; as well as The parameters of the artificial neural network are iteratively adjusted to reduce the difference between the output of the artificial neural network and the desired output, which includes the corresponding 3D coordinates of the plurality of 3D anatomical maps.

9. The method of claim 7, further comprising the plurality of 3D anatomical maps for generating the training data in response to signals received from electrodes of at least one catheter inserted into the body part of the respective living subject.

10. The method of claim 7, wherein each set of 2D fluoresce images in the plurality of sets of 2D fluoresce images comprises only two 2D fluoresce images.

11. The method of claim 10, wherein the plurality of 2D fluorescent mirror images include corresponding anterior and posterior projections and corresponding left anterior oblique projections of the corresponding body parts.

12. A medical system comprising: A fluorescence imaging device configured to capture a set of two-dimensional 2D fluorescence images of body parts of a live subject; monitor; and Processing circuit, the processing circuit being configured to: The trained artificial neural network is applied to: (a) a set of two-dimensional 2D fluorescence microscopy images of body parts of a living subject; and (b) the corresponding first 3D coordinates in coordinate space for each 2D fluorescence microscopy image in the set of 2D fluorescence microscopy images, thereby generating second 3D coordinates for a 3D anatomical mapping; and The 3D anatomical mapping is rendered onto the display in response to the second 3D coordinates.

13. The system of claim 12, wherein the set of 2D fluorescence images comprises only two 2D fluorescence images.

14. The system of claim 13, wherein the set of 2D fluorescent images includes anterior and posterior projections of the body part and a left anterior oblique projection of the body part.

15. The system of claim 12, wherein the second 3D coordinate comprises one or more of the following: grid vertices of a 3D mesh; and a 3D point cloud.

16. The system of claim 12, further comprising a catheter including electrodes and configured for insertion into the body part of the living subject, the processing circuitry being configured to improve the 3D anatomical mapping in response to signals received from the electrodes of the catheter.

17. The system of claim 12, wherein the first 3D coordinate and the second 3D coordinate are in the same coordinate space.

18. The system according to claim 12, wherein: The fluorescence imaging device is configured to capture multiple sets of two-dimensional (2D) fluorescence images of corresponding body parts of a live subject; and The processing circuitry is configured to train the artificial neural network to generate 3D anatomical mappings in response to training data, the training data including: The aforementioned multiple sets of 2D fluorescence microscopy images of corresponding body parts of the corresponding live subjects; The corresponding 3D coordinates of the multiple sets of 2D fluorescent mirror images; and The corresponding 3D coordinates of multiple 3D anatomical mappings of the corresponding body parts of the corresponding live subject.

19. The system of claim 18, wherein the processing circuitry is configured to: The multiple sets of 2D fluorescence microscopy images of corresponding body parts of the corresponding live subjects and the corresponding 3D coordinates of the multiple sets of 2D fluorescence microscopy images are input into the artificial neural network; and The parameters of the artificial neural network are iteratively adjusted to reduce the difference between the output of the artificial neural network and the desired output, which includes the corresponding 3D coordinates of the plurality of 3D anatomical maps.

20. The system of claim 18, further comprising at least one catheter including an electrode and configured to be inserted into the body part of the respective live subject, the processing circuitry being configured to generate the plurality of 3D anatomical maps of the training data in response to a signal received from the electrode of the at least one catheter inserted into the body part of the respective live subject.

21. The system of claim 18, wherein each set of 2D fluorescein images in the plurality of sets of 2D fluorescein images comprises only two 2D fluorescein images.

22. The system of claim 21, wherein the plurality of 2D fluorescent images include corresponding anterior-posterior projections and corresponding left anterior oblique projections of the corresponding body parts.

23. A software product comprising a non-transitory computer-readable medium storing program instructions that, when read by a central processing unit (CPU), cause the CPU to: The trained artificial neural network is applied to: (a) a set of two-dimensional 2D fluorescence microscopy images of body parts of a living subject; and (b) the corresponding first 3D coordinates in coordinate space for each 2D fluorescence microscopy image in the set of 2D fluorescence microscopy images, thereby generating second 3D coordinates for a 3D anatomical mapping; and The 3D anatomical mapping is rendered to the display in response to the second 3D coordinates.

Images