Modality-based transverse vascular imaging side branch matching

Abstract

This disclosure provides a method for matching side branches from different imaging modalities. Side branches detected from a series of intravascular images can be matched with side branches detected from extravascular images. Matching pairs can be determined based on a first identified reference pair and then a technique that takes into account side branch characteristics such as diameter, orientation, relative size, and / or relative order.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to angiography and intravascular or extravascular imaging modalities, and to matching side branches identified with one imaging modality to side branches identified with another imaging modality.

BACKGROUND ART

[0002] Ultrasonic devices insertable into a patient's vasculature have proven to have diagnostic capabilities for various diseases and disorders. For example, intravascular ultrasound (IVUS) imaging systems are used as intravascular imaging modalities to provide information for diagnosing occluded blood vessels and assisting physicians in tasks such as stent selection and deployment, and site selection for atherectomy procedures.

[0003] An IVUS imaging system includes a control module (having a pulse generator, image acquisition and processing components, and a monitor), a catheter, and a transducer disposed within the catheter. The catheter containing the transducer is placed in a lumen or cavity within or adjacent to the region to be imaged, such as the vessel wall or patient tissue near the vessel wall. The pulse generator in the control module generates electrical pulses that are delivered to the transducer and converted into acoustic pulses that are transmitted through the patient tissue. The patient tissue (or other structure) reflects the acoustic pulses, and the reflected pulses are absorbed by the transducer and converted into electrical pulses. The converted electrical pulses are delivered to the image acquisition and processing components and converted into images that can be displayed on the monitor.

[0004] Coronary angiography (angiography) or CT coronary angiography (CTA or CCTA) is a radiographic imaging modality used to evaluate the coronary arteries of the heart. Typically, the patient receives an intravenous injection of contrast agent, after which the heart is scanned using a high-speed CT scanner. CTA and IVUS are often used in combination. For example, physicians typically use radiographic imaging modalities and IVUS to assess the degree of occlusion (or multiple occlusions) in the coronary arteries to diagnose coronary artery disease.

[0005] To assist physicians in reviewing these images, they are often co-registered with each other. For example, each image in a series of IVUS images may be mapped to or co-located with the location of the vessel represented in the X-ray image. Co-registration of angiographic and IVUS images is a groundbreaking technique that dramatically improves the assessment of coronary artery disease by providing a comprehensive understanding of plaque composition, severity, and vascular anatomy. This advance significantly improves the accuracy of stent placement, the development of treatment strategies, and lesion monitoring.

[0006] Therefore, improved co-registration technologies and workflows are needed. [Overview of the Initiative]

[0007] This disclosure provides a system configured to match a side branch detected by one imaging modality (e.g., IVUS) with a side branch detected by another different imaging modality (e.g., angiography) and to adjust the side branches based on the matching. Generally, this disclosure provides side branch matching based on the position, size, and orientation of the side branches. This disclosure provides an advance in co-registration technology by enabling automated co-registration, thereby reducing the need for manual intervention and improving the accuracy of co-registration.

[0008] In some embodiments, the present disclosure may be implemented as a method for cross-modality image co-registration. The method includes the steps of: a computing device identifying a first side branch in a first image frame corresponding to a first imaging modality; the computing device identifying a second side branch in a plurality of second image frames corresponding to a second imaging modality different from the first imaging modality; the computing device matching some of the first side branches with some of the second side branches without user input; and, based on the matching, co-registering the plurality of second image frames with the first image frame, wherein the step of co-registering the plurality of second image frames includes adjusting the position of the second side branches on the first image frame.

[0009] In further embodiments of the method, the first imaging modality is radiographic angiography, and the second imaging modality is intravascular ultrasound, optical coherence tomography, or computed tomography (CT) angiography.

[0010] In a further embodiment of the method, the step of matching some of the first side branches with some of the second side branches includes the steps of: the computing device identifying one of a plurality of first side branches and one of a plurality of second side branches as a reference side branch pair; the computing device identifying a plurality of candidate side branch pairs based in part on the reference side branch pair, wherein each of the plurality of candidate side branch pairs includes at least one of the plurality of first side branches and at least one of the plurality of second side branches; and the computing device identifying a plurality of matched side branch pairs from the plurality of candidate side branch pairs based in part on the characteristics of the plurality of first side branches and the plurality of second side branches, wherein each of the plurality of matched candidate pairs includes one of the plurality of first side branches and one of the plurality of second side branches.

[0011] In further embodiments of the method, the characteristics include one or more of the diameter of the lateral branch, the orientation of the lateral branch, the relative size of the lateral branch, or the relative position of the lateral branch. In a further embodiment of the method, identifying one of a plurality of first collateral branches and one of a plurality of second collateral branches as a reference collateral pair includes a computing device determining the distance along the midline of the vessel between the branch in the first collateral branch and the branch in the second collateral branch, determining whether the distance between the trifoliation in the first collateral branch and the trifoliation in the second collateral branch is less than a threshold distance, and designating the pair of the first collateral branch and the second collateral branch as a reference collateral pair based on the determination that the distance between the trifoliations is less than a threshold distance.

[0012] In a further embodiment, the method may include the steps of determining whether two or more pairs of first and second lateral branches are identified as having a trifle distance less than a threshold distance, and, based on the determination that two or more pairs of first and second lateral branches have a trifle distance less than a threshold distance, designating the pair of first and second lateral branches having the smallest trifle distance as the reference lateral branch pair.

[0013] In a further embodiment, the method includes the steps of determining whether two or more pairs of first and second lateral branches are identified as having a trifle distance less than a threshold distance, and designating the pair of first and second lateral branches having the smallest weight as the reference lateral branch pair, based on the determination that two or more pairs of first and second lateral branches have a trifle distance less than a threshold distance.

[0014] In a further embodiment of the method, the weight is defined as the sum of the distance difference between the three branches and the diameter difference between the side branches. In a further embodiment, the method includes the steps of: determining the distance along the centerline between a first lateral branch and a second lateral branch based on the determination that the distance between the three branches is not less than a threshold distance; determining whether the distance along the centerline between the first lateral branch and the second lateral branch is less than a threshold distance; and designating the first lateral branch and the second lateral branch as a reference lateral branch pair based on the determination that the distance between the first lateral branch and the second lateral branch is less than a threshold distance.

[0015] In a further embodiment, the method includes the step of increasing the threshold distance based on the determination that the distance between the first side branch and the second side branch is not less than the threshold distance. In a further embodiment, the method includes the steps of: determining whether two or more pairs of first and second lateral branches are identified as having a distance between them along a centerline that is less than a threshold distance; and, based on the determination that two or more pairs of first and second lateral branches have a distance between them along a centerline that is less than a threshold distance, designating the pair of first and second lateral branches having the minimum distance between them along a centerline as a reference lateral branch pair.

[0016] In a further embodiment, the method includes the steps of: determining whether two or more pairs of first and second lateral branches are identified as having a distance between them along a centerline that is less than a threshold distance; and, based on the determination that two or more pairs of first and second lateral branches have a distance between them along a centerline that is less than a threshold distance, designating the pair of first and second lateral branches having the smallest weight as a reference lateral branch pair, where the weight is defined as the sum of the difference between the distances between the lateral branches along a centerline and the difference between the diameters of the lateral branches.

[0017] In a further embodiment of the method, the confidence level is determined in part on the difference between the diameter of one of a plurality of first side branches and the diameter of one of a plurality of second side branches in each of a plurality of matched side branch pairs.

[0018] In some embodiments, the present disclosure is implemented as a computer-readable storage device. The computer-readable storage device comprises instructions that can be executed by a processor of a computing device connected to an intravascular imaging device and / or a fluoroscopy device, and when executed, the instructions cause the computing device to perform one of the methods disclosed herein.

[0019] In some embodiments, the present disclosure may be implemented as an apparatus. The apparatus comprises a processor configured to be connected to an intravascular imaging apparatus and / or a radiofluoroscopy apparatus, the apparatus further comprises a memory containing instructions, the processor configured to execute instructions for performing any of the methods disclosed herein.

[0020] In some embodiments, the present disclosure may be implemented as an apparatus for a cross-modality image co-registration system. The apparatus comprises a processor configured to be connected to an intravascular imaging apparatus and / or a radiofluoroscopy apparatus, and a memory storage device having instructions executable by the processor, which, when executed by the processor, cause the apparatus to identify a first side branch in a first image frame corresponding to a first imaging modality, identify a second side branch in a plurality of second image frames corresponding to a second imaging modality different from the first imaging modality, match some of the first side branches with some of the second side branches without user input, and, based on the matching, co-register the plurality of second image frames with the first image frame, the co-registration of the plurality of second image frames including adjusting the position of the second side branches on the first image frame.

[0021] In further embodiments of the apparatus, the first imaging modality is radiographic angiography, and the second imaging modality is intravascular ultrasound, optical coherence tomography, or computed tomography (CT) angiography.

[0022] In a further embodiment of the device, during execution, an instruction for matching some of the first side branches with some of the second side branches further causes the device to identify one of a plurality of first side branches and one of a plurality of second side branches as a reference side branch pair, identify a plurality of candidate side branch pairs based in part on the reference side branch pair, where each candidate side branch pair includes at least one of a plurality of first side branches and at least one of a plurality of second side branches, and to identify a plurality of matched side branch pairs from the plurality of candidate side branch pairs based in part on the characteristics of the plurality of first side branches and a plurality of second side branches, where each pair of the plurality of matched side branch pairs includes one of a plurality of first side branches and one of a plurality of second side branches.

[0023] In further embodiments of the apparatus, the characteristics include one or more of the diameter of the lateral branches, the orientation of the lateral branches, the relative size of the lateral branches, or the relative position of the lateral branches. In a further embodiment of the device, during execution, an instruction to identify one of a plurality of first collaterals and one of a plurality of second collaterals as a reference collateral pair further causes the device to have a computing device determine the distance along the midline of the vessel between the bifurcation in the first collateral and the bifurcation in the second collateral, determine whether the distance between the trifurcation in the first collateral and the trifurcation in the second collateral is less than a threshold distance, and, based on the determination that the distance between the trifurcations is less than a threshold distance, designate the pair of the first collateral and the second collateral as a reference collateral pair.

[0024] In a further embodiment of the device, the instruction causes the device to, at execution, determine whether two or more pairs of first and second side branches are identified as having a tri-branch distance less than a threshold distance, and, based on the determination that two or more pairs of first and second side branches have a tri-branch distance less than a threshold distance, designate the pair of first and second side branches with the smallest tri-branch distance as the reference side branch pair.

[0025] In a further embodiment of the apparatus, the instructions, when executed, further cause the apparatus to determine whether two or more pairs of the first side branch and the second side branch are identified as having an inter-trifurcation distance less than a threshold distance, and based on a determination that the two or more pairs of the first side branch and the second side branch have an inter-trifurcation distance less than the threshold distance, designate the pair of the first side branch and the second side branch having the minimum weight as a reference side branch pair, wherein the weight is defined as the sum of the inter-trifurcation distance difference and the side branch diameter difference.

[0026] In a further embodiment of the apparatus, the instructions, when executed, further cause the apparatus to identify the distance along the centerline between the first side branch and the second side branch based on a determination that the inter-trifurcation distance is not less than the threshold distance, determine whether the distance along the centerline between the first side branch and the second side branch is less than the threshold distance, and designate the first side branch and the second side branch as a reference side branch pair based on a determination that the distance between the first side branch and the second side branch is less than the threshold distance.

[0027] In a further embodiment of the apparatus, the instructions, when executed, further cause the apparatus to increase the threshold distance based on a determination that the distance between the first side branch and the second side branch is not less than the threshold distance.

[0028] In a further embodiment of the apparatus, the instructions, when executed, further cause the apparatus to determine whether two or more pairs of the first side branch and the second side branch are identified as having an inter-side-branch distance along the centerline that is less than a threshold distance, and based on a determination that the two or more pairs of the first side branch and the second side branch have an inter-side-branch distance along the centerline that is less than the threshold distance, designate the pair of the first side branch and the second side branch having the minimum inter-side-branch distance along the centerline as a reference side branch pair.

[0029] In a further embodiment of the apparatus, the instructions, when executed, further cause the apparatus to determine whether two or more pairs of a first side branch and a second side branch are identified as having a distance between the side branches along the centerline that is less than a threshold distance, and based on a determination that the two or more pairs of the first side branch and the second side branch have a distance between the side branches along the centerline that is less than the threshold distance, designate the pair of the first side branch and the second side branch having the minimum weight as a reference side branch pair, wherein the weight is defined as the sum of the difference in the distance between the side branches along the centerline and the difference in the diameters of the side branches.

[0030] In a further embodiment of the apparatus, the confidence level is determined, for each of the plurality of matched side branch pairs, based in part on the difference between the diameter of one of the plurality of first side branches and the diameter of one of the plurality of second side branches.

[0031] In some embodiments, the present disclosure is implemented as a computer-readable storage device. The computer-readable storage device comprises instructions executable by a processor of a computing device connected to an intravascular imaging apparatus and / or a fluoroscopy apparatus, wherein when executed, the instructions cause the computing device to identify a first side branch in a first image frame corresponding to a first imaging modality, identify second side branches in a plurality of second image frames corresponding to a second imaging modality different from the first imaging modality, match some of the first side branches with some of the second side branches without user input, and co-register the plurality of second image frames with the first image frame based on the matching, wherein co-registering the plurality of second image frames comprises adjusting the positions of the second side branches on the first image frame.

[0032] In a further embodiment of the computer-readable storage device, the first imaging modality is X-ray angiography, and the second imaging modality is intravascular ultrasound, optical coherence tomography, or computed tomography (CT) angiography.

[0033] In a further embodiment of the computer-readable storage device, instructions for matching some of the first side branches with some of the second side branches at runtime further cause the computing device to identify one of a plurality of first side branches and one of a plurality of second side branches as a reference side branch pair, identify a plurality of candidate side branch pairs based in part on the reference side branch pair, where each candidate side branch pair includes at least one of a plurality of first side branches and at least one of a plurality of second side branches, and identify a plurality of matched side branch pairs from the plurality of candidate side branch pairs based in part on the characteristics of the plurality of first side branches and the plurality of second side branches, where each pair of the plurality of matched side branch pairs includes one of a plurality of first side branches and one of a plurality of second side branches.

[0034] In further embodiments of the computer-readable storage device, the characteristics include one or more of the diameter of the side branches, the orientation of the side branches, the relative size of the side branches, or the relative position of the side branches. In a further embodiment of the computer-readable storage device, at runtime, an instruction for identifying one of a plurality of first collaterals and one of a plurality of second collaterals as a reference collateral pair further causes the computing device to determine the distance along the midline of the blood vessel between the branch in the first collateral and the branch in the second collateral, to determine whether the distance between the trifoliation in the first collateral and the trifoliation in the second collateral is less than a threshold distance, and, based on the determination that the distance between the trifoliations is less than a threshold distance, to designate the pair of the first collateral and the second collateral as a reference collateral pair.

[0035] In a further embodiment of the computer-readable storage device, the instruction further causes the computing device to determine, at runtime, whether two or more pairs of first and second side branches are identified as having a tri-branch distance less than a threshold distance, and, based on the determination that two or more pairs of first and second side branches have a tri-branch distance less than a threshold distance, to designate the pair of first and second side branches with the smallest tri-branch distance as the reference side branch pair. [Brief explanation of the drawing]

[0036] To facilitate the identification of any element or operation, the leading digit of the reference number refers to the drawing number in which that element first appears. [Figure 1] This figure shows a multimodality side-branch matching system according to at least one embodiment. [Figure 2] This figure shows a routine for matching collateral branches identified from a series of intravascular images with collateral branches identified from extravascular images, according to at least one embodiment. [Figure 3] This figure shows the identification of exemplary angiographic image frames and reference pairs according to at least one embodiment. [Figure 4] This figure shows a routine for identifying a reference pair, according to at least one embodiment. [Figure 5] This figure shows the identification of exemplary angiography image frames and candidate matches according to at least one embodiment. [Figure 6] This figure shows the identification of exemplary angiographic image frames and matched pairs according to at least one embodiment. [Figure 7A] This figure shows an exemplary angiographic image frame and mapped medial vascular branch according to at least one embodiment. [Figure 7B] This figure shows the adjustment of the angiographic image frame and the mapped medial branch of the vessel. [Figure 8]This figure shows a computer-readable storage medium according to at least one embodiment. [Figure 9] This figure shows an exemplary vascular imaging system according to at least one embodiment. [Figure 10] This is a schematic diagram of a machine in the form of a computer system, in which a set of instructions can be executed to cause the machine to perform any one or more of the methods described herein, according to exemplary embodiments. [Modes for carrying out the invention]

[0037] As described above, this disclosure provides a system and technique for matching side branches identified from different imaging modalities (e.g., IVUS and angiography). While this disclosure uses IVUS and angiography as examples to illustrate the technique, it should be noted that any suitable extravascular and intravascular imaging modalities may be used. Furthermore, this disclosure relies on the identified side branches and their information (e.g., location, size, orientation, etc.). The actual identification of side branches is outside the scope of this disclosure.

[0038] Figure 1 shows a side branch matching system 100 according to one embodiment of the present disclosure. Generally, the side branch matching system 100 is a system configured to match side branches detected from extravascular images (e.g., angiography) with side branches detected from intravascular images (e.g., IVUS). For example, the side branch matching system 100 may be configured to match side branches detected from an angiography image frame 118 with side branches detected from an IVUS image frame 120. To this end, the side branch matching system 100 may include a vascular imaging system 102 or be coupled to a vascular imaging system 102. The vascular imaging system 102 can be any of various vascular imaging systems configured to acquire images from multiple imaging modalities (e.g., angiography, IVUS, intravascular OCT, etc.). An example of a vascular imager configured to acquire both external (angiography) and internal (IVUS) vascular images is described with reference to the combined internal and external imaging system 900 shown in Figure 9.

[0039] The side branch matching system 100 includes a computing device 104. The computing device 104 can be any of a variety of computing devices. In some embodiments, the computing device 104 is integrated into and / or implemented by the console of the vascular imaging system 102. In some embodiments, the computing device 104 can be a tablet, laptop, workstation, or server communicatively coupled to the vascular imaging system 102. In yet other embodiments, the computing device 104 may be provided by a cloud-based computing device such as a Computing as a Service (CaaS) system accessible over a network (e.g., the Internet, an intranet, or a wide area network). The computing device 104 may include a processor 106, memory 108, input and / or output (I / O) devices 110, and a network interface 114.

[0040] Processor 106 may include, for example, circuitry or processor logic such as one of various commercially available processors. In some examples, processor 106 may include multiple processors, multithreaded processors, multicore processors (whether the multiple cores coexist on the same or separate dies), and / or other types of multiprocessor architectures in which multiple physically separate processors are linked in some way. In addition, in some examples, processor 106 may include a graphics processing portion, as well as dedicated memory, multithreading, and / or other parallel processing capabilities. In some examples, processor 106 may be an application-specific integrated circuit (ASIC) or a field-programmable integrated circuit (FPGA).

[0041] Memory 108 may include logic, part of which may include an array of integrated circuits forming a non-volatile memory for persistently storing data, or a combination of non-volatile and volatile memory. It should be understood that memory 108 may be based on any of various technologies. In particular, the array of integrated circuits included in memory 108 may be arranged to form one or more types of memory, such as dynamic random-access memory (DRAM), NAND memory, or NOR memory.

[0042] The I / O device 110 can be any of a variety of devices for receiving input and / or providing output. For example, the I / O device 110 may include a keyboard, mouse, joystick, foot pedal, haptic feedback device, LED, etc. The display 112 may be a conventional display or a touch-enabled display. Furthermore, the display 112 can utilize a variety of display technologies, such as liquid crystal displays (LCDs), light-emitting diodes (LEDs), or organic light-emitting diodes (OLEDs).

[0043] The network interface 114 may include logic and / or functions to support communication interfaces. For example, the network interface 114 may include one or more interfaces that operate according to various communication protocols or standards for direct communication or communication over a network communication link. Direct communication may be performed via the use of communication protocols or standards described in one or more industry standards (including their derivatives and variations). For example, the network interface 114 may enable communication over buses such as Peripheral Component Interconnect Express (PCIe), Non-Volatile Memory Express (NVMe), Universal Serial Bus (USB), System Management Bus (SMBus), SAS (e.g., Serial Attached Small Computer System Interface (SCSI)) interface, and Serial AT Attachment (SATA) interface. Furthermore, the network interface 114 may include logic and / or functions that enable communication over various wired or wireless network standards (e.g., 802.11 communication standards). For example, the network interface 114 may be configured to support wired communication protocols or standards such as Ethernet®. As another example, the network interface 114 may be configured to support wireless communication protocols or standards such as Wi-Fi, Bluetooth®, ZigBee®, LTE, 5G, etc.

[0044] Memory 108 may include instructions 116, angiography image frames 118, IVUS image frames 120, angiography image side branch information 122, IVUS image side branch information 124, reference side branch pair 126, candidate side branch pair 128, matched side branch 130, low-confidence side branch matching 132, and confirmed side branch matching 134.

[0045] During operation, the processor 106 executes instruction 116 to cause the computing device 104 to receive angiography image frames 118 and IVUS image frames 120 from the vascular imaging system 102. In some examples, the processor 106 executes instruction 116 to cause the computing device 104 to receive the angiography image frames 118 and IVUS image frames 120 simultaneously with their acquisition. In other examples, the processor 106 can execute instruction 116 to receive either the angiography image frame 118 or the IVUS image frame 120 simultaneously with their acquisition, while other image modalities have been previously acquired. In yet another example, the processor 106 executes instruction 116 to cause the computing device 104 to receive angiography image frames 118 and IVUS image frames 120, both of which have been previously acquired. In each of these examples, the processor 106 executes instruction 116 to cause the computing device 104 to receive a data structure of information containing the indices of the angiography image frame 118 and the IVUS image frame 120 (for example, directly from the vascular imaging system 102, from the network interface 114, from a memory location, etc.).

[0046] Generally, angiography image frame 118 may be a CT image of the patient's heart (or a portion of the patient's heart) acquired after the injection of contrast agent into the patient's vascular system. In some examples, angiography image frame 118 may be a frame from a series of angiography image frames (e.g., a cine-loop). Similarly, IVUS image frame 120 may be a series of ultrasound images acquired from within the blood vessels of the patient's heart as the ultrasound probe is withdrawn through a portion of the vessel. In some embodiments, angiography image frame 118 may be acquired while IVUS image frame 120 is being acquired. In other embodiments, IVUS image frame 120 may be acquired first, followed by angiography image frame 118, or vice versa.

[0047] As described above, this disclosure provides a way to "match" side branches detected from extravascular images (e.g., angiography image frame 118) with side branches detected from intravascular images (e.g., IVUS image frame 120). To this end, angiography image side branch information 122 may be identified from angiography image frame 118, and IVUS image side branch information 124 may be identified from IVUS image frame 120. For example, the processor 106 executes an instruction 116 to cause the computing device 104 to identify angiography image side branch information 122 from angiography image frame 118 and IVUS image side branch information 124 from IVUS image frame 120. In other examples, the angiography image side branch information 122 and the IVUS image side branch information 124 may be identified by another system and / or computing device, and the processor 106 may execute instruction 116 to receive a data structure containing the instructions for the angiography image side branch information 122 and the IVUS image side branch information 124 (for example, from a network interface 114, from a memory location, or similar).

[0048] The precise techniques and systems used to identify angiography image side branch information 122 and IVUS image side branch information 124 are outside the scope of this disclosure, however, angiography image side branch information 122 and IVUS image side branch information 124 include indications of information about side branches. For example, angiography image side branch information 122 may include indications of the position of each side branch defined relative to the angiography image frame 118 (e.g., two-dimensional (2D) position). Furthermore, angiography image side branch information 122 may include indications of the size of each side branch (e.g., in millimeters) and the orientation of each side branch (e.g., left, right). Similarly, IVUS image side branch information 124 may include indications of the size and orientation (angle rotation, etc.) of each side branch detected based on the IVUS image frame 120.

[0049] This disclosure provides a method for "matching" side branches detected based on IVUS image frames 120 with side branches detected based on angiography image frames 118, for the purpose of co-registering images acquired by these different image modalities.

[0050] Generally, the processor 106 executes instruction 116 to cause the computing device 104 to identify the side branches from the angiography image side branch information 122 and the side branches from the IVUS image side branch information 124, and to designate these as the reference side branch pair 126. Furthermore, the processor 106 can execute instruction 116 to identify candidate side branch pairs 128 from the angiography image side branch information 122 and the IVUS image side branch information 124. The candidate side branch pair 128 includes multiple side branch "pairs," each pair referring to a side branch from the angiography image side branch information 122 and a side branch from the IVUS image side branch information 124. Note that some side branches from either the angiography image side branch information 122 or the IVUS image side branch information 124 may constitute multiple "pairs." In other words, candidate branch pairs 128 do not necessarily define branch pairs that have a one-to-one branch mapping between angiography image branch information 122 and IVUS image branch information 124.

[0051] If the candidate side branch pair 128 includes a pair that references a side branch in another pair, the processor 106 executes instruction 116 to cause the computing device 104 to generate a matched side branch 130 based on the candidate side branch pair 128. If the candidate side branch pair 128 does not include a pair that references a side branch in another pair, the processor 106 executes instruction 116 to cause the computing device 104 to specify the candidate side branch pair 128 and the matched side branch 130. In other words, if the candidate branch pair 128 consists of a 1:N (n>1) relationship between the paired branch, for example, if one branch from the IVUS image branch information 124 is paired with two branch from the angiography image branch information 122, or vice versa, the processor 106 executes instruction 116 to cause the computing device 104 to remove one or more pairs from the candidate branch pair 128 and generate a branch 130 having a 1:1 pairing (or mapping) between the branch from the angiography image branch information 122 and the IVUS image branch information 124.

[0052] The processor 106 executes instruction 116 to cause the computing device 104 to identify low-confidence side-branch matching 132 and generate confirmed side-branch matching 134 based in part on the matched side-branch 130 and the low-confidence side-branch matching 132. For example, pairs identified in the matched side-branch 130 with confidence values ​​below a threshold are removed from the matched side-branch 130, resulting in confirmed side-branch matching 134.

[0053] Figures 2 and 4 show routines 200 and 400 according to several embodiments of the present disclosure, respectively. Routine 200 is performed by a side branch matching system 100 or another computing device, as outlined herein, to match angiographic side branches with IVUS side branches and adjust the position of the IVUS side branches on the angiographic image based on the matching. For example, routine 200 may be performed to match angiographic side branches with IVUS side branches, and routine 400 may be performed as part of routine 200 to identify a reference side branch pair.

[0054] Routine 200 may begin in block 202, "In a computing device, receive information on a number of angiographic collaterals detected based on angiographic image frames of the patient's blood vessels," and information on collaterals detected based on angiographic image frames may be received by the computing device. For example, processor 106 executes instruction 116 to cause computing device 104 to receive angiographic collateral information 122. In some examples, processor 106 also executes instruction 116 in block 202 to cause computing device 104 to receive angiographic image frames 118.

[0055] Proceeding to block 204, "In the computing device, information on numerous medial vascular branches detected based on a series of intravascular image frames of the patient's blood vessels," information on side branches detected based on a series of IVUS image frames may be received by the computing device. For example, processor 106 executes instruction 116 to cause computing device 104 to receive IVUS image side branch information 124. In some examples, processor 106 also executes instruction 116 in block 204 to cause computing device 104 to receive IVUS image frames 120.

[0056] Proceeding to block 206, "Computing device maps side branches onto the centerline of the vessel in the angiographic image frame," the IVUS side branches can be mapped onto the centerline of the vessel in the angiographic image frame. For example, processor 106 can execute instruction 116 to map side branches onto the centerline of the vessel in the angiographic image frame 118. This is shown in more detail in Figure 3.

[0057] Figure 3 shows examples of mapping side branches onto the midline of a vessel and identifying reference side branch pairs according to several embodiments. This figure is shown with reference to the side branch matching system 100 and angiography image frame 118. As shown, the angiography image frame 118 includes a display of a vessel 302 having a midline 304. Generally, the midline 304 may correspond to a retraction path associated with the IVUS image frame 120. Identification of the midline is outside the scope of this disclosure. Furthermore, Figure 3 shows IVUS side branches 306 and angiography side branches 308a and 308b mapped onto the midline 304. Initial mapping and placement of side branches are also outside the scope of this disclosure. However, generally, any of the various techniques for identifying the midline for angiography / IVUS co-registration and aligning IVUS side branches onto the midline can be utilized.

[0058] As described above, the angiography image side branch information 122 and the IVUS image side branch information 124 include indications of information beyond simply the position of the side branch. For example, the angiography image side branch information 122 and the IVUS image side branch information 124 include indications of the orientation of the side branch (e.g., left or right, angular rotation, etc.). In the image shown in Figure 3, the angiography side branch 308a is a rightward-oriented side branch, and 308b is a leftward-oriented side branch.

[0059] Returning to Figure 2, routine 200 can proceed to block 208, "The computing device identifies one of several angiographic collaterals and one of several medial vascular collaterals as a reference collateral pair," where one of the collaterals associated with angiographic image collateral information 122 and one of the collaterals associated with IVUS image collateral information 124 may be identified as the reference collateral pair. Figure 4 shows routine 400 that may be implemented in block 208 of routine 200 to identify the reference pair, which will be described in more detail below. However, generally, in block 208, the computing device 104 can identify a "pair" of angiographic collaterals and IVUS collaterals that has the highest confidence that they are the same collateral. For example, the processor 106 can execute instruction 116 to identify this collateral pair and designate it as the reference collateral pair 126. The reference collateral pair 126 enables correlation of which IVUS angle corresponds to which angiographic collateral orientation (e.g., left, right, etc.).

[0060] Figure 4 shows routine 400, which can begin in block 402 and provides a technique for identifying a reference side branch pair. In block 402, “Computing device obtains distance threshold,” the computing device may obtain the distance threshold. For example, processor 106 may obtain the distance threshold (e.g., from memory, from a register location, or similar) by executing instruction 116. In some embodiments, the distance threshold is 2 millimeters. In other embodiments, the distance threshold is 5 millimeters. In some embodiments, the distance threshold can be between 2 and 10 millimeters.

[0061] Proceeding to block 404, "Computing device determines distance along the midline between the trifoliation in the angiographic side branch and the trifoliation in the IVUS side branch," the distance between the trifoliation in the angiographic side branch and the trifoliation in the IVUS side branch can be determined. For example, processor 106 can execute instruction 116 to determine the distance between the angiographic trifoliation and the IVUS trifoliation based on angiographic image side branch information 122 and IVUS image side branch information 124.

[0062] When the process proceeds to determination block 406, "Is the distance between any three branches less than the distance threshold?", a determination may be made as to whether the distance between the angiographic three branches and the IVUS three branches is less than the distance threshold. For example, processor 106 can execute instruction 116 to determine whether the distance along the centerline 304 between any angiographic three branches and the IVUS three branches identified in block 404 is less than the distance threshold. From determination block 406, routine 400 can proceed to either block 408 or determination block 414. Based on the determination in determination block 406 that the distance between the angiographic three branches and the IVUS three branches is not less than the distance threshold, routine 400 can proceed from determination block 406 to determination block 408, or based on the determination in determination block 406 that the distance between the angiographic three branches and the IVUS three branches is less than the distance threshold, routine 400 can proceed from determination block 406 to determination block 414.

[0063] In some embodiments, if multiple trifids are identified in the determination block 406, the processor 106 can execute instruction 116 to reduce the distance threshold until a single pair of IVUS and angiography branches is identified as having a trifid difference less than the threshold distance. That is, the branch with the smallest distance difference can be identified.

[0064] In block 408, "Computing device determines distance along the centerline between IVUS side branch and angiographic side branch," the distance along the centerline between the IVUS side branch and the angiographic side branch can be determined. For example, processor 106 can execute instruction 116 to determine the distance along the centerline 304 between IVUS side branch 306 and angiographic side branch 308a or 308b. Proceeding to determination block 410, "Is the distance between any side branches less than the distance threshold?", a determination can be made as to whether the distance between the angiographic side branch and the IVUS side branch is less than the distance threshold. For example, processor 106 can execute instruction 116 to determine whether the distance along the centerline 304 between IVUS side branch 306 and angiographic side branches 308a and 308b, as determined in block 408, is less than the distance threshold. From determination block 410, routine 400 can proceed to either block 412 or determination block 414. Routine 400 may proceed from determination block 410 to block 412 based on the determination in determination block 410 that the distance between the angiographic side branch and the IVUS side branch is not less than the distance threshold, and routine 400 may proceed from determination block 410 to determination block 414 based on the determination in determination block 410 that the distance between the angiographic side branch and the IVUS side branch is less than the distance threshold.

[0065] In block 412, "Increase the distance threshold by the increment value," the distance threshold can be increased by the increment value (e.g., 0.2 millimeters). For example, processor 106 can execute instruction 116 to increase the distance threshold by the increment value. From block 412, routine 400 can return to block 404 and continue identifying side branch pairs as described above.

[0066] In determination block 414, "Have two or more side branch pairs been identified?", a determination is made regarding whether two or more side branch pairs have been identified in either determination block 406 or determination block 410. For example, processor 106 can execute instruction 116 to determine whether two or more side branch pairs (e.g., multiple 1:1 pairs, a combination of 1:1 pairs and 1:n pairs, or at least one 1:n pair) have been identified. From determination block 414, routine 400 can proceed to either block 416 or block 418. Based on the determination that two or more side branch pairs were identified in determination block 414, routine 400 can proceed from determination block 414 to block 416, and based on the determination that only one side branch pair was identified in determination block 414, routine 400 can proceed from determination block 414 to block 418.

[0067] In block 416, “Computing device identifies weights for each of the side branch pairs,” weights for each of the side branch pairs can be identified. For example, processor 106 can execute instruction 116 to identify weights for each of the side branch pairs identified in either decision block 406 or decision block 410. In some embodiments, the weights may be the sum of the distance difference between the IVUS side branch 306 and the angiography side branch 308a or 308b along the centerline 304 and the diameter difference between the IVUS side branch 306 and the angiography side branch 308a or 308b.

[0068] When the process proceeds to block 420, "Designate the pair with the lowest weight as the reference pair," a pair with a lower weight may be designated as the reference pair. For example, the processor 106 may execute instruction 116 to designate the pair of IVUS side branch 306 and angiography side branch 308a or 308b with a lower weight as the reference side branch pair 126.

[0069] In block 418, "Specify one pair as a reference pair," one pair may be specified as a reference pair. For example, the processor 106 can execute instruction 116 to specify the pair of IVUS side branch 306 and angiography side branch 308a or 308b as the reference side branch pair 126.

[0070] Figure 3 shows a reference branch pair 126 that can be identified based on routine 400. Returning to Figure 2, routine 200 can proceed to block 210 “The computing device identifies, for each IVUS branch, one or more candidate matches to angiographic branches, based in part on the reference pair,” and candidate or potential matches between angiographic branches and IVUS branches can be identified. For example, processor 106 can execute instruction 116 to identify a potential match between IVUS branch 306 and angiographic branch 308a or 308b based on the reference branch pair 126 and the angiographic image branch information 122 and IVUS image branch information 124. Furthermore, processor 106 can execute instruction 116 to designate the potential match as a candidate branch pair 128. In some embodiments, processor 106 can execute instruction 116 to identify a candidate branch pair 128 based on the distance along the centerline 304 between the branches and / or the orientation of the branches. For example, orientations can be keyed based on a reference side branch pair 126. Thus, the processor 106 can execute instruction 116 to identify, for each IVUS side branch 306, angiographic side branches 308a or 308b having similar orientations and located within a threshold distance along the centerline 304.

[0071] Figure 5 shows an example of identifying potential or candidate side branch pairs. This figure is shown with reference to the side branch matching system 100 and an angiography image frame 118. For example, the illustrated IVUS side branch 306 may potentially match with any of the adjacent angiography side branches 308a. That is, the processor 106 can execute instruction 116 to identify a potential match between the IVUS side branch 306 identified in this figure and any of the adjacent angiography side branches 308a (for example, based on the distance between side branches along the centerline 304), and designate this potential match as one of the candidate side branch pairs 128.

[0072] Returning to Figure 2, routine 200 can proceed to block 212 “Computing device identifies matched side branch pairs based on matching candidates,” and 1:1 side branch matching pairs may be identified based on matching candidates. For example, processor 106 can execute instruction 116 to identify matching side branch pairs from candidate side branch pairs where each pair matches 1:1 between IVUS and angiographic side branch, based on angiography image side branch information 122 and IVUS image side branch information 124. In some embodiments, processor 106 can perform a 1:n matching correction in candidate side branch pairs 128 based on the matching weights of each candidate (e.g., the sum of distance difference and diameter difference). For example, referring to Figure 5, a designated IVUS side branch 306 may be designated as a pair of matched side branches 308a having the adjacent angiographic side branch 130 with the minimum weight; that is, it may be designated as a pair of matched side branches 308a having one adjacent angiographic side branch 308a where the sum of the distance difference between the IVUS side branch 306 and the angiographic side branch 308a and the diameter difference between the IVUS side branch 306 and the angiographic side branch 308a is minimized. Furthermore, the processor 106 can execute instruction 116 to designate these 1:1 matchings as matched side branches 130.

[0073] Proceeding to block 214, "Computing device identifies low-confidence matches," pairs of matched side branches 130 with low confidence may be identified. For example, the processor 106 may execute instruction 116 to identify the confidence of pairs in the matched side branches 130, determine which of the pairs have a confidence level below a threshold, and designate these pairs as low-confidence side branch matches 132. In some embodiments, the confidence level may be the difference between diameters, the difference between distances, or a weight (e.g., the sum of the distance difference and the diameter difference).

[0074] Proceeding to block 216, “A computing device identifies a set of verified matchings based in part on low-confidence pairs,” a set of verified matching pairs may be identified by removing low-confidence pairs from matched side branches. For example, processor 106 may identify a verified side branch matching 134 by executing instruction 116 to remove a low-confidence side branch matching 132 from a matched side branch 130.

[0075] Figure 6 shows an example of verified side branch matching 134 shown on an angiography image frame 118. Returning to Figure 2, routine 200 can proceed to block 218, “Computing device adjusts the position of the medial vascular branch based in part on verified side branch matching,” and the position of the IVUS side branch may be adjusted based on the verified side branch matching. For example, processor 106 can execute instruction 116 to adjust the position of the side branch associated with the IVUS image side branch information 124 (e.g., position index, grid coordinates, pixel reference, etc.) based on the verified side branch matching 134. In particular, these positions may be shifted to be linearly correlated with the matched angiography side branch.

[0076] Figures 7A and 7B show examples of adjusting some positions 702 or position indices of the IVUS side branches 306 based on verified side branch matching 134. For example, the position 702 of the IVUS side branch 306 shown in Figure 7A is adjusted based on verified side branch matching 134, as shown in Figure 7B.

[0077] Figure 8 shows a computer-readable storage medium 800. The computer-readable storage medium 800 may comprise any non-temporary computer-readable or machine-readable storage medium, such as an optical storage medium, a magnetic storage medium, or a semiconductor storage medium. In various embodiments, the computer-readable storage medium 800 may constitute a manufactured product. In some embodiments, the computer-readable storage medium 800 may store computer-executable instructions 802 that a circuit (e.g., a processor 106) can execute. For example, the computer-executable instructions 802 may include instructions for implementing operations described with respect to the side-branch matching system 100, which can improve the functionality of the side-branch matching system 100, as detailed herein. For example, the computer-executable instructions 802 may include instructions that cause a computing device to execute routine 200 in Figure 2 or routine 400 in Figure 4. As another example, the computer-executable instructions 802 may include instruction 116. Examples of computer-readable storage media 800 or machine-readable storage media may include any tangible medium capable of storing electronic data, including volatile or non-volatile memory, removable or non-removable memory, erasable or non-erasable memory, writable or rewritable memory, etc. Examples of computer executable instructions 802 may include any appropriate type of code, such as source code, compiled code, interpreted code, executable code, static code, dynamic code, object-oriented code, visual code, etc.

[0078] Figure 9 shows a combined internal and external imaging system 900 that includes both an intraluminal imaging system 902 (e.g., an IVUS imaging system) and an extravascular imaging system 904 (e.g., an angiography imaging system). The combined internal and external imaging system 900 further includes a computing device 906, which optionally includes circuits, controllers, and / or a processor, memory, and software. In some embodiments, a side branch matching system 100 may be incorporated into the computing device 906, or the side branch matching system 100 may incorporate the computing device 906. Generally, the intraluminal imaging system 902 may be configured to generate intravascular imaging data (e.g., IVUS images), while the extravascular imaging system 904 may be configured to generate extravascular imaging data (e.g., angiography images).

[0079] The extravascular imaging system 904 may include a table 908 that can be positioned to provide sufficient space for positioning the C-arm 910 of the angiography / fluoroscopy unit in an operating position relative to the patient 912 in contact with the drive unit. The C-arm 910 may be configured to acquire fluoroscopic images when no contrast agent is present in the patient's 912 blood vessels, and / or to acquire angiographic images when a contrast agent is present in the patient's 912 blood vessels.

[0080] Raw radiographic image data acquired by the C-arm 910 may be transmitted to an extravascular data input port 914 via a transmission cable 916. The input port 914 may be a separate component, integrated into a computing device 906, or part of the computing device 906. The input port 914 may include a processor that converts the raw radiographic image data received by the input port 914 into extravascular image data (e.g., angiography data / fluoroscopy image data) in the form of, for example, live video, DICOM, or a series of individual images. The extravascular image data may first be stored in memory within the input port 914 or in memory of the computing device 906. If the input port 914 is a separate component from the computing device 906, the extravascular image data may be transferred to the computing device 906 via the transmission cable 916 and input to an input port (not shown) of the computing device 906. In some alternative configurations, communication between devices or processors may be performed via wireless communication rather than by cables as shown.

[0081] Intravascular imaging data may be, for example, IVUS data or OCT data acquired by an intravascular imaging system 902. The intravascular imaging system 902 may include an intravascular imaging device such as an imaging catheter 920. The imaging catheter 920 is configured to be inserted into the body of the patient 912 such that the distal end of the imaging catheter 920, including a diagnostic assembly or probe 922 (e.g., an IVUS probe), is near a desired imaging location in the blood vessel. Radiopaque material or a marker 924 placed on or near the probe 922 may provide an indicator of the current location of the probe 922 in the radiographic image. In some embodiments, the imaging catheter 920 and / or probe 922 may include a guide catheter (not shown) inserted into the lumen of the subject (e.g., a blood vessel such as a coronary artery) on a guidewire (not shown). However, in some embodiments, the imaging catheter 920 and / or probe 922 may be inserted into the blood vessel of the patient 912 without a guidewire.

[0082] In some embodiments, the imaging catheter 920 and / or probe 922 may also include other data acquisition functions in addition to imaging functions. For example, data related to FFR and / or iFR data, pressure, flow rate, temperature, electrical activity, oxygenation, biochemical composition, or any combination thereof. In some embodiments, the imaging catheter 920 and / or probe 922 may further include therapeutic devices such as stents, balloons (e.g., angioplasty balloons), grafts, filters, valves, and / or different types of therapeutic intraluminal devices.

[0083] The imaging catheter 920 is coupled to a proximal connector 926 to connect the imaging catheter 920 to the image acquisition device 928. The image acquisition device 928 may be connected to a computing device 906 via a transmission cable 916 or a wireless connection. Intravascular image data may first be stored in the memory of the image acquisition device 928 or in the memory of the computing device 906. If the image acquisition device 928 is a separate component from the computing device 906, the intravascular image data may be transferred to the computing device 906, for example, via the transmission cable 916.

[0084] The computing device 906 may also include one or more additional output ports for transferring data to other devices. For example, the computer may include an output port for transferring data to a data archive or memory device 932. The computing device 906 may also include a user interface (described in more detail below) which includes circuits, processing components, and combinations of instructions executable by the processing components and / or circuits, in order to enable the image recognition and vascular routing or pathfinding described herein, and / or the dynamic co-registration of intravascular and extravascular images using identified vascular routes.

[0085] In some embodiments, the computing device 906 may include user interface devices such as a keyboard, mouse, joystick, touchscreen device (such as a smartphone or tablet computer), touchpad, trackball, voice command interface, and / or other types of user interfaces known in the art.

[0086] The user interface may be rendered and displayed on a display 934 connected to a computing device 906 via a display cable 936. Although the display 934 is shown as separate from the computing device 906, in some examples the display 934 may be part of the computing device 906. Alternatively, the display 934 may be located separately from the computing device 906 and connected wirelessly. In another example, the display 934 may be part of a different computing device from the computing device 906, such as a tablet computer, which may be connected to the computing device 906 via a wired or wireless connection. In some applications, the display 934 includes a head-up display and / or a head-mounted display. In some applications, the computing device 906 generates output on different types of visual, textual, graphical, haptic, audio, and / or video output devices, such as speakers, headphones, a smartphone, or a tablet computer. In some applications, the user interface rendered on the display 934 functions as both an input and output device.

[0087] Figure 10 shows a schematic diagram of machine 1000 in the form of a computer system in which a set of instructions can be executed to cause the machine to perform any one or more of the methods described herein. Specifically, Figure 10 shows a schematic diagram of machine 1000 in the exemplary form of a computer system in which a set of instructions 1008 (e.g., software, programs, applications, applets, apps, or other executable code) can be executed within machine 1000 to cause machine 1000 to perform any one or more of the methods described herein. For example, a set of instructions 1008 could cause machine 1000 to execute instruction 116, routine 200 in Figure 2, routine 400 in Figure 4, etc. More generally, a set of instructions 1008 could cause machine 1000 to match IVUS side branches with angiographic side branches, as described herein.

[0088] Multiple instructions 1008 transform a general, unprogrammed machine 1000 into a specific machine 1000 programmed to perform the functions described and illustrated in a particular manner. In alternative embodiments, machine 1000 may operate as a standalone device or be coupled to other machines (e.g., networked). In a networked configuration, machine 1000 may operate as a server or client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. Machine 1000 may include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a PDA, an entertainment media system, a mobile phone, a smartphone, a mobile device, a wearable device (e.g., a smartwatch), a smart home device (e.g., a smart appliance), another smart device, a web appliance, a network router, a network switch, a network bridge, or any machine capable of sequentially or otherwise executing multiple instructions 1008 that specify the actions performed by machine 1000. Furthermore, although only a single machine 1000 is illustrated, the term “machine” can also be interpreted to include a set of machines 200 that individually or collaboratively execute multiple instructions 1008 to perform any one or more of the methods described herein.

[0089] Machine 1000 may include processors 1002, memory 1004, and I / O components 1042, which may be configured to communicate with each other via a bus 1044, etc. In exemplary embodiments, processor 1002 (e.g., a central processing unit (CPU), a reduced instruction set arithmetic (RISC) processor, a composite instruction set arithmetic (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an ASIC, a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, processors 1006 and 1010, which are capable of executing multiple instructions 1008. The term “processor” is intended to include multicore processors, which may include two or more independent processors (sometimes called “cores”) capable of executing instructions simultaneously. Figure 10 shows multiple processors 1002, but machine 1000 may include a single processor with a single core, a single processor with multiple cores (e.g., a multicore processor), multiple processors with single cores, multiple processors with multiple cores, or any combination thereof.

[0090] Memory 1004 may include main memory 1012, static memory 1014, and storage unit 1016, which are accessible to processor 1002 via bus 1044, etc. Main memory 1004, static memory 1014, and storage unit 1016 store a plurality of instructions 1008 that embody any one or more of the plurality of methods or functions described herein. The plurality of instructions 1008 may also be fully or partially present in main memory 1012, static memory 1014, machine-readable media 1018 in storage unit 1016, at least one of processor 1002 (e.g., in the processor's cache memory), or any suitable combination thereof during their execution by machine 1000.

[0091] The I / O component 1042 may include a wide variety of components for receiving inputs, providing outputs, generating outputs, transmitting information, exchanging information, capturing measurements, and so on. The specific I / O component 1042 included in a particular machine depends on the type of machine. For example, portable devices such as mobile phones are likely to include touch input devices or other such input mechanisms, while headless server machines are unlikely to include such touch input devices. The I / O component 1042 may also include many other components not shown in Figure 10. The I / O component 1042 is grouped according to function simply to simplify the following description, and this grouping is not limiting. In various exemplary embodiments, the I / O component 1042 may include an output component 1028 and an input component 1030. The output component 1028 may include visual components (e.g., displays such as plasma display panels (PDPs), light-emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tubes (CRTs)), acoustic components (e.g., speakers), tactile components (e.g., vibration motors, resistors), and other signal generators. The input component 1030 may include alphanumeric input components (e.g., keyboards, touchscreens configured to receive alphanumeric input, photo-optical keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touchpads, trackballs, joysticks, motion sensors, or other pointing devices), tactile input components (e.g., physical buttons, touchscreens that provide the position and / or force of touch or touch gestures, or other tactile input components), and audio input components (e.g., microphones).

[0092] In further exemplary embodiments, the I / O component 1042 may include, among many other components, a biometric component 1032, a motion component 1034, an environmental component 1036, or a position component 1038. For example, the biometric component 1032 may include components for detecting facial expressions (e.g., hand expressions, facial expressions, voice expressions, gestures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or electroencephalography), and identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or electroencephalography-based recognition). The motion component 1034 may include acceleration sensor components (e.g., accelerometers), gravity sensor components, rotation sensor components (e.g., gyroscopes), and the like. The environmental component 1036 may include, for example, an illuminance sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers for detecting ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an acoustic sensor component (e.g., one or more microphones for detecting background noise), a proximity sensor component (e.g., an infrared sensor for detecting nearby objects), a gas sensor (e.g., a gas detection sensor for detecting the concentration of harmful gases for safety or for measuring airborne pollutants), or other components that can provide displays, measurements, or signals corresponding to the surrounding physical environment. The position component 1038 may include a position sensor component (e.g., a GPS receiver component), an altitude sensor component (e.g., an altimeter or barometer for detecting atmospheric pressure from which altitude can be derived), a compass sensor component (e.g., a magnetometer), and the like.

[0093] Communication can be implemented using a wide variety of technologies. The I / O component 1042 may include a communication component 1040 capable of operating to connect machine 1000 to network 1020 or device 1022 via connections 1024 and 1026, respectively. For example, the communication component 1040 may include a network interface component or another suitable device for interface connection with network 1020. In further embodiments, the communication component 1040 may include a wired communication component, a wireless communication component, a cellular communication component, a near-field communication (NFC) component, a Bluetooth® component (e.g., Bluetooth® Low Energy), a Wi-Fi® component, and other communication components for communicating via other modalities. Device 1022 may be another machine or one of a wide variety of peripheral devices (e.g., a peripheral device connected via USB).

[0094] Furthermore, the communication component 1040 may include components that can detect identifiers or are operable to detect identifiers. For example, the communication component 1040 may include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting one-dimensional barcodes such as Universal Product Code (UPC) barcodes, Quick Response (QR) Code®, Aztec Code, Data Matrix, Dataglyph, MaxiCode, PDF417, Ultra Code, UCC RSS-2D barcodes, and other optical codes), or an acoustic detection component (e.g., a microphone for identifying tagged audio signals). In addition, various types of information, such as location via Internet Protocol (IP) geolocation, location via Wi-Fi® signal triangulation, and location by detection of NFC beacon signals that can indicate a specific location, can be derived via the communication component 1040.

[0095] Various memories (i.e., memory 1004, main memory 1012, static memory 1014, and / or the memory of processor 1002) and / or storage unit 1016 may store one or more sets of instructions and data structures (e.g., software) that embody or are utilized by any or more of the methods or functions described herein. When these instructions (e.g., a number of instructions 1008) are executed by processor 1002, they trigger a variety of operations for carrying out the disclosed embodiments.

[0096] As used herein, the terms “mechanical storage medium,” “device storage medium,” and “computer storage medium” mean the same thing and may be used interchangeably in this disclosure. These terms refer to one or more storage devices and / or media (e.g., centralized or distributed databases, and / or associated caches and servers) that store executable instructions and / or data. Accordingly, these terms are to be interpreted as including, but are not limited to, solid-state memory and magneto-optical media, including internal or external memory of a processor. Specific examples of mechanical storage mediums, computer storage mediums, and / or device storage mediums include non-volatile memory, such as semiconductor memory devices including erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGAs, and flash memory devices, as well as magnetic disks, magneto-optical disks, such as internal hard disks and removable disks, and CD-ROM and DVD-ROM disks. The terms “mechanical storage medium,” “computer storage medium,” and “device storage medium” specifically exclude carrier waves, modulated data signals, and other such mediums, although at least some of them are included under the term “signaling medium,” as described below.

[0097] In various exemplary embodiments, one or more parts of network 1020 could be an ad-hoc network, an intranet, an extranet, a VPN, a LAN, a WLAN, a WAN, a WWAN, a MAN, the Internet, part of the Internet, part of a PSTN, a basic telephone services (POTS) network, a cellular telephone network, a wireless network, a Wi-Fi® network, another type of network, or a combination of two or more such networks. For example, network 1020 or part of network 1020 could include a wireless or cellular network. Connection 1024 could be a code division multiple access (CDMA) connection, a global system for mobile communications (GSM®) connection, or another type of cellular or wireless connection. In this example, connection 1024 may implement any of the various types of data transfer technologies, including single-carrier radio transmission technology (1xRTT), Evolution Data Optimization (EVDO) technology, General-Purpose Packet Radio Service (GPRS) technology, Extended Data Rate (EDGE) technology for GSM® Evolution, 3G Partnership Projects (3GPP®) including 3G, 4G wireless networks, Universal Mobile Telecommunications System (UMTS), High Speed ​​Packet Access (HSPA), Global Interoperability for Microwave Access (WiMAX®), Long-Term Evolution (LTE) standards, others defined by various standards-setting bodies, other long-range protocols, or other data transfer technologies.

[0098] Multiple instructions 1008 may be transmitted and received over network 1020 using a transmission medium via a network interface device (e.g., a network interface component included in communication component 1040) and utilizing one of several well-known transfer protocols (e.g., Hypertext Transfer Protocol (HTTP)). Similarly, multiple instructions 1008 may be transmitted and received using a transmission medium via a connection 1026 to device 1022 (e.g., a peer-to-peer connection). The terms “transmission medium” and “signaling medium” mean the same thing and may be used interchangeably in this disclosure. The terms “transmission medium” and “signaling medium” are to be interpreted as including any intangible medium, including digital or analog communication signals or other intangible medium, that is capable of storing, encoding, or carrying multiple instructions 1008 for execution by machine 1000 and facilitating communication of such software. Accordingly, the terms “transmission medium” and “signaling medium” are to be interpreted as including any form, such as modulated data signals, carrier waves, etc. The term “modulated data signal” means a signal in which one or more of its properties are set or modified in such a way as to encode information within the signal.

[0099] Terms used herein are given their common meanings in the relevant technical field or the meanings indicated by their use in the context, but where a clear definition is provided, that meaning shall prevail.

[0100] In this specification, references to “one embodiment” or “a certain embodiment” do not necessarily refer to the same embodiment, but may refer to the same embodiment. Unless the context clearly indicates otherwise, throughout this specification and the claims, words such as “equipped with,” “equipped with,” etc., are interpreted comprehensively, as opposed to exclusive or exhaustive, meaning “including but not limited to.” Terms used singular or plural include singular or plural, respectively, unless expressly limited to one or more. Also, when used in this application, words such as “in this specification,” “above,” “below,” and similar words refer to the entire application, not to any part thereof. When the claims use the word “or” in relation to a list of two or more items, the word includes any item in the list, all items in the list, and any combination of items in the list, unless expressly limited to one or the other. Any term not expressly defined in this specification has its conventional meaning as generally understood by those skilled in the art.

Claims

1. A method for cross-modality image co-registration, The computing device identifies a first side branch in a first image frame corresponding to a first imaging modality, The computing device identifies a second side branch in a plurality of second image frames corresponding to a second imaging modality different from the first imaging modality, The computing device performs the step of matching some of the first side branches with some of the second side branches without user input, The step of registering the plurality of second image frames together with the first image frame based on the matching, A method comprising the step of co-registering a plurality of second image frames, wherein the step of adjusting the position of the second side branch on the first image frame.

2. The method according to claim 1, wherein the first imaging modality is X-ray angiography, and the second imaging modality is intravascular ultrasound, optical coherence tomography, or computed tomography (CT) angiography.

3. The step of matching some of the first side branches with some of the second side branches is: The computing device identifies one of a plurality of first side branches and one of a plurality of second side branches as a reference side branch pair. The computing device identifies a plurality of candidate branch pairs based in part on the reference branch pair, wherein each candidate branch pair includes at least one of the plurality of first branch pairs and at least one of the plurality of second branch pairs. The method according to claim 1, wherein the computing device identifies a plurality of matched side branch pairs from the plurality of candidate side branch pairs, based in part on the characteristics of the plurality of first side branches and the plurality of second side branches, wherein each of the plurality of matched side branch pairs includes one of the plurality of first side branches and one of the plurality of second side branches.

4. The method according to claim 3, wherein the characteristics include one or more of the diameter of the lateral branches, the orientation of the lateral branches, the relative size of the lateral branches, or the relative position of the lateral branches.

5. Identifying one of the plurality of first lateral branches and one of the plurality of second lateral branches as the reference lateral branch pair means that The computing device determines the distance along the midline of the blood vessel between the branch in the first side branch and the branch in the second side branch. To determine whether the distance between the three branches in the first lateral branch and the three branches in the second lateral branch is less than the threshold distance, The method according to claim 4, further comprising designating the pair of the first lateral branch and the second lateral branch as the reference lateral branch pair based on the determination that the distance between the three branches is less than a threshold distance.

6. The steps include determining whether two or more pairs of the first lateral branch and the second lateral branch are identified as having a distance between three branches less than a threshold distance, The method according to claim 5, further comprising the step of designating the pair of the first and second lateral branches having the smallest distance between the three branches as the reference lateral branch pair, based on the determination that two or more pairs of the first lateral branch and the second lateral branch have a distance between the three branches less than a threshold distance.

7. The steps include determining whether two or more pairs of the first lateral branch and the second lateral branch are identified as having a distance between three branches less than a threshold distance, The method according to claim 5, further comprising the step of designating the pair of the first and second lateral branches having the smallest weight as the reference lateral branch pair, based on the determination that two or more pairs of the first and second lateral branches have a distance between the three branches that is less than a threshold distance.

8. The method according to claim 7, wherein the weight is defined as the sum of the distance difference between the three branches and the diameter difference of the lateral branches.

9. Based on the determination that the distance between the three branches is not less than a threshold distance, the step of determining the distance along the center line between the first side branch and the second side branch, A step of determining whether the distance along the center line between the first side branch and the second side branch is less than a threshold distance, The method according to claim 5, further comprising the step of designating the first lateral branch and the second lateral branch as the reference lateral branch pair based on the determination that the distance between the first lateral branch and the second lateral branch is less than a threshold distance.

10. The method of claim 9, further comprising the step of increasing the threshold distance based on the determination that the distance between the first lateral branch and the second lateral branch is not less than the threshold distance.

11. The steps include determining whether two or more pairs of the first lateral branch and the second lateral branch are identified as having a distance between the lateral branches along the center line that is less than a threshold distance, The method according to claim 9, comprising the step of designating a pair of first and second side branches having the minimum distance between side branches along the center line as the reference side branch pair, based on the determination that two or more pairs of first side branches and second side branches have a distance between side branches along the center line that is less than a threshold distance.

12. The steps include determining whether two or more pairs of the first lateral branch and the second lateral branch are identified as having a distance between the lateral branches along the center line that is less than a threshold distance, The process further includes the step of designating the pair of the first and second side branches having the smallest weight as the reference side branch pair, based on the determination that the distance between the side branches along the center line is less than a threshold distance, among two or more pairs of the first side branches and the second side branches, The method according to claim 9, wherein the weight is defined as the sum of the difference in distances between lateral branches along the center line and the difference in diameters between lateral branches.

13. The method according to claim 12, wherein the confidence level is determined in each of the plurality of matched lateral branch pairs based in part on the difference between the diameter of one of the plurality of first lateral branches and the diameter of one of the plurality of second lateral branches.

14. A computer-readable storage device comprising instructions executable by a processor of a computing device connected to an intravascular imaging device and / or an X-ray fluoroscopy device, wherein, during execution, the instructions cause the computing device to perform the method according to any one of claims 1 to 13.

15. An apparatus comprising a processor configured to be connected to an intravascular imaging apparatus and / or an X-ray fluoroscopy apparatus, wherein the apparatus further comprises a memory having instructions, and the processor is configured to execute the instructions to perform the method according to any one of claims 1 to 13.

Images