Lateral branch detection for intravascular image and extravascular image co-registration
The machine learning model recognizes the lateral branch positions in the intravascular image and selects a high confidence frame subset, which solves the problem of registration of IVUS images with extravascular images or multiple groups of intravascular images, and improves the accuracy and diagnostic efficiency of image association.
Patent Information
- Application Number
- CN202410617874.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-05-18
- Filing Date
- 2024-05-17
- Publication Date
- 2024-11-22
AI Technical Summary
It is difficult for physicians to effectively register intravascular ultrasound (IVUS) images with extravascular images or multiple sets of intravascular images, especially in the association of pre- and post-treatment images.
Using machine learning (ML) models, including convolutional neural networks (CNNs) and vision transformer networks, identify the lateral branch positions in the intravascular image, and select a subset of frames with high confidence from the image frame set through an analysis algorithm. to assist in co-registration of images.
Improves the registration accuracy of intravascular images with extravascular images or another set of intravascular images, and simplifies the image correlation process of physicians in diagnosis and treatment.
Smart Images

Figure CN119006539A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates generally to intravascular ultrasound (IVUS) imaging systems. In particular, but not exclusively, the present invention relates to identifying side branches in a series of intravascular images to assist in co-registering the intravascular images with extravascular images or registering multiple sets of intravascular images of the same vessel. Background Art
[0002] Ultrasound devices that can be inserted into a patient's body have demonstrated diagnostic capabilities for a variety of diseases and conditions. For example, intravascular ultrasound (IVUS) imaging systems have been used as an imaging modality for diagnosing blocked blood vessels and providing information to help medical practitioners select and place stents, select sites for atherectomy surgery, etc.
[0003] The IVUS imaging system includes a control module (which has a pulse generator, image acquisition and processing components and a monitor), a catheter and a transducer disposed in the catheter. The catheter containing the transducer is positioned in a lumen or cavity in or near the area to be imaged, such as a vessel wall or in patient tissue close to the vessel wall. The pulse generator in the control module generates electrical pulses, which are delivered to the transducer and converted into acoustic pulses, which are transmitted through the patient tissue. The patient tissue (or other structure) reflects the acoustic pulses, and the reflected pulses are absorbed by the transducer and converted into electrical pulses. The converted electrical pulses are delivered to the image acquisition and processing components and converted into images that can be displayed on the monitor.
[0004] However, it may be difficult for a physician to correlate IVUS images with other external imaging of the vessel to be treated. For example, it may be difficult to correlate the overall structure of a vessel depicted in an angiogram with the structure of the same vessel depicted in an IVUS image. In addition, it may be difficult to correlate multiple sets of IVUS images of the same vessel. For example, it may be difficult to correlate a pre-treatment IVUS run with a post-treatment IVUS run.
[0005] Therefore, there is a need for systems and methods that can achieve co-registration of an external image with an IVUS image or an IVUS image with another set of IVUS images. Summary of the invention
[0006] This Summary is provided to introduce a group of concepts in a simplified form that are further described in the Detailed Description below. This Summary is not intended to necessarily identify key features or essential features of the claimed subject matter, nor is it intended to be an aid in determining the scope of the claimed subject matter.
[0007] In general, the present invention provides methods for identifying the location of side branches in a series of intravascular images (e.g., pre-treatment IVUS pullback, post-treatment IVUS pullback, etc.) to assist in co-registering the IVUS images with extravascular images (e.g., angiograms, etc.) or with another set of IVUS images. The present invention provides methods for training a machine learning (ML) model to infer side branch locations from IVUS images, and for extracting frames representing side branches from IVUS images.
[0008] In some embodiments, the present invention may be implemented as a method. The method may include receiving, at a processor, a series of intravascular ultrasound (IVUS) images of a patient's blood vessel, the series of IVUS images including a plurality of image frames; identifying, by the processor via a first machine learning (ML) model, a set of frames in the plurality of image frames, wherein the frames in the set of frames are associated with one or more side branches of the blood vessel; identifying, by the processor via a second ML model, a location of at least one of the one or more side branches in one or more frames in the plurality of image frames; and selecting, by the processor, a subset of frames from the set of frames based at least on an output from the first ML model and an output from the second ML model.
[0009] In some embodiments of the method, the first ML model is trained to infer the set of frames from a plurality of image frames.
[0010] In some embodiments of the method, the first ML model is configured to receive as input one or more adjacent frames from a plurality of image frames.
[0011] In some embodiments of the method, the first ML model is a convolutional neural network (CNN), a vision transformer network, or a combination of a CNN and a vision transformer network.
[0012] In some embodiments of the method, the first ML model is configured to apply the convolution window on a single frame or multiple adjacent frames from the plurality of image frames until all frames in the plurality of image frames have been received as input.
[0013] In some embodiments of the method, a second ML model is trained to determine, for each frame in a plurality of image frames, whether a frame represents a side branch in one or more side branches; and for each frame determined to represent a side branch in one or more side branches, identify a location of a side branch in the frame in one or more side branches.
[0014] In some embodiments of the method, the first ML model is configured to output, for each frame in the frame set, a confidence score representing a confidence that one or more side branches have been detected in the frame, and wherein selecting a subset of frames from the frame set comprises: identifying frames from the frame set having confidence scores greater than or equal to a threshold; and selecting the identified frames to include in the subset of frames.
[0015] In some embodiments of the method, the second ML model is trained to generate indications of locations as bounding boxes.
[0016] In some embodiments of the method, selecting frames from some of the set of frames of the series of IVUS images includes identifying adjacent frames from a plurality of image frames where bounding boxes in each frame are within a threshold distance of each other; and merging side branches associated with the identified frames.
[0017] In some embodiments of the method, the first ML model is configured to output, for each frame in the frame set, a confidence score representing a confidence that one or more side branches are detected in the frame, and wherein merging the side branches associated with the identified adjacent frames comprises: identifying one of the adjacent frames from the frame set having a highest confidence score, wherein the bounding boxes are within a threshold distance of each other; and selecting the identified one of the adjacent frames having the highest confidence score as the frame from the plurality of image frames to include in the frame subset.
[0018] In some embodiments of the method, the second ML model is a convolutional neural network (CNN).
[0019] In some embodiments of the method, the second ML model is a vision transformer network.
[0020] In some embodiments of the method, the second ML model is a combination of a CNN and a vision transformer network.
[0021] In some embodiments, the present invention may be implemented as a device for intravascular imaging apparatus. The device may include a processor; and a memory, the memory including instructions, in response to being executed by the processor, the instructions cause the device to implement any method of the embodiments described herein.
[0022] In some embodiments, the present invention may be implemented as at least one machine-readable storage device comprising a plurality of instructions which, in response to being executed by a processor of an intravascular ultrasound (IVUS) imaging system, cause the processor to implement the method of any of the embodiments described herein.
[0023] In some embodiments, the present invention may be implemented as a device for an intravascular imaging apparatus. The device may include a processor; and a memory including instructions, which, in response to being executed by the processor, cause the device to receive a series of intravascular ultrasound (IVUS) images of a patient's blood vessel, the series of IVUS images including a plurality of image frames; identify a frame set in the plurality of image frames via a first machine learning (ML) model, wherein the frames in the frame set are associated with one or more side branches of the blood vessel; identify a position of at least one of the one or more side branches in one or more frames in the plurality of image frames via a second ML model; and select a subset of frames from the frame set based at least on an output from the first ML model and an output from the second ML model.
[0024] In some embodiments of the device, the first ML model is trained to infer the set of frames from a plurality of image frames.
[0025] In some embodiments of the device, the first ML model is configured to receive as input one or more adjacent frames from the plurality of image frames.
[0026] In some embodiments of the device, the first ML model is a convolutional neural network (CNN), a visual transformer network, or a combination of a CNN and a visual transformer network, and wherein the second ML model is a convolutional neural network (CNN), a visual transformer network, or a combination of a CNN and a visual transformer network.
[0027] In some embodiments of the device, the first ML model is configured to apply the convolution window on a single frame or multiple adjacent frames from the plurality of image frames until all frames in the plurality of image frames have been received as input.
[0028] In some embodiments of the device, the second ML model is trained to determine, for each frame in a plurality of image frames, whether a frame represents a side branch of the one or more side branches; and for each frame determined to represent a side branch of the one or more side branches, identify a location of the side branch of the one or more side branches in the frame.
[0029] In some embodiments of the device, the first ML model is configured to output a confidence score for each frame in the frame set, the confidence score representing a confidence that one or more side branches have been detected in the frame, and wherein selecting a subset of frames from the frame set includes: identifying frames from the frame set having confidence scores greater than or equal to a threshold; and selecting the identified frames to include in the subset of frames.
[0030] In some embodiments, the present invention may be implemented as at least one machine-readable storage device, the at least one machine-readable storage device comprising a plurality of instructions, in response to being executed by a processor of an intravascular ultrasound (IVUS) imaging system, the plurality of instructions causing the processor to receive a series of IVUS images of a patient's blood vessel, the series of IVUS images comprising a plurality of image frames; identify a set of frames from the plurality of image frames via a first machine learning (ML) model, wherein the frames in the set of frames are associated with one or more side branches of the blood vessel; identify a location of at least one of the one or more side branches in one or more frames of the plurality of image frames via a second ML model; and select a subset of frames from the set of frames based at least on an output from the first ML model and an output from the second ML model.
[0031] In some embodiments of at least one machine-readable storage device, in response to being executed by a processor of the IVUS imaging system, the plurality of instructions further cause the processor to identify adjacent frames from a plurality of image frames, wherein bounding boxes in each frame are within a threshold distance of each other; and to merge side branches associated with the identified frames. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] To easily identify the discussion of any component or activity, the highest digit or digits in a reference number refer to the figure number in which the component is first introduced.
[0033] Figure 1 An IVUS imaging system is shown in accordance with at least one embodiment.
[0034] Figure 2 An example blood vessel image is shown.
[0035] Figure 3A An example series of intravascular images is shown.
[0036] Figure 3B Another example from a series of intravascular images is shown.
[0037] Figure 4 A combined internal and external vascular imaging system is shown in accordance with at least one embodiment.
[0038] Figure 5 A side branch detection system in accordance with at least one embodiment is shown.
[0039] Figure 6 A logic flow for detecting side branches in a series of IVUS images in accordance with at least one embodiment is shown.
[0040] Figure 7 Another logic flow for detecting side branches in a series of IVUS images in accordance with at least one embodiment is shown.
[0041] Figure 8 A machine learning (ML) system suitable for use with at least one embodiment is shown.
[0042] Fig. 9 A computer-readable storage medium is shown in accordance with at least one embodiment.
[0043] Fig.10 A diagram of a machine according to at least one embodiment is shown. DETAILED DESCRIPTION
[0044] The features and technical advantages of the present invention have been summarized above so that the following detailed description of the present invention can be better understood. It should be understood by those skilled in the art that the disclosed embodiments can be easily used as a basis for modifying or designing other structures to achieve the same purpose as the present invention. When considered in conjunction with the accompanying drawings, the novel features of the present invention regarding its organization and operation and further objects and advantages will be better understood according to the following description. However, it should be clearly understood that each of the accompanying drawings is provided for the purpose of illustration and description and is not intended to be a definition of the limitation of the present invention.
[0045] As described above, the present disclosure relates to identifying side branches in a series of intravascular images (e.g., IVUS images) to assist in co-registering the intravascular images with one or both of an extravascular image (e.g., angiogram) or another set of intravascular images. Thus, an example IVUS imaging system, a patient vessel, a series of IVUS images, and a combined IVUS / external imaging system are described.
[0046] Suitable IVUS imaging systems include, but are not limited to, one or more transducers disposed on a distal end of a catheter configured and arranged to be percutaneously inserted into a patient. Examples of IVUS imaging systems with catheters are found in, for example, U.S. Patent Nos. 7,246,959; 7,306,561; and 6,945,938, and U.S. Patent Application Publication Nos. 2006 / 0100522; 2006 / 0106320; 2006 / 0173350; 2006 / 0253028; 2007 / 0016054; and 2007 / 0038111; all of which are incorporated herein by reference.
[0047] Figure 1 One embodiment of an IVUS imaging system 100 is schematically illustrated. The IVUS imaging system 100 includes a catheter 102 that can be coupled to a control system 104. The control system 104 can include, for example, a processor 106, a pulse generator 108, and a drive unit 110. In at least some embodiments, the pulse generator 108 forms electrical pulses that can be input to one or more transducers (not shown) disposed in the catheter 102.
[0048] In some embodiments, mechanical energy from the drive unit 110 can be used to drive an imaging core (also not shown) disposed in the catheter 102. In at least some embodiments, electrical signals transmitted from one or more transducers can be input to the processor 106 for processing. In at least some embodiments, the processed electrical signals from one or more transducers can be used to form a series of images, as described in more detail below. For example, a scan converter can be used to map scan line samples (e.g., radial scan line samples, etc.) to a two-dimensional Cartesian grid, which can be used as the basis for a series of IVUS images displayed to a user.
[0049] In at least some embodiments, the processor 106 may also be used to control the operation of one or more of the other components of the control system 104. For example, the processor 106 may be used to control at least one of the frequency or duration of the electrical pulses transmitted from the pulse generator 108, the rotation rate of the imaging core driven by the drive unit 110. Additionally, where the IVUS imaging system 100 is configured for automatic pullback, the drive unit 110 may control the speed and / or length of the pullback.
[0050] Figure 2 An image 200 of a patient's blood vessel 202 is shown. As described, an IVUS imaging system (e.g., IVUS imaging system 100, etc.) is used to capture a blood vessel, such as a series of images or "recordings" of blood vessel 202. For example, an IVUS catheter (e.g., catheter 102) is inserted into blood vessel 202, and a record or series of IVUS images is captured as catheter 102 is retracted from distal end 204 to proximal end 206. Catheter 102 may be retracted manually or automatically (e.g., under control of drive unit 110, etc.).
[0051] Figure 3A and Figure 3B A two-dimensional (2D) representation of an IVUS image of a blood vessel 202 is shown. For example, Figure 3A An IVUS image 300a depicting a longitudinal view of an IVUS recording of the blood vessel 202 between the proximal end 206 and the distal end 204 is shown.
[0052] Figure 3B An image frame 300b is shown depicting an on-axis (or short-axis) view of the vessel 202 at point 302. In other words, the image frame 300b is a single frame or a single image from a series of IVUS images that may be captured between the distal end 204 and the proximal end 206, as described herein. As introduced above, the present invention provides for processing raw IVUS images to identify side branches of a vessel and extracting (or identifying) frames associated with the identified side branches from the IVUS images to assist in co-registering the IVUS images with one or both of the angiographic image or another set of IVUS images of the vessel.
[0053] Figure 4 A combined IVUS external imaging system 400 is shown, which includes both an IVUS imaging system 402 (e.g., IVUS imaging system 100, etc.) and an extravascular imaging system 404 (e.g., an angiographic imaging system). The combined IVUS external imaging system 400 also includes a computing device 406, which includes circuits, controllers and / or processors and memory and software, which is configured to execute a method for vascular imaging registration of acquired extravascular imaging data and acquired intravascular imaging data. Generally, the IVUS imaging system 402 can be arranged to generate IVUS intravascular imaging data (e.g., IVUS image 300a and image frame 300b), while the extravascular imaging system 404 can be arranged to generate extravascular imaging data (e.g., image 200).
[0054] The extravascular imaging system 404 may include an angiography table 408, which may be arranged to provide sufficient space for positioning an angiography / fluoroscopy unit C-arm 410 in an operating position relative to the patient 412 on the drive unit 110. The raw radiological image data acquired by the C-arm 410 may be transmitted to the extravascular data input port 414 via a transmission cable 416. The input port 414 may be a separate component, or may be integrated into or become part of the computing device 406. The input port 414 may include a processor that converts the raw radiological image data received thereby into extravascular image data (e.g., angiography / fluoroscopy image data), for example, in the form of real-time video, DICOM, or a series of individual images. The extravascular image data may initially be stored in a memory within the input port 414 or may be stored in a memory of the computing device 406. If input port 414 is a separate component from computing device 406, the extravascular image data may be transferred to computing device 406 via transmission cable 418 and transferred to an input port (not shown) of computing device 406. In some alternatives, communication between devices or processors may be via wireless communication rather than via cables as described.
[0055] The intravascular imaging data can be, for example, IVUS data or OCT data obtained by the IVUS imaging system 402. The IVUS imaging system 402 can include an intravascular imaging device, such as an imaging catheter 420. The imaging catheter 420 is configured to be inserted into the patient 412 so that its distal end, including a diagnostic component or probe 422 (e.g., an IVUS probe), is located near the desired imaging location of the blood vessel. A radiopaque material or marker 424 located on or near the probe 422 can provide an indication of the current position of the probe 422 in the radiographic image.
[0056] The imaging catheter 420 is coupled to a proximal connector 426 to couple the imaging catheter 420 to an image acquisition device 428. The image acquisition device 428 may be coupled to the computing device 406 via a transmission cable 430 or a wireless connection. The intravascular image data may initially be stored in a memory of the image acquisition device 428 or may be stored in a memory of the computing device 406. If the image acquisition device 428 is a separate component from the computing device 406, the intravascular image data may be transferred to the computing device 406 via, for example, the transmission cable 430.
[0057] The computing device 406 may also include one or more additional output ports for transferring data to other devices. For example, the computer may include an output port for transferring data to a data archive or storage device 432. The computing device 406 may also include a user interface (described in more detail below) that includes a combination of circuits, processing components, and instructions executable by the processing components and / or circuits to achieve dynamic co-registration of intravascular and extravascular images.
[0058] The user interface may be presented and displayed on a display 434, which is coupled to the computing device 406 via a display cable 436. Although the display 434 is depicted as being separate from the computing device 406, in some examples, the display 434 may be part of the computing device 406. Alternatively, the display 434 may be remote from the computing device 406 and wirelessly connected thereto. As another example, the display 434 may be part of another computing device different from the computing device 406, such as a tablet computer, which may be coupled to the computing device 406 via a wired or wireless connection.
[0059] Figure 5 A side branch detection system 500 according to some embodiments of the present invention is shown. In general, the side branch detection system 500 is a system that uses a machine learning (ML) model to select a set of frames in a series of IVUS images that represent side branches and uses an analysis algorithm to process the set of frames and sub-select from the set of frames to detect side branches in the series of IVUS images, wherein the frames are associated with "critical" or "unique" side branches. The ML model can be trained using labeled IVUS images.
[0060] The side branch detection system 500 can be used with a commercial IVUS guidance or navigation system, such as, for example, the one available from Boston Scientific. Purchased Guidance system and external imaging system, such as, for example, the above combined Figure 4 The extravascular imaging system is implemented in combination.
[0061] The side branch detection system 500 includes a computing device 502. The computing device 502 can be any of a variety of computing devices. In some embodiments, the computing device 502 can be incorporated into and / or implemented by the computing device 406. In some embodiments, the computing device 502 can be a workstation or server that is communicatively coupled to the IVUS imaging system 100 and an external imaging system (e.g., an extravascular imaging system 404, etc.). In other embodiments, the computing device 502 can be provided by a cloud-based computing device, such as, provided by a computing as a service system that can be accessed via a network (e.g., the Internet, an intranet, a wide area network, etc.). The computing device 502 may include a processor 504, a memory 506, an input and / or output (I / O) device 508, a network interface 510, and an IVUS imaging system acquisition circuit 512.
[0062] Processor 504 may include circuitry or processor logic, such as, for example, any of a variety of commercial processors. In some examples, processor 504 may include multiple processors, multi-threaded processors, multi-core processors (whether the multiple cores coexist on the same die or on separate chips), and / or some other kind of multi-processor architecture, with multiple physically separate processors linked in some way through the above multi-processor architecture. Additionally, in some examples, processor 504 may include a graphics processing portion, and may include dedicated memory, multi-threaded processing, and / or some other parallel processing capabilities. In some examples, processor 504 may be an application specific integrated circuit (ASIC) or a field programmable integrated circuit (FPGA).
[0063] The memory 506 may include logic circuits, a portion of which includes an integrated circuit array, thereby forming a non-volatile memory to persistently store data or a combination of non-volatile memory and volatile memory. It should be understood that the memory 506 can be based on any of a variety of technologies. In particular, the integrated circuit array included in the memory 120 can be arranged to form one or more types of memory, such as, for example, dynamic random access memory (DRAM), NAND memory, NOR memory, etc.
[0064] I / O device 508 may be any of a variety of devices for receiving input and / or providing output. For example, I / O device 508 may include a keyboard, a mouse, a joystick, a foot pedal, a display, a touch-enabled display, a tactile feedback device, an LED, etc.
[0065] The network interface 510 may include logic circuits and / or features for supporting a communication interface. For example, the network interface 510 may include one or more interfaces operating according to various communication protocols or standards to communicate via direct or network communication links. Direct communication may occur via the use of a communication protocol or standard described in one or more industry standards (including descendants and variants). For example, the network interface 510 may facilitate communication through a bus, such as, for example, peripheral component interconnect high speed (PCIe), non-volatile memory high speed (NVMe), universal serial bus (USB), system management bus (SMBus), SAS (e.g., serially attached small computer system interface (SCSI)) interface, serial AT attachment (SATA) interface, etc. Additionally, the network interface 510 may include logic circuits and / or features for enabling communication through various wired or wireless network standards (e.g., 704.11 communication standards). For example, the network interface 510 may be arranged to support wired communication protocols or standards, such as, Ethernet, etc. As another example, the network interface 510 may be arranged to support wireless communication protocols or standards such as, for example, Wi-Fi, Bluetooth, ZigBee, LTE, 5G, etc.
[0066] The IVUS imaging system acquisition circuit 512 may include circuitry, including custom manufactured or specially programmed circuitry, configured to receive or receive and send signals between the IVUS imaging system 100, the signals including indications of IVUS operation, a series of IVUS images, or a frame or multiple frames of IVUS images; and to receive or receive and send signals between external imaging systems (e.g., angiographic imaging systems, extravascular imaging systems 404, etc.).
[0067] The memory 506 may include instructions 514. During operation, the processor 504 may execute the instructions 514 to cause the computing device 502 to detect side branches in a series of IVUS images, as outlined herein. For example, the processor 504 may execute the instructions 514 to receive the IVUS images 516 from the IVUS imaging system 402. In an alternative embodiment, the processor 504 may execute the instructions 514 to receive the IVUS images 516 from a memory device storing captured IVUS images.
[0068] The processor 504 may execute the instructions 514 and the ML model 518a to generate a frame set 520 of IVUS images from the IVUS image 516, wherein the frame set 520 of IVUS images includes frames of the IVUS image 516 in which the side branch is detected. In some embodiments, the processor 504 may execute the instructions 514 to derive the frame set 520 of IVUS images from the ML model 518a using the IVUS image 516 as an input to the ML model 518a. In some embodiments, the ML model 518a may be a convolutional neural network (CNN) configured to acquire a number of frames (e.g., 2, 3, 4, etc.) from the IVUS image 516 at a time and output an indication of a frame set of the IVUS image 516 that depicts or captures the side branch. In some embodiments, the ML model 518a may be a visual transformer or a combination of a CNN and a transformer network configured to acquire the IVUS image 516 and output an indication of a frame set (e.g., the frame set 520 of the IVUS image, etc.) that depicts or captures the side branch.
[0069] In addition, in some embodiments, the processor 504 may execute instructions 514 to generate a side branch frame 522 of an IVUS image from the frame set 520 of the IVUS image or the IVUS image 516, wherein the side branch frame 522 of the IVUS image includes an image frame in which a side branch is detected (e.g., from the frame set 520 of the IVUS image or the IVUS image 516) and a position on the frame where the side branch is detected. In some embodiments, the processor 504 may execute instructions 514 to derive the side branch frame 522 of the IVUS image from the ML model 518b using the frame set 520 of the IVUS image as an input to the ML model 518b. In other embodiments, the processor 504 may execute instructions 514 to derive the side branch frame 522 of the IVUS image from the ML model 518b using the IVUS image 516 as an input to the ML model 518b. In some embodiments, the ML model 518b can be a CNN configured to acquire an IVUS frame or a set of IVUS frames and (1) identify whether a side branch is represented in the frame and (2) generate a location marker (e.g., a bounding box, etc.) indicating the region of the frame where the side branch is identified. In addition, in some embodiments, the ML model 518b can be a visual transformer or a combination of a CNN and a transformer network configured to acquire an IVUS image frame or a plurality of frames and (1) identify whether a side branch is represented in the frame and (2) generate a location marker (e.g., a bounding box, etc.) indicating the region of the frame where the side branch is identified. The generated location marker can be stored in the memory 506 as the side branch location 524.
[0070] In some embodiments, the processor 504 may execute instructions 514 to analytically process the output of either or both of the ML model 518a and the ML model 518b to generate a subset of frames 526 of the IVUS image. For example, the processor 504 may execute instructions 514 to process the set of frames 520 of the IVUS image and the side branch frames 522 of the IVUS image and the side branch locations 524 to determine a subset of frames 526 of the IVUS image, wherein the frames in the subset of frames of the IVUS image include frames in which the same side branch is indicated by both the ML models 518a and 518b. As a specific example, the processor 504 may execute instructions 514 to select a frame with the highest side branch detection confidence from the set of frames 520 of the IVUS image based on the output from the ML model 518a as a frame to be included in the subset of frames 526 of the IVUS image.
[0071] As another example, the processor 504 may execute instructions 514 to discard frames from the side branch frames 522 of the IVUS image, where the side branch locations 524 do not indicate the locations of the side branches in these frames. In other words, the processor 504 may execute instructions 514 to exclude some frames from the subset of frames 526 of the IVUS image, where for these frames in the side branch frames 522 of the IVUS image, the side branch locations 524 are not indicated.
[0072] Figure 6 A logic flow 600 is shown that can be implemented to detect side branches in a series of IVUS images in order to co-register the IVUS images with an angiogram or another series of IVUS images. The logic flow 600 can be implemented by the side branch detection system 500 and for clarity of presentation, will be described with reference to the side branch detection system 500. However, it is noted that the logic flow 600 can also be implemented by an IVUS co-registration system other than the side branch detection system 500.
[0073] Logic flow 600 may begin at block 602. At block 602, "At the processor receives a series of IVUS images of a patient's blood vessel, the series of IVUS images comprising a plurality of image frames," a series of IVUS images may be received by the processor. For example, processor 504 may execute instructions 514 to receive IVUS images 516 (e.g., from IVUS imaging system 402, from a memory storage location, etc.). As described in detail above, IVUS images 516 include several frames of images from a "run" or pullback through the blood vessel.
[0074] Continuing to block 604, “Identifying, by the processor via a first machine learning (ML) model, a set of frames in a plurality of image frames, wherein the frames in the set of frames are associated with one or more side branches of a blood vessel,” a set of frames in a plurality of image frames may be identified, wherein the frames in the set of frames are associated with a side branch or multiple side branches of a blood vessel. For example, the processor 504 may execute instructions 514 to identify some frames from the IVUS image 516 in which side branches are depicted or represented. More specifically, the processor 504 may execute instructions 514 to derive the set of frames 520 of IVUS images from the ML model 518a using the IVUS image 516 as input. In some embodiments, the ML model 518a may be configured to output the set of frames 520 of IVUS images from the IVUS image 516, and a confidence level for each frame in the set of frames 520 of IVUS images, wherein the confidence level includes a confidence level that the frame depicts or represents a side branch.
[0075] Continuing to block 606, “Identifying, by the processor via a second ML model, a location of at least one of the one or more side branches in one or more of the plurality of image frames,” the location of at least one side branch of the one or more branches in one or more of the plurality of image frames may be identified. For example, the processor 504 may execute instructions 514 to identify the location of a side branch or multiple side branches in a frame or multiple frames 520 of an IVUS image, and store the location or multiple locations as a side branch location 524. More specifically, the processor 504 may execute instructions 514 to derive a side branch frame 522 and a side branch location 524 of an IVUS image from the ML model 518b using the frame set 520 of the IVUS image as input. In some embodiments, the ML model 518b may be configured to output an indication (e.g., a bounding box, etc.) of the side branch location 524 in each frame of the frame set 520 of the IVUS image in which the side branch is identified. Further, the side branch frame 522 of the IVUS image may be formed from the frame 520 of the IVUS image in which the location of the side branch is identified by the ML model 518b.
[0076] Continuing to block 608, “Selecting, by the processor, a subset of frames from the set of frames based at least on an input from a first ML model and an output from a second ML model,” frames for each of the one or more side branches may be selected from the series of IVUS images based on the set of frames and the location of the side branch in some of the set of frames. For example, the processor 504 may execute instructions 514 to determine a frame representing a side branch, and if both the first ML model and the second ML model identify the frame as representing a side branch, include the frame in the subset of frames 526. In another example, the processor 504 may execute instructions 514 to determine that multiple frames of the series of IVUS images 516 represent the same side branch based on the side branch location 524 and “merge” the frames, or more specifically, select only one of the frames to include in the subset of frames 526 of the IVUS images based on an output from ML model 518a (e.g., a confidence level above a threshold, etc.) and / or an output from 518b (e.g., a bounding box indicating the same location, etc.).
[0077] Logic flow 600 may also include a block for co-registering the series of IVUS images received at block 602 with an angiogram of the vessel and / or with another series of IVUS images of the vessel (see Figure 7 ). For example, the processor 504 may execute the instructions 514 to implement the logic flow 600 to detect frames with side branches in a first series of IVUS images of a blood vessel captured before percutaneous intervention (PCI) treatment, and detect frames with side branches in a second series of IVUS images of the blood vessel captured after PCI. Subsequently, the processor 504 may execute the instructions 514 to co-register the first series of IVUS images with the second series of IVUS images, wherein the frames corresponding to the side branches in each series are used to facilitate co-registration.
[0078] Figure 7 A logic flow 700 is shown that can be implemented to select frames representing side branches from a series of IVUS images based on a set (or subset) of frames identified as representing side branches using an ML model. For example, the logic flow 700 can be implemented at block 608 of the logic flow 600. The logic flow 700 can be implemented by the side branch detection system 500 and for clarity of presentation, will be described with reference to the side branch detection system 500. However, it is worth noting that the logic flow 700 can also be implemented by an IVUS co-registration system other than the side branch detection system 500.
[0079] The logic flow 700 may begin at decision box 702. At decision box 702, "For each frame in the frame set, is the confidence score greater than or equal to a threshold?", a determination may be made regarding the confidence of the side branch detection for each frame in the frame set. For example, the processor 504 detailed above may execute instructions 514 to derive a frame set 520 of IVUS images from an ML model 518a using an IVUS image 516 as input. In addition to inferring the frame set 520 of IVUS images from the IVUS image 516, the ML model 518a may also output a confidence level that the detected side branch is an actual side branch for each frame in the frame set 520 of IVUS images. In other words, the ML model 518a may output a confidence score or measure whether the inference is accurate. At decision box 702, the processor 504 may execute instructions 514 to determine whether the confidence score for each frame is above (e.g., greater than or equal to) a threshold level.
[0080] From decision block 702, the logic flow 700 may continue to block 704 or decision block 706. For example, based on determining that the confidence score of the frame in the frame set is not above a threshold level, the logic flow 700 may continue from decision block 702 to block 704, and based on determining that the confidence score of the frame in the frame set is above a threshold level, the logic flow 700 may continue from decision block 702 to decision block 706. At block 704, "Discard frames with confidence scores below threshold from the frame set," frames with confidence scores below the threshold may be discarded from the frame set. In some embodiments, decision blocks 702 and blocks 704 may be performed iteratively frame by frame. At block 704, the processor 504 may execute instructions 514 to form a subset of frames 526 of an IVUS image from the frame set 520 of the IVUS image by including only frames from the frame set 520 of the IVUS image in the subset of frames 526 of the IVUS image where the frames have a confidence score above the threshold level.
[0081] At decision box 706, "Are side branches detected at the same location on multiple adjacent frames?", a determination may be made as to whether a side branch is detected at the same location on multiple adjacent frames. For example, the processor 504 may execute instructions 514 to determine whether the side branch locations 524 of multiple frames that are adjacent to each other (e.g., side by side, within a certain number of frames, within a set distance, etc.) are the same. As a specific example, the processor 504 may execute instructions 514 to determine whether the side branch locations 524 of frames in a subset of frames of the IVUS image 522 that are within a specified distance (e.g., less than or equal to 1 / 3 mm, etc.) are the same.
[0082] From decision block 706, logic flow 700 may continue to block 708 or block 710. For example, logic flow 700 may continue from decision block 706 to block 708 based on a determination that a side branch is detected at the same location on multiple adjacent frames, while logic flow 700 may continue from decision block 706 to block 710 based on a determination that a side branch is not detected at the same location on multiple adjacent frames.
[0083] At block 708, “Identify a frame having the highest confidence score from a plurality of adjacent frames to represent a side branch at the location and include the identified frame in the subset of frames,” a single frame from the adjacent frames having the same side branch location (e.g., based on the side branch location 524, etc.) may be selected to represent the side branch. For example, the processor 504 may execute instructions 514 to select a single frame from a set of adjacent frames (e.g., frames within a specified distance from each other, etc.) identified at decision block 706 as having the same side branch location to include in the subset of frames 526 of the IVUS image. In some embodiments, the processor 504 may execute instructions 514 to select the frame having the highest confidence score output from the ML model 518a as the frame to be included in the subset of frames 526 of the IVUS image.
[0084] At box 710 "Co-register the series of IVUS images with angiographic images and / or another series of IVUS images based on some of the frames in the set", some of the frames in the set can be used to co-register the series of IVUS images with angiographic or another series of IVUS images of the blood vessel, as outlined above. For example, the processor 504 can execute instructions 514 to implement the logic flow 600 to detect frames with side branches in a first series of IVUS images of the blood vessel captured before PCI, and detect frames with side branches in a second series of IVUS images of the blood vessel captured after PCI. Subsequently, the processor 504 can execute instructions 514 to co-register the first series of IVUS images with the second series of IVUS images, where frames corresponding to side branches in each series are used to facilitate co-registration.
[0085] As described above, in some embodiments, the processor 504 of the computing device 502 may execute instructions 514 to generate the frame set 520 of the IVUS image, the side branch frames 522 of the IVUS image, and / or the side branch locations 524 using the ML model or models. In such examples, the ML model may be stored in the memory 506 of the computing device 502. It should be understood that prior to deployment, the ML model is trained. Figure 8An ML environment 800 is shown that can be used to train an ML model that can later be used to generate (or infer) the frame set of IVUS images 520 and / or a frame subset of IVUS images 522, as described herein. The ML environment 800 can include an ML system 802, such as a computing device that applies an ML algorithm to learn a relationship. In this example, the ML algorithm can learn the relationship between a set of inputs (e.g., a series of IVUS images 816) and a set of outputs (e.g., side branch annotations on the series of IVUS images 818) to be able to infer (e.g., during deployment, etc.) the outputs (e.g., frame set 520 of IVUS images, side branch frames 522 of IVUS images, side branch locations 524) based on unseen or new inputs (e.g., IVUS images 516, frame set 520 of IVUS images, etc.).
[0086] The ML system 802 may utilize experimental data 808 collected during several previous procedures. The experimental data 808 may include a series of IVUS images 816 for several patients. The experimental data 808 may be collocated with the ML system 802 (e.g., stored in a storage device 810 of the ML system 802), may be remote from the ML system 802 and accessed via a network interface 804, or may be a combination of local and remote data.
[0087] The experimental data 808 can be used to form training data 812. As described above, the ML system 802 can include a storage device 810, which can include a hard drive, a solid-state storage device, and / or a random access memory. The storage device 810 can hold the training data 812. Generally, the training data 812 can include an information element or data structure that includes an indication of a series of IVUS images 816 for a number of patients. In addition, the training data 812 can optionally include side branch annotations on the series of IVUS images 818. For example, the side branch annotations on the series of IVUS images 818 can include a series of IVUS images 816 that are annotated to indicate frames that depict or represent side branches.
[0088] The training data 812 may be applied to train an ML model 814. Depending on the application, different types of models may be used to form the basis of the ML model 814. For example, in the present example, an artificial neural network (ANN), such as a CNN and / or a visual transformer may be particularly well suited for learning associations between a series of IVUS images (e.g., a series of IVUS images 816) and side branches depicted or represented in the series (e.g., frames with side branches, side branch locations in frames, etc.).
[0089] Any suitable training algorithm 820 may be used to train the ML model 814. Figure 8The example depicted in may be particularly well suited for supervised training algorithms or reinforcement learning training algorithms. For supervised training algorithms, the ML system 802 can apply the series of IVUS images 816 as model input 822, to which the side branch annotations on the series of IVUS images 818 can be mapped to learn the association between the model input 822 and the side branch information 824 (e.g., frames representing side branches, bounding boxes around side branch locations, etc.) output by the ML model 814. In a reinforcement learning scenario, the training algorithm 820 can attempt to maximize some or all (or a weighted combination) of the model inputs 822 mapped to the side branch information 824 to produce an ML model 814 with minimal error. In some embodiments, the training data 812 can be divided into "training" data and "testing" data, where a subset of the training data 812 can be used to adjust the ML model 814 (e.g., the model's internal weights, etc.), and another non-overlapping subset of the training data 812 can be used to measure the accuracy of the ML model 814 to infer (or generalize) side branch information 824 based on "unseen" training data 812 (e.g., training data 812 that was not used to train the ML model 814).
[0090] The ML model 814 may be applied using processor circuitry 806, which may include suitable hardware processing resources to operate on logic and structures in storage device 810. The development of training algorithm 820 and / or trained ML model 814 may rely, at least in part, on hyperparameters 826. In an exemplary embodiment, model hyperparameters 826 may be automatically selected based on hyperparameter optimization logic 828, which may include any known hyperparameter optimization techniques appropriate to the selected ML model 814 and training algorithm 820 to be used. In an alternative embodiment, the ML model 814 may be retrained over time to adapt to new knowledge and / or updated experimental data 808.
[0091] Once the ML model 814 is trained, it can be applied (e.g., by the processor circuit 806, the processor 504, etc.) to new input data (e.g., IVUS images 516, etc.). The input to the ML model 814 can be formatted according to a predefined model input 822 that reflects the manner in which the training data 812 is provided to the ML model 814. The ML model 814 can generate side branch information 824, which can be, for example, a set of frames 520 of IVUS images in which side branches are detected, the location of the side branches in the frames where the side branches are located, etc.
[0092] The above description relates to a particular kind of ML system 802 that applies supervised learning techniques when given available training data with input / outcome pairs. However, the present invention is not limited to use with a particular ML paradigm, and other types of ML techniques may be used. For example, in some embodiments, the ML system 802 may apply, for example, an evolutionary algorithm or other types of ML algorithms and models to generate side branch information 824 based on a series of IVUS images 816. For example, user interactions with the output (e.g., deleting an identified side branch, etc.) may be recorded, and such recorded information may be used to retrain the ML model 814.
[0093] Fig. 9 A computer-readable storage medium 900 is shown. The computer-readable storage medium 900 may include any non-temporary computer-readable storage medium or machine-readable storage medium, such as an optical, magnetic or semiconductor storage medium. In various embodiments, the computer-readable storage medium 900 may include an article of manufacture. In some embodiments, the computer-readable storage medium 900 may store computer-executable instructions 902 executable by a circuit (e.g., processor 106, processor 504, IVUS imaging system acquisition circuit 512, etc.). For example, the computer-executable instructions 902 may include instructions for implementing the operations described with respect to the logic flow 600 and / or the logic flow 700. Examples of computer-readable storage media 900 or machine-readable storage media may include any tangible medium capable of storing electronic data, including volatile or non-volatile memory, removable or non-removable memory, erasable or non-erasable memory, writable or rewritable memory, and the like. Examples of computer-executable instructions 902 may include any suitable type of code, such as source code, compiled code, interpreted code, executable code, static code, dynamic code, object-oriented code, visual code, and the like.
[0094] Fig.10 A diagram of a machine 1000 in the form of a computer system is shown within which a set of instructions may be executed to cause the machine to perform any one or more of the methodologies discussed herein. More specifically, Fig.10 A diagram of a machine 1000 in the example form of a computer system is shown within which instructions 1008 (e.g., software, programs, applications, applet, applications, or other executable code) may be executed to cause the machine 1000 to perform any one or more of the methodologies discussed herein. For example, the instructions 1008 may cause the machine 1000 to perform Figure 6 The logic flow 600, Figure 7More generally, the instructions 1008 may cause the machine 1000 to detect side branches in a series of IVUS images to co-register the IVUS images with either or both of the angiogram and another series of IVUS images.
[0095] Instructions 1008 transform a general, non-programmed machine 1000 into a specific machine 1000 that is programmed to perform the functions described and shown in a specific manner. In alternative embodiments, the machine 1000 operates as a standalone device or can be coupled (e.g., networked) to other machines. In a networked deployment, the machine 1000 can operate in the capacity of a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 1000 may include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a PDA, an entertainment media system, a cellular phone, a smart phone, a mobile device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a network device, a network router, a network switch, a bridge, or any machine capable of sequentially or otherwise executing instructions 1008 specifying the actions to be taken by the machine 1000. Further, while a single machine 1000 is illustrated, the term "machine" shall also be taken to include any collection of machines 1000 that individually or jointly execute instructions 1008 to perform any one or more of the methodologies discussed herein.
[0096] The machine 1000 may include a processor 1002, a memory 1004, and an I / O component 1042, which may be configured to communicate with each other, such as via a bus 1044. In an example embodiment, the processor 1002 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an ASIC, a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, a processor 1006 that may execute instructions 1008 and a processor 1010. The term "processor" is intended to include multi-core processors, which may include two or more independent processors (sometimes referred to as "cores") that may execute instructions simultaneously. Although Fig.10 Multiple processors 1002 are shown, but the machine 1000 may also include a single processor having a single core, a single processor having multiple cores (e.g., a multi-core processor), multiple processors having a single core, multiple processors having multiple cores, or any combination thereof.
[0097] The memory 1004 may include a main memory 1012, a static memory 1014, and a storage unit 1016, both of which may be accessed by the processor 1002, such as via a bus 1044. The main memory 1004, the static memory 1014, and the storage unit 1016 store instructions 1008 that embody any one or more of the methods or functions described herein. During execution of the instructions 1008 by the machine 1000, the instructions 1008 may also reside, in whole or in part, within the main memory 1012, within the static memory 1014, within the machine-readable medium 1018 within the storage unit 1016, within at least one of the processors 1002 (e.g., within a cache of the processor), or any suitable combination thereof.
[0098] I / O components 1042 may include a variety of components to receive input, provide output, generate output, transmit information, exchange information, capture measurements, and the like. The specific I / O components 1042 included in a particular machine will depend on the type of machine. For example, a portable machine such as a mobile phone will likely include a touch input device or other such input mechanism, while a headless server machine will likely not include such a touch input device. It should be understood that I / O components 1042 may include Fig.10 1042 are grouped according to function, which is merely to simplify the following discussion, and the grouping is by no means limiting. In various example embodiments, the I / O components 1042 may include output components 1028 and input components 1030. The output components 1028 may include visual components (e.g., displays such as plasma display panels (PDPs), light emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tubes (CRTs)), acoustic components (e.g., speakers), tactile components (e.g., vibration motors, resistance mechanisms), other signal generators, and the like. The input components 1030 may include alphanumeric input components (e.g., keyboards, touch screens configured to receive alphanumeric input, optical keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touch pads, trackballs, joysticks, motion sensors, or another orientation meter), tactile input components (e.g., physical buttons, touch screens that provide the location and / or force of touch or touch gestures, or other tactile input components), audio input components (e.g., microphones), and the like.
[0099] In other example embodiments, the I / O component 1042 may include a biometric component 1032, a motion component 1034, an environment component 1036, or a position component 1038, as well as various other components. For example, the biometric component 1032 may include a component for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition), etc. The motion component 1034 may include an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, a rotation sensor component (e.g., a gyroscope), etc. The environment component 1036 may include, for example, an illumination sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers that detect ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an acoustic sensor component (e.g., one or more microphones that detect background noise), a proximity sensor component (e.g., an infrared sensor that detects nearby objects), a gas sensor (e.g., a gas detection sensor used to detect dangerous gas concentrations to ensure safety or measure pollutants in the atmosphere), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment. The position component 1038 may include a position sensor component (e.g., a GPS receiver component), an altitude sensor component (e.g., an altimeter or barometer that detects air pressure from which altitude can be derived), an orientation sensor component (e.g., a magnetometer), etc.
[0100] The communication may be implemented using a variety of techniques. The I / O components 1042 may include a communication component 1040 operable to couple the machine 1000 to the network 1020 or the device 1022 via the coupling 1024 and the coupling 1026, respectively. For example, the communication component 1040 may include a network interface component or another suitable device for interfacing with the network 1020. In other examples, the communication component 1040 may include a wired communication component, a wireless communication component, a cellular communication component, a near field communication (NFC) component, Parts (e.g. Low power consumption), Device 1022 may be another machine or any of a variety of peripheral devices (eg, a peripheral device coupled via USB).
[0101] In addition, the communication component 1040 can detect the identifier or include a component operable to detect the identifier. For example, the communication component 1040 can include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., for detecting a one-dimensional barcode, such as a universal product code (UPC) barcode, a multi-dimensional barcode, such as a Quick Response (QR) code, an Aztec code, a Data Matrix, a Dataglyph, a MaxiCode, a PDF417, an Ultra Code, a UCCRSS-2D barcode, and other optical codes), or an acoustic detection component (e.g., a microphone for identifying an audio signal of a tag). In addition, various information can be derived via the communication component 1040, such as a location via Internet Protocol (IP) geolocation, a location via Internet Protocol (IP), ... The location of signal triangulation, the location of an NFC beacon signal via detection that can indicate a specific location, and the like.
[0102] Various memories (i.e., memory 1004, main memory 1012, static memory 1014, and / or memory of processor 1002) and / or storage unit 1016 may store one or more sets of instructions and data structures (e.g., software) that embody or are utilized by any one or more of the methods or functions described herein. These instructions (e.g., instructions 1008) cause various operations when executed by processor 1002 to implement the disclosed embodiments.
[0103] As used herein, the terms "machine storage medium", "device storage medium", "computer storage medium" mean the same thing and are used interchangeably in the present invention. The terms refer to a single or multiple storage devices and / or media (e.g., centralized or distributed databases and / or associated caches and servers) that store executable instructions and / or data. Thus, the terms should be considered to include, but are not limited to, solid-state memory and optical and magnetic media, including memory internal or external to the processor. Specific examples of machine storage media, computer storage media, and / or device storage media include non-volatile memory, such as semiconductor memory devices, such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGA, and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms "machine storage medium", "computer storage medium", and "device storage medium" expressly exclude carrier waves, modulated data signals, and other such media, at least some of which are covered under the term "signal media" discussed below.
[0104] In various example embodiments, one or more portions of network 1020 may be an ad hoc network, an intranet, an extranet, a VPN, a LAN, a WLAN, a WAN, a WWAN, a MAN, the Internet, a portion of the Internet, a PSTN, a plain old telephone service (POTS) network, a cellular telephone network, a wireless network, 1020, another type of network, or a combination of two or more such networks. For example, network 1020 or a portion of network 1020 may include a wireless or cellular network, and coupling 1024 may be a code division multiple access (CDMA) connection, a global system for mobile communications (GSM) connection, or another type of cellular or wireless coupling. In this example, coupling 1024 may implement any of a variety of types of data transmission technologies, such as single carrier radio transmission technology (1xRTT), evolution data optimized (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rates for GSM evolution (EDGE) technology, third generation partnership project (3GPP) (including 3G), fourth generation wireless (4G) network, universal mobile telecommunications system (UMTS), high speed packet access (HSPA), world wide interoperability for microwave access (WiMAX), long term evolution (LTE) standard, other standards defined by various standards development organizations, other remote protocols, or other data transmission technologies.
[0105] The instructions 1008 may be transmitted or received over the network 1020 using a transmission medium via a network interface device (e.g., a network interface component included in the communication component 1040) and using any of several well-known transmission protocols (e.g., the Hypertext Transfer Protocol (HTTP)). Similarly, the instructions 1008 may be transmitted or received via a coupling 1026 (e.g., a peer coupling) to the device 1022 using a transmission medium. The terms "transmission medium" and "signal medium" mean the same thing and may be used interchangeably in the present invention. The terms "transmission medium" and "signal medium" should be deemed to include any intangible medium that can store, encode, or carry instructions 1008 for execution by the machine 1000; and include digital or analog communication signals or other intangible media to facilitate the communication of such software. Therefore, the terms "transmission medium" and "signal medium" should be deemed to include any form of modulated data signal, carrier wave, etc. The term "modulated data signal" refers to a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal.
[0106] Terms used herein should be given their ordinary meaning in the relevant art or the meaning indicated by their usage in context, but if an explicit definition is provided, that meaning shall prevail.
[0107] Herein, references to "an embodiment" or "an embodiment" do not necessarily refer to the same embodiment, although they may refer to the same embodiment. Unless the context clearly requires otherwise, throughout the specification and claims, the words "include", "comprises", etc. should be interpreted as inclusive, rather than exclusive or exhaustive; that is, "including but not limited to". Words using the singular or plural also include the plural or singular, respectively, unless explicitly limited to one or more. Additionally, the words "herein", "above", "below", and words of similar meaning, when used in this application, refer to the entirety of this application, rather than to any part of this application. When a claim uses the word "or" to refer to a list of two or more items, the word covers all of the following interpretations of the word: any one of the items in the list, all of the items in the list, and any combination of the items in the list, unless explicitly limited to one or another. Any term not explicitly defined herein has the conventional meaning commonly understood by those skilled in the relevant art.
[0108] By using true anatomical models, more accurate surgical plans can be made than through statistical modeling.
[0109] Terms used herein should be given their ordinary meaning in the relevant art or the meaning indicated by their usage in context, but if an explicit definition is provided, that meaning shall prevail.
[0110] Herein, references to "an embodiment" or "an embodiment" do not necessarily refer to the same embodiment, although they may refer to the same embodiment. Unless the context clearly requires otherwise, throughout the specification and claims, the words "include", "comprises", etc. should be interpreted as inclusive, rather than exclusive or exhaustive; that is, "including but not limited to". Words using the singular or plural also include the plural or singular, respectively, unless explicitly limited to one or more. Additionally, the words "herein", "above", "below", and words of similar meaning, when used in this application, refer to the entirety of this application, rather than to any part of this application. When a claim uses the word "or" to refer to a list of two or more items, the word covers all of the following interpretations of the word: any one of the items in the list, all of the items in the list, and any combination of the items in the list, unless explicitly limited to one or another. Any term not explicitly defined herein has the conventional meaning commonly understood by those skilled in the relevant art.
Claims
1. A method comprising: receiving, at a processor, a series of intravascular ultrasound (IVUS) images of a blood vessel of a patient, the series of IVUS images comprising a plurality of image frames; identifying, by the processor via a first machine learning (ML) model, a frame set of the plurality of image frames, wherein frames in the frame set are associated with one or more side branches of the blood vessel; identifying, by the processor via a second ML model, a location of at least one of the one or more side branches in one or more frames of the plurality of image frames; as well as A subset of frames is selected from the set of frames by the processor based at least on an output from the first ML model and an output from the second ML model.
2. The method of claim 1 , wherein the first ML model is trained to infer the set of frames from the plurality of image frames.
3. The method according to any one of claims 1 to 2, wherein the first ML model is configured to receive one or more adjacent frames from the plurality of image frames as input.
4. The method according to any one of claims 1 to 3, wherein the first ML model is a convolutional neural network (CNN), a visual transformer network, or a combination of a CNN and a visual transformer network.
5. The method of claim 4, wherein the first ML model is configured to apply a convolution window on a single frame or multiple adjacent frames from the plurality of image frames until all frames in the plurality of image frames have been received as input.
6. The method according to any one of claims 1 to 5, wherein the second ML model is trained to: determining, for each frame of the plurality of image frames, whether the frame represents a side branch of the one or more side branches; and For each frame determined to be representative of the side branch of the one or more side branches, a position of the side branch of the one or more side branches in the frame is identified.
7. The method of any one of claims 1 to 6, wherein the first ML model is configured to output, for each frame in the frame set, a confidence score representing a confidence that the one or more side branches were detected in the frame, and wherein selecting a subset of frames from the frame set comprises: identifying frames from the set of frames having a confidence score greater than or equal to a threshold; as well as The identified frames are selected for inclusion in the subset of frames.
8. The method of any one of claims 1 to 7, wherein the second ML model is trained to generate the indication of the location as a bounding box.
9. The method of claim 8, wherein selecting frames from some of the set of frames of the series of IVUS images comprises: identifying adjacent frames from the plurality of image frames, wherein the bounding boxes in each frame are within a threshold distance of each other; as well as The side branches associated with the identified frame are merged.
10. The method of claim 9, wherein the first ML model is configured to output a confidence score for each frame in the set of frames, the confidence score representing a confidence that the one or more side branches were detected in the frame, and wherein merging the side branches associated with the identified adjacent frames comprises: identifying, from the set of frames, one of the adjacent frames having a highest confidence score, wherein the bounding boxes are within a threshold distance of each other; as well as The identified one of the adjacent frames having the highest confidence score is selected as the frame from the plurality of image frames for inclusion in the subset of frames.
11. The method according to any one of claims 1 to 10, wherein the second ML model is a convolutional neural network (CNN).
12. The method according to any one of claims 1 to 11, wherein the second ML is a visual transformer network.
13. The method according to any one of claims 1 to 12, wherein the second ML model is a combination of a CNN and a Vision Transformer Network.
14. A device for an intravascular imaging apparatus, the device comprising: processor; as well as A memory comprising instructions which, in response to being executed by the processor, cause the apparatus to implement the method according to any one of claims 1 to 13.
15. At least one machine-readable storage device comprising a plurality of instructions which, in response to being executed by a processor of an intravascular ultrasound (IVUS) imaging system, cause the processor to implement the method of any one of claims 1 to 13.
Citation Information
Patent Citations
Piezocomposite transducers
US20060100522A1
Systems and methods for evaluating objects with an ultrasound image
US6945938B2
Image formation apparatus and sublimation printer
US7246959B2
