Systems and methods for classification of arterial image regions and features thereof

Abstract

To provide methods and systems suitable for evaluating image data from a patient on a real time or substantially real time basis using machine learning methods and systems.SOLUTION: There are provided systems and methods for improving diagnostic tools for end users such as cardiologists and imaging specialists using machine learning techniques applied to specific problems associated with intravascular images that have polar representations. Further, given the use of rotating probes to obtain image data for OCT, IVUS, and other imaging data, dealing with the two coordinate systems associated therewith creates challenges. The present disclosure addresses these and numerous other challenges relating to solving the problem of quickly imaging and diagnosing a patient such that stenting and other procedures may be applied during a single session in the catheter lab.SELECTED DRAWING: Figure 1C

Description

[Technical field]

[0001] In part, this application relates to imaging arteries and segmenting their components. Specifically, in some embodiments, the present invention relates to The present application relates to characterizing arterial tissue and associated arterial regions and features of interest and / or On the application of machine learning to classification.

[0002] [CROSS REFERENCE TO RELATED APPLICATIONS] This application is a joint venture with U.S. Provisional Patent Application No. 62 / 791,876, filed January 13, 2019. This application claims priority to and the benefit of, and is hereby incorporated by reference in its entirety. It is incorporated herein by reference. [Background technology]

[0003] Optical coherence tomography (OCT) is a technique used in ophthalmology, cardiology, and It is an interferometric imaging technique that has wide applications in endoscopic surgery, gastroenterology and other medical fields. The optical fiber probe allows for high-resolution visualization of subsurface structures. OCT has therefore become particularly useful for minimally invasive imaging of internal tissues and organs. OCT systems can generate images at up to 100 frames per second, allowing This allows imaging of coronary arteries in the beating heart within seconds. time domain OCT (TD-OCT) and frequency domain (Fourier domain OCT or optical frequency The method will be implemented in both OFDI (optical frequency domain imaging) and OFDI (optical frequency domain imaging). OCT is a method for measuring blood flow using intravascular ultrasound (IVUS). Various other imaging techniques, such as endoscopic imaging, fluoroscopy, x-ray based imaging systems, and other imaging techniques. It can be used in conjunction with imaging techniques.

[0004] OCT imaging of various parts of the patient's body can help doctors determine the best type and course of treatment. For example, imaging of coronary arteries by intravascular OCT can help locate stenosis. ion, the presence of vulnerable plaque, or the type of atherosclerotic plaque. This information can be used by cardiologists to determine whether or not to administer medication (e.g., cholesterol-lowering drugs). ), catheter-based procedures such as angioplasty and stent placement, or coronary bypass surgery. Choosing which procedure, such as an invasive surgical procedure like endoscopic surgery, will best benefit the patient In addition to its applications in clinical medicine, OCT has been used in animal studies and clinical trials. It is also extremely useful for drug development in the field.

[0005] Normal arteries have a consistent layered structure consisting of the tunica intima, tunica media and tunica adventitia. As a result of the atherosclerotic process, the intima becomes pathologically thickened and becomes packed with fibrous, proteoglycan Various types of proteins, including macrophages and other inflammatory cells, are involved in the synthesis of proteins, lipids and calcium. These tissue types can be easily measured by manual measurement and imaging techniques. These have different optical properties that can be measured by the The plaques in question are so-called vulnerable plaques, which have a fibrous cap with an underlying lipid pool. do.

[0006] In a typical OCT imaging system, an optical probe mounted on a catheter is used to image the inside of a coronary blood vessel, etc. The light beam is then sent out and the backscattered signal is , are received through coherent detection using an interferometer. As the area is scanned, many lines of data can be collected. The image (2D or 3D) is then reconstructed using known techniques. Cardiologists visually analyze the blood vessels and identify pathological features such as vessel wall thickening and plaque composition. Be appreciated. Summary of the Invention [Problem to be solved by the invention]

[0007] A tissue type is identified by its appearance on the screen, so certain Certain information may not be readily visible, which may lead to errors in the analysis. Arteries, organs, or other body parts of a given patient that are difficult to accurately classify by visual inspection Various other components or areas of interest in the area are found in virtually every medical field. Therefore, it is possible to detect various targets in the image data and What is needed is a system and method for representing a user's experience and the content of that experience to an end user. and address other challenges. [Means for solving the problem]

[0008] A system of one or more computers is a set of computers that perform actions on the system during operation. The software, firmware, hardware, or combination thereof that enables the When installed on a system, it is designed to perform a specific operation or action. The one or more computer programs may be configured to be executed by a data processing device. The device may be configured to perform a particular action by containing instructions that, when executed by the device, cause the device to perform an action. One general aspect is to provide a method for performing a process or an action. The above machine learning system (MLS) was used to evaluate the coronary arteries. The present invention includes a method for

[0009] In one embodiment, the disclosure relates to various method steps. A set of image data including frames of polar images is acquired, and Each region or feature identified is a ground truth annotation. one or more regions or features of interest in each polar image of the set of image data, and running a machine learning system using the set of annotated polar images. training a neural network of the system, Each of the regions is identified by a class, and the trained neural network Inputting polar image data into the network and generating a set of color-coded regions, each of which represents a and displaying a predicted output image corresponding to the class.

[0010] In one embodiment, the system includes one or more It contains an AI processor and one or more trained software-based neural networks. The machine learning system includes a network of machines that run on one or more AI processors. , a convolutional neural network. The method may include: In one embodiment, the method may include acquiring a set of image data comprising: The image data includes intravascular image data. In one embodiment, the image data The method further comprises: one or more regions or features of interest in each polar image of the set of images to be annotated; The ground truth annotations may include annotating the ground truth features. The method may store the group of approximations in memory as a set of truth masks. Use a set of annotated polar images, e.g. through a set of round truth masks This could include training a neural network for a machine learning system using In one embodiment, a ground truth mask may be generated based on a particular feature or feature set. contains the region of interest corresponding to the channel.

[0011] Therefore, the first region of the ground truth mask corresponds to the first feature. and a second region of the ground truth mask can correspond to a second feature. This correspondence can be applied to M features and P regions, where each feature can be one or more As an example, two or more regions of calcium, a region of the lumen, and a region of the intima A region can be part of a ground truth mask and can be a class or Their classification by type has the effect of making each region so classified / segmented. In one embodiment, the ground truth The mask is a representation of which regions in the input image data from the user correspond to a particular feature or channel. It is used to train a neural network to detect / predict what will happen to the In one embodiment, the method comprises the steps of: A ground truth mask is input to the neural network to and training a neural network, the neural network comprising a plurality of features. It takes one or more epochs until the feature / channel can be validly detected within the error threshold. In one embodiment, the method is trained using image data, such as polar image data. The image data is fed into a trained neural network and the machine learning system and displaying a predicted output image from the ground system. Truth image masking allows the user to annotate the representation of an image in a user interface. In various embodiments, the polar image is generated in response to such user input. The neural network is annotated in the interface. One or more software applications such as PyTorch, LibTorch, etc. Another embodiment of this aspect is a method for performing the method in a multi-threaded manner, the method comprising: A corresponding computer system configured to execute the method and apparatus, and one or more and a computer program recorded on a computer storage device.

[0012] The embodiment may include one or more of the following processes or steps: In an embodiment, each image includes a plurality of image data elements relating to said coronary arteries. In one embodiment, annotating comprises identifying groups or regions of pixels that define a feature of interest. This is done using a graphical user interface that includes user control to select the 2D boundary. In one embodiment, the training of the neural network is performed by The cost function is then repeated until the output of the cost function is equal to or less than a threshold. We compare the prediction output of a machine learning system (MLS) with the ground truth input. In various embodiments, a cross-sectional area is used to measure the error associated with the cost function. The method further comprises: The method may further include classifying the features as a type or class. In the above, the type or class is selected from the group consisting of intima, media, adventitia, lumen, EEL, IEL, and protease. In one embodiment, the calcium plaque is selected from the group consisting of calcium plaque, calcium plaque, calcium sulphate ... wherein the image data used with the systems and methods disclosed herein comprises: Carpet view image, scan line, pixel, 2D image, 3D image, angiography image , intravascular images, CT scan images, X-ray images, and other images of arteries, veins, organs, or other parts of the circulatory system. The above features, regions, channels, classes, etc. are trained on these. It can be detected using a trained neural network.

[0013] In one embodiment, the feature, the region, the type and / or the class are , one or more side branches, a lumen, a guidewire, a stent strut, a stent, Folded stent, bioresorbable vascular scaffold (BVS) fold), drug eluting stents (DES), Blooming Artificial Intravascular Coagulation Fact, pressure wire, guidewire, lipid, atherosclerotic plaque, stenosis, calcium ,Calcified plaque, tissue containing calcium, lesions, fat, Malapose stent and sufficient A subexpanded stent, an overexpanded stent, radiopaque markers, and the arterial tree the branching angle of the probe, the calibration element of the probe, the doped membrane, the light scattering particles, the sheath, and the doped sheath and a fiducial alignment point, a diameter measure, a calcium arc measure, and a thickness of a region or feature of interest. , radial scale, guide catheter, shaded area, guidewire segment, length, thickness and and others disclosed herein.

[0014] In one embodiment, each data element, each image, and each output is processed by the machine learning system. In one embodiment, the annotation is stored in a machine-readable memory in electronic communication with the system. A set of polar images is an image that contains one or more imaging artifacts or unwanted imaging conditions. In one embodiment, the one or more imaging artifacts or the unwanted Unfavorable imaging conditions include incomplete clearing of the artery prior to intravascular imaging of the artery, insufficient contrast, Insufficient contrast solution; light intensity below the average level for intravascular imaging; Contrast cloud, non-uniform rotational distortion (NURD), blooming Reflections from artifacts, jailed side branches, and imaging probe components In one embodiment, the method and system disclosed herein is selected from the group including The system further comprises: one or more arc-based metrics, Ca and E, in the predicted output image; The similarity measures of both EL and detected EEL diameter and detected Ca depth are identified. In one embodiment, the method further comprises the steps of: The neural network is a convolutional neural network, and the first node or The number of input channels of the layer is four. The method and system includes generating a carpet view using line projection and and filtering the screen view to reduce noise in the predicted output image. Operable or capable of performing the following:

[0015] In one embodiment, the above (and other aspects relating to identifiable elements in the input image data) Each of the above (and other examples disclosed herein) is a region of interest for training the MLS. Region of interest (ROI) or feature of interest (FOI) can be used as a marker and can be detected by a trained MLS. In one embodiment, each of the above has an associated mass data channel. or a ground truth mask or output In one embodiment, the output matrix is ​​an element in an image mask, such as a The block includes a plurality of regions, and a multi-channel segmentation representation is generated for the input data. As generated, different regions correspond to different channels. Then, a first frame of image data is processed using the neural network of the MLS. generating a first output mask corresponding to a first frame of the image data; In the mask, regions / features of interest are identified using indicia such as color coding, hatching, etc. Therefore, the first output mask is changed to include overlay, image data, and or changes to the mask region that identifies the class / type, or specific features, class The input pixel number may be further divided into two groups, each group having a different indicia relating to the area of ​​the pixel that is classified as a pixel. Includes force image data.

[0016] In one embodiment, the predicted output image shows boundaries of predicted or classified features. The method further comprises converting the predicted output image into a polar form. The method further includes converting the detected The output polar image mask with indicia corresponding to the ROI / FOI is the Cartesian image (Ca The indicia can be converted to a Cartesian image, In various embodiments, the annotations for the Cartesian image are The acquisition and processing involves the generation of ground truth images / masks and neural network traces. Instead, ground truth polar images are generated for a given niche. The resulting image is then processed and used to train a neural network. In, carpet view masks, and subsets thereof, can be used. In one embodiment, the carpet view has a first axis corresponding to the frame number; and a second axis corresponding to the scan line number.

[0017] In one embodiment, the neural network is conformal. l) A neural network. In one embodiment, the MLS is an AI program. In one embodiment, the AI ​​processor includes one or more parallel processing elements. The AI ​​processor includes n parallel processing elements and a dedicated AI processor memory. In one embodiment, the dedicated AI processor memory is approximately 8 GB. In one embodiment, the dedicated AI processor memory is in the range of about 64 GB. In one embodiment, the AI ​​program is in the range of about 64 GB to about 128 GB. In one embodiment, the parallel processing element is a graphics processing unit. The core is a CUDA (Compute Unified Device Architecture) core processor. processor, tensor core processor, and stream processor. In one embodiment, the AI ​​processor is an edge network application. In one embodiment, the device is operated locally through a The computing device disposed within the housing of the data collection / imaging system. Includes 8GB or more of dedicated memory with 32GB or more of on-board RAM as part of the An AI processor such as a graphical processing unit is used.

[0018] In one embodiment, the method includes using MLS when classifying user image data. Processing time is reduced by smoothing the image data before inputting it into the neural network. This further includes reducing the number of features / regions of interest during the training phase. It can be applied to patient image data when classifying, detecting, and / or identifying. The method further comprises: reducing the processing time of the MLS when classifying user image data by using a neural network; Reducing the image by resizing or excluding areas before inputting it into the work The method further includes performing a circular shift once, twice, or or three times. The method further comprises performing a left-right scan on one or more of the polar images. The method further includes performing a right flip. The method may be performed using image processing or machine learning methods. The method further includes performing lumen detection using the detected lumen boundary data set and generating a set of detected lumen boundary data. The method further comprises: determining whether the patient is a patient; determining whether the patient is a patient; and determining whether the patient is a patient. and generating an image mask. any method or process, or computer software on a computer-accessible medium The method and system may include a variety of bus and interface components. may include.

[0019] One general embodiment includes performing lumen detection to detect the lumen boundary. In this embodiment, the luminal boundary data detected on an image-by-image basis is also stored together with the polar image data of the patient. Another embodiment of this aspect is that each Corresponding computer systems, devices configured to perform the actions of the methods. and a computer program stored on one or more computer storage devices. .

[0020] Implementations may include one or more of the following features. In one embodiment In accordance with the present invention, inputting the detected luminal boundary data includes inputting the detected luminal boundary data into the polar image data of the patient. Reduces latency in classifying regions and features of interest. Software, methods or processes, or computer software on a computer-accessible medium In one embodiment, the training software includes patient data. Or each input image for processing is converted into multiple versions, the multiple versions being , which is generated by left-right flip and circular shift. This provides an extended dataset. ,And this expanded dataset reduces the error and improves the ,performance of neural networks / MLS. In various embodiments, MLS refers to neural networks. This also includes q and vice versa.

[0021] One general aspect is a data collection and / or imaging and region / feature characterization system. The system also includes a housing. The system receives polar data, ultrasonic data, and , one or more of the following image data: optical image data, X-ray image data, and intravascular image data The intravascular system also includes a frame grabber for receiving the signal. The intravascular system also includes a power supply. The system also includes one or more electronic memory storage devices in electrical communication with the power supply. The intravascular system is executable on a processor and includes one or more electronic memory stores. The intravascular imaging system also includes one or more image processing software modules stored in the intravascular imaging system. The system also includes a computing device including a first processor, A power supply is in electronic communication with the power supply and the first processor. In an embodiment, one or more AI processors and dedicated AI processor memory are It may be located in a housing or connected to the hardware through one or more ports, buses, or networks. In one embodiment, the MLS and its trained The neural network developed is suitable for client / server implementations, edge computing, through cloud or software as a service implementation It is operated remotely.

[0022] In one embodiment, the system further comprises: The system also includes one or more software programs stored in the one or more functions. Machine learning system with neural network including machine learning software module The intravascular system also includes one or more AI processors, The machine learning software module above is executable on the one or more AI processors. One or more AI processors, a bus, an AI processor memory, and image data are stored in the and an interface for transmitting and receiving data to and from a first processor, The system is in electronic communication with the power supply and is connected to the machine learning system, the computing device, The chair and the one or more electronic memory storage devices are disposed within the housing. In one embodiment, the bus is a PCIe bus. corresponding computers each configured to perform the actions of the method. A system, an apparatus, a computer program recorded on one or more computer storage devices program, AI processor, dedicated ASIC, circuitry and circuitry components. In an embodiment, the bus is connected to the AI ​​processor and to on-board memory and diagnostics. Connect the processor / imaging system.

[0023] Implementations can include one or more of the following features: The housing includes an optical connector. The housing of a thermography tomography imaging system or an intravascular ultrasound imaging system. The one or more image processing software modules may generate polar intravascular images. Software to convert Cartesian intravascular images to polar images and software to label regions or features of interest when displayed to the end user. The tissue classification overlay software, lumen detection software module, and image leveling software A smoothing pre-processing software module, an image resizing software module, and a graphics A GU that labels or marks training images using ground truth data Image annotation software with I, pre-processing software module, and circular shift software and one or more of the machine learning software modules. The module is a neural network interface, a lumen contour prediction, and a side branch. Prediction, Image Resizing Module, User Interface and Input Processing Software The neural network parameters are controlled and set by the MLS interface. Interface software module, MLS memory manager software, and pre-processing software A software module, a stent strut prediction software module, and a jail and a guidewire prediction software module. , and an interface module for exchanging data with an imaging system. The implementation of the described techniques may be implemented in hardware, a method or process, or a computer. The computer software may be included on a data-accessible medium.

[0024] In part, this disclosure uses machine learning (ML) methods and systems. and adapted to evaluate image data from a patient in real time or substantially real time. In various embodiments, the present invention relates to a computer-based method and system for detecting intravascular A set of image data such as data pullback is fed to a convolutional neural network Classification is done virtually in real time using a trained neural network In various embodiments, the image data set comprises between about 400 frames and about 600 frames. 00 frames, which are acquired from memory or by imaging the patient using an imaging system. In one embodiment, about 400 frames are acquired. A set of image data containing about 600 frames is classified. In one embodiment, In this embodiment, substantially real time is in the range of about 1 second to about 60 seconds. In this embodiment, substantially real time is in the range of about 1 second to about 30 seconds. In this embodiment, substantially real time is in the range of about 1 second to about 20 seconds. In this embodiment, substantially real time is in the range of about 1 second to about 15 seconds. In this embodiment, substantially real time is in the range of about 1 second to about 10 seconds. In this case, the substantial real time is less than about 10 seconds. and using machine learning techniques applied to the specific problem related to intravascular images with polar representations. The aim of the project is to improve diagnostic tools for end users such as cardiologists and imaging professionals. In addition, the rotating probe can be used to capture OCT, IVUS image data, and other imaging data. When considering how to obtain the two coordinate systems, the challenge is to deal with the two associated coordinate systems. The present disclosure addresses these challenges and provides a method for performing stent placement during a single session in a cath lab. The problem of rapidly imaging and diagnosing a patient so that imaging and other procedures can be applied is and numerous other problems related to solving multiple features or regions of interest. The ability to segment images into Inform evaluation of atherectomy and other surgical options and provide patient By providing clinicians with diagnostic information to assess changes in the patient's condition, and reducing the time spent during subsequent treatment procedures.

[0025] In one embodiment, MLS system training is conducted by an expert. The annotated polar image is generated using the M The MLS is used to train the MLS by processing new polar images from the patient. , which still produces an output of classified image regions in polar format. After using S for prediction and inference, the polar form of the prediction output from MLS is converted to Cartesian form. The tissue regions were classified (lumen, intima, side branch, guidewire, stent, Images having the following properties (e.g., chromosome number, chromosome number, chromosome number, etc.) are displayed in Cartesian format. In,coordinates can be reversed and the image is in Cartesian form when,annotated. and finally converted to polar form after a given MLS processing and prediction.

[0026] In various embodiments, user interface feedback of various workflows is provided. In addition, a probability map and a texture map are generated that provide a The trained neural networks can be combined, compared, convolved, and otherwise used to generating output results that classify regions and features of interest using neural networks; In various embodiments, a given neural network preferably includes: It is trained using annotated polar images.

[0027] In part, the present disclosure provides one or more compositions based on characterized tissue of a body lumen, such as a coronary artery. A user interface that uses a structured map representation to facilitate improved information and time management. In various embodiments, the various detected ROIs / FOIs are Co-registered with endoscopic data and used as part of an imaging system or other diagnostic system The above user interfaces can be used to display the information.

[0028] In part, the present disclosure uses an imaging processing pipeline to identify tissue types and side brands. Other features, such as catheters, stents, guidewires and other features, including vascular properties and materials. It detects regions of interest in blood vessels and uses neural networks to Other R such as calcium, lumen, media, intima, lipids and others disclosed herein. The present invention relates to a method for identifying a region of interest within the blood vessel for detecting OI / FOI.

[0029] In one embodiment, a segmentation in one or more masks, images, or outputs is performed. and / or the tissue type or tissue characteristic, region of interest, selected for detection and expression. The ROI, feature of interest, class or type, or vessel feature may be any of the following: cholesterol, fiber, lipid pool, lipid, fibrofatty, calcification, calcium nodules, calci The present invention relates to a method for treating pulmonary artery disease, including the use of pulmonary artery disease, ... The various systems disclosed herein include tissue maps that include one or more of the above. The system includes a dedicated circuit, a controller, an FPGA, an AI processor, and the components disclosed herein. and other components such as the above to carry out all of the methods and processes disclosed herein. The device is operable to perform the

[0030] While the present disclosure is directed to various aspects and embodiments, the various The aspects and embodiments may be integrated, combined, or combined. As an integrated system or as part of separate components, devices and systems where appropriate It is understood that the present invention can be used together as a system. Each of the embodiments described herein may incorporate the above aspects to varying degrees as necessary for a given implementation. can be incorporated into each one.

[0031] The figures are not necessarily to scale, emphasis instead generally being placed upon illustrative principles. The figures are to be considered in all respects illustrative and not intended to limit the present disclosure. , the scope of which is defined only by the claims.

[0032] The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be available upon request and payment of the necessary fee. If paid, it will be provided by the US Patent and Trademark Office. [Brief description of the drawings]

[0033] [Figure 1]FIG. 1 is a schematic diagram of a system suitable for training a neural network, such as a network that uses ground truth masks obtained on arterial image data, such as image frames, to generate predicted outputs that classify particular features or regions of interest, in accordance with an exemplary embodiment of the present disclosure. [Figure 1A] FIG. 1 is a schematic diagram of a machine learning system suitable for use with image data, such as image data acquired of arteries, during a training phase, in accordance with an exemplary embodiment of the present disclosure. [Figure 1B] FIG. 1 is a schematic diagram of a machine learning system suitable for use with user image data, such as image data acquired of arteries, to detect, predict, and / or classify features of interest in the user image data, according to an exemplary embodiment of the present disclosure. [Figure 1C] FIG. 1 is a schematic diagram of a machine learning system suitable for use with user image data showing input polar image data, such as OCT image data, that is processed to generate ground truth image data used to train an MLS, and predicted image data obtained using a trained MLS, according to an exemplary embodiment of the present disclosure. [Figure 1D] FIG. 1 is a schematic diagram of a machine learning system suitable for use with user image data to generate classified or characterized features or regions of interest for the user image data, according to an exemplary embodiment of the present disclosure. [Figure 1E] FIG. 1 is a schematic diagram of a machine learning system suitable for use with user image data that generates classified or characterized features or regions of interest for the user image data, including a lumen detection system and a pre-processing system, according to an exemplary embodiment of the present disclosure. [Figure 1F] 1 illustrates an example process flow of a training process and a subsequent prediction process of certain example embodiments of the present disclosure. [Figure 1G] FIG. 1 illustrates image data obtained from imaging a patient given an exemplary embodiment of the present disclosure, and three output probability maps corresponding to detecting the media, calcium, and lumen as separate channels. [Diagram 2]FIG. 1 illustrates a multi-layer neural network architecture suitable for training and generating predictive results using ground truth annotations, according to an exemplary embodiment of the present disclosure. [Figure 2A] FIG. 1 is a schematic diagram showing a set of outputs from an MLS including indicia corresponding to various color-coded classes, and their use to obtain projections to generate a carpet view and optionally a tissue map, in accordance with an exemplary embodiment of the present disclosure. [Figure 2B] 1 is a plot of an MLS system showing loss function values ​​for testing data and training data, according to an example embodiment of the present disclosure. [Figure 2C] 1 is a plot of an MLS system showing mismatch percentage values ​​for test data and training data, according to an exemplary embodiment of the present disclosure. [Figure 2D] 1A-1C show intravascular polar images of a patient's artery, polar images annotated or augmented with classified ground truth regions, and polar images with features and / or regions of interest classified using MLS, according to exemplary embodiments of the present disclosure. [Figure 2E] 1A-1D show, from left to right, polar image data of an artery, a polar image annotated with ground truth lumen data, lumen detection results performed with MLS with a 2D neural net, and lumen detection results performed with MLS with a 3D neural net, in accordance with an exemplary embodiment of the present disclosure. [Figure 2F] 1A-1D show, from left to right, polar image data of an artery, a polar image annotated with ground truth lumen data, lumen detection results performed with MLS with a 2D neural net, and lumen detection results performed with MLS with a 3D neural net, in accordance with an exemplary embodiment of the present disclosure. [Figure 2G]1A-1D show, from left to right, polar image data of an artery, a polar image annotated with ground truth lumen data, lumen detection results performed with MLS with a 2D neural net, and lumen detection results performed with MLS with a 3D neural net, in accordance with an exemplary embodiment of the present disclosure. [Figure 2H] 1A-1D show, from left to right, polar image data of an artery, a polar image annotated with ground truth lumen data, lumen detection results performed with MLS with a 2D neural net, and lumen detection results performed with MLS with a 3D neural net, in accordance with an exemplary embodiment of the present disclosure. [Figure 3A] FIG. 1 illustrates a graphical user interface of an MLS system suitable for navigating and annotating image data for training an MLS or a component thereof, according to an exemplary embodiment of the present disclosure. [Figure 3B] FIG. 13 illustrates a polar frame of image data from an imaging probe pullback through an artery, annotated using a graphic user interface-based training tool for MLS, according to an exemplary embodiment of the present disclosure. [Figure 3C] FIG. 1 is a schematic diagram of a Cartesian format of a segmented arterial image in which tissue regions are classified by a predictive operation of MLS, according to an exemplary embodiment of the present disclosure. [Figure 3D] FIG. 13 illustrates an output image from MLS converted to Cartesian form showing the input image and a semantic segmentation of the image showing the MLS classified media region or mask, lumen region or mask, and Ca region or mask, according to an exemplary embodiment of the present disclosure. [Figure 3E] FIG. 13 illustrates an output image from MLS converted to Cartesian form showing the input image and a semantic segmentation of the image showing the MLS classified media region or mask, lumen region or mask, and Ca region or mask, according to an exemplary embodiment of the present disclosure. [Figure 4A]FIG. 1 illustrates an example imaging diagnostic system that includes one or more machine learning systems that are or can be integrated with an imaging system, or that are or can be otherwise combined with an imaging system. [Figure 4B] FIG. 1 illustrates an example imaging diagnostic system that includes one or more machine learning systems that are or can be integrated with an imaging system, or that are or can be otherwise combined with an imaging system. [Figure 4C] FIG. 1 illustrates an example imaging diagnostic system that includes one or more machine learning systems that are or can be integrated with an imaging system, or that are or can be otherwise combined with an imaging system. [Figure 5A] FIG. 1 is a schematic diagram of a system suitable for performing guided atherectomy using an optical imaging system, such as an OCT imaging system, according to an exemplary embodiment of the present disclosure. [Figure 5B] 1 is a schematic diagram of a method suitable for performing guided atherectomy using an optical imaging system, such as an OCT imaging system, according to an exemplary embodiment of the present disclosure. [Figure 5C] FIG. 1 illustrates an OCT image segmented to highlight calcium plaque regions using an MLS system, such as a deep learning-based system, according to an exemplary embodiment of the present disclosure. [Figure 5D] FIG. 1 illustrates an OCT image segmented to highlight calcium plaque regions using an MLS system, such as a deep learning-based system, according to an exemplary embodiment of the present disclosure. [Figure 5E] 1A-1C show user interface representations of calcium plaque in 3D, cross-sectional and longitudinal sections, according to exemplary embodiments of the present disclosure. [Figure 5F] FIG. 1 illustrates a calcium map generated on an angiography system using angiography co-registration, according to an exemplary embodiment of the present disclosure. [Figure 5G] FIG. 1 illustrates an annotated Cartesian OCT image according to an exemplary embodiment of the present disclosure. [Figure 5H] FIG. 1 illustrates an annotated Cartesian OCT image according to an exemplary embodiment of the present disclosure. [Figure 5I] FIG. 1 illustrates an expanded Cartesian OCT image identifying various layers of an artery, according to an exemplary embodiment of the present disclosure. [Figure 5J] FIG. 5D illustrates a histology image of the artery of FIG. 5C with various layers of the artery identified as part of a ground truth or training set review suitable for annotating image data for use with a given MLS embodiment, according to an exemplary embodiment of the present disclosure. [Figure 6A] 13A-13C show Cartesian OCT images and predicted output of MLS identifying lumen, media, and Ca plaque, and exemplary diameter measurements, respectively, according to an exemplary embodiment of the present disclosure. [Figure 6B] 13A-13C show Cartesian OCT images and predicted output of MLS identifying lumen, media, and Ca plaque, and exemplary diameter measurements, respectively, according to an exemplary embodiment of the present disclosure. [Figure 6C] 13A-13C show additional OCT images, a ground truth mask, and an output mask showing tissue classification and prediction using MLS, according to an exemplary embodiment of the present disclosure. [Figure 6D] 13A-13C show additional OCT images, a ground truth mask, and an output mask showing tissue classification and prediction using MLS, according to an exemplary embodiment of the present disclosure. [Figure 6E] 13A-13C show additional OCT images, a ground truth mask, and an output mask showing tissue classification and prediction using MLS, according to an exemplary embodiment of the present disclosure. [Figure 6F] 13A-13C show additional OCT images, a ground truth mask, and an output mask showing tissue classification and prediction using MLS, according to an exemplary embodiment of the present disclosure. [Figure 7A]FIG. 2 illustrates an exemplary original OCT image in Cartesian form as acquired using a combined imaging MLS-based system, according to an exemplary embodiment of the present disclosure. [Figure 7B] FIG. 2 illustrates a grayscale polar image, according to an exemplary embodiment of the present disclosure. [Figure 7C] FIG. 1 illustrates a ground truth mask according to an exemplary embodiment of the present disclosure. [Figure 7D] FIG. 7B illustrates an example embodiment of the present disclosure, showing an annotation using a trained MLS, and a deep learning output of the MLS embodiment generated using the image of FIG. 7B as input for training and prediction. [Figure 8A] FIG. 1 illustrates pre-processing operations on a polar OCT image before annotation with expert or ground truth data, according to an exemplary embodiment of the present disclosure. [Figure 8B] 1A-1C illustrate various exemplary polar image masks for detecting lumen, media, and calcium, according to exemplary embodiments of the present disclosure. [Figure 8C] 1A-1C illustrate various exemplary polar image masks for detecting lumen, media, and calcium, according to exemplary embodiments of the present disclosure. [Figure 8D] 1A-1C illustrate various exemplary polar image masks for detecting lumen, media, and calcium, according to exemplary embodiments of the present disclosure. [Figure 9A] 1A-1C show polar OCT images illustrating a circular shift and left-right flip applied to the original OCT polar image as part of a pre-processing step, according to an exemplary embodiment of the present disclosure. [Figure 9B] 1A-1C show polar OCT images illustrating a circular shift and left-right flip applied to the original OCT polar image as part of a pre-processing step, according to an exemplary embodiment of the present disclosure. [Figure 9C] 1A-1C show polar OCT images illustrating a circular shift and left-right flip applied to the original OCT polar image as part of a pre-processing step, according to an exemplary embodiment of the present disclosure. [Figure 10A]FIG. 1 illustrates an image showing ground truth annotated images and predicted outputs, according to an exemplary embodiment of the present disclosure. [Figure 10B] FIG. 1 illustrates an image showing ground truth annotated images and predicted outputs, according to an exemplary embodiment of the present disclosure. [Figure 11A] 1A-1C are images illustrating various pre-processing steps developed to increase MLS processing speed and reduce patient wait times, according to exemplary embodiments of the present disclosure. [Figure 11B] 1A-1C are images illustrating various pre-processing steps developed to increase MLS processing speed and reduce patient wait times, according to exemplary embodiments of the present disclosure. [Figure 11C] 1A-1C are images illustrating various pre-processing steps developed to increase MLS processing speed and reduce patient wait times, according to exemplary embodiments of the present disclosure. [Figure 11D] 1A-1C are images illustrating various pre-processing steps developed to increase MLS processing speed and reduce patient wait times, according to exemplary embodiments of the present disclosure. [Figure 12A] 1 is a flowchart of exemplary processing steps in which raw image data is fed and processed along with annotation using ground truth data, according to an exemplary embodiment of the present disclosure. [Figure 12B] 1 is a flowchart of exemplary processing steps in which raw image data is fed and processed along with annotation using ground truth data, according to an exemplary embodiment of the present disclosure. [Figure 13A] 1A-1C illustrate exemplary user interfaces of diagnostic tools that support stent planning through virtual stent placement incorporating tissue classification using MLS systems and methods, according to exemplary embodiments of the present disclosure. [Figure 13B] 1A-1C illustrate exemplary user interfaces of diagnostic tools that support stent planning through virtual stent placement incorporating tissue classification using MLS systems and methods, according to exemplary embodiments of the present disclosure. [Figure 14]1A-1C show representations of an artery including various cross sections of the boundary from an OCT scan of a subject's artery and generation of a tissue map using various sets of image data acquired during pullback, according to an exemplary embodiment of the present disclosure. [Figure 15] 1A-1C show schematic representations of an artery acquired using one or more imaging modalities and various cut planes suitable for generating a tissue map, according to an exemplary embodiment of the present disclosure. [Figure 16] FIG. 13 illustrates various tissue map representations generated using OCT imaging pullback of an artery with various indicia integrated into a user interface displaying the various tissue maps to support diagnostic and treatment planning such as stent placement and atherectomy, according to an exemplary embodiment of the present disclosure. [Figure 17] FIG. 13 illustrates various tissue map representations generated using OCT imaging pullback of an artery with various indicia integrated into a user interface displaying the various tissue maps to support diagnostic and treatment planning such as stent placement and atherectomy, according to an exemplary embodiment of the present disclosure. [Figure 18] FIG. 13 illustrates various tissue map representations generated using OCT imaging pullback of an artery with various indicia integrated into a user interface displaying the various tissue maps to support diagnostic and treatment planning such as stent placement and atherectomy, according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0034] A variety of data acquisition and analysis systems are available to obtain information about the coronary system. Data obtained from a blood vessel using a device or intravascular measurements related to a blood vessel may be used. Data derived from extravascular measurements or extravascular measurements may be analyzed or used to assist researchers and clinicians. Optical coherence tomography (OCT) uses an interferometer to measure blood An imaging modality that obtains distance measurements for objects placed within a tube or blood vessel. Intravascular ultrasound (IVUS) can also be used with a probe to image portions of blood vessels. Angiography and fluoroscopy systems are also capable of making diagnostic decisions and providing stenosis. To image the patient, various possible treatment options, such as placement of a catheter, These and other imaging systems image the patient from the outside or inside. It can be used to capture images from a variety of sources and obtain raw data. The present disclosure provides an improved method for extracting components of medical imaging data. Various machines that contain one or more networks, such as neural networks, provide classification An exemplary MLS-based system is shown in FIG. 1B, 1C, 1D, 1E, 4A, 4B, and those disclosed and illustrated herein. As shown with respect to other figures.

[0035] Of particular interest is image data acquired relating to body lumens, such as coronary arteries. Considering the benefits of intravascular imaging for diagnostics, flow measurement, stent planning, etc., the MLS system Obtaining timely image analysis and tissue typing and typing using the system is of great value. These systems often require significant amounts of computation time to perform image processing tasks. Therefore, the MLS system is used to process the patient image data sets. Reducing the amount of time required to process a data set is one feature of various embodiments of the present disclosure.

[0036] In general, the present disclosure relates to a method for collecting image data, etc., from a blood vessel using an intravascular data collection device. Any intravascular data that can be used to generate and receive signals containing diagnostic information. These devices can be applied to optical or ultrasonic probes. imaging devices such as probes, pressure sensor devices, and other components of the blood vessels or cardiovascular system. This may include, but is not limited to, other devices suitable for collecting data relating to It is not something that can be done.

[0037] Figure 1 shows the results of annotating the ground truth mask to generate it. collecting image data for classification using the neural network; and To carry out various steps of the present disclosure, such as displaying such prediction results to a user, Various user interfaces for viewing and evaluating visual representations of arterial information are provided. These user interfaces are described herein as It can be controlled using a mouse, joystick, or other control device and has one One or more mobile devices that can be operated using the above processors and memory storage elements. It can contain elements.

[0038] During the stent delivery planning procedure, users can refer to the OCT and annotated angiograms. The stent may then be further expanded or moved as part of the delivery plan. The features and methods of the present invention can be implemented using a system 3 shown in FIG.

[0039] FIG. 1 illustrates a method for collecting data or detecting a characteristic or condition of a subject 4. Various data acquisition subsystems suitable for detecting or otherwise diagnosing subject 4. In one embodiment, the subject is seated on a table, a bed, 19 or a suitable support such as a chair or other suitable support. The subject 4 is a human or another animal having a particular region of interest 25 .

[0040] The data acquisition system 3 may be a nuclear magnetic resonance, x-ray, computer-aided tomography, or other suitable Non-invasive imaging systems, including non-invasive imaging techniques, are also included. As a non-limiting example, angiography suitable for producing cines may be used. An angiography system 20 is shown. The angiography system 20 may include a fluoroscopy system. The angiography system 20 may include one or more imaging techniques, such as OCT or IVUS. The probe is connected to the subject 4 so that blood vessels in the region 25 of the subject 4 are imaged using an angiogram in 30, typically in the form of a frame of image data while the pullback procedure is being performed. The subject 4 is non-invasively imaged such that frames of angiographic data are generated in the It is composed of.

[0041] The angiography system 20 is in communication with an angiography data storage and image management system 22. The angiography data storage and image management system 22 is, in one embodiment, a workstation. In one embodiment, the collected Data processing related to the angiographic signal is performed directly on the detector of the angiography system 20. Images from the system 20 are stored and managed by an angiography data storage and image manager 22. and managed.

[0042] In one embodiment, the system server 50 or workstation 85 is connected to the system 22 In one embodiment, the entire system 20 is configured to handle the functions of detecting electromagnetic radiation, such as x-rays. The system 20 also receives such radiation after it has passed through the subject 4. The data processing system 22 then uses the signals from the angiography system 20 to , one or more regions of subject 4 are imaged, including region 25 .

[0043] As shown in this particular example, the region of interest 25 may be a region of the vascular system, such as a particular blood vessel, or a region of the peripheral vascular system. This subset can be imaged using OCT. A thermocouple-based data collection probe 30 is introduced into the subject 4 to acquire specific blood vessels, such as the coronary arteries. The probe 30 is placed in the lumen of a blood vessel. a probe that combines two or more of the above features; and a vascular The probes may be of various types, including other probes suitable for imaging within the The probe 30 typically includes a probe tip, one or more radiopaque The probe tip includes a marker, an optical fiber, and a torque wire. directors, acoustic beam directors, pressure detector sensors, other transducers or detectors, and the above The system may include one or more data acquisition subsystems, such as a combination of

[0044] In the case of a probe including an optical beam director, the optical fiber 33 is connected to the The torque wire is in optical communication with the probe through a hole in which the optical fiber is disposed. In FIG. 1, the optical fiber 33 is shown without the torque wire surrounding it. In addition, the probe 30 may also include a polymer sheath (not shown) or the like that forms part of a catheter. In the context of an OCT system, it is part of the sample arm of the interferometer. The optical fiber 33 is connected to a patient interface unit (PIU) as shown in the figure. The optical fiber is optically coupled to the optical interface unit 35.

[0045] The patient interface unit 35 receives the end of the probe 30 and is optically coupled thereto. Typically, the data collection probe 30 is used The PIU35 is suitable for use based on the type of data collection probe used. For example, the assembly of OCT and IVUS data collection probes. The alignment requires OCT and IVUS PIU. PIU35 typically uses torque The wire, sheath, and optical fiber 33 disposed therein may be removed as part of a pullback procedure. In addition to being pulled back, the probe tip also includes a motor adapted to pull back. Typically, the blood vessel of the subject 4 is rotated by the PIU 35. The probe 30 can also image coronary flow reserve. Measure specific parameters such as fractional flow reserve (FFR) or other pressure measurements. can be used to determine

[0046] The PIU 35 is then connected to one or more intravascular data acquisition systems 40. The data acquisition system 40 may be an OCT system, an IVUS system, another imaging system, and combinations of the above. For example, probe 30 may be an OCT probe. The system 40 in this context includes a sample arm of an interferometer, a reference arm of an interferometer, and a nce arm), a photodiode, a control system, and a patient interface unit. Similarly, as another example, in the context of an IVUS system, intravascular data The acquisition system 40 includes ultrasonic signal generating and processing circuitry, noise filters, a rotatable jaw, In one embodiment, the device may include an input, a motor, and an interface unit. The data acquisition system 40 and the angiography system 20 are connected to an angiography video frame time. and a shared client configured to synchronize the stamp and the OCT image frame timestamp. clock or other timing signal.

[0047] In addition to the invasive and non-invasive image data collection systems and devices of FIG. Various other types of data may be collected regarding area 25 and other parameters of interest for the subject. For example, the data collection probe 30 may include one or more pressure sensors, such as a pressure wire. The pressure wire can be used without the addition of OCT or ultrasound components. The pressure readings can be taken along a segment of a blood vessel within region 25 of subject 4. You can obtain it by doing the following.

[0048] Such readings may be relayed by a wired connection or via a wireless connection. As shown in the fractional flow reserve FFR data collection system, the wireless transceiver 48 Receives pressure readings from the probe 30 and transmits them to the system to The device is configured to generate FFR measurements at one or more locations along a blood vessel. The upper displays 82, 83 also show angiogram frames of data, OCT frames, and User interfaces for T and angiographic data, as well as other controls and features of interest It can be used to indicate.

[0049] An intravascular image, such as a frame of intravascular data, generated using the data collection probe 30 The data is sent via the PIU 35 to a data acquisition and processing system 40 coupled to the probe. Non-invasive image data generated using the angiography system 20 can be registered in a joint One or more servers or workstations, such as the recording server 50 and the workstation 85 The angiogram may be transmitted from the system 22 to the computer, stored therein, and processed thereby. A video frame grabber device, such as a computer board, configured to capture image data. The vice 55 can be used in a variety of embodiments.

[0050] In one embodiment, the server 50 is stored in a memory 70 and operated by a processor 80. The server includes one or more co-registration software modules 67 executed by the A trained neural network suitable for implementing the various embodiments of the present invention. 2. In one embodiment, A, such as a graphical processing unit 53, The I processor is included in the server 50 and is in electrical communication with the memory 70. The computing device / server 50 is a processor-based computing server Alternatively, one or more databases 90, etc. The database includes the generated image data, the subject's parameters, and the system shown in FIG. generated by one or more of the devices or components and received by the database 90; The system may be configured to receive other information that is forwarded to database 90. do.

[0051] The database 90 is stored in the memory of the workstation 85 while the server 50, this is just one example configuration. The software module 67 may be executed on a processor of the workstation 85. The database 90 may be located in the memory of the server 50. The use of a device or system to execute a software module is provided as an example. The hardware and software described herein in various combinations may be used in any combination. The software acquires frames of image data, processes such image data, and Various software modules can be used to register various image data. one or more regions of interest (ROIs) and / or detected or characterized tissue or The method generates one or more tissue maps showing arterial material such as calcium. The system may also include tissue map generation software suitable for performing the above steps.

[0052] Unless otherwise specified herein, software module 67 may be referred to as pre-processing software. Other software processes the image data, transforms, matrices, and responds to patient triggers. The database component 67 facilitates co-registration of different types of image data or is also used to perform annotation of image data to generate ground truth. Software, such as other software-based components, as well as various embodiments of the present disclosure. The present invention may include other software, modules, and functions suitable for implementing the present invention. These modules can be used to measure the luminal area using scanline-based or image-based techniques. Detection, stent detection using scanline-based or image-based techniques, Indicator generation, juxtaposition bar generation for stent planning, discrepancies, side branches and missing data Guidewire shadow indicators to prevent confusion due to may include.

[0053] The database 90 is generated by the angiography system 20 and stored in the server 50. Receive and store angiographic image data 92, such as image data acquired by grabber 55. The database 90 may be configured to include the images generated by the OCT system 40. The OCT image data, such as image data acquired by the frame grabber 55 of the server 50, The device may be configured to receive and store data 95.

[0054] In addition, the subject 4 may be connected to one or more monitors, such as monitor 49, via one or more electrodes. The monitor 49 may be electrically coupled to a cardiac monitor, such as, but not limited to, an electrocardiogram configured to generate data indicative of various states of the subject, such as systole and diastole; A graphics monitor may be included.

[0055] The use of directional arrows, or the lack thereof, in a given diagram indicates how information can flow. No limitation or requirement of orientation is intended. For example, the elements shown in FIG. For a given connector, such as an arrow or line shown, the information may vary for a given embodiment. The light may flow in more than one direction or in only one direction, as appropriate. Any suitable data transmission connection, such as a wired connection, a power connection, a wireless connection, or an electrical connection, may be used. It can include.

[0056] One or more software modules may be implemented in an angiography system, such as system 20 shown in FIG. The frame of angiographic data received from the You may not use the Software, any of its components, or any software-based or program-based Various software modules may include one or more steps of the processor-executed method. The rules may be used in any given embodiment of the present disclosure.

[0057] In part, the present disclosure relates to a processor-based method for detecting intravascular data collected by an intravascular probe. Intravascular data acquisition systems and related systems that can be converted or analyzed by the The results of such analysis and transformation are used to generate nuclei for classifying components of medical images. The end user can then use various representations of the display to communicate with a given MLS with a global network. In one embodiment, the OCT, IVUS, and X-ray images can be displayed to the end user. A given imaging system, such as an imaging system at a base station, is in electronic communication with the MLS and receives a given type of imaging signal. The modified version of image data acquired using the imaging system is It can be displayed during the same session in which it was acquired. V-net, U-net, CU MedVision1, CUMedVision2, VGGNet, multi-stage multi-recursive input Fully convolutional network (M 2 FCN:Multi-stage Multi-recursive-input Fully Co nvolutional Network, Coarse-to-Fine Stacked Fully convolutional nets, deep active learning frameworks, ResNet, and Combinations of these, as well as other neural networks and Various neural network architectures, such as software-based machine learning frameworks, The architecture can be used for image segmentation.

[0058] In one embodiment, the MLS provides results based on rapid processing, such as within about 2 seconds to about 30 seconds. The necessary machine learning operations and the training process are handled to obtain In one embodiment, the results are approximately 45 A dedicated hardware system for a given MLS embodiment can be used to perform AI / ML The machine learning system may include multiple processors, such as a processor for detecting an image. Constituent tissues, tissue types, and other regions of interest are detected and classified based on type or another parameter. The image may be segmented or processed so that it can be characterized according to the This can be implemented by training a classifier on the The lumen, intima, media and plaque were detected, and the boundaries corresponding to these different tissues were identified. Training a given MLS / neural network This involves teaching the network with known inputs and known outputs.

[0059] The present disclosure provides one or more memory devices including a memory with an increased amount of memory allocated per processor. The present invention relates to an advanced machine learning system including the AI ​​processor. , is designed to support multi-channel segmentation techniques. Various channels can be selected for different areas of interest and characteristics. For example, In one embodiment, the first channel, the second channel, the third channel and the fourth channel channels, one each in the lumen, intima, media, and plaque. Other classes / types can be associated with different channels to allow security. It can also facilitate fragmentation.

[0060] In one embodiment, the plaque type is classified. In addition, the plaque type can be classified as calcified. The presence of plaque and other detectable features in the vascular section indicates the presence of a constriction from a stenosis or the like. Given that a single plaque or stenosis may be present, another feature of the present disclosure is that the single plaque or stenosis may be present. To obtain the above scores quickly and automatically to facilitate the end user's judgment. For example, image data and its machine learning-based analysis can help The given score determined using the method determines whether emergency action is not recommended or not, or whether the stenosis is Decide whether a stent should be placed or whether other procedures such as atherectomy or bypass are warranted. This can help determine whether a product will be approved or not.

[0061] In healthy patients, arteries appear to be arranged in a coherent structure that includes the tunica intima, tunica media, and tunica adventitia. As a result of the atherosclerotic process, the intima undergoes pathological thickening. It also contains fiber, proteoglycans, lipids and calcium, as well as macrophages and other inflammatory cells. These tissue types may contain plaques composed of different types of tissues, including diseased cells. A pool is a set of training data for one or more of the machine learning systems of the present disclosure. When imaged using a variety of imaging systems that can be used to establish Plaques, which are considered to be the most pathologically important, have the following characteristics: The so-called vulnerable plaque consists of a fibrous cap with a layer of plaque. Plaques have different geometries. For example, foam cells are usually composed of large lipid pools. The shoulders form ribbon-like features, the media appears as a ring around the vessel, etc. Currently, it is used for qualitative evaluation of OCT images. The neural net was designed to identify fibrous capsules and / or fibrous capsules with underlying lipid pools. In various embodiments, calcium is referred to herein as Also includes, but is not limited to, calcified plaque and other calcium-containing tissues.

[0062] The patient is still catheterized and ready to receive a stent or other treatment option. While the patient is in the ventricle, the imaging procedure is quickly performed to obtain images of the arteries, and the machine Being able to process images using a learning system can save a lot of time and Improved patient outcomes are achieved.

[0063] The outer edge of the tunica media, called the external elastic lamina, determines the size of the stent during the intervention. Used by physicians to locate the tunica media and measure the diameter of partially diseased tissue This is time-consuming and difficult to do, and requires training in image interpretation. Automated detection and measurement of EEL diameter allows for the determination of treatment options at the time of diagnosis or otherwise. This paper addresses these technical challenges faced when evaluating patients with An example of the determination is shown in FIG. 6B. FIG. 3C shows a Cartesian view of an artery. 1 shows a set of labeled masks and ROI masks for the 3D image.

[0064] The ROIs are generally shown as examples, and may be either calcium or side branch or stenotic. The region containing the lumen L, intima I, and proximal end struts may correspond to other features of interest. Each region or feature corresponding to Lark Q, adventitia ADV, imaging probe P, media M, etc. is represented by CNN. The NN trained in Fig. 3C can be used to generate the NN by MLS. Image 365 is a diagnostic tool that helps guide and inform clinicians. In various embodiments, the detection of the various features in FIG. , and the angiogram, as shown in Figure 5F, is displayed using one or more user interfaces. In some embodiments, one or more features can be co-registered with the data. The region, type, and class prediction results are co-registered with the angiography data. The authors report that calcium plaques detected in OCT are referred to angiography for review and planning. In some embodiments, the tissue image is mapped / co-registered to the shadow image. Maps and other image data representations are used to support diagnosis and treatment recommendations. In one embodiment, each of the identified features, such as ADV, EEL, IEL, L, P, I, Q, etc. The features or classes are the trained neural networks disclosed herein. One or more of them can be used to generate a mask or a prediction mask. The indicia corresponding to the results are color-coded and other indicia are used. It can be shown that

[0065] In one embodiment, an annotated mask region corresponding to a set or group of pixels The domain may be used to train one or more neural networks as disclosed herein. Define the ground truth mask to be used. Once the image is generated, a set of pixels corresponding to the regions of user data and the lumen, calcium, A feature or characteristic of a region, such as the EEL, or another class or characteristic disclosed herein. A prediction or detection mask is generated that includes the class identifier. A prediction is generated for each class, and then the OCT, IVUS, or other image data is All of the predictions for a given image data input, such as an input frame of Then, a final prediction result for all classes is generated. In one embodiment, the prediction result is Output with regions corresponding to particular classes, indicated as such by indicia such as Summarizing the image mask and which indicia map to which classes Displayed as one or more legends.

[0066] Plaque detection and plaque type classification allow the physician to choose his or her intervention strategy. Doctors recommend atherectomy if the calcium burden is too high. Alternatively, stents can be landed in different zones depending on the underlying plaque composition. Automating the process of plaque identification and characterization This makes image interpretation easier for physicians and improves physician workflow.

[0067] The MLS disclosed herein can be implemented using a variety of neural networks. It can be integrated with a variety of imaging diagnostic systems. The system and method are implemented using a deep learning framework. In an embodiment, the MLS is implemented using neural networks, rule engines, fuzzy logic systems, etc. stem, comparator, flattening module, shift module, e.g. heuristic system Image processing modules such as size change modules for image processing systems, pattern matching systems, and a pattern recognition system, including one or more of these software implementations.

[0068] In one embodiment, the MLS uses a convolutional neural network (CNN). Using neural networks (NN) such as holistic neural networks CNNs contain multiple nodes or neurons, and they use image data, Any changes to the submitted data and / or image data and / or image data components In one embodiment, the system may receive and output classification information about the segments. In one embodiment, the CNN performs semantic image segmentation. In the above, semantic segmentation using a given MLS embodiment is performed on an image Detect whether the pixel has calcium and EEL, and detect whether the pixel has calcium and EEL. This can be used to assist physicians in selecting and identifying treatment options. In one embodiment, the present invention helps to solve various problems related to guiding certain procedures. MLS uses 3D CNNs such as PyTorch 3D CNN or V-NET. The neural networks described herein are implemented using PyTorch, Microsoft tCognitive Toolkit (CNTK), TensorRT, Tenso rFlow, and other similar proprietary and open source ML frameworks and tools The implementation can be implemented using a variety of frameworks, including

[0069] In one embodiment, the tool selected for data annotation is a tool that allows the user to select pixels. Such an exemplary user interface allows the user to select, move, and / or draw boundaries. The interface tool 305 is shown in Figures 3A and 3B. Support consistency of annotations across systems (especially in 3D), GUI-based annotation tools, etc. If standardized tools are used, the constraints of the tools and the actions to annotate them The set of supports consistent training data, which is a set of images that are generated by different users. This is in contrast to manually annotating the image and having it scanned or processed. In this embodiment, a useful feature of the GUI is the ability to transfer data from a previous frame to a subsequent frame. Furthermore, in one embodiment, the points are at the poles. The vertices are annotated with the vertices, eliminating the Cartesian-to-polar transformation during preprocessing.

[0070] In one embodiment, the input to the MLS is approximately 450 expert annotated images. The training set consists of one or more extracts for each tissue type in each image pixel. In one embodiment, the training center The kit was used to perform lumen detection before using the image data as a training set. , can be preprocessed by the training set of another MLS. In one embodiment, the first set of training data is preprocessed using one or more preprocessing techniques. These preprocessing techniques use the annotated lumen region or segment Includes lumen detection using MLS pre-trained using a training set. Preprocessing techniques can include training the network and / or backpropagation. To speed up the prediction speed of the trained network during ward propagation. Therefore, the pre-processing techniques include image data flattening, circular shift ( circular shift process, below noise floor Excluding portions of the image data, such as depth data, where every other scanline of the image is removed or data removal, which can be done every other time so that every other column is removed. Pixel filtering and other pre-processing can be used to remove defects and improve the uniformity of the region. A processing step may also be included.

[0071] In one embodiment, the training set is an individual image or a group of images, Approximately 400 to 600 clusters, or subsets, scan lines, or pixels, etc. In one embodiment, the training data includes a training set of image data elements ("image data elements"). The imaging set includes about 500 to about 700 images or image data elements. In one embodiment, the training set is about 700 to about 800 images or image data elements. In one embodiment, the training set comprises about 800 to about 900 images. In one embodiment, the training set comprises about 90 In one embodiment, the tray contains from 0 to about 1000 images or image data elements. A survey set contains about 1000 to about 1100 images or image data elements. In an embodiment, the training set comprises about 1000 to about 5000 images or Contains data elements.

[0072] In one embodiment, a given CNN used with a particular MLS may include multiple nodes. Such a network may contain various layers, including hidden layers. The network elements are trained to detect OCT, IV, It learns about input image data, such as US, angiograms, CT scans, or other image data. The weights, filters, or biases may change over time as the In one embodiment, the network used in a given MLS implementation is over time in response to a training set including data inputs and / or Cartesian image data inputs. The filter includes a number of filters that vary dynamically.

[0073] In one embodiment, the system includes one or more graphical user interface components. A CNN interface that can be used to facilitate batch input of image data. The given image is an example of image data. The image data may include the intravascular pullback and the Scanlines from other sources of medical imaging data suitable for segmentation and classification In one embodiment, the ground truth image data may also include: Each image that makes up the image is classified into different parts, classes or types, and such data is used to identify such parts, classes or Supports training the network to detect types. In an embodiment, the ground truth and training sets are the same as those used in the MLS run. In various embodiments, the method automatically predicts regions or features of interest without human intervention while To detect or classify regions of interest, a trained MLS is used to classify and generate is the mask generated for the features of interest. Training set / ground truth Humans or machine learning can be used to generate the data. Each data channel may be implemented in various implementations. The state of the

[0074] As shown in FIG. 1A, a given set of medical image data may be segmented into tissues and other data types. 1 shows a schematic diagram of a system 5 for performing the training phase of a MLS suitable for classifying Various features, regions, types, and / or classes of tissue within an image, as well as regions, pictorial Cells, contours and boundaries are tagged or identified in the image data to generate ground truth Image data with annotations such as masks can be obtained. The team processes the image data and uses a trained network to generate the tissue Classify and analyze features, regions, types, and / or classes, as well as regions, pixels, contours, and boundaries. The present invention can generate an output of image data having indicia suitable for display and visualization. It can be used to train a neural network, such as A ground truth match is a region that corresponds to a particular channel or feature, or may contain a group of regions corresponding to a particular channel or feature. A region comprises a set of pixels in a given ground truth mask. In this state, the ground truth mask includes calcium, plaque, EEL boundary, etc. The specific types or classes, as well as various other classes, types, and It can be annotated to include regions or features that correspond to the classification.

[0075] In one embodiment, the image data, the annotated image data, and / or the ground truth The concentration of the image data set is calculated by performing one or more transformations on the false mask. These transformations include circular shift, left flip-flop, Left flip, right flip, flatten, resize, crop, flip The image data may include one or more of filtering, binarization, and normalization. OCT data, IVUS data, or other data acquired using one or more rotational elements When the data is amplified, the data augmentation must avoid transformations that are inconsistent with the imaging being performed. This is carried out as:

[0076] In one embodiment, the MLS includes one or more inputs for receiving image data; It includes a CNN that generates output based on MLS, which processes image data. In an embodiment, the MLS and / or CNN comprises a deep learning classifier. In an embodiment, for example, about 400 OCT images, IVUS images, or angiogram images, Approximately 2000 image data elements are annotated by experts or The MLS is provided with image data. This image data is organized as image data elements and , obtained from one or more experts and / or a first MLS for image pre-processing such as lumen detection. and ground truth annotations for such data elements. In this embodiment, each image data element pixel (or a subset thereof) is a pixel of the first MLS. Annotated by Kispert.

[0077] Therefore, an image data element and a ground truth annotation for this image data element are The image data comprising the training set is provided as input to the MLS. In an embodiment, the training set is generated using CNN and / or deep learning. As shown in Figure 1A, the image data / image data elements and The inputs of the MLS are the ground truth annotations and the ground truth annotations. Image data elements combined with truth annotations and / or ground truth annotations; In one embodiment, the error value is the output of the cost function. In one embodiment, the error value is a training error value. ,scores,and tests are used to measure the change in training error over time to an acceptable threshold,and Convergence can be evaluated. In various embodiments, Dice, Jacquard, (Jaccard), cosine similarity, and overlap distance, metrics, scores, or coefficients can be an output, an input, or a component of the cost function. The details are described herein. Other systems use various pre-processing steps to further improve training and / or processing times. may include.

[0078] FIG. 1F is suitable for use with various regions / features of interest such as lumen, calcium, media, and intima. In accordance with an exemplary method 80 of the present disclosure, which includes a training process and a subsequent prediction process, In one embodiment, the neural network being trained is a The system was trained using ground truth annotations including tunica media, intima, lumen, and media. In one embodiment, the neural network that is trained is a calculus. Ground truth annotations consisting of tunica intima, lumen, and media, or any of these It is trained using ground truth annotations that specifically target one or more The method includes acquiring image data, which may include polar image data, such as endovascular data. First, a ground truth mask or other ground truth data is generated (step 100). A set of data is generated by reviewing and annotating a set of image data. In one embodiment, the method further comprises: One or more regions of interest or related regions in each polar image of the set of images are identified as truth annotations. The method includes annotating the cardiac features (step 102) corresponding to the ground truth annotations. The pixels can be stored in a persistent electronic memory, such as the database of FIG. All of the data and annotations described in this specification are based on the neural network of a given imaging system. network and image processing software modules, and It can be stored in a data structure.

[0079] In one embodiment, the ground truth annotations are as described herein with respect to FIG. 3A etc. This can be done using a user interface which will be shown and discussed in more detail. In an embodiment, the annotations are stored as a ground truth mask, and regions or features are of a region or feature of interest identified by a user along with the boundaries or pixels that make up the region or feature of interest. In one embodiment, the mask may include a set of an original image, such as a polar image, and In one embodiment, the mask includes a specific Contains the address / list of pixels corresponding to the ground truth annotations or prediction results of .

[0080] Train the neural network of a machine learning system using a set of annotated polar images Then, a training is performed (step 104). The ground truth masks and the annotated images are used to obtain these ground truth masks. The ground truth mask can be generated by using the image data. Applies to circular flip, right flip, left flip, and rotational imaging modalities This disclosure can be extended using other possible extension transformations. The ground truth mask has been modified to increase the number of ground truth masks. In one embodiment, the ground truth mask is Lumen detection, including luminal boundaries or luminal features or luminal regions, is performed as an initial detection step. In one embodiment, training is performed using a cost function or other measure of error. This is performed until one or more metrics such as the error are reduced to an acceptable level. As the number of MLS samples increases, the detection / prediction accuracy of MLS improves.

[0081] Once the MLS is trained, image data is fed into the neural network. A set of image data is generated using prediction, detection, classification, etc. of various features / regions of interest. (Step 105). K probability masks are generated for each of the K classes / types. A probability map for three different classes, K being 3, is created (step 106). An example of a filter output / probability mask is shown in Figure 1G. Specifically, Figure 1G shows the filter output / probability mask from a patient's imaging. The following shows the image data obtained from the experiment and the three output probability maps / probability masks for different classes. The output mask / map is shown as separate channels given the exemplary embodiment of the present disclosure. These correspond to the detection of the media (M), calcium (Ca), and lumen (L) of the entire lumina, as shown in the figure. For each channel, the detected classes are shown in red. The media M is red in the media channel and the two detected calcium regions Ca , calcium channels are red, and luminal L is red in the luminal channel. Other indicia can be used to detect / predict different areas / features of interest. The classes can be color coded or represented. In one embodiment, each class of feature of interest is The type or sequence of the predictions that the network makes about the classification of each part of the image it was trained on. Assign the mask, map, and associated channel to track the final output. do.

[0082] Calcium, media, lumen, and intima are the features / regions that the neural network classifies. In one implementation, K is 4 if the class / type is trained to be In this state, the output of the neural network is one probability map per class / type. The method includes: A final predicted output may be generated (step 107). For a given K classes / types, each of the K probability maps is Each pixel in the image frame is compared and evaluated pixel by pixel, and The final prediction result is then assigned to each pixel. The prediction results are processed to generate a final prediction output. A predicted output mask that contains one or more indicia corresponding to a type / class. In an embodiment, the method comprises: The method includes displaying the force image along with the class / type indicia (step 108).

[0083] In one embodiment, various indicia are trained using MLS. The segmentation between the different regions and features of interest identified by the In one embodiment, the neural network / Modified to include features / regions of interest identified / predicted using machine learning systems The output image data is displayed. The above steps can be implemented using the computer-based method disclosed herein. and AI processor-based systems. It is possible.

[0084] Various neural network architectures may be used in conjunction with the embodiments disclosed herein. For example, V-net, U-net, CUMedVision1 , CUMedVision2, VGGNet, multi-stage multi-recurrent input fully convolutional network ark(M 2 FCN), coarse-to-fine stacked fully convolutional net, deep ac Interactive learning frameworks, ResNet, their combinations, and other neural networks The neural networks and software-based machine learning frameworks described herein are Suitable for performing feature / region of interest image segmentation and classification as shown. Figure 2 shows the results of the analysis using the annotated ground truth mask. The system performs a 3D scan of the regions of interest / features, generating probability maps suitable for evaluating prediction output. A convolutional neural network model suitable for displaying classified image data with features The multi-layer neural network architecture 115 in FIG. , which is suitable for training using ground truth annotations and generating prediction results. The neural network architecture in Figure 2 is a set of various types / classes (calculations). The numerous groups of membranes, lumen, media, intima, and others disclosed herein. The network can be trained using ground truth masks from a The network architecture 115 is a network architecture disclosed herein or a modified version thereof. In one embodiment, Thus, network 115 is a convolutional network.

[0085] Neural network 115 includes input 111 and output 117. Output 117 is When K classes are specified to classify the features and regions of interest, each image The K probability maps are the K probability maps of the input data. For each frame of the data 111, a comparison is made to determine which pixels are classified as being in one of the K classes. A scoring system or method used to determine which has a higher relative probability than are evaluated using a weighting system. The evaluation results show that the various K classes and related Final prediction output showing the features and regions that were selected, along with indicia and legends distinguishing the classes. is generated.

[0086] Various nodes (OPs) are operations and transformations that can be applied to the channels of image data. The network 115 includes a set of associated nodes N1 through N9. In various embodiments of network 115, the number of channels T is typically 16 or more. In order to efficiently speed up the processing of the network 115, applicants set T to 4, so that T= 4. Training the network to classify and evaluate arterial image data Therefore, in various embodiments, the T channels are The channel supports four channels, and can be set to 2T channels, 4T channels, or 8T channels. 16T channels are available in 8, 16, and 32 channel configurations. These various channels correspond to the number of instances of the input data. How should the service be handled with respect to different functional OPs in different nodes of the network? It is set at each layer / hidden layer of the network to specify The network 115 uses functions / operators such as adaptive learning algorithms to generate the graph. In one embodiment, the training is performed using gradient descent. The based method trains the network115 with annotated ground truth masks. Used for training.

[0087] The various nodes of the network 115 are OP1, OP2, OP3, OP4, OP5, The operations and transformations of nine nodes N1 to N9 are shown by OP6, OP7, OP8, and OP9. As shown by network elements 128a and 128b, Other nodes and layers can be added between nodes N2, N3 and nodes N7, N8. A network including nodes N1, N2, N3, and N4 and an incoming arrow from node N4 to node N5. The left side of circuit 115 performs one or more downsampling / downconversion operations DC. In contrast, the network including the outgoing arrow from node N5 to N6 and the nodes N6 to N9 is To the right of network 115, this part of the network has one or more upsampling In one embodiment, this new The neural network architecture combines an encoder that captures context information and an accurate and a symmetric decoder path to enable localization.

[0088] The left part of the network 115 consists of multiple convolution blocks and downsampling operations. The downsampling is used to encode the input image into a feature representation at different levels of The right part of the network is the upsampling convolution operation. and concatenation operations. The right part of the network outputs an image with the same dimensions as the input image. In one embodiment, the network architecture operates to There are four instances of sampling or upsampling operations, and therefore ,The input image dimensions need to be divided by 16 to avoid dimensional mismatch. Herein, the image data has pixel dimensions of M×N.

[0089] In one embodiment, the M×N dimensions are 912×512, but other specified dimensions may be used. In one embodiment, images other than the specified M×N size can be used. is padded or cropped to match the specified dimensions. In this embodiment, the OP operations and functions OP1 to OP9 executed at each node are convolutions. , deconvolution, additive process, concatenation process, upconversion process, downconversion process A process selected from the group consisting of a softmax process, a PReLu process, and a In one embodiment, the Softmax process and the parametric rectification linear unicast The Parametric Rectified Linear Unit (PReLu) process extracts the input image data. The network parameters transmitted between these nodes are then added to the output probability map. In one embodiment, the layer is implemented between one or more nodes as a layer that converts The output of the neural network can be constructed using the Softmax function. It includes a probability assignment layer for each pixel, each scanline, or each time. Each mask region is based on a set of trained classes, also called classes or labels. The probabilities according to different groupings such as , , etc. are expressed as multiple outputs 117 for each of the K classes. In one embodiment, each node can operate as a layer. ,A layer can also be defined by two horizontal nodes and an arrow between ,the nodes. In one embodiment, arrows 130a-130d are drawn from the left to the right of the network. The transfer between nodes, denoted by It corresponds to combining or concatenating two or more channels.

[0090] In one embodiment, the network architecture of FIG. Three or more types / classes of pixel labels, such as lumen, as well as corresponding features of interest imaged The method is configured to characterize other classes and types of features and regions of interest. In the above, the network 115 may allocate all of the T channels shown, where T is 4. The network processing speed is about 400 to 600 frames of image data. Reduce channel count so that imaging and other Given the goal of reducing waiting time in the cath lab between procedures, Complex neural networks, such as networks with large T, can predict outcomes / probabilities Although the inference time is longer when generating a new model, there are other issues in the training process such as overfitting. In various embodiments, T is set to 4 in the network 115 of FIG. , the result is a reduction in inference time and improved network accuracy. In one embodiment, where T is equal to 4, the channel of FIG. channels, 2T channels, 4T channels, 8T channels, and 16T channels The available options are 4, 8, 16, 32, and 64, corresponding to the channels.

[0091] In one embodiment, a given node or forwarding operation 130a-130d may Each horizontal layer, such as two nodes linked together, is divided into OP1 to OP9 and / or vertical DC operations between the nodes. These layers are convolutional layers that perform convolution operations such as one or more of the following: Some features of the input ground truth data are extracted or learned. For a convolutional layer with a scalar, we can set different strides for each node. The stride provides control over the downsizing of the image during the convolution operation. For example, if the image size is 256x256, the output size after convolution is 128x1. The deconvolution on the right side of the network is Upscaling is performed for every UC operation.

[0092] FIG. 2A illustrates various color-coded classes Ca (red), A set of outputs from the MLS, including indicia corresponding to the lumen (blue) and media (green). These are used to obtain axial projections, and the carpet view and optional FIG. 1 is a schematic diagram showing the optional generation of a tissue map.

[0093] In various embodiments, the neural network architecture is a 2D network. It can be a network or a 3D network, and therefore can process 2D and 3D data. In one embodiment, each frame is a 2D polar image. The 3D of the pullback of F frames is displayed using the carpet view representation. In various embodiments, a filter is applied to the 2D carpet view to remove noise or It is possible to remove other unnecessary features that would increase processing time compared to using the D operator. Further details regarding the use of the carpet view are provided with respect to FIG. 2A. .

[0094] After the detection process of the tunica media and calcium, each frame in the polar space contains the tunica media M and calcium These frames have corresponding frames / masks of the regions of interest / features of Calcium Ca. / Mask is used to train the neural network for lumen L and ROI / FOI detection. The four output image masks / frames can be used to A set of frames 190 is shown as an example. In various embodiments, The set contains about 400 to 600 frames. Blue for lumen, red for calcium. A color code indicia was used with green for the tunica media and green for the frame and The output image mask is shown in the figure. In one embodiment, the entire pullback can be covered when the tube is All of the frames of the media and calcium masks, including the cavity, are ORed on the x-axis or y-axis. The lines of the mask are then projected along the x, y, and z axes, i.e., Shown as four lines 193. Projection with green for tunica media and red for calcium. The color coding of the pixels in the lines can be seen. All lines are arranged in a single sequence based on the left-to-right proximal-to-distal frame order. This is combined with a binary media mask and a binary calcium mask.

[0095] This combination of line projections 193 is shown as the carpet view 195. Optionally, in some embodiments, a carpet view or line projection 193 is shown. The outermost ring of the organizational map is used to create an organizational map 198. corresponds to the proximal direction, while the innermost ring indicates the distal direction. The tissue map 198 is created by performing a polar transformation on the carpet view. The carpet view is essentially a 3D data set that contains all the frames from the pullback. In one embodiment, a binary morphological reconstruction function is included. A filter was applied to the media and calcium carpet view images 198 to remove noise and small In one embodiment, the predicted output frame from the neural network is Such a carpet view filter for frame / mask based carpet views. The ring step advantageously removes small detection areas in the carpet view image. while the large detection area remains unchanged.

[0096] In one embodiment, the processed carpet view image is subjected to a convolution process, It is applied to the media mask and calcium mask through a method process or a comparative process, It is possible to remove noise in 3D polar space. Operations on 2D carpet views By restricting to 2D filters, we can use 3D operators on the 3D data space. The processing time for the operations is improved. The various processes and operations shown in FIG. Computational methods such as pipeline and neural network-based feature / region detection and classification This can be implemented using database-based systems and software modules. In one embodiment, a trained neural network is used to estimate calcium. The method to detect ROI / FOI such as tunica media, lumen, and intima during OCT pullback To increase the processing rate of a set of simultaneously acquired image frames, Instead of using neural networks, we use an image processing pipeline to generate side-border images. The method includes performing one or more of: guidewire detection, guidewire detection, and stent detection.

[0097] In one embodiment, the carpet view is a 3D image of the intravascular pullback. D representation. The intravascular image data acquired during the pullback procedure using the probe , of the scan line by developing a cross-sectional view or a polar view along the longitudinal section. A representation can be created and displayed to the user. The carpet view is a 2D In one embodiment, the carpet view is a cross-sectional OCT image. However, it is not intended to be unfolded or deployed in the same manner as the rolled up cylinder of a car pit is unfolded. This shows what was done.

[0098] A carpet view is a representation of a group of polar image data or a sub-group of its components in one or more directions. For example, in one embodiment, the carpet The view reduces the radial offset dimensions of the 3D intravascular data set to a single intensity. The data can be expressed in (Z, θ) coordinates as follows. In-projection can be used to generate the carpet view. The method for reducing the radial offset dimension is to use the near offset estimate The next step is to sum the intensity value between the far offset estimate and the By summing, the number of specific regions in the carpet view / OCT image is calculated. These result in increased detectable contrast as well as improved resolution. 5E and 13A and 13B. Used to remove noise from the output prediction mask displayed using one or more panels It is possible.

[0099] In one embodiment, the carpet view or OCT data is A carpet view mask of the endovascular data can be used to generate a The display is displayed via a user interface such as that shown in FIG. 5E, FIG. 13A, and FIG. 13B. To facilitate filtering of noise and artifacts from the final prediction output, The binarization process performed on the carpet view image is advantageously In one embodiment, the carpet view mask is used to remove unwanted pixel fragments. The image is then used as input for a subsequent intravascular image processing stage where the signal intensity or other noise is removed. do.

[0100] In one embodiment, the carpet view is A 2D data set generated from the scan lines of the pullback where dimensions are removed or reduced. In the carpet view, the intensity value of the shadow area is low and the intensity value of the tissue is high. The pet view is typically a grayscale image, but in some embodiments, A color version can be displayed.

[0101] In one embodiment, the training and experimentation are performed using three In one embodiment, the training code is generated using one or more AI processors. The training was performed in Python using the PyTorch framework. Before feeding the images and masks into the neural network for training, all images are was normalized to the range (0.0 to 1.0). All images and masks were randomly ordered. The first part is the training model. 90% of the total dataset used. During the training session, images and masses were The algorithm uses vertical scaling of (-256, +256) to extend the training samples for each epoch. The second part compares the performance of the trained model with that of the In one embodiment, the cross-entropy loss was used to evaluate the is calculated over the model output using a Softmax function that operates pixel-by-pixel The training and testing results are shown in Figure 2B and 2C. As shown in Figure 2B, the actual cross-sectional area calculated from the prediction and ground truth is It refers to the entropy loss. The mismatch percentage is the difference between the accuracy of each image compared to the ground truth. The percentage of misclassified pixels in the image is shown in Fig. 2B and Fig. 2C. This shows that the training process converged well after 300 epochs, and the overfitting problem was eliminated. There wasn't.

[0102] As referred to herein, lumen detection is performed by detecting ground truth data image data and This is done on user image data for which features / regions of interest need to be detected / classified. In some embodiments, lumen detection involves scanning for discontinuities such as start and stop pairs. In another embodiment, lumen detection is performed by analyzing the The method is trained using annotated images such as ground truth masks with the lumen boundary defined. The model is implemented using a 2D neural network or a 3D neural network. FIG. 2E shows polar image data of an artery 230 with identified lumen L and tissue T. Figure 2F shows the polar image data annotated using the ground truth luminal data. 233, which has lumen L and tissue T corresponding to two different regions / channels of interest. Figure 2G shows the results of the luminal examination performed using MLS with 2D neural nets. FIG. 2H shows the output image data 235 resulting from the 3D neural network. 2 shows output image data 240 resulting from lumen detection performed using MLS with is doing.

[0103] 3A and 3B are diagrams illustrating a method for navigating frames of image data and adding annotations to the image data. A user interface for a system suitable for annotating and generating ground truth FIG. 3B shows that region 357 has a boundary with pixels far from the center of the region. By moving the boundary points 360a, 360b, and 360c, the ground truth is obtained point by point. A defined polar image 350 is shown. In one embodiment, various drawing editing tools are The raw image data can be annotated using the annotation tool. These annotated images are The class or feature of the annotated region is defined using the interface 305 of FIG. 3A. Once stored in memory, it can be used to generate a ground truth mask. In Figure 3A, 271 frames of the intravascular imaging pullback are available for annotation. Frame 146 provides ground truth data for training the MLS. The user-selected area for annotation in FIG. 3B is selected for annotation in the same manner as in FIG. It corresponds to the tunica media as indicated by the class identifier selected in A. Thus, Any feature / class can be selected for labeling, and this feature / class is the annotation. The annotations are stored in memory with the annotations. The annotations may include, but are not limited to, specifics such as lumen, media, calcium, etc. The present invention relates to a method for detecting a pixel boundary / region that defines a mask that is matched against a class or feature of the pixel boundary / region. Any of the disclosed classes, features, or regions may be included.

[0104] As an example, FIG. 2C shows a graph spanning a period (from 0 epoch to over 600 epochs). In Figure 2C, we show plots suitable for evaluating the change in training error. The x-axis shows the number of training epochs, and the y-axis shows the Dice coefficient, a measure of error. The blue and orange curves show the network performance on the training and testing data. This shows the performance of the network.

[0105] In one embodiment, the training data, such as ground truth data, is Image data elements that can include various image formats such as grayscale images of Both are provided as inputs to the MLS CNN. MLS reduces the training error. over the duration of the epoch until it is either In one embodiment, this period is about 100 epochs to about 1000 epochs. This is the range of the

[0106] In one embodiment, the training includes raw images and ground truth annotations. All data are in polar coordinates. Ground truth annotations are based on existing software for lumen detection. The lumen detection from this software can be extended by software algorithms. The output was combined with expert user annotations of media, plaque and plaque type. , a ground truth image is generated. In one embodiment, the input to the MLS is The ground truth image is the output, which is a classified image. When the true image is the same as the true true image, the error is approximately 0.

[0107] FIG. 1B illustrates a MLS and CN training process as described herein or otherwise. Train MLS15 using one or more of the training processes suitable for use with N. This shows the use of MLS15 after training. After the raw images are processed, image segmentation is performed to obtain images containing classified or characterized tissues. In one embodiment, the pullback of the intravascular imaging probe can be Polar images from the cloud are processed by the trained MLS to generate classified images. The system of FIG. 1B may be modified in various ways to further improve training and / or processing times. In one embodiment, the pre-processing step / action 16 may include In one embodiment, the normalization step includes the step of normalizing the image data, the graph, Truth annotations, masks, frames, scanlines, neural network output and one or more or all of the intensities of other intensity-based data, with intensities ranging from about 0 to about 1, etc. or another applicable range.

[0108] An example of an unclassified / uncharacterized OCT image in polar format corresponding to a cross-sectional image of a coronary artery is shown in the first image of Figure 2D. The second image of Figure 2D shows the polar image with the tunica media, Ca , and a ground truth mask showing the ROI / FOI of the lumen. The third image in 2D shows color-coded images with the media in green, calcium in red, and the lumen in blue. 1 is a polar OCT image with inference or prediction results shown in the CT scan. Acquired using optical coherence tomography images classified or characterized by MLS Examples of the user image data are shown in Figs. 3D and 3E. It is clear that both images are Cartesian images. In this study, MLS training and prediction are performed on polar images, and after classification, polar The figure is converted to a Cartesian diagram, and the annotations generated by the trained MLS are then color coded. or other suitable indicia or visualization on the image. As shown, the inner lumen area is shown as blue, and the plaque area is shown as a 9 o'clock position. The calcium is shown in red, and the medial region is the area extending past the midpoint between the top and bottom of the lumen. This is roughly indicated as the thin curved area on the right side of the image.

[0109] A given cost function is calculated based on the ground truth input / output while processing the patient data. Evaluate the output of a machine learning system by comparing the training set to the expected output. The goodness of fit between the training data and the output data is a function of cost. The output of the cost function is a value that corresponds to the error metric. This output can be used by a machine learning system or neural network or These other operational components generate the predicted results and Given the ground truth, how well does it perform in terms of accurate prediction? A comparison metric such as difference or distance to summarize. If the output of the cost function is 0, Therefore, the system is in perfect working order. Iterative modifications are used to reduce the cost or error function of the system and improve its predictive accuracy. It can be improved.

[0110] In addition, a pixel-wise cost function is added to measure the distance between the prediction and the ground truth, or Another suitable metric or score may be specified to measure. Backpropagation is used to iterate the network-based values ​​derived from the cost function. In one embodiment, the weights of the cost function are updated. The derivatives are used to update the weights during backpropagation. The process has the advantage that the actual prediction results are closer to the ground truth. This reduces or minimizes the error of each output neuron / node of the neural network. In one embodiment, the neural network is a convolutional neural network. The first is the Neural Network (CNN).

[0111] Figure 1C shows the results of training with the augmented dataset of annotated ground truth masks. A machine that includes a trained neural network 34, such as a trained network. FIG. 1 is a schematic diagram of a training system 30. As shown, the OCT polar image data in the illustrated example are Input image data such as image data is processed by a trained neural network 34. The input polar OCT image is processed and the media (green), calcium (red), and Color-coded indicia corresponding to different arterial features / regions such as the endothelium (color), the endothelium (blue), and the lumen (blue). The predicted value / prediction output is generated in the form of a polar image including the following: MLS Neural Network 3 4, the format of the annotated ground truth image masks used to train Examples of ground truth images are also shown. The output of the system is an arc-based metric of similarity for both Ca and EEL. / scale, detected EEL diameter, and detected Ca depth. In some embodiments, these values ​​include EEL, media, calcium, luminal, and other values. are measured on the image data after classifying the regions and features of interest.

[0112] Continuing to refer to Figure 1C, once the MLS is trained, backpropagation The program can run from time 0 to the backpropagation time BPT. Then, the predictions corresponding to the annotated images or masks are compared with raw patient images, such as OCT pullbacks. Input the image into the MLS and propagate the data through the network to generate a predicted output. In one embodiment, the BPT time is greater than 0 seconds and is about In one embodiment, the BPT time is less than about 180 seconds. In one embodiment, the BPT time is less than about 90 seconds. , the BPT time may, in various examples, be substantially real-time as disclosed herein. Various neural networks may be used to implement the systems and methods disclosed herein. These networks can be used with various hidden layers, functions, and The method may include a combination of operators and operable combinations for multiple channels. In one embodiment, a channel is a set of pixels from a given frame or It corresponds to input image data, such as a subset of pixels from a given frame.

[0113] FIG. 1D operates in conjunction with an image processing component of the imaging system to determine the image In one embodiment, the image processing is CT image processing. MLS uses neural networks for AI detection. This can be done using AIP / GPU. The processing time T1 is In one embodiment, the AIP is in the range of about 20 seconds to about 40 seconds. GeForce GPUs from the GeForce 8000 family or higher. AI detection is Cavity, calcium, EEL, and any of the ROIs / FOIs disclosed herein The system 65a of FIG. 1E can be configured to detect lumen detection through processing. The luminal AI detection is separated as a separate processing stage having a time T3. The system in Figure 1E is shown in Figure 1E. It also includes various pre-processing steps that reduce downstream processing time while the device is being prepared for the procedure. Tissue detection / prediction is performed before the pre-processing. Pre-processing may have processing time. In an embodiment, the pre-processing includes a data flattening step.

[0114] Finally, AI detection can be performed on lumen detection and pre-processed image data. The AI ​​detection process may take a period T4. In one embodiment, T3 is from about 1 second to about In one embodiment, T4 is in the range of about 5 seconds to about 12 seconds. In one embodiment, the total processing time T2 ranges from about 8 seconds to about 20 seconds. In an embodiment, T2 is from about 5 seconds to about 15 seconds.

[0115] In one embodiment, the MLS and the method steps and operations performed are Using an end-trained neural network to process OCT images The training data can include semantic image segmentation. Approximately 900 OCT images with regions of interest labeled with motion are included in polar space, and each interest The regions correspond to given ground truths. Each given ground truth is a A trained network is used to map the channels found in the image data. In one embodiment, various input images and their corresponding A channel-specific mask is used to train the network using one or more algorithms. In one embodiment, a stochastic gradient descent optimization algorithm is used. will be done.

[0116] In part, the present disclosure relates to tissues and the various constituent layers and materials (such as those of the exemplary tissue types or tissues described above). are tissue properties) and how they change over time. and an exemplary output Cartesian image of a patient's arteries classified by one or more associated methods is shown in FIG. 3D and shown in images B and D of Fig. 3E.

[0117] Figures 2E-H show polar images of an artery and the annotated images using ground truth luminal data. The results of lumen detection performed using MLS with 2D neural nets on the polar images obtained by the CT scan. The results of lumen detection performed using MLS with a 3D neural network are shown in Fig. 1. There are.

[0118] The various layers of the coronary artery are shown. An internal region P of the blood vessel L is shown in the lumen of the blood vessel L. In particular, the present disclosure relates to tissue typing for arteries. and detecting and identifying tissue types and regions and / or features of interest. Displaying the tissue characterization / classified arteries This prediction or inference from the MLS depicted in the image can be used by the end user to make diagnostic and treatment decisions. The methods and systems disclosed herein can be used to detect Some non-limiting examples of tissue types that can be used include internal areas where blood flows, lumens, The tunica intima, tunica media, external elastic lamina (EEL) (also called external elastic membrane), and internal elastic lamina (IEL) al elastic lamina), tunica adventitia, plaque, calcium or calcified tissue, etc. It is bounded by the IEL and the EEL. The inner membrane is bounded by the lumen and the IEL.

[0119] The present disclosure relates to one or more machine learning systems or artificial intelligence (AI) systems. e) Using the system to detect or image an artery or other structure. Various embodiments for segmenting the image into various constituent tissue types or regions of interest. In part, the machine learning system may be used in an intravascular imaging system, an ultrasound system, or In conjunction with an imaging system such as an angiography system or an X-ray system such as a fluoroscopy system In one embodiment, the device is designed to be mounted or combined with The present disclosure provides a method for detecting calcium ions in the lumen, EEL, media, and calcium ions using MLS. The present invention relates to characterizing tissue to detect one or more of the following: Further details regarding the vasculature and calcium plaque are shown in Figures 5G-5J.

[0120] In one embodiment, one or more imaging systems (OCT, IVUS, angiography, Image data, which can be raw image data from X-ray, fluoroscopy, etc. All truth annotations are in one or more coordinate systems. Ground truth, ROI, FO I, and the prediction results can each correspond to a channel, given a training image Or the patient image may have an associated image mask. Polar coordinates of OCT data, etc. Processing formula data is challenging for a variety of systems.

[0121] Such images were obtained immediately after image data was acquired from a patient who still had a catheter inserted. Given the real-time requirements associated with generating a classified set of data, There is a real need for timely processing of image data using MLS. Machine learning techniques and To enable the use of AI techniques, various pre-processing or MLS design options are implemented. This can facilitate data processing times in the range of about 1 second to about 20 seconds.

[0122] In general, the systems and methods disclosed herein allow a physician to determine whether or not to treat a given patient. Helps determine whether and, if so, which lesions / strictures should be treated In one embodiment, the system provides a variety of automated diagnostic tools for detecting physiology. It provides guidance on treating the most important lesions based on the plaque type. Using the details of the flow and other MLS detected features, the shortest possible flow rate is determined to provide maximum flow recovery. In addition, the stent can be selected based on tissue classification and other factors as disclosed herein. The stent is then fitted to a catheter that is adapted for placement in a known artery by measuring the catheter length. A virtual stent placement can be performed, providing interactive planning.

[0123] The systems and methods disclosed herein allow clinicians to perform a variety of stent landing zones. This provides guidance to allow the assessment of lesion landing zones. Supported by virtual stent placement with virtual flow reserve to aid in the selection of Co-registration with angiography and OCT data further supports this. By using MLS detection, normal frames (frames with healthy tissue) are more likely to be detected. Thus, as embodied herein, Such MLS solves the problem of guiding stent planning and helping to improve patient outcomes. .

[0124] Furthermore, automated EEL measurements obtained using the MLS described herein What stent size should be considered and what type of stent should be used? Calcium detection via MLS supports the choice of atherectomy over stenting. In addition, calcium detection provides information about pre-lesion treatment and treatment options, such as ,Provides input parameters when deciding between calcium detection to aid in the decision of BVS vs DES do.

[0125] 4A-4C are integrated or capable of being integrated with an imaging system, or One or more other devices that are or can be combined with the imaging system in another manner. 1 is an exemplary imaging diagnostic system including the above machine learning system. These can include OCT, IVUS, angiography, fluoroscopy, x-ray based imaging systems etc. do.

[0126] Referring to FIG. 4A, a data acquisition / imaging probe 407 and an intravascular diagnostic system / device Imaging system 410 including MLS system 442 suitable for use as a data acquisition system. The system 442 includes one or more AI processors, such as GPUs, in serial or parallel. P and an AI memory AIM. The system may include For use in implementing one or more of the AI / ML-based techniques, systems, and methods The system 410 may be a stent planning system that suggests stent placement options. It can be used as a system to simulate VFR, IFR, FFR, etc. based on flow measurements. Graphical user interface for localized or virtual stent planning Some exemplary graphical user interfaces for such a system are provided. The components of the VFR are shown in Figures 13A and 13B. In addition, the mask is shown to reduce calcium (C a / red), media (M / green) and lumen (L / blue). The simulated stent was also generated using a neural network. , is shown in a longitudinal section of the artery representation in the user interface shown in Figure 13B.

[0127] In various embodiments, the intravascular probe 407 may be used to perform x-ray imaging such as angiography. With the stem in place, the artery can be imaged using CT scans, MRIs, and X-ray-based Imaging systems, as well as other imaging systems, such as other 2D and 3D imaging systems The image data may be generated using and / or the image data may be stored in one or more media. The probe 407 in various embodiments may be stored in a memory storage device. Other imaging modalities may be included, such as OCT, intravascular ultrasound (IVUS), etc. The probe 407 is in optical communication with an intravascular diagnostic / data acquisition system 410. An OCT optical system or subsystem that connects to the probe 407 via an optical fiber 414. The system 431 includes a light source such as a laser, an interferometer having a sample arm and a reference arm, and various The optical path includes a clock generator, a photodiode, and other OCT system components. nothing.

[0128] The system 410 includes one or more diagnostic software tools for MLS-based image detection. The software further includes a program or module 412. The software may be implemented in one or more memory devices such as a memory device 445. The instructions may be stored as non-transitory instructions on one or more memory devices, and may be used to by one or more computing devices, such as the processing device 440 or the MLS 442 In one embodiment, the MLS can be implemented using one or more AI processors. The stent planning software tool includes a pre-sten A user may compare the stent profile and the post-stent profile or other profiles. , a target profile generated by a comparator or other comparison software routine In general, the software 412 may include one or more vascular profiles, such as: The endovascular data set was processed and analyzed using Fig. 1F, Fig. 2, Fig. 2A, Fig. 2B, Fig. 4B, Fig. 4C, Fig. 1 2A and 12B and other method steps depicted and disclosed herein. The various method steps described herein can be performed, such as: In one embodiment, the software 412 is stored in the AIM and executed by the AIP. It is possible.

[0129] The software 412 may include a controller for controlling an intravascular data set and an intravascular probe or angiography system. The present invention is designed to process other vascular data from other detectors or data sources, such as 1 In one embodiment, vascular data can be recorded during the pullback procedure and electronic The training, pre-processing, and ground truth can be stored in a memory device. mask generation, detected / predicted tissue classification, neural nets and other features and software The software components may be executed on the AIP or computing device 440. The software includes various MLS training modules, pre-processing modules, These include a lumen contour prediction module 412A, , side branch prediction module 412B, resizing module 412C, image flattening Module 412D, Lumen Flattening Module, Stent Strut Prediction Module 412 E, user interface and input processing module 412F, pre-processing module 412G , MLS interface module 412H, MLS memory manager module 412 I. GUI training for annotating image data and generating ground truth masks The filter module 412 may include a filter selection module 412J, an intensity normalization module 412K, etc.

[0130] In one embodiment, the software module processes the endovascular data to identify tissue. Characterize regions of interest such as calcium regions, taper regions, lipid pools, and other tissue features. The software 412 is designed to determine the fractional coronary flow reserve, the vascular resistance ratio (VR R: Vascular Resistance Ratio), and other measured and calculated intravascular data collection parameters. It is also possible to compare parameters from the stented to non-stented state. To the extent that the state of the It is possible to generate a

[0131] In one embodiment, an OCT system 431 can be used. The system is a balanced photodiode-based detector that receives the light returned by the probe 407. The optical receiver includes a computer, processor, ASIC, or system 410. or another subsystem in electrical or optical communication with system 410. The computing device 440, such as other devices included as a probe 407 In various embodiments, the computing device 440 receives an electronic signal from , local memory, bus and data processing and stent visualization and stent crimping Software for image data processing etc. configured for stent malapposition detection 44 In one embodiment, the PCIe bus or Other high bandwidth, low latency buses may be used to communicate with a given imaging system, MLS, or both. It is used to connect various components of a combination system including:

[0132] The stent deployment planning tool 412 may be part of the software 444 or These tools can also exchange data with the software 444. 7 is used to place a virtual stent within the luminal area that is positioned against the vessel wall. FIG. 13B illustrates a method for deploying one or more virtual stents and displaying the virtual stent on a user interface. 1 illustrates an example area of ​​a segment of a pullback that may be displayed on a

[0133] As shown in FIG. 4A, the display 446 also uses the collected endovascular data to A system 410 for displaying information 447 such as transverse and longitudinal cross-sectional views of blood vessels that are generated. Intravascular data may be acquired using a probe 407 and stored in a memory 445. Once stored in the endovascular data, it is processed to obtain the lateral distribution of the vessel along the length of the pullback region. Generate and display information such as cross-sections, longitudinal sections, and / or three-dimensional views 447 or a subset thereof It is possible to use 2D or 3D image masks to The loose data and the predicted results can be shown or stored. These figures are shown in the following figures. As shown in and described below, A vascular map generated using distance measurements obtained from the system 410 can be used. The images are processed using the tools and software modules described herein. and the lumen contour, vessel diameter, vessel cross-sectional area, landing zone, and In one embodiment, the present invention provides information about a blood vessel including a virtual stent partitioned by the In this case, the MLS442 is a computer system that includes one or more computing devices and one or more software. Software programs or modules. Tasks, methods, steps, processes and The present specification is operable to perform other features described herein with respect to each of them. There are a variety of devices, components, systems, and subsystems disclosed in the document.

[0134] The MLS442 performs MLS training and processing of patient image data, such as image data elements. One or more AI processors and / or GPUs and / or The ML may include processing cores and / or stream processors and dedicated memory. Further details of the S-based system are shown in Figure 4B.

[0135] As shown in FIG. 4B, a combined imaging and MLS system 450 is shown. The system uses a set of ground truth data to train the MLS. A data set that can support annotating raw image data with points and contours The display includes a display 420. The display may include a MLS 425 or an imaging subsystem 43. 0. The MLS 425 and the imaging system 430 can be connected to a data transfer interface. In one embodiment, the MLS 425 is a multi-layered, ... One or more buses / motherboards that provide communication channels and connections for the memory and other components. In one embodiment, the MLS 425 includes several AI processors. These can be arranged in a server configuration. As shown, the three AI processors AIP1, AIP2, and AIP3 are serial, The MLS and imaging system communicate in a single system. In one embodiment, the housing is provided by an AIP. In one embodiment, the AIP includes a cooling system CS for managing the high temperatures generated. Contains one or more graphics processing units (GPUs). In one embodiment, the housing is configured to ensure continued operation of the AIP within the housing. In one embodiment, the cooling system CS includes a temperature sensor TS for adjusting the temperature of the cooling system CS. The AIP is connected to the imaging system and / or MLS via a network. Transferring data to be processed externally.

[0136] In one embodiment, the system includes a motherboard that interfaces with the AI ​​processor. This motherboard is placed in the system housing. The housing is or to move the housing and its imaging MLS system components to another location. The system may be a cart including wheels to carry the OCT probe or the I. A probe interface unit 4 including a coupler for an optical imaging probe such as a VUS probe 70. This PIU is also referred to as a dock in one embodiment.

[0137] In various embodiments, the trained neural network is Graphics processor / graphics located inside the data acquisition system housing The AI / ML processor is a processing unit with N In one embodiment, N is in the range of about 2000 to about 2500. In one embodiment, N is in the range of about 2000 to about 3000. In one embodiment, N ranges from about 3000 to about 4000. In one embodiment, N is in the range of about 4000 to about 5000. In one embodiment, N is in the range of about 5000 to about Parallel processors are divided into individual hardware units called compute units. In one embodiment, the computing units can be grouped on a single wear element. , ranging from about 20 to about 80 computational units for a given AI processor. The computational unit may include multiple parallel processors. Compute Unified Device Architecture (CUDA) used with CUDA cores ranging from about 2000 to about 10000 cores In one embodiment, the Compute Unified Device Architecture (CUDA) is approximately 200 Used with CUDA cores ranging from 0 to about 2500 cores. Examples of processors include CUDA core and tensor core processors provided by Nvidia, and AM D and stream processors provided by D. In one embodiment, about 200 to about 300 tensor cores are used in the AI ​​processor. Included in processor / graphics processor.

[0138] In one embodiment, the AI ​​processor memory (AIM) is greater than about 8 GB. In one embodiment, the AI ​​processor memory is greater than about 16 GB. In one embodiment, the AI ​​processor memory is greater than about 32 GB. Therefore, the AI ​​processor memory is greater than about 64 GB. In one embodiment, the AI ​​processor memory is greater than about 128 GB. The memory ranges from about 4 GB to about 256 GB. In one embodiment, the AI ​​processor The memory ranges from about 8 GB to about 128 GB. The processor memory ranges from about 16 GB to about 64 GB. In one embodiment, the processor memory ranges from about 8 GB to about 32 GB. In one embodiment, the processor memory ranges from about 32 GB to about 64 GB. One or more of the memory storage devices can speed up the process and provide a better understanding of the patient's condition while in the cath lab. Includes an NVMe™ interface that reduces data parsing time for MLS operations. In an embodiment, 16 GB or more of on-board RAM is required for the imaging system / data collection system. In one embodiment, 32 of the on-board RAM is used on the motherboard of the system. GB or more are used on the motherboard of the imaging system / data acquisition system.

[0139] In one embodiment, the AI ​​processor memory ranges from about 1 GB to about 2 GB. In one embodiment, the AI ​​processor memory ranges from about 2 GB to about 4 GB. In one embodiment, the AI ​​processor memory ranges from about 4 GB to about 6 GB. In this embodiment, the AI ​​processor memory is in the range of about 6 GB to about 8 GB. In an embodiment, the AI ​​processor memory ranges from about 8 GB to about 10 GB. In an embodiment, the AI ​​processor memory ranges from about 10 GB to about 12 GB. In this embodiment, the AI ​​processor memory ranges from about 12 GB to about 14 GB. In one embodiment, the AI ​​processor memory ranges from about 14 GB to about 16 GB.

[0140] FIG. 4C illustrates two methods for identifying / segmenting features / regions of interest in image data. Software module suitable for processing image data using alternative process flows - Patents.com In one embodiment, the image data is collected from a polar OC The primary image data used is arterial image data such as 1T image data, but other image data from various sources can also be used. The upper process flow option uses artificial intelligence-based lumen detection4. 87 were trained on the lumen boundary / lumen mask as ground truth. This AI lumen detection is performed by using an AI processor such as a GPU. The image data is stored in a memory that is accessible to the user. A forward propagation algorithm is then used to generate a mask or other representation of the detected lumen boundary for the frame. Segment / classify images using classification and backpropagation A NN that generates the patterns, features of interest, regions of interest, and other detections / predictions that the NN was trained on. As a result, a prediction result 495 such as an output predicted image mask is generated.

[0141] In contrast, in other embodiments, the side branch 479, the guidewire 481, and The results of any initial detections, such as lumen 482, are then used to process the arterial data and train the ML along with arterial image data as input to MLS for pre-detection using S Before that, other detections / predictions are performed in various sequences and orders. MLS-based detection / prediction of a given set of classes, FOIs, ROIs, etc., for image data and The prior set of side branches, guide wires, lumens, etc. The detections can be stored as image masks. In one embodiment, the MLS is In the illustrated exemplary embodiment, the implementation of side branches, guidewires, lumens, etc. It is trained against a ground truth image dataset that includes pre-detections can be.

[0142] In part, the present disclosure relates to a method for treating tissue, such as calcified tissue, or calcified tissue, including calcium plaque. Detect the nodules / areas of interest associated with atherectomy and Using a virtually real-time guided method to perform atherectomy using a laser In various embodiments, the OCT imaging probe includes a swept source laser. The probe is in optical communication with an imaging laser, such as a hologram. As a result, light is received from the imaging laser through the probe. The light is directed to a light directing element such as a unitary lens, a GRIN lens, or a beam director. Light is transmitted through one of the optical elements to the tissue, and then the tissue The light from the optics is received and returned to the OCT imaging system. interferes with the light generated by the imaging laser.

[0143] FIG. 5A is a schematic diagram of an exemplary system for performing OCT-guided atherectomy. FIG. 5B is a schematic diagram of an exemplary method for performing OCT-guided atherectomy. 5A includes one or more MLS systems that include trained feature detection software such as neural networks. A system 500 is shown for detecting features of interest using an OCT system 505 that includes a stem. The imaging probe 510 includes an optical fiber that is part of the sample arm of an interferometer. The reference arm of the interferometer may be included in the OCT system 500 or may be The patient interface unit 470 communicates with the imaging probe and its optical fiber. It includes a rotary coupler or other mechanism for coupling to the OCT system 505 .

[0144] One or more optical switches 520 may be in optical communication with the PIU 470. 520 may be controlled by controller software or a controller. The OCT system then includes an imaging laser 527, which may include an ablation laser or The control software or controller 535 communicates with the optical scanner. The switch includes a GRIN lens, beam director, and microcontroller that direct light to the probe 510. Lenses or other optical elements are used to control the imaging laser and target ablation for OCT imaging. Then, the MLS and the ablation laser are switched between Calcium detection software such as Calcium Detection Software 525 is used to detect calcium. Identify areas of concern that contain tumors or other materials where ablation is the preferred treatment option It is possible.

[0145] In part, the present disclosure works in conjunction with OCT software or other imaging software. Calcium using MLS systems such as deep learning-based artificial intelligence systems MLS is used to detect the presence of calcium plaques or other ablation targets. Deep learning networks / MLS are used by expert users 450-1000 annotated OCs with calcium regions marked by This trained network was trained using T images. Then, through deep learning networks, we segment the images at the pixel level. Raw, filtered OCT images are provided.

[0146] In one embodiment, a deep learning based system that operates in conjunction with OCT software is used. Calcium detection using a trained MLS system, such as the artificial intelligence of the company, One embodiment of the disclosure is the MLS, which is used to treat the presence of calcium plaque or other dislodgable tissue. Deep learning networks are trained by expert users to Train using 450 annotated OCT images with marked calcium regions This trained network includes deep learning The raw OCT images are segmented at pixel level through a filtering network. is being supplied.

[0147] Figures 5C and 5D show the results of using the MLS system to highlight calcium plaque areas. Figure 5E shows the segmented OCT image. A user interface that displays detected areas of calcium, such as calcified plaques, in a longitudinal section and FIG. 5F then shows the OCT data co-registered with the angiography data. By displaying frames containing calcium in relation to the angiographic data, This is a calcium map created on the contrast data. Calcium plaque detected in the angiogram is displayed in a graphical user interface for review and planning. The calcium regions detected in the OCT pullback were also mapped onto the interface. , are mapped onto the angiogram data for review by the physician. The physician then: Identify calcium nodule regions of interest that are candidates for atherectomy or ablation procedures.

[0148] Detect and exclude tissue of interest using an OCT imaging probe, as outlined in Figure 5B. A method 560 is described. First, a first OCT pullback is performed (563). Using the initial set of T data, calcium detection is performed using the MLS embodiment, or This is done using a different method of detecting endothelial cells (565). OCT data are collected during angiographic imaging sessions. The OCT and angiographic images are then acquired during the imaging session. A calcium map is generated (575), as shown in Figure 5D. reviewed imaging data and calcium detection to determine plaque status as a basis for ablation. Once the area of ​​atherectomy is identified, a second OCT pullback is performed. During this inter-acquisition time, calcium detection 581 is also performed. The ablation system is controlled by software-based controls, such as calcium Align high power laser for atherectomy using map and target calcium nodules The system also provides real-time calcium detection to provide a first-order OCT probe. This corrects any changes in the alignment between the back image and the current image. This can be made easier by narrowing the search window of the frames you need to search. In one embodiment, the catheter tip reaches the target calcium plaque nodule. Then, a high-power ablation laser is turned on (584) and delivers a short burst of energy. The ablation procedure is called atherectomy. 7 is user controlled in various embodiments.

[0149] 5G and 5H show annotated Cartesian OCT images according to an exemplary embodiment of the present disclosure. These figures show the position of the imaging probe P in the lumen, the vessel wall W, the adventitia AD, and Calcium plaque and details regarding the calcium angle are shown. In this case, MLS can be used to detect and output calcium angle measurements. FIG. 1 is a magnified Cartesian OCT image identifying various layers of an artery, according to an exemplary embodiment of the present disclosure. The image shows.

[0150] FIG. 5J is a block diagram of a method for using the MLS embodiment according to an exemplary embodiment of the present disclosure. Reviewing ground truth or training sets suitable for annotating image data FIG. 5C shows a histology image of the artery with the various layers of the artery identified as part of the 5G and 5H show an in-plane view of a blood vessel B having a blood vessel wall VW defining a lumen L. Arterial cross-sections obtained by using an intravascular probe P shown in vivo The oval / circular reflection and shadow of vessel B includes the side branch SB. nothing.

[0151] 6A and 6B are Cartesian OCT images and the lumen (blue indicia), media ( MLS to identify Ca / Ca plaques (red indicia) The prediction output and the neural network trained according to one or more embodiments disclosed herein The results were obtained using an MLS-based system with neural networks (NNs). 6C and 6D show an example diameter measurement of 31 mm. T images and polar images generated using scan lines collected using an imaging probe Figure 6E shows ground truth masks of Ca, lumen, intima, and media. FIG. 6F shows the prediction results output from the deep learning MLS of the present disclosure. It shows.

[0152] FIG. 7A shows a decal as acquired using a combined imaging MLS-based system. Figure 1 shows an example original OCT image in a 3D format. The imaging probe is in the upper right quadrant of the image. FIG. 7B shows a Cartesian representation of FIG. 7A. 7C and 7D show the corresponding grayscale polar images of the ground-to-ground image. Annotation using loose masks and trained MLS for training and prediction The deep learning output of the MLS embodiment is generated using the image of FIG. 7B as input for Each mask or each detection corresponds to a channel. For use in tissue classification using image masks corresponding to round truth and prediction results Regarding the designed multi-channel MLS.

[0153] In one embodiment, the semantic segmentation is based on tissue type, calcification, um, side branch, lumen, guide wire, intima, media, calcium, fibrous, stainless A plurality of stent struts corresponding to a plurality of arterial features such as stents, stent struts, stenoses, and other arterial features. The ground truth of various features is obtained by convolutional neural network. In one embodiment, two neural networks are used to train the Two neural networks were used, the first for lumen detection, and the second for luminal detection. After the lumen detection is performed by the first neural network, the second neural network The network can be used to detect other arterial features. In either the first network or the second network, or both networks In some embodiments, the network is a convolutional neural network. Detection may also be performed on other features of interest such as calcium, media, intima, and other features disclosed herein. Lumen detection is performed on the image data before detecting the lumen. The patent application "Lumen morphology image reconstruction based on the scan line data" was filed in The system and method disclosed in U.S. Patent No. 9,138,147, entitled "Optical Computed Tomography (OCT)," The present invention may be implemented using a variety of systems and methods, including those described in detail in this U.S. patent. The contents of which are incorporated herein by reference in their entireties.

[0154] In one embodiment, a first neural network and a first image processing pipeline is used to reduce waiting time while the catheter is inserted between imaging sessions. It is advantageous to generate an output from the region of interest / feature classification system. It is therefore desirable to reduce the time it takes to The network uses classes of M and / or more than M types of features / regions of interest. The image processing system is trained using frames of annotated image data. In some embodiments, M is in the range of 2 to 3. In some embodiments, M is in the range of 2 to 4. In some embodiments, M ranges from 2 to 5. In some embodiments, M ranges from 2 to 6. In some embodiments, M ranges from 2 to 7. In some embodiments, M ranges from 2 to 8. In some embodiments, M ranges from 2 to 9. In some embodiments, M ranges from 2 to 10. In some embodiments, M ranges from 3 to 4. In one embodiment, M is 3, and the neural network The types / classes used to train the work are calcium, luminal, and medial. The ground truth used to train the neural network. A ground truth mask is a mask that has a group of annotated regions. At least one of the annotated regions in one ground truth mask includes The three include a media annotation, a calcium annotation, and a lumen annotation. Any collection of cardiac feature / region types and classes may be used in accordance with the types and classes disclosed herein. The class may include any of the following:

[0155] In various embodiments, image processing is performed with a trained neural network. The processing pipeline allows for the speeding up of image data through each of these. In addition, the annotation of all possible detectable features in the image is supported. Training a neural network to predict features / regions of interest in a given scene is Excessive network training time when using the network to predict outcomes , excessive network complexity, and excessive processing time.

[0156] FIG. 8A is annotated using expert data / ground truth data. 8 illustrates the pre-processing operations on a previous polar OCT image 800. As shown, the pre-processing steps As part of the strategy, useful imaging data or detectable tissue or ROI or FOI Depth pixels that do not encode or capture the depth dimension are effectively reduced or constrained. are selectively excluded and used for iterative expansion or direct annotation using expert truth data. In one embodiment, reduced image data elements 805 are generated that can be Multiple data augmentation processes are implemented, such as circular shifts, left-right flips, and other image-based transformations. It is used to convert one image or image data element into multiple versions of itself through do.

[0157] For example, the original image is circularly shifted by multiple phases such as 90 degrees, 180 degrees, and 270 degrees. When a new version and the original image or image data element are created, the new version and the original image or image data element are divided into four Each of these four versions is then flipped left and right to create eight In addition, each of these can be used to change the pixel count of a pixel. Select a set of filters, pixel region boundaries, and other geometric elements to measure calcium, lumen, Annotate image data using ground truth information such as side branch classification. The user interface software shown in FIG. 3A and FIG. 3B can be used. The image can be annotated with ground truth annotations by: In addition, they use cost functions or other metrics to measure detection accuracy. It can also be evaluated by a trained MLS. The average results or statistical analysis can be used to improve the detection accuracy.

[0158] 8B-8D are schematic diagrams showing the structure of a luminal, medial, and calcium-detecting endothelial cell according to an exemplary embodiment of the present disclosure. 1 shows various exemplary image masks for detecting the lumen, intima, media, calcium, Various image masks, such as a mask of the fiber and any ROI or FOI disclosed herein. A mask may be used. In one embodiment, the mask is a binary mask. In one embodiment, each mask is generated to reflect a ground truth detection. In some embodiments, given a ground truth mass, A cluster is a set of one or more regions that correspond to ground truth classes such as calcium, lumen, and intima. and other areas of the mask that do not correspond to one or more of the classes. In, the ground truth mask is the first set of pixels that correspond to one or more classes. and a second set of pixels that do not correspond to one or more classes. The truth mask is generated by a given MLS, such as the neural network shown in FIG. 1C and FIG. The method can be used to train a neural network of In this embodiment, resizing and other steps such as flattening as described herein may be performed on a given generation. This is performed before generating an extended or additional version of the image. The pixel count may include a subset of pixels corresponding to a channel, the channel being any of the pixel counts described herein. The masks correspond to the features of interest such as calcium, intima, etc. as shown. It is used to train a database or to process image data to obtain an output. The pixel count of a given channel is used as a result of using the MLS trained for In one embodiment, the multiple masks can be used to identify multiple channels. and the processing of a given image is done by processing a given In one embodiment, the ground truth note The interpretation includes multiple channels of information. For example, to train a neural network The given ground truth annotations / masks for the purpose of classification are calcium region, media region, lumen region, and the intimal region.

[0159] 9A-9C are run before training the MLS neural network. Circular shift and left flip applied to the original OCT polar image as part of the preprocessing step 9A-9C are polar OCT images showing a right flip. These processing steps or transformations are performed on image data, such as raw polar OCT images. As a result, one raw polar image is transformed into seven additional modifications or versions. The dilation process can be implemented in a variety of ways. Figure 9B shows a circular dilation of the original image of Figure 9A. FIG. 9C shows a left-right flip of the shifted image of FIG. 9B. A given such expansion can produce eight modifications of the image of a single frame. This includes one original and three circular shifts (each 90 degrees). A left-right flip of each of these four copies results in a total of eight. Additional operations, or a subset thereof, may be performed using MLS training and / or It can be used to preprocess data for prediction.

[0160] 10A and 10B show the ground truth annotated image and the predicted output, respectively. The image shown in Fig. 10A shows the inner blue ground truth mask. The yellow layers are designated as separate channels or classifications such as lipid LPs. The orange area corresponds to calcium. In this case, it is desirable to implement a penalty or restriction system to avoid "false positives". In FIG. 10B, the background channel "B" is shown. The background is particularly noticeable in the prediction results. If a background is specified, this can be used to distinguish it from one of the other interest classes or types. In various embodiments, the present invention can effectively prevent a user from being erroneously classified as one of the above-mentioned types of information. The channels can be generated using ground truth annotations. It can be established using features corresponding to the facts, background, non-uniform rotation distortion effects, etc. Thus, in some embodiments, the neural network is Remove from or display in a given user interface representation of It is desirable to prevent artifacts, imaging system errors, false positives, and It can be trained to identify other imaging phenomena. The mask showing the predicted results is , as displayed in Figure 10B.

[0161] 11A-11D are diagrams showing a system developed to increase MLS processing speed and reduce patient wait times. 11A and 11B are images showing the various pre-processing steps performed on the Ground truth shown via green for membrane, red for Ca, and blue for lumen. The OCT image with is devoid of useful data along 976 pixels in the vertical direction. These images reveal the presence of regions such as region 300 along the depth dimension. The image can be flattened as shown in FIG. 11B. The white luminal contour in Fig. 11B is flattened to an almost straight line. Thus, MLS / NN selects images in a transformed state that is more likely to contain relevant tissue data. Similarly, in FIG. 11C, A lines (scan Half-resolution resizing is done by omitting lines to generate a central image. Additionally, depth pixels in the image are omitted or removed to further resize the image. This results in the preprocessed image on the right side of Figure 11C. These steps are the basis of the MLS operation. Improved real-time imaging when the patient is catheterized after OCT or IVUS imaging These processes can be performed in real time or substantially in real time. This can save time and improve patient outcomes. The OCT image 244 shows calcium plaque area in red, luminal area in blue, and The same flattened OC with inference or prediction results in the form of an image mask showing the media region of 1 and 2. Flattening can be used as a pre-processing step in some embodiments. This polar image can be converted to Cartesian form, as shown in FIG. It can be displayed to the end user in a diagram similar to a cult image.

[0162] 12A and 12B show how raw image data is fed and the ground truth data is used to FIG. 12A is a flow chart of exemplary processing steps performed with annotations. One or more of steps A1 to A8 may be performed by one or more computers as disclosed herein. This can be performed using a mobile device-based system as shown in FIG. Thus, according to one embodiment, the system and method perform one or more of the following: The imaging system and / or the MLS may be implemented using a computing device or Through IP, multi-point annotation of image data or image data elements can be performed. The GUI places and connects points to effectively label a given tissue type within an image. The method facilitates defining a boundary that can be converted into an image mask. The method may include detecting / updating lumen detection such as annotation, lumen detection, or contour. Output, raw image data or data elements are transferred from one system, such as an imaging system, to the MLS / to an interface or intermediate system that provides input to the neural network. In one embodiment, resizing the image data elements may include This is done to remove non-useful image data to improve processing time. In an embodiment, the method includes generating a binary mask for tissue labeling / annotation. Additionally, data can be formatted as an array or in another format for input. Finally, a composite image containing the image mask and the resized raw image data or data elements can be generated. The input data that can be input to MLS / NN for processing such as training or prediction is will be done.

[0163] FIG. 12B illustrates another embodiment, which may be performed by various systems disclosed herein. In FIG. 12B, one or more of the steps disclosed herein may be used. Using the computing device-based system, among steps A1 to A8 These steps may include one or more of the following: One or more imaging systems (OCT, IVUS, X-ray, Anglo, etc.) The system components acquire the raw image data set. They then perform various pre-processing steps. The luminal area of ​​the raw image and the mask image with ground truth data can be calculated. A flattening of the image area and / or the image region may be performed. Every other scan line is removed. In one embodiment, the lumen contour or boundary is flat. The ROI / FOI data is then enhanced and meaningful ROI / FOI data is highlighted, as shown in Figures 11B and 11D. In other embodiments, the image may be circularly shifted, flipped left / right, or transformed in a different way. This can increase the number of entries in a given training set. It is possible to run MLS / NN so that the results of ML can be monitored and tested. It is possible that different versions will be produced.

[0164] Clip scanlines / data / pixels from raw and annotated mask images Adjusting rows (in polar or Cartesian format) by omitting, omitting, or removing For example, as shown in FIG. 11A, the pixels corresponding to the depth on the vertical axis are removed. The image can be resized as shown in Figures 11B and 11C. ,Add scan lines / data / pixels to generate raw images and It may be useful to standardize the sequence of the raw image data and the annotated mask image. The data and mask image data are formatted as needed for predictive analysis or additional training. This may include inputting the data into an MLS / neural network for filtering.

[0165] 13A and 13B show a stent planning methodology using virtual stent placement incorporating tissue classification. 1 shows an exemplary user interface of a diagnostic tool that supports these The user interface is designed to provide the most critical pathology and / or maximum flow restoration based on physiology. This helps to delineate the MLS analysis that can identify the shortest stent that can be used. In one embodiment, the stent is formed from semicircles, as shown by the white stent region. ,To perform interactive virtual stent placement, a virtual stent is placed against the lumen representation. Interactive virtual stent placement allows for the placement of the stent S. The user can obtain predicted and current VFR values. The lower section showing the percent diameter stenosis and the minimum luminal area The MLA diagram in the panel shows the VFR which changes based on the stenosis and / or virtual stent selection. p-values ​​and VFR values ​​are also shown. The angiographic images shown in the upper right panel are In FIG. 13B, the artery is co-registered with a Cartesian or polar view or a longitudinal section of the artery. , predicted VFRp and current VFR are calculated using ROI / FO of lumen, calcium plaque and media. A virtual stent Vs and Vs masked or characterized with various indicia corresponding to I and Cartesian arterial images.

[0166] The methods and systems disclosed herein provide a diagnostic planning tool for a user. For example, the method and system may perform virtual imaging into an artery on image data from the pullback. Such diagnostic graphs include tools that can automatically place stents. Further details of the graphical user interface based tool are shown in Figs. 13A and 13B. Such a system combining imaging and machine learning suitable for diagnostic planning can be seen in the figure below. An exemplary user interface from the system is shown in Figures 13A and 13B. Moreover, automated deployment of such stents allows for the placement of such stents in optimal locations. , a process for displaying the appropriate stent size identified for the end user, the user This includes the interface and associated software-based features.

[0167] The present disclosure does not provide a method for optimizing certain parameters, such as the optimal location or otherwise. The present invention includes various embodiments of stent planning software for placing a stent at a location In one embodiment, the parameters that are optimized to facilitate stent planning are: , the flow rate that can be achieved by deploying a stent of a particular length. The proximal and distal landing zone locations of the stent and the size of the stent are These are provided to end users with a set of possible stents and stent deployment locations. The improvement in flow rate that can be achieved using the

[0168] In one embodiment, the training data is generated by a ground truth expert It is created using guidance from the MLS for guidance or lumen detection, etc. The scanning data is divided into masks. In one embodiment, the masks are divided into one or more individual These are the individual channels that are used to train the training set. Once the training set is passed through the neural network, the Each of them essentially contributes to the various weights and filters of the network. The NN adapts / changes in response to the mask / training data. When the raw data is fed into the trained network, the raw image data is The individual channels are tagged or labeled with different channels, all of which are luminal, Intima, media, adventitia, EEL, IEL, plaque, calcium, calcium plaque, ste The different features used in the training set, such as images of the guidewire, Respond.

[0169] In general, the MLS system disclosed herein performs multi-channel segmentation. For the process, each tissue type, each region of interest, each arterial layer, etc. are included in the training set and the prediction Processed as a separate data channel with its own image mask to generate a measurement output will be done.

[0170] One exemplary method for assessing flow restoration as a result of stent deployment is described in METHOD AND APPARATUS FOR AUTOMATED DETERMINATION OF A LUMEN CONTOUR OF A STENTED BLOOD Using the method described in U.S. patent application Ser. No. 14 / 115,527 entitled "VESSEL" The contents of this U.S. patent application are hereby incorporated by reference in their entirety. This specification is incorporated herein by reference in its entirety. Other techniques, including those not listed herein, may also be used. can be used to understand some aspects of flow changes and behavior in arteries. Then, a virtual stent is created based on the identified landing zone and stent length(s). The features shown in Figures 13A and 13B, which illustrate the constriction and various features related to tent selection and positioning, are It is helpful to consider the symptoms.

[0171] In various aspects, the present disclosure provides a machine learning system, neural network, and a method for detecting coronary artery disease using polar images of the coronary arteries. network, and convolutional neural network for training one or more In various aspects, the present disclosure relates to a ground truth method for performing a coronary artery polar image. Machine learning systems, neural networks, and convolutional neural networks using source annotations In various aspects, the present disclosure relates to training one or more of the network of neural networks. Various tissue types and features of interest are annotated in the training set. Neural networking is a machine learning system that uses ground truth annotations made to images. training of one or more of the following: Regarding.

[0172] In various aspects, the present disclosure provides a method for detecting a variety of tissue types and features / regions of interest in a training set. Ground truth analysis is performed on Cartesian or non-polar images of coronary arteries annotated at the Machine learning systems, neural networks, and convolutions using do-truth annotations In various embodiments, the present invention relates to the training of one or more of the neural networks. The disclosure relates to trained machine learning systems, neural networks, and / or convolutional neural networks. Polar or non-polar models using convolutional neural networks, as well as combinations of these. Classify / characterize input patient data in a standard or Cartesian format and classify it into various tissue types The procedure is performed on polar images of the coronary arteries where the morphology and features of interest are annotated in the training set. Identify different tissue types and features / regions of interest using ground truth annotations generated by Pertaining to doing.

[0173] The present disclosure uses ground truth tissue types and features / regions of interest to generate patient image data. Classify and identify various tissue types and features / regions of interest in data and image data elements Various embodiments, aspects, and / or features of one or more of M In the LS embodiment, the tissue type and feature / region of interest (FOI) / ROI are Coronary artery, one coronary artery, OCT image, IVUS image, X-ray image, ultrasound image, angiogram image Image, graph or plot of arterial tree, side branch, lumen, guidewire, stent, Jailed stent, bioabsorbable scaffold (BVS: Bioresorbable ble Vascular Scaffold), drug eluting stent (DES), fiber Quality, Blooming artifact, Pressure wire, Lipid, Calcium, Atherosclerotic plaque plaque, stenosis, calcium, calcified plaque, tissue containing calcium, lesion, lipid Fat, malapose stents, underinflated stents, excessive Over inflated stent, radiopaque marker, arterial tree branching angle, PET Calibration elements such as dope membranes, sheaths, dope sheaths, reference alignment points, changes in arterial diameter, motion The radial change of the blood vessels, flow measurements obtained using imaging data, FFR values ​​of the images, bifurcation models del, combinations of the above, classes or types of the above, Cartesian images, and polar images , a Cartesian image of a non-polar image or a portion thereof, and one or more of a polar image and a non-polar image. It can include.

[0174] In one embodiment, the probe includes or is in optical communication with an optical fiber. The optical fiber and tip of the probe are connected to one or more catheter sheaths, etc. The probe tip is fitted with an angled beam director or lens case. The imaging modalities may include a variety of elements such as transducers of the imaging modalities and the like. The optical fiber of lobe 407 may also include a torque wire disposed around the fiber. The probe transmits light into the lumen L and receives light scattered from the vessel wall VW.

[0175] In one embodiment, the optical fiber is part of the sample arm of an interferometer. The data collection probe 407, such as a CT probe, collects depth information suitable for imaging a sample such as a blood vessel. It can be used to collect information such as image data, for example polar or Cartesian images. A set of frames of the data are shown in 3D. A cross-sectional image of the blood vessel is generated based on the optical signals transmitted and received by the probe. The scan lines are collected as the image moves (see Figs. 3D, 3E, 6A, and 6B). (See the Cartesian images shown in the figure.) Cross-sectional images such as images 6A and 6B are Cartesian images. 7B and 8A show examples of polar maps or polar forms of medical images. Cartesian and Polar Images The image may be referred to as a frame in various embodiments.

[0176] OCT images, such as the cross-sectional images of FIGS. 3B, 3C, and 3D, are typically taken one scan at a time. The scan line is acquired along the ray originating from the center of the catheter to the maximum imaging depth. A sequence of samples is called a scanline in one embodiment. In morphology, the smallest data unit in an OCT image is called a sample. The sequence of samples along a ray originating from the center to the maximum imaging depth is called a scan line. OCT images are typically acquired one scan line at a time. Cross-sectional images are can be formed from a set of scan lines collected as the probe rotates. Additionally, to image a segment of an artery or other blood vessel, the catheter can be rotated. The probe is then moved longitudinally while scanning the transverse image in a spiral pattern. The images are acquired by taking various scans related to the slice of the vessel or artery of interest. The scan lines have angles between them like the spokes of a wheel. The scanlines are acquired in polar format in one embodiment. do.

[0177] [User interface features for mapping organizational characteristics] In part, this disclosure relates to image navigation, user interface design, time-saving enhancements The present invention is directed to a medical diagnostic system including a catheter and other design considerations. to address time and information management issues during time-critical medical procedures, such as procedures performed in This disclosure describes a method for converting the results of tissue characterization into a tissue map display. Tissue characterization is used to characterize pathology that may exist, be in development, or be developed. Research findings, artificial intelligence techniques, machine learning techniques, attenuation and backscattering based techniques, and image processing This can be done using a variety of techniques, including logical detection and enhancement techniques.

[0178] Embodiments of the tissue maps disclosed herein are useful for measuring the volume of tissue in a subject, such as an artery, blood vessel, or other body lumen. Various characterized tissues and boundaries and their relative positions on one or more parts of the examiner To provide a user interface that improves the visibility of image data including the layout. Doctors, technicians, and other end users can view and analyze a set of images with characterized regions of tissue. When given, diagnosis can be made more quickly and with more information than would otherwise be possible. Planning decisions can be reached based on analytical information, allowing for informed decision making .

[0179] Tissue characterization generally involves identifying tissue types or other tissue characteristics for a given set of patient tissues. Thus, an artery is imaged and multiple images are generated from the image data. When forming a set of data, the tissues in each image are characterized to determine whether they are fat. It is possible to determine with a high degree of accuracy whether the tissue is of a particular type, such as fat or muscle. In addition, calcification, intima, EEL, and other types or imaged tissue as disclosed herein may be detected. It is also possible to determine at a more detailed level what is present in a given subject. Typically, one or more detection processes identify the type of tissue of interest in the image region. It is used to detect the presence of a machine learning inference process or an image processing pipeline based process. A separate detection process, such as a Calcium Detection Process, detects calcium and tunica media in the image data and then The umbilical plaque mask and the media mask are available in a set of image frames. As discussed above, the ring is created by projecting each mask along the A line. 16-18 show a stainless steel structure according to an exemplary embodiment of the present disclosure. Various tissue maps are available to support diagnostic and treatment planning, such as catheter placement and atherectomy. The OC of the artery having various indicia integrated into a user interface that displays 13A-13C show various tissue map representations generated using T imaging pullback.

[0180] In Figures 14, 15, 16, 17, and 18, each ring on the plane is a 3D projection. Figure 14 shows one frame of tissue characterization results from the pullback OCT data. In the example, a three-dimensional representation of the artery 1405 is obtained using a guidewire GW, MLS, or another technique. The area of ​​detected calcium is shown along with the three cross-sectional boundaries R1, R2, and R3. These boundaries or rings are then projected onto a plane, as shown, to generate a tissue map TM. Alternative versions of the tissue map are also shown in FIG. 2A and FIGS. 16 and 17. FIG. 15 illustrates various arterial structures shown along a 3D arterial representation 1425 extending in the proximal and distal directions. 1 shows a schematic representation of cut planes / rings / boundaries R1-Rn. In one embodiment, O Imaging data, such as CT, IVUS, x-ray, or other imaging modality, may be used to image the artery 142. 5 such 3D representations can be generated.

[0181] Intravascular optical coherence tomography (OCT) imaging provides high-resolution visualization of coronary artery morphology. In part, the present disclosure provides a method for treating intracoronary plaque (calcium, lipid, fibrosis, and The automatic detection and / or classification of internal and external calcification boundaries is also performed in one implementation. The detection and classification process improves the interpretation of OCT images. and provide targeting information to a diagnostician. The results are clear and easy to interpret, and the results are consistent, as shown in the tissue map TM and the examples in Figures 14 to 18. The results of data analysis applied to endovascular data sets are presented to the user in a manner that aids the examiner in their diagnosis. The present invention relates to a system and method for displaying information to a user.

[0182] In part, the present disclosure relates to a user interface and a method for determining whether a given artery or another body lumen (intestine, esophagus, etc.) is in a state where the artery is in a vascular tract. A global tissue map can be generated from a set of imaging data (such as a tract, a vein, etc.) Describe a Graphic User Interface (GUI) that provides a graphical data representation. The organizational map responds to user selections, such as user selection of a particular region of the organizational map. In response to a user clicking or otherwise selecting an organizational map region, The user interface software module of the applicable imaging system may include a user-selected The user is then able to display the underlying image or frame of image data specific to the selection. It is possible to redirect the information displayed to the user.

[0183] In this way, rather than scrolling through a long series of images or frames of image data, An organizational map is presented to the user. Interactions and selections on the organizational map are dynamic. By quickly guiding the end user to an area of ​​interest such as calcium plaque or lipids in the vein, This reduces the need for diagnostics and analysis. and while blood flow is temporarily stopped to image a section of the blood vessel. Stent planning can be expedited, thus making faster decisions and improving patient outcomes. The selection of the stent landing zone and the appropriate stent placement can improve Calcified areas where arterial expansion is restricted by the presence of calcium or another unfavorable arterial condition The favorable and unfavorable areas of the artery are identified by color, graphics, and the like. Fixes, visual cues, or other indicia such as animations or other cues or using the features of the user interface to determine proper or improper stent landing. It can be flagged as a zone.

[0184] In part, the present disclosure relates to a system suitable for use in a cath lab, such as an optical coherence tomography system. In part, this disclosure relates to data collection systems, such as intravascular data collection systems. The present invention relates to a data acquisition system including a processor adapted to display endoscope image data. The image data to be processed includes data or images generated based on the depth measurements. In an embodiment, the image data is generated using optical coherence tomography. The system also includes a user interface that displays intravascular information, such as data on intravascular plaque. It can be displayed.

[0185] In part, the diagnostic systems, methods, navigation tools, and compositions disclosed herein The texture map and associated features are then processed by one or more of the texture characterization processes and image data detection. After the analysis software module is run, plaque composition, lesion characteristics, tissue type, and better tools for interpretation and decision-making based on relevant tissue and vascular information. Provides improvements in the form of a tissue map interface and a scan for a given subject. Characterized tissue data compiled from the underlying image data acquired during the study are Demonstrate improved image views, e.g., for OCT and other intravascular imaging modalities. Thus, a given tissue map represents the volumetric map (vo) of the full imaging probe pullback in a single image. It is possible to provide a lumetric view.

[0186] In this case, the imaging probe moves through the blood vessel as it rotates to acquire imaging data. The end user can then use the given tissue map to identify calcium plaques. and arc extend and thickness of healthy landing zone for stent implementation In addition, other details regarding the tissue type can be assessed. In addition, a given tissue map may be able to detect excessive amounts of calcium or other undesirable tissue conditions. To the extent that this information reveals the amount of A given tissue map can facilitate the selection of alternative treatment options. A given method may be used to generate a CT scan of a patient using an imaging system. It may vary based on the source of the image data obtained from the scan.

[0187] In the pre-intervention assessment stage, the physician As shown in the tissue map display 1425 of FIG. Pixels with red indicate areas where the tunica media was detected in the tissue characterization, while It shows the presence of calcium plaques Ca. In the given tissue map user interface The colours mentioned may vary and in some cases are indicated by hatching or other indicators. This can be replaced by using a ndicia.

[0188] The dotted lines in the tissue map represent R1, R2, and R3 in FIG. 14 along the length of the arterial representation. R3 ring is shown to provide a match with the ring of R3. The tissue map can show the extent of calcium plaque. The intensity values ​​are approximately 0.5 m Calcium plaque in millimeters as indicated by the symbols ranging from m to about 1.5 mm The thickness of the ROI / FOI detected using the methods disclosed herein corresponds to the thickness of the Display any class or type using an organizational map representation, such as organizational map 1425 In addition, other information such as plaque containing fiber and lipids can be also shown in FIG. The tissue map can be added to the stent deployment as shown in the exemplary tissue map in During the stage, the doctor can easily identify the landing zone from this plot. The green intensity indicates the thickness of the tunica media, but the two end points of the EEL are opposite each other at 180 degrees. This can then be modified to show the effective distance for the stent size. It can be used for useful information.

[0189] FIG. 17 shows the tissue map of FIG. 16 with the identified tunica media (M), lumen (L), and calcium (Ca). FIG. 1 shows cross-sectional frames R1, R2, and R3 shown in FIG. In this study, these regions of interest were labeled with blue for the lumen, red for calcium, and red for the median luminal area. Next, FIG. 18 shows the guide wire GW (gray (light blue), tunica media M (green), calcium Ca (red), lipid LP (blue), and side blood vessels 14 shows a texture map 1490 including indicia of the SB (boundary shown). These different representations can be used to support different workflows and diagnostics. Cut.

[0190] In part, the present disclosure relates to a diagnostic system having one or more imaging and tissue detection methodologies applied thereto. The interface of this diagnostic system makes it easy to navigate the vascular representation. For a given blood vessel or other body lumen, such as a coronary artery, one or more tissue types or other Regions of interest can be identified using a variety of techniques, particularly to identify areas of interest such as calcified regions within blood vessels. May represent calcium nodules, calcified tissues and other calcium-related tissues. A tissue map representation of the tissue is generated and used to display the characterized tissues and regions of interest to the user. Features suitable for detection and inclusion on one or more tissue maps can be used. The tissues and / or regions of interest identified were: lipid regions, luminal regions, stem Strut, Side Branch, Guidewire, External Elastic Layer (EEL), Internal Elastic Layer (IE L), boundaries and associated volumes and other arterial characteristics as disclosed herein, It may include one or more of the tissue types.

[0191] In part, the present disclosure provides one or more compositions based on characterized tissue of a body lumen, such as a coronary artery. A user interface that uses a structured map representation to facilitate improved information and time management. In the various organizational maps shown, you can move in the direction of the page. The OCT imaging probe is positioned and moved distally to the pulled back location. The length of the pullback is between R1 and RN, where RN is the frame of the pullback. The lowest ring count is the most proximal. In one embodiment This can be seen in the ring configuration in Figure 15. can be done.

[0192] The present disclosure provides OCT image data that can detect calcium in blood vessels and other tissues. IVUS image data, CT scan image data, MRI image data, angiography image data and acquiring characterized image data relating to blood vessels, such as from other image data sources. The findings are based in part on the discovery that some imaging modalities can detect areas of calcification. The vascular tissue is visible as discrete dark features against the lighter vascular tissue background in the OCT image. This is true for OCT images which show calcium as darker areas.

[0193] In part, the present disclosure relates to tissue types and lumen, side branches, stents, guidewires, and other features, such as other characteristics, properties and materials of the blood vessel. This relates to a method for identifying

[0194] In one embodiment, a tissue type analysis and / or tissue type segmentation The representation of the stimulated vessel may be a 2D cross-sectional rendering of the vessel or a 3D longitudinal rendering of the vessel. In one embodiment, the representation of a blood vessel or a blood vessel-related The resulting image data, with characterization of the underlying tissue, is converted into a tissue map. In one embodiment, the detected tissue types and / or regions of interest are displayed in a tissue map. Various colors, shapes, hatches, masks, boundaries, etc. can be used for identification or segmentation. and other graphical elements or overlays may be used.

[0195] In part, the present disclosure relates to a system for identifying a region of interest in a blood vessel, the system comprising: A processor in communication with the memory, the memory being configured to, when executed, cause the processor to generate an image of a blood vessel. Image data is acquired, and a number of filters are applied to the image data to identify characteristics or types of tissue, etc. In one embodiment, the image data includes instructions to generate a scan type. In one embodiment, the image data is X-ray based data. In an embodiment, the image data is a polar image. In one embodiment, one or more polar Images are sampled. The samples are combined to generate a tissue-characterized representation of the blood vessels. The tissue-characterized representation of the blood vessels is in polar form. The tissue map derives the polar tissue-characterized representation. In one embodiment, the scalar is obtained by converting the scalar into a cult representation. In the present study, the tissue map is a series of rings, circles, or circles arranged in sequence along a proximal to distal axis. , or an ellipse.

[0196] In part, one embodiment of the present disclosure comprises one or more detection processes as described herein. an intravascular data acquisition system and one or more software programs for implementing the process and the display process; This paper relates to a software-based graphic user interface and software modules. In one embodiment, the intravascular data is acquired while the angiographic data is simultaneously acquired. In other embodiments, angiograms, CT scans, X-ray based imaging, photographs, A radiography or other imaging modality acquires imaging data that is used to generate tissue maps. It is used to obtain.

[0197] In part, the present disclosure provides a method for performing a stenting procedure by visualizing a subject's blood vessels, such as one or more coronary arteries. The present invention relates to a system and method for treatment evaluation, including imaging planning and surgical options. The data can be acquired using an intravascular data collection probe. The blood vessel can be pulled back through the catheter and data can be collected about the blood vessel. Such pullbacks and associated data collection may be useful in determining whether a stent is being deployed or not. The resulting intravascular data from the pullback is used to evaluate the In a variety of ways, such as for visualization of vascular areas, features, and stents deployed therewith. The image data used to generate the tissue map can be used to The tissue map can then be co-registered with the vascular data. The area can be selected and used to generate maps (OCT, IVUS, X-ray, etc.) The underlying image data can be viewed, as well as the location of the selected vessel on the tissue map. Angiographic data may also be viewed with highlighting or other indicia indicating the region.

[0198] The stent may be visualized with respect to the side branch in various embodiments of the present disclosure. This is an important feature because side branches can be easily removed during stent deployment. This is because it is usually the case that it is desirable to avoid the placement of a stent in the artery. In this way, the tissue map can show side branches and the The frame may be flagged as unsuitable for use in the stent landing zone. The systems and methods described herein can be used for different types of tissue and regions of interest. The tissue map facilitates visualization of the stent landing zone for various users. The collected data are presented at the interface and at the representation of the stent struts and side branches. Based on the detection of these features in the endovascular data, the images are co-registered with the angiographic data. It is possible.

[0199] In part, this disclosure relates to intravascular data collection, such as OCT, IVUS, and other imaging modalities. The collection system and the stent landing zone in the vessel as part of the tissue map, side For generating and visualizing diagnostic information such as branches, regions of interest, and characterized tissue regions. A suitable graphical element to represent such diagnostic information of interest is the tissue map. User selection in tissue map to facilitate navigation to underlying images summarized in a graph It functions as an element.

[0200] Also disclosed herein are methods for determining the composition of a stent, tissue type, tissue volume, and tissue boundary. A system and method for visualizing a world using one or more software modules. ,Detection of side branch location, lumen contour, and stent strut position, vascular Generation of a representation, generation of a tissue map, and navigation of imaging based on user selection of the tissue map. The systems and methods disclosed herein can be used to control navigation. Angles, thicknesses, volumes, widths, frame sizes, and other measurements for various tissue types including calcium, lipid, fibrous, etc. US Patent 5,993,966, and US Patent 5,993,966. This also includes the following features.

[0201] In various embodiments, such a measurement tool can be used to measure a given area of ​​interest of a particular tissue type. These can be used to measure the above parameters and any geometric properties of the area. The measurements are used to generate various ratings or scores suitable for review by end users. For example, calcium burden in specific regions of blood vessels can be visualized on tissue maps. If the calcium is present but only a small amount of surface calcium is present overall, measurements of this are not required. should guide users and not exclude such areas as potential landing zones. can help.

[0202] For the sake of clarity, this disclosure will be omitted wherever convenient or appropriate. Various aspects of embodiments of the applicant's teachings are described below without omitting certain specific details. For example, the discussion of the same or similar features in alternative embodiments is intended to be construed as limiting the scope of the present invention. Known ideas or concepts may be omitted in any form for the sake of brevity. Some details may not be discussed. Those skilled in the art will appreciate that some embodiments of the applicant's teachings may be Any embodiment described herein is provided solely for the purpose of providing a thorough understanding of the embodiments. Please be aware that you may not require some of the details specifically stated in the Likewise, the described embodiments may be modified without departing from the scope of the present disclosure. It is obvious that the present invention can be modified or changed in accordance with the general knowledge of the art. This detailed description of the embodiments is not intended to limit the scope of the applicants' teachings in any way. It shouldn't be.

[0203] As used herein, the terms "about" and "substantially the same" refer to, for example, the same Through measurement or handling procedures in the This can occur through electrical losses, through manufacturing differences / defects in electrical elements. Variations in the quantities of the invention that are included in the invention, as well as variations that are not necessarily consistent with the prior art. Unless the invention encompasses known values, variations that would be recognized as equivalent by those of ordinary skill in the art are not intended to be limiting. Typically, the term "about" refers to a range within the stated range by 1 / 10, e.g., ±10%, of the stated value. It means that the voltage is greater than or less than a specified value or range of values. For example, a voltage of about +3V DC Applying voltage to the element means a voltage of +2.7V DC to +3.3V DC. Similarly, when values ​​are said to be "substantially the same", those values ​​may differ by up to 5%. These terms may vary by up to about 100%. Any quantitative value recited in the claims, regardless of whether it is a For example, there may be cases where the above-mentioned This includes fluctuations in quantities such as:

[0204] [NON-LIMITING SOFTWARE FEATURES AND CHARACTERIZATION OF MLS AND TISSUE CHARACTERIZATION / CLASSIFICATION SYSTEMS AND METHODS Embodiment The following description is directed to device hardware suitable for carrying out the methods of the present disclosure described herein. This document is intended to provide an overview of the hardware and other operating components of the This description is not intended to limit the applicable environments or the scope of the present disclosure. Similarly, the hardware and other operational components may be suitable as part of the apparatus described above. The present disclosure relates to a personal computer, a multiprocessor system, Microprocessor-based or programmable electronic devices, network PCs, minicomputers It may be implemented using other system configurations, including microcomputers, mainframe computers, etc. The present disclosure also provides for communication between different rooms, such as in a catheterization lab or cath lab. A distributed computing system in which tasks are performed by remote processing devices that are linked through a network. The method may also be implemented in a computing environment.

[0205] Some parts of the detailed description refer to operations on data bits within a computer memory. These algorithmic descriptions and symbolic representations are presented in , can be used by those skilled in the computer and software related fields. In embodiments of the present invention, an algorithm is generally defined herein as a method for producing a desired result. It can be thought of as a self-consistent sequence of operations. Operations performed as steps or otherwise described herein may refer to operations involving physical quantities. These are operations that require logical manipulations. Usually, these quantities are stored, transferred, combined, transformed, compared, etc. These may take the form of electrical or magnetic signals that can be transmitted, received, transmitted, or otherwise manipulated. do.

[0206] As will become apparent from the discussion below, unless otherwise specified, throughout this specification: "Processing" or "Calculation" or "Classification" or "Characterization" or "Correlation" or "Detection" or "Evaluation" " or "convolution" or "deconvolution" or "classification" or "segmentation" or " "Training" or "Annotation" or "Alignment" or "Measurement" or "Calculation" or "Ratio" "Compare," "Generate," "Detect," "Determine," "Indicate," "Boolean logic," or "Relate" A discussion using terms such as a set of operations or other similar terms may be used to describe a computer system or electronic device. The data represented as physical (electron) quantities in the registers and memory of electronic memory devices is called or register or other such information storage, transmission or display device trained to manipulate and convert data into other data represented as physical quantities in similar devices; The operation of the MLS, computer system, AI processor, GPU, or electronic device and processes.

[0207] The present disclosure also relates in some embodiments to an apparatus for performing the operations herein. This equipment may be specially constructed for the required purpose or may be The device is selectively activated or restarted by a computer program stored in the computer. The present invention may include a general-purpose computer configured to perform various functions using various circuits and their components. to perform some of the data collection and transformation and processing described herein. It is possible.

[0208] The algorithms and displays presented herein essentially implement the principles of any particular computer. The present disclosure is not intended to be limiting, nor is it intended to be limiting, to any particular device, system, or equipment. It may therefore be used in conjunction with a program to perform the required method steps. In some cases it may prove convenient to build a more specialized device. The required structure for these systems will appear from the description below. It is not described with reference to any particular programming language, but rather may be implemented in various ways. The embodiments can thus be implemented using a variety of programming languages.

[0209] Embodiments of the present disclosure include a processor (e.g., a microprocessor, a microcontroller, for use with a computer (such as a digital signal processor, Data-programmed logic, programmable logic devices (e.g., field programmed with FPGAs or other programmable logic devices For use with programmable logic, discrete components, integrated circuit mechanisms (e.g. (e.g., an application specific integrated circuit (ASIC)), or any other device, including any combination thereof. The present invention may be embodied in many different forms, including but not limited to, means. In exemplary embodiments of the present disclosure, an OCT probe, a 2D imaging or 3D imaging system, and Some or all of the processing of data collected using computer-based and processor-based systems may be performed by a The set of instructions is implemented as a set of computer program instructions that The program is converted into an executable form, stored in a computer readable medium, and executed by an operating system. The query response is executed by a microprocessor under the control of the operating system. and the input data may include training sets, image masks, and other methods disclosed herein. The inputs and outputs are converted into processor understandable instructions suitable for generating the inputs and outputs. The computer program logic implementing all or part of the functionality described above may be The following formats are included: code format, computer executable format, and various intermediate formats (e.g., assembler, compiler, (including, but not limited to, formats produced by a compiler, linker, or locator) The source code may be embodied in a variety of forms, including but not limited to the operating system. Various programming languages ​​are provided for use with the operating system or environment. Language (e.g. object code, assembly language, or Python, Perl, High-level languages ​​such as Go, FORTRAN, C, C++, JAVA, or HTML It may include a series of computer program instructions implemented in either a The source code may define and use a variety of data structures and communication messages. Source code is in a form that can be executed by a computer (e.g., via an interpreter). or the source code may be modified (e.g. by a translator, assembler, or compiler) It can be converted into a computer-executable format (via Pyra).

[0210] Various embodiments described herein, or components or portions thereof, may be implemented using software. Many different versions of software, firmware, and / or hardware, or modules thereof The present invention can be implemented in various embodiments. The software code or dedicated control hardware used in the present invention does not limit the present invention. For example, the embodiments described hereinabove are not, for example, conventional techniques or objects. Any suitable component such as .NET, SQL, or MySQL using object-oriented techniques. The present invention can be implemented in computer software using any suitable computer programming language. do.

[0211] Computer software and other computer-implemented instruction programming languages can be translated into machine code by a compiler or assembler before execution, and / or can be directly translated at run-time by the interpreter. Examples include ARM, MIPS, and x86, and examples of high level languages ​​include Ada, BASIC, C, C++, C#, COBOL, Fortran, LUA, Clojure, Java, Examples of scripting languages ​​include Lisp, Pascal, and Object Pascal. Bourne script, JavaScript, Python, Ruby, P Includes HP and Perl.

[0212] The operation and behavior of the embodiments may be implemented using actual software code or dedicated hardware components. The absence of such specific reference is not intended to be construed as limiting the present invention. It is possible, since one of ordinary skill in the art can easily and with only moderate effort and without undue experimentation, A software implementation of the present disclosure may be designed based on the description in the specification; This is because it is clearly understood that the hardware can be controlled.

[0213] Various machine learning systems, deep learning neural networks, 3D neural networks Neural Networks, Convolutional Neural Networks, 2D Neural Networks, N Layered neural network, feedforward neural network, feedforward Word Network, Feedback Word Network, Radial Basis Function Neural Network Network, Kohonen self-organizing neural network, recurrent neural network recurrent neural network (RNN), modular neural network, deep learning Learning networks, artificial intelligence-based systems and frameworks, and Related neural networks such as combinations of the above can be used.

[0214] The various machine learning system software and methods described herein Other computer functions supported include .NET, C, C++, Python, C♯, and Matlab programs. programming modules and tools in any suitable computer programming language Use traditional, functional, or object-oriented techniques For example, various machine learning systems can be implemented in computer software. The information may be stored in a computer readable medium, such as a RAM, a ROM, a secondary storage device, or another The machine learning system may be implemented using a software module stored in the method. One or more processing cores (e.g., CPU, GPU, and / or AI accelerator cores) of the system ) then executes the software module to generate the Functions (e.g., network 107, encoders 111-114, learning coach 110, etc.) can be implemented.

[0215] Computer programs are stored in semiconductor memory devices (e.g. RAM, ROM, PRO M, EEPROM, or Flash programmable RAM), magnetic memory devices (e.g. diskette or fixed disk), optical memory devices (e.g., CD-ROM), P Tangible storage media such as C cards (e.g., PCMCIA cards) or other memory devices in any form (e.g., source code form, computer executable form, or intermediate form) Computer programs can be fixed permanently or temporarily in various communication technologies. The signal may be fixed in any form to a signal that can be transmitted to a computer using any of the techniques available. These communication technologies include analog technology, digital technology, optical technology, and wireless technology. (e.g. Bluetooth), networking technologies, and Internetworking These include, but are in no way limited to, computer programs. The document may be stored on a removable storage medium (e.g., a package ( You may distribute the software in any format, such as a shrink wrapped software, or It may also be preloaded into the computer system (e.g., in the system ROM or on a fixed disk). and from the server or bulletin board to a communication system (e.g., the Internet or World Wide Web). It can also be distributed via the Internet.

[0216] Hardware logic (programs) implementing all or part of the functionality described herein. (including programmable logic used in conjunction with a programmable logic device) They can be designed using traditional manual methods, or through computer-aided design (CAD), Hardware description language (e.g., VHDL or AHDL), or PLD programming language Electronically, using various tools such as language (e.g., PALASM, ABEL, or CUPL) It can also be designed, captured, simulated, or documented.

[0217] Programmable logic is a semiconductor memory device (e.g., RAM, ROM, PRO M, EEPROM, or Flash programmable RAM), magnetic memory devices (e.g. a hard disk (e.g., diskette or fixed disk), an optical memory device (e.g., CD-ROM), or may be fixed, either permanently or temporarily, on a tangible storage medium such as another memory device. Programmable logic is a combination of analog, digital, optical, and wireless technologies. (e.g. Bluetooth), networking technologies, and Internet working using any of a variety of communication technologies, including but not limited to Programmable logic can be attached to a signal that can be transmitted to a computer. Removable storage media (e.g., packaged software) carrying printed or electronic documentation It may be distributed as software or as part of a computer system (e.g., as part of system ROM). The software can be preloaded onto a computer (such as a hard disk or a fixed disk) or can be downloaded to the communication system from a server or bulletin board. It may also be distributed via a public system (e.g., the Internet or the World Wide Web). Cut.

[0218] Various examples of suitable processing modules are discussed in more detail below. When a module is connected to a network, the module is adapted to perform a particular data processing or data transmission task. Generally, the preferred embodiment refers to software, hardware, or firmware that performs the In the module, a command, a resistance change, a voltage change, a current change, a guide wire Various types of data such as base probe data, intravascular pressure data, ratios, indices and other information of interest are Suitable software routines, programs and methods for receiving, transforming, distributing and processing data of any type. ROM or other memory resident applications.

[0219] The computers and computer systems described herein acquire data: The software applications used to process, store and / or communicate The method may include operably linking a computer readable medium, such as a memory for storing Such memory may be used in conjunction with a computer or a Understand that a system can be internal, external, remote, or local. It is possible.

[0220] Memory may also be any type of storage device, including, but not limited to, a hard disk, an optical disk, a floppy disk, Disks, DVDs (digital versatile discs), CDs (compact discs), memories tick, flash memory, ROM (read only memory), RAM (random access memory) memory), DRAM (Dynamic Random Access Memory), PROM (Programmable ROM), EEPROM (Extended Erasable Programmable Read Only Memory), and / or other similar computer programmable "includes any means for storing software or other instructions, including a data-readable medium" can be done.

[0221] Generally, computer-implemented methods and systems that are applied in conjunction with the embodiments of the present disclosure described herein. The readable memory medium is any suitable medium capable of storing instructions that are executed by the programmable device. Any memory medium may be included in the method steps described herein, if applicable. The steps are embodied as instructions stored on one or more computer-readable memory media. These instructions can be written in a variety of programming languages, including C++, C, Java, etc. Various programming languages ​​and / or methods that may be applied to create instructions according to embodiments of the present disclosure. It may be software embodied in other kinds of software programming languages. Cut.

[0222] The storage medium may be non-transitory or may include a non-transitory device. Thus, a non-transitory storage medium or device may include a tangible device, This means that a device may change its physical state but has a concrete physical form. Thus, for example, non-transient means that the data will not change even if there is a change in this state. This refers to the fact that the vice remains tangible.

[0223] The aspects, embodiments, features, and examples of the present disclosure are considered to be illustrative in all respects and are not to be construed as limiting the present disclosure. is not intended to limit the scope of the present disclosure, the scope of which is defined only by the claims. Other embodiments, modifications, and uses are within the spirit and scope of the claimed disclosure. It will be apparent to one skilled in the art without departing from the scope of the present invention.

[0224] The use of headings and paragraphs in this application is not meant to limit the disclosure. may be applied to any aspect, embodiment, or feature of the present disclosure. Only patent claims using the phrase "means for" are to be interpreted under 35 U.S.C. ...112, 6 U.S.C. It is intended that the invention be In such cases, such claims should not be construed under 35 U.S.C. § 112. Any limitations from the specification shall be deemed to be included in the claims unless such limitations are expressly included in the claims. Nor is it intended that any of the foregoing be read into the claims of this application.

[0225] Where values ​​or ranges of values ​​are given, the endpoints of each value and the ranges given and between them are The values ​​will remain within the teachings of the present disclosure unless any different range is specifically stated. However, it can be increased or decreased by 20%.

[0226] Throughout this application, compositions are referred to as having, including, or comprising certain components. Where stated or a process has, includes, or comprises certain process steps When described as "a" or "an" it is understood that the composition of the present teachings consists essentially of or consists of the recited components. and the process of the present teachings is essentially the process recited. It is contemplated that the method may comprise any of the process steps or may comprise the process steps listed.

[0227] In this application, an element or component may be included in a recited list of elements or components. When referring to being selected from and / or from a list, the element or component refers to the It can be any one of the elements or components listed, and the elements or components listed It should be understood that may be selected from the group consisting of two or more of the members. Furthermore, the elements and / or features of the compositions, devices, or methods described herein may be , whether expressly or implicitly stated herein, deviates from the spirit and scope of the present teachings. It should be understood that the components can be combined in various ways without any need for modification.

[0228] The terms "include", "includes", "including", The use of "have," "has," or "having" shall not be construed as limiting the scope of the invention. Unless otherwise specified, the terms and conditions of the present application should generally be understood to be open-ended and non-limiting. .

[0229] Use of the singular herein includes the plural (and vice versa) unless otherwise specified. Further, unless the context clearly indicates otherwise, the singular forms "a," "an," and "the" are used interchangeably. "The" includes the plural. Additionally, use of the term "about" is intended to provide a quantification. When preceding a value, the present teachings also include the specific quantitative value itself, unless otherwise specified.

[0230] The order of steps or order for performing certain actions may be changed without departing from the spirit and scope of the present teachings. It should be understood that the number of steps is not critical, so long as the number of steps is not critical. The actions can be performed simultaneously.

[0231] Where a range or list of values ​​is provided, the value between the upper and lower limits of that range or list of values Each intervening value is individually contemplated and is treated as if each value were specifically recited herein. Further, any smaller amounts between and including the upper and lower limits of a given range are encompassed within the present disclosure. Small ranges are contemplated and encompassed within the present disclosure. A list of example values ​​or ranges is provided for any given It is not intended to exclude other values ​​or ranges between and including the upper and lower limits of the range.

[0232] The figures and descriptions of the disclosure may be omitted for clarity, while other elements may be omitted to provide a clear understanding of the disclosure. It will be understood that the present invention has been simplified to show elements that are relevant for It will be appreciated, however, that these and other factors may be desirable. However, such elements are known in the art and may be incorporated herein by reference in their entirety. In order not to facilitate a better understanding, a discussion of such elements is not provided herein. It should be understood that the figures are presented for illustrative purposes and not as structural diagrams. Omitted details and modifications or alternative embodiments are within the knowledge of one of ordinary skill in the art. do.

[0233] In certain aspects of the present disclosure, to provide an element or structure, or to A single component may be replaced by multiple components to perform a given function. It is understood that multiple components may be replaced with a single component. Such substitutions may be used to implement certain embodiments of the present disclosure. Except where not possible, such substitutions are considered to be within the scope of the present disclosure.

[0234] The examples presented herein are intended to illustrate possible and specific embodiments of the present disclosure. It is intended that the examples be primarily for illustration of the present disclosure for those skilled in the art. It is understood that the same is true of these figures or the specification without departing from the spirit of the present disclosure. There may be variations on the operations described in. For example, in certain cases The method steps or actions may be performed or executed in a different order or the actions may be , can be added, deleted or modified.

Claims

1. acquiring a set of image data including a frame of polar images; The image is annotated such that each annotated region or feature becomes a ground truth annotation. Annotating one or more regions or features of interest in each polar image of the image data set. And, The set of annotated polar images is used to generate a machine learning system (MLS). training a neural network for each of the regions in each polar image; Each is identified by a class, inputting polar image data into the trained neural network; displaying a predicted output image including color-coded regions, each color corresponding to a class; A method for evaluating a coronary artery, comprising:

2. The method of claim 1 , wherein each image includes a plurality of image data elements relating to the coronary arteries.

3. The annotating step comprises annotating a group of pixels or a two-dimensional boundary that defines the feature of interest. A graphical user interface including user controls for selecting Item 1. The method according to item 1.

4. Training the neural network comprises: The cost function is then used to iterate over the predicted output of the MLS to the ground truth input. The method of claim 1 , further comprising comparing the force.

5. The one or more annotated regions or features of interest of each polar image are classified into a type or class. and classifying the neural network of the machine learning system as Training includes: The method of claim 1 further comprising classifying.

6. The types or classes include intima, media, adventitia, lumen, EEL, IEL, plaque, 6. The method of claim 5, wherein the calcium plaque is selected from the group consisting of calcium and calcium plaque.

7. The type or class may be a side branch, a lumen, a guidewire, a stent stratum, stents, jailed stents, bioabsorbable scaffolds (BVS), drug-dissolving Displacement stent (DES), fibrous, blooming artifact, pressure wire, guide Includes wires, lipids, atherosclerotic plaque, stenosis, calcium, calcified plaque, and tissue Calcium, lesions, fat, malapose stents, under-inflated stents, excess The inflated stent, radiopaque marker, branching angle of the arterial tree, calibration element of the probe, doped membrane, light scattering particles, sheath, doped sheath, reference alignment point, diameter scale, radial scale , guide catheter, shaded area, guidewire segment, length, and thickness. The method of claim 5 , wherein the β-amino acid is selected from the group consisting of β-amino acid and β-amino acid.

8. Each data element, each image, and each output is stored in a machine-readable memory in electronic communication with the MLS. The method of claim 1 , wherein the first and second inputs are stored in the memory.

9. In the predicted output image, one or more arc-based metrics, Ca and EEL, The method further identifies the similarity measure of the detected EEL diameter and the detected Ca depth. The method of claim 1 , comprising:

10. The neural network is a convolutional neural network, and a first node The method of claim 1 , wherein the number of input channels of the layer is four.

11. The predicted output image may include one or more indicia indicating boundaries of predicted or classified features. The method of claim 1 , comprising:

12. generating a carpet view using line projection; and fracturing the carpet view.

2. The method of claim 1, further comprising: filtering the predicted output image to reduce noise in the predicted output image. The method described.

13. 2. The method of claim 1, wherein the neural network is a conformal neural network. method.

14. The MLS comprises an AI processor, the AI ​​processor having one or more parallel processing elements. The method of claim 1 comprising:

15. The AI ​​processor comprises N parallel processing elements and further comprises a dedicated AI processor memory. The method of claim 14 comprising:

16. The method of claim 14 , wherein the AI ​​processor is a graphics processing unit.

17. The parallel processing elements include a CUDA core processor, a tensor core processor, and a stream 15. The method of claim 14, wherein the processor is selected from the group consisting of:

18. 16. The method of claim 15, wherein the dedicated AI processor memory is in the range of 8 GB to 64 GB. How to.

19. The dedicated AI processor memory is in the range of 64 GB to 128 GB. The method described.

20. The MLS processing time for classifying user image data is calculated by using the neural network. and reducing the image data by flattening the image data before inputting the image data to the 1. The method according to claim 1.

21. The MLS processing time for classifying user image data is calculated by using the neural network. and reducing the image by resizing or excluding areas of the image before inputting the image into the The method of claim 1 .

22. performing one, two or three circular shifts on one or more of said polar images; The method of claim 1 , further comprising augmenting the training data by:

23. 4. The method of claim 3, further comprising: performing a left-right flip on one or more of the polar images.

1. The method according to claim 1.

24. Prior to inputting the polar image data of the patient into the neural network, The method of claim 1 further comprising removing a subset of the scanlines.

25. Performing lumen detection using an image processing method or a machine learning method, and providing detected lumen boundary data The method of claim 1 , further comprising generating a set of:

26. Instead of using a trained neural network, we use the Uses an image processing pipeline to increase the processing rate of sets of image frames acquired between and performing one or more of side branch detection, guidewire detection, and stent detection. The method of claim 1 , further comprising:

27. The input of the detected lumen boundary data includes a region of interest and a region of interest in the polar image data of the patient.

26. The method of claim 25, reducing latency for classifying cardiac features.

28. One or more image masks for each region or feature of interest identified in the patient image. The method of claim 1 , further comprising generating:

29. Each input image for training or processing patient data is converted into multiple versions and the multiple versions are generated by left-right flips and circular shifts.

1. The method according to claim 1.

30. Housing and a frame grabber for receiving one or more of x-ray image data and intravascular image data; A power supply device; one or more electronic memory storage devices in electrical communication with the power supply; executable on the processor and stored in the one or more electronic memory storage devices. one or more image processing software modules; A computing device comprising a first processor, the power supply and a a computing device in electronic communication with the first processor; One or more software programs stored on said one or more electronic memory storage devices. And, A machine equipped with a neural network that includes one or more machine learning software modules. A machine learning system (MLS), wherein the neural network is a ground truth for polar images. The ground truth annotations are trained using the calculus. a machine learning system, One or more AI processors, the one or more machine learning software modules is executable on the one or more AI processors, the one or more AI processors being dedicated one or more AI processors, each processor having a memory for an interface for transmitting and receiving image data to and from the first processor; It is equipped with The machine learning system is in electronic communication with the power supply. a computing device and the one or more electronic memory storage devices are mounted in the housing. The trained neural network is placed in a substantially real-time and an intravascular imaging and tissue characterization system operable to classify the image data in a time domain. Stem.

31. The housing may be adapted to accommodate an optical coherence tomography imaging system or an intravascular ultrasound imaging system.

31. The method of claim 30, wherein the substantially real time is less than 10 seconds. The system is as follows:

32. The one or more image processing software modules convert the polar intravascular image into a Cartesian image. software to convert Cartesian intravascular images to polar images; A tissue classification overlay that labels regions or features of interest when displayed to the user. software, a lumen detection software module, and an image flattening pre-processing software module. Using the image resizing software module and ground truth data, Image annotation software having a GUI for labeling or marking training images using a a pre-processing software module and a circular shift software module 31. The system of claim 30, comprising one or more of the following:

33. The one or more machine learning software modules include a neural network interface. A luminal contour prediction module, a side branch prediction module, an image resizing module, and a user interface. a user interface and input processing software module, and said neural network an MLS interface software module for controlling and setting parameters of the ML S memory manager software, pre-processing software module, and stent strut and a jailed stent prediction software module. A guidewire prediction software module and an interface for exchanging data with the imaging system.

31. The system of claim 30, comprising one or more of: a face module; and

Images