AI-based risk assessment of medical procedures

Abstract

Systems and methods for clinical decision support are provided. One or more input medical images of an anatomical object of a patient are received. Characteristics of an anatomical object are extracted from one or more input medical images. A risk associated with a medical procedure to be performed on the anatomical object is determined based on the one or more input medical images and the extracted characteristics of the anatomical object using one or more machine learning based risk assessment models. A simulation of the medical procedure is performed on the anatomical object based on the one or more input medical images, the extracted characteristics of the anatomical object, and the risk associated with the medical procedure. One or more clinical decisions associated with the medical process are automatically made based on the risk associated with the medical process and the results of the simulation. And outputting one or more clinical decisions.

Description

Technical Field

[0001] This invention relates generally to AI-based risk assessment of medical procedures, and more particularly to AI-based risk assessment of stent underexpansion for clinical decision support. Background Art

[0002] Risk assessment associated with medical procedures is important for providing support for clinical decision-making. For example, PCI (percutaneous coronary intervention) is a medical procedure often performed as a treatment for CAD (coronary artery disease). CAD is a cardiovascular disease that occurs when plaque builds up in the arteries that supply blood to the heart, causing the arteries to narrow and restrict blood flow. This narrowing of the arteries is called stenosis. PCI can be performed to relieve symptoms such as angina, improve blood flow to the heart, and reduce the risk of heart attack. During PCI, a stent is placed at the site of the stenosis to keep the artery open. To place the stent, it is initially rolled onto a deflated balloon at the tip of the catheter. Once the catheter reaches the stenosis, the balloon inflates, expanding the stent. This expansion forces plaque to build up against the artery wall, widening the artery and restoring blood flow.

[0003] One risk associated with PCI is under-dilation of the stent. Proper stent dilation is important to ensure optimal stent attachment to the arterial wall and to minimize the risk of complications such as stent thrombosis or restenosis. Traditionally, invasive intracoronary imaging techniques such as IVUS (intravascular ultrasound) and OCT (optical coherence tomography) have been proposed to assess the risk of under-dilation. However, IVUS / OCT increases the duration and cost of the procedure and introduces additional risks to the patient. Summary of the Invention

[0004] According to one or more embodiments, systems and methods for clinical decision support are provided. The system receives one or more input medical images of a patient's anatomical object. Characteristics of the anatomical object are extracted from the one or more input medical images. Using one or more machine learning-based risk assessment models, based on the one or more input medical images and the extracted characteristics of the anatomical object, the system determines the risks associated with a medical procedure to be performed on the anatomical object. Based on the one or more input medical images, the extracted characteristics of the anatomical object, and the risks associated with the medical procedure, a simulation of the medical procedure is performed on the anatomical object. Based on the risks associated with the medical procedure and the results of the simulation, one or more clinical decisions associated with the medical procedure are automatically made. One or more clinical decisions are output.

[0005] In one embodiment, one or more clinical decisions include decisions to perform a medical procedure. One or more preoperative medical images of an anatomical object of a patient for which the medical procedure is to be performed are received. Additional characteristics of the anatomical object are extracted from the one or more preoperative medical images. Using one or more machine learning-based risk assessment models, based on the one or more preoperative medical images and the extracted additional characteristics of the anatomical object, an updated risk associated with the medical procedure to be performed on the anatomical object is determined. Based on the one or more preoperative medical images, the extracted additional characteristics of the anatomical object, and the updated risk associated with the medical procedure, an additional simulation of the medical procedure is performed on the anatomical object. Based on the updated risk associated with the medical procedure and the results of the additional simulation, one or more additional clinical decisions associated with the medical procedure are automatically made. One or more additional clinical decisions are output.

[0006] In one embodiment, a medical procedure is performed based on one or more additional clinical decisions. Post-procedural risks associated with the performed medical procedure can be determined.

[0007] In one embodiment, the following steps are iteratively repeated to optimize the cost function: 1) determining the risks associated with the medical procedure based on the results of the simulation, and 2) performing a simulation of the medical procedure.

[0008] In one embodiment, the anatomical object includes a stenosis. Geometric features associated with the stenosis and plaque features associated with the stenosis are extracted from one or more input medical images.

[0009] In one embodiment, characteristics of a medical procedure are received. Further simulation of the medical procedure is performed on an anatomical object based on these characteristics.

[0010] In one embodiment, the anatomical object includes a stenosis, the medical procedure includes a PCI (percutaneous coronary intervention) procedure to place a stent at the stenosis, and the risks include the risk of insufficient stent expansion.

[0011] In one embodiment, one or more input medical images include at least one of photon-counting computed tomography (CT) images, X-ray angiography images, intravascular ultrasound images, or optical coherence tomography (OCT) images.

[0012] These and other advantages of the present invention will be apparent to those skilled in the art from the following detailed description and accompanying drawings. Attached Figure Description

[0013] Figure 1 A workflow for minimizing the risk of stent underexpansion is illustrated according to one or more embodiments;

[0014] Figure 2A method for clinical decision support according to one or more embodiments is shown;

[0015] Figure 3 A method for updating clinical decision support according to one or more embodiments is illustrated;

[0016] Figure 4 An exemplary artificial neural network is shown that can be used to implement one or more embodiments;

[0017] Figure 5 A convolutional neural network that can be used to implement one or more embodiments is shown;

[0018] Figure 6 A data flow diagram of an embodiment using a generative adversarial network according to one or more embodiments is shown;

[0019] Figure 7 A schematic structure is shown that can be used to implement a recursive machine learning model of one or more embodiments; and

[0020] Figure 8 A high-level block diagram of a computer that can be used to implement one or more embodiments is shown. Detailed Implementation

[0021] This invention generally relates to AI-based risk assessment methods and systems for medical processes used in clinical decision support. Embodiments of the invention are described herein to provide an intuitive understanding of these methods and systems. Digital images typically comprise digital representations of one or more objects (or shapes). Here, the digital representation of an object is generally described in terms of identifying and manipulating the object. This manipulation is a virtual manipulation performed in the memory or other circuitry / hardware of a computer system. Therefore, it should be understood that embodiments of the invention can be performed within a computer system using data stored within the computer system. Furthermore, the reference to pixels in an image herein can be equivalent to voxels in the image, and vice versa.

[0022] Risk assessment associated with medical procedures is important for clinical decision support. For example, a PCI procedure may be performed to place a stent at the site of stenosis to treat CAD. Traditional methods for risk assessment of stent underexpansion involve invasive intracoronary imaging techniques, which leads to increased duration and cost of the procedure and introduces additional risks to the patient. The embodiments described herein provide a non-invasive, AI-based risk assessment method for stent underexpansion. This AI-based risk assessment can be performed prior to the PCI procedure for stent placement, allowing the PCI procedure to be tailored to the patient's specific needs and resulting in better patient outcomes, reduced costs, shorter procedure time, and reduced patient risk.

[0023] Figure 1 A workflow 100 for minimizing the risk of stent underexpansion is illustrated according to one or more embodiments. Workflow 100 includes a phase 102 for pre-PCI decision support and planning and a phase 104 for real-time PCI assistance. Phase 102 for pre-PCI decision support and planning is performed before the PCI procedure for stent placement, and phase 104 for real-time PCI assistance is performed during and after the PCI procedure for stent placement. (The last sentence appears to be incomplete and possibly refers to a different workflow.) Figure 2 and Figure 3 To describe Figure 1 Executed during Phase 102 Figure 2 Method 200 is used for PCI pre-decision support and planning. Figure 3 Method 300 is executed during phase 104 for real-time PCI assistance.

[0024] Figure 2 A method 200 for clinical decision support according to one or more embodiments is illustrated. The steps and sub-steps of method 200 can be performed by one or more suitable computing devices, such as… Figure 8 Computer 802.

[0025] exist Figure 2 Step 202 involves receiving one or more input medical images of the patient's anatomical objects. In one example, such as... Figure 1 As shown in workflow 100, one or more input medical images are PCCT (photon-counting computed tomography) medical images received in step 106. In one embodiment, the anatomical object is a stenosis (i.e., narrowing or blockage of a blood vessel) of the patient. However, the anatomical object can be any other suitable anatomical object of interest to the patient, such as, for example, an organ, blood vessel, bone, tumor, or other abnormality.

[0026] In one embodiment, one or more input medical images include PCCT images. PCCT imaging utilizes a photon counting detector, which can distinguish individual photons and measure their energy levels. Compared to conventional CT (computed tomography), PCCT offers improved spatial resolution, reduced radiation dose, and better material resolution. In one embodiment, one or more input medical images include CT images. However, one or more input medical images may include images of any other suitable modality, such as, for example, MRI (magnetic resonance imaging), US (ultrasound), X-ray, or any other medical imaging modality or combination of medical imaging modalities. One or more input medical images may include 2D (two-dimensional) images and / or 3D (three-dimensional) volumes, and may include a single image or multiple images.

[0027] In one embodiment, optionally, in Figure 2Step 202 also receives medical process characteristics and / or patient characteristics. For example, in the case of PCI, medical process characteristics may include, for example, specific characteristics of stent delivery (e.g., deployment location, balloon type, inflation pressure, pre-stent and post-stent expansion, proximal and / or distal landing sites, etc.), stent properties (e.g., length, size, material properties, type (e.g., open cell or closed cell), maximum diameter at a given pressure, etc.). Patient characteristics may include, for example, demographics, medical history, diagnosis, treatment, test / laboratory results, or any other characteristics of the patient.

[0028] One or more input medical images (and optionally medical process characteristics and / or patient characteristics) can be obtained, for example, from an image acquisition device (e.g., when the image is acquired). Figure 8 The image acquisition device 814) directly receives one or more input medical images, and transmits them from the storage device or memory of the computer system (e.g., ...). Figure 8 The computer 802 loads one or more input medical images (as well as medical process characteristics and / or patient characteristics) from its storage device 812 or memory 810, or from a remote computer system (e.g., Figure 8 The computer (802) receives one or more input medical images (as well as medical process characteristics and / or patient characteristics) to receive. Such a computer system or remote computer system may include one or more patient databases, such as, for example, EHR (Electronic Health Record), EMR (Electronic Medical Record), PHR (Personal Health Record), HIS (Health Information System), RIS (Radiology Information System), PACS (Picture Archiving and Communication System), LIMS (Laboratory Information Management System), or any other suitable database or system.

[0029] exist Figure 2 Step 204 involves extracting features of anatomical objects from one or more input medical images. In one example, such as... Figure 1 As shown in workflow 100, in step 106, the characteristics of the anatomical object are extracted based on AI-based image analysis.

[0030] The characteristics of an anatomical object can include any suitable features of the anatomical object. In one embodiment, where the anatomical object is a stenosis, the characteristics can include geometric features associated with the stenosis and plaque features associated with the stenosis. Geometric features can include anatomical and morphological features, such as, for example, identification of the vascular centerline, identification and assessment of the lumen, identification of the outer wall, etc. Given high-resolution 3D image data of the interior and periphery of the stenosis, PCCT images allow for a more accurate assessment of geometric features, thereby avoiding artifacts such as smearing. Plaque features can include, for example, location, volume / number, type (e.g., calcification, fibrosis, fibrofatty, or nodular), extent, etc. The multi-energy capabilities and high spatial resolution of PCCT provide better characterization of tissue and material composition. In one embodiment, plaque features can include the results of an assessment of pericoronary adipose tissue performed using PCCT images.

[0031] In one embodiment, the characteristics of the anatomical object may optionally include features extracted from radiomics analysis of one or more input medical images. For example, radiomics analysis can be performed by overlaying markers on one or more input medical images to identify the beginning and end of a stenosis, identifying a region of interest around the lesion based on the markers, and extracting radiomics GLCM (gray-level co-occurrence matrix) features from one or more input medical images within the region of interest.

[0032] One or more machine learning-based feature extraction networks can be used to extract features of anatomical objects. The feature extraction network can be implemented using any suitable (e.g., well-known) machine learning-based network. The feature extraction network receives one or more input medical images as input and generates features of the anatomical object as output. The feature extraction network is trained using training data during a prior offline or training phase, according to any suitable (e.g., well-known) method. Once trained, the feature extraction network is applied during the online or training phase, for example, by performing... Figure 2 Step 204.

[0033] exist Figure 2 Step 206 involves determining the risks associated with the medical procedure to be performed on the anatomical object based on extracted characteristics of one or more input medical images and the anatomical object. In one example, such as... Figure 1 As shown in workflow 100, risk is identified in AI-based stent expansion insufficiency risk assessment step 108.

[0034] In one embodiment, the medical procedure is a PCI procedure to place a stent at the narrowing site, and the risks associated with PCI include the risk of insufficient stent expansion. However, the medical procedure and the risks associated with it can include any suitable medical procedure and / or any suitable risk. For example, a risk might include the risk of collateral closure.

[0035] In one embodiment, risk is represented as a risk score. However, risk can also be represented as a binary risk / no risk, a risk probability, a risk classification into multiple categories, or any other suitable manner. One or more machine learning-based risk assessment models are used to determine risk. The risk assessment model can be implemented using any suitable (e.g., well-known) machine learning-based risk assessment model. The risk assessment model receives one or more input medical images, extracted features of anatomical objects, and optional patient features as input, and generates the risk associated with the medical procedure as output.

[0036] The risk assessment model is trained using training data during a previous offline or training phase, according to any suitable (e.g., well-known) method. The training data may include training images labeled to identify observed stent underexpansion, stent thrombosis, and in-stent restenosis. Observed stent underexpansion may be defined, for example, at the frame level or lesion site, automatically based on the lumen diameter or area after PCI (e.g., determined by coronary angiography or IVUS / OCT imaging), and / or defined by the user (e.g., an expert). Labels may be evaluated immediately during or after the PCI procedure, or they may be evaluated long-term (e.g., for in-stent restenosis). In one embodiment, the training data may include synthetic training images showing rare anatomical configurations. For example, a GAN (Generative Adversarial Network) may be used to generate synthetic training images. The synthetic training images may be used to pre-train the risk assessment model. Once trained, the risk assessment model is applied, for example, during an online or training phase, such as performing [a specific procedure]. Figure 2 Step 206.

[0037] exist Figure 2 Step 208 involves performing a simulation of the medical procedure on the anatomical object based on one or more input medical images, extracted characteristics of the anatomical object, and risks associated with the medical procedure. For example, the simulation could involve performing PCI at a stenosis. In one example, in step 110, the simulation of the medical procedure is performed via a mechanical modeling framework for optimizing stent deployment.

[0038] In one embodiment, simulations are performed using one or more mechanical models. The mechanical model is a mathematical model or equation based on natural laws (e.g., biological, physical, and / or chemical laws). The mechanical model receives one or more input medical images, extracted characteristics of the anatomical object, risks associated with the medical procedure, and optional characteristics of the medical procedure as input, and generates a simulation of the medical procedure as output.

[0039] In some embodiments, the simulated medical procedure additionally or alternatively includes passing a stent through the stenosis. If passing the stent through the stenosis is difficult, pre-dilation and / or plaque removal may be performed.

[0040] In one embodiment, an iterative workflow is provided by iteratively repeating steps 206 and 208 to optimize (e.g., minimize) a cost function. Step 206 is iteratively repeated using the results of a simulation of the medical procedure (determined in step 208), and step 208 is iteratively repeated using the risks associated with the medical procedure (determined in step 206). The iterative workflow is provided by varying parameters of the medical procedure. For example, in the case of PCI where a stent is placed at a narrow site, parameters may include different stent deployment locations, different stent types (e.g., size, length, material properties, etc.), varying the number of stents, different procedural settings (e.g., inflation pressure), pre-stent expansion, and post-stent expansion, etc.

[0041] The cost function can be defined based on one or more metrics of interest (e.g., lumen size after stent deployment, stent attachment, etc.) and is optimized as part of an iterative workflow. The cost function can be further defined based on one or more of the following: risks associated with the medical procedure (e.g., risk of stent underexpansion and risk of collateral closure), the degree of edge stripping and perforation, and hemodynamic characteristics after stent deployment (e.g., using a hemodynamic model to estimate the post-PCT FFR (flow reserve score), or using a machine learning-based model to predict the post-PCI FFR value for a given anatomical model after stent deployment).

[0042] In one embodiment, the iterative workflow is further extended by incorporating mechanical modeling of plaque resection. Thus, if a satisfactory risk is not achieved in the initial iterations, plaque resection can be considered and simulated to modify the anatomical properties of the coronary arteries, thereby improving the risk (e.g., the risk of insufficient stent expansion). Therefore, one output could be an indication to perform plaque resection prior to medical procedures.

[0043] exist Figure 2Step 210, based on the risks associated with the medical procedure and the results of simulations, automatically makes one or more clinical decisions related to the medical procedure. These decisions can be made using any suitable (e.g., well-known) method, based on the risks associated with the medical procedure and the results of simulations. One or more clinical decisions could be whether or not to perform the medical procedure. In one example, such as... Figure 1 As shown in workflow 100, the clinical decision support system 112 for PCI determines whether to perform planned PCI 116 or not to perform PCI 114 (and instead perform CAGB (coronary artery bypass surgery) or optimal medical treatment).

[0044] In one embodiment, one or more clinical decisions may also include characteristics of the medical procedure, such as, for example, the deployment location of one or more stents, stent characteristics (length, size, type (e.g., open loop for bifurcation or closed loop for ostial lesions), pre-dilation, inflation pressure during stent deployment and post-dilation, requirements for pre-stent dilation and post-stent dilation, requirements for plaque resection, etc.

[0045] In one embodiment, virtual angiography can also be generated from one or more input medical images (e.g., PCCT images) covering pre- and post-stent implantation scenarios to support clinicians in understanding recommended clinical decisions for taking the best course of action.

[0046] exist Figure 2 Step 212 outputs one or more clinical decisions. For example, this can be done via a display device on a computer system (e.g., Figure 8 One or more clinical decisions are displayed on the I / O of the computer 802 (808), in the memory or storage device of the computer system (e.g., Figure 8 One or more clinical decisions are stored on the memory 810 or storage device 812 of the computer 802, or by transmitting one or more clinical decisions to a remote computer system (e.g., Figure 8 The computer (802) is used to output one or more clinical decisions.

[0047] For example, in one embodiment, where one or more clinical decisions involve decisions to perform a medical procedure, one or more preoperative medical images of the patient are acquired for the purpose of performing the procedure. Additional clinical decisions are made by applying the same network / model, but using one or more preoperative medical images as one or more input medical images. Figure 2 Method 200 was repeated as Figure 3Method 300. For example, if one or more clinical decisions involve scheduling a patient for coronary angiography and PCI, steps 206 and 208 can be iterated to update the risk and simulation using information extracted from CT images acquired during coronary angiography. These CT images can provide additional insights because: 1) coronary angiography allows for more precise analysis of luminal information, and certain changes in anatomy may occur between the acquisition times of one or more input medical images (e.g., PCCT images) and coronary angiography images; and 2) patient characteristics may change over time.

[0048] Because one or more machine learning-based models are used to determine risk (in) Figure 2 In step 206), the updated risk can be determined substantially in real time. However, mechanical modeling can involve longer runtimes and may not be suitable for performing simulations substantially in real time during medical procedures. Therefore, in one embodiment, mechanical modeling can be replaced by a simulation network based on physical information machine learning, which has the same inputs and outputs as the mechanical model.

[0049] Figure 3 A method 300 for updating clinical decision support according to one or more embodiments is illustrated. The steps and sub-steps of method 300 can be performed by one or more suitable computing devices, such as… Figure 8 Computer 802.

[0050] exist Figure 3 Step 302 involves receiving one or more preoperative medical images of the patient for the medical procedure. In one embodiment, the one or more preoperative medical images are X-ray images, IVUS images, and / or OCT images acquired during coronary angiography to guide the performance of PCI. For example, as... Figure 1 As shown in workflow 100, one or more preoperative medical images may be acquired during coronary angiography 118. However, the one or more preoperative medical images may have any other suitable modality or modality. The one or more preoperative medical images may include 2D images and / or 3D volumes, and may include a single image or multiple images.

[0051] One or more preoperative medical images can be obtained, for example, directly from the image acquisition device during image acquisition (e.g., Figure 8 The image acquisition device 814) receives one or more preoperative medical images from the storage device or memory of the computer system (e.g., ...). Figure 8 The computer 802 loads one or more preoperative medical images from its storage device 812 or memory 810, or from a remote computer system (e.g., Figure 8 The computer (802) receives one or more preoperative medical images. Such a computer system or remote computer system may include one or more patient databases.

[0052] exist Figure 3 Step 304 involves extracting additional properties of the anatomical object from one or more preoperative medical images. These additional properties can include any suitable characteristics of the anatomical object. Examples of such properties include... Figure 2 Step 204 describes one or more machine learning-based feature extraction networks that extract additional features from one or more preoperative medical images.

[0053] exist Figure 3 Step 306 involves determining updated risks associated with the medical procedure to be performed on the anatomical object, based on one or more preoperative medical images and extracted additional characteristics of the anatomical object. In one example, such as... Figure 1 As shown in workflow 100, the updated risk is determined in AI-based scaffold expansion insufficiency risk assessment step 120. The updated risk is similar to that in... Figure 2 Step 206 identifies the risks. These can be addressed using methods described above. Figure 2 Step 206 describes one or more machine learning-based risk assessment models to determine the updated risk.

[0054] exist Figure 3 Step 308 involves performing an additional simulation of the medical procedure on the anatomical object based on one or more preoperative medical images, extracted additional features of the anatomical object, and updated risks associated with the medical procedure. In one example, this additional simulation is performed via the framework for optimizing stent deployment established in step 122. One or more machine learning-based simulation models can be used to perform the additional simulation, which have similar features to those described above. Figure 2 The one or more machine models described in step 208 have the same inputs and outputs. The one or more machine learning-based simulation models can be trained using training data during a previous offline or training phase. The training data may include synthetic data and outputs computed by the one or more machine models; therefore, the one or more machine learning-based simulation models can be alternative models to the one or more machine models.

[0055] In one embodiment, steps 306 and 308 are repeated iteratively to optimize the cost function, as described above. Figure 2 As described in steps 206 and 208.

[0056] exist Figure 3Step 310, based on the updated risk and additional simulation results associated with the medical process, automatically makes one or more additional clinical decisions associated with the medical process. One or more additional clinical decisions can be made using any suitable (e.g., well-known) method, based on the updated risk and additional simulation results associated with the medical process. The one or more additional clinical decisions can be similar to those described above. Figure 2 Step 210 describes one or more clinical decisions. In one example, such as Figure 1 As shown in workflow 100, one or more additional clinical decisions can be made automatically by the clinical decision support system determined by PCI124.

[0057] exist Figure 3 Step 312 outputs one or more additional clinical decisions. For example, this can be done via a display device on a computer system (e.g., [device name missing]). Figure 8 One or more additional clinical decisions are displayed on the I / O 808 of the computer system (e.g., memory or storage device). Figure 8 One or more additional clinical decisions are stored on the memory 810 or storage device 812 of the computer 802, or by transmitting one or more additional clinical decisions to a remote computer system (e.g., Figure 8 The computer (802) is used to output one or more additional clinical decisions.

[0058] In one embodiment, a medical procedure (e.g., PCI) may be performed based on one or more additional clinical decisions.

[0059] In one embodiment, one or more supplemental risk assessment models can be used to determine post-procedure risks associated with a medical procedure. For example, in the case of PCI involving stent placement, post-procedure risks may include risks resulting from insufficient stent expansion, such as in-stent thrombosis or lesion re-occlusion. In-stent thrombosis refers to the formation of a blood clot or thrombus within a stent after its placement in a coronary artery, which can lead to vessel occlusion, impaired blood flow, and adverse clinical outcomes such as myocardial infarction, stent thrombosis, and death. The risk of in-stent thrombosis can be minimized by optimizing stent placement, ensuring adequate lesion preparation, using antiplatelet and anticoagulant drugs, and closely monitoring the patient. One or more machine learning-based supplemental risk assessment models can be used to determine post-procedure risks. Supplemental risk assessment models can be implemented using any suitable (e.g., well-known) machine learning-based models. Machine learning-based supplemental risk assessment models receive medical images (e.g., X-ray angiography, OCT, IVUS) of the patient acquired after the medical procedure as input. For example, such medical images may be acquired immediately after stent placement or via subsequent CT, for example, one year after stent placement. In addition, machine learning-based risk assessment models can also receive blood tests from patients some time after the medical procedure has been performed, which indicate inflammation in the body.

[0060] The embodiments described herein relate to a claimed system and a claimed method. The features, advantages, or alternative embodiments described herein can be assigned to other claims and vice versa. In other words, the system claims and embodiments can be modified using features described or claimed in the context of the corresponding method. In this case, the functional features of the method are implemented by the physical units of the system.

[0061] Furthermore, certain embodiments described herein relate to methods and systems for utilizing trained machine learning models, as well as methods and systems for providing trained machine learning models. Features, advantages, or alternative embodiments described herein can be assigned to other claimed objects, and vice versa. In other words, the claims and embodiments for providing trained machine learning models can be modified by features described or claimed in the context of utilizing trained machine learning models, and vice versa. Specifically, the dataset used in the methods and systems for utilizing trained machine learning models can have the same properties and characteristics as the corresponding dataset used in the methods and systems for providing trained machine learning models, and the trained machine learning model provided by the corresponding method and system can be used in the methods and systems for utilizing trained machine learning models.

[0062] Generally, trained machine learning models mimic human cognitive functions associated with other human thought processes. Specifically, through training on training data, machine learning models are able to adapt to new environments and detect and infer patterns. Another term for a “trained machine learning model” is a “trained function.”

[0063] Typically, the parameters of a machine learning model can be tuned through training. Specifically, supervised training, semi-supervised training, unsupervised training, reinforcement learning, and / or active learning can be used. Furthermore, representation learning (an alternative term is "feature learning") can be used. Specifically, the parameters of a machine learning model can be iteratively adapted through several training steps. Specifically, within training, a certain cost function can be minimized. Specifically, within the training of a neural network, the backpropagation algorithm can be used.

[0064] In particular, machine learning models, such as, for example, in Figure 1 Steps 106, 108, 120, and 126 utilize machine learning-based models / networks, in Figure 2 Steps 204 and 206 utilize machine learning-based models / networks, and in Figure 3 Steps 304 and 306 utilize machine learning-based models / networks, which may include, for example, neural networks, support vector machines, decision trees, and / or Bayesian networks, and / or the machine learning model may be based on, for example, k-means clustering, Q-learning, genetic algorithms, and / or association rules. Specifically, the neural network may be, for example, a deep neural network, a convolutional neural network, or a convolutional deep neural network. Furthermore, the neural network may be, for example, an adversarial network, a deep adversarial network, and / or a generative adversarial network.

[0065] Figure 4 An embodiment of an artificial neural network 400 is shown, which can be used to implement one or more machine learning models described herein. Alternative terms for “artificial neural network” are “neural network,” “artificial neural network,” or “neural network.”

[0066] The artificial neural network 400 includes nodes 420…432 and edges 440…442, where each edge 440…442 is a directed connection from a first node 420…432 to a second node 420…432. Typically, the first node 420…432 and the second node 420…432 are different nodes 420…432, but they may also be the same. For example, in… Figure 4In the diagram, edge 440 is a directed connection from node 420 to node 423, and edge 442 is a directed connection from node 430 to node 432. The edges 440...442 from the first node 420...432 to the second node 420...432 are also represented as the "incoming edges" of the second node 420...432 and the "outgoing edges" of the first node 420...432.

[0067] In this embodiment, nodes 420…432 of the artificial neural network 400 can be arranged in layers 410…413, wherein the layers may include an inherent order introduced by edges 440…442 between nodes 420…432. Specifically, edges 440…442 can only exist between adjacent node layers. In the illustrated embodiment, the input layer 410 includes only nodes 420,…,422 without any incoming edges, the output layer 413 includes only nodes 431,432 without any outgoing edges, and hidden layers 411,412 are located between the input layer 410 and the output layer 413. Typically, the number of hidden layers 411,412 can be arbitrarily chosen. The number of nodes 420…422 in the input layer 410 is typically related to the number of input values ​​of the neural network, and the number of nodes 431,432 in the output layer 413 is typically related to the number of output values ​​of the neural network.

[0068] Specifically, (real) numbers can be assigned as values ​​to each node 420...432 of the neural network 400. Here, x (n) i This represents the value of the i-th node 420...432 in the nth layer 410...413. The values ​​of nodes 420...422 in the input layer 410 are equivalent to the input values ​​of the neural network 400, and the values ​​of nodes 431 and 432 in the output layer 413 are equivalent to the output values ​​of the neural network 400. Furthermore, each edge 440...442 may include a real number weight, specifically a real number within the interval [-1, 1] or the interval [0, 1]. Here, w... (m，n) i，j This represents the weight of the edge between the i-th node (420...432) of layer m (410...413) and the j-th node (420...432) of layer n (410...413). Furthermore, for the weight w... (n，n+1) i，j Define the abbreviation w (n) i，j .

[0069] Specifically, in order to calculate the output value of neural network 400, the input value is propagated through the neural network. Specifically, the values ​​of nodes 420...432 in the (n+1)th layer 410...413 can be calculated based on the values ​​of nodes 420...432 in the nth layer 410...413 using the following formula.

[0070] Here, function f is the transfer function (another term is "activation function"). Known transfer functions are step functions, sigmoid (S-type) functions (e.g., logic functions, generalized logic functions, hyperbolic tangent functions, arctangent functions, error functions, smooth step functions), or rectifier functions. Transfer functions are primarily used for normalization purposes.

[0071] Specifically, the value is propagated layer by layer through the neural network, wherein the value of the input layer 410 is given by the input of the neural network 400, wherein the value of the first hidden layer 411 can be calculated based on the value of the input layer 410 of the neural network, wherein the value of the second hidden layer 412 can be calculated based on the value of the first hidden layer 411, and so on.

[0072] To set the value of the edge Training data must be used to train the neural network 400. Specifically, the training data includes the training input data and the training output data (denoted as t). i For the training step, neural network 400 is applied to the training input data to generate computational output data. Specifically, the training data and the computational output data include multiple values, the number of which is equal to the number of nodes in the output layer.

[0073] Specifically, the comparison between the calculated output data and the training data is used to recursively adapt the weights within the neural network (backpropagation algorithm). Specifically, the weights change according to the following formula: Where γ is the learning rate, and δ (n) j It can be based on δ (n+1) j Recursively calculate as If the (n+1)th layer is not an output layer, and If the (n+1)th layer is the output layer 413, where f' is the first derivative of the activation function, and t (n+1) j It is the comparison training value of the j-th node of the output layer 413.

[0074] A convolutional neural network (CNN) is a neural network that uses convolution operations instead of general matrix multiplication in at least one layer (a so-called "convolutional layer"). Specifically, a convolutional layer performs a dot product of one or more convolutional kernels with the input data / image of the convolutional layer, where the entries for the one or more convolutional kernels are parameters or weights adapted through training. Specifically, Frobenius inner product and ReLU activation functions can be used. A CNN may include additional layers such as pooling layers, fully connected layers, and normalization layers.

[0075] By using convolutional neural networks, input images can be processed very efficiently because convolution operations based on different kernels can extract various image features. This allows relevant image features to be found during training by adapting the weights of the convolution kernels. Furthermore, due to weight sharing within the convolution kernels, fewer parameters need to be trained, preventing overfitting during training and allowing for faster training or more layers in the network, thus improving network performance.

[0076] Figure 5 An embodiment of a convolutional neural network 500 is illustrated, which can be used to implement one or more machine learning models described herein. In the illustrated embodiment, the convolutional neural network 500 includes an input node layer 510, convolutional layers 511, pooling layers 513, fully connected layers 514, and an output node layer 516, as well as hidden node layers 512 and 514. Alternatively, the convolutional neural network 500 may include several convolutional layers 511, several pooling layers 513, and several fully connected layers 515, as well as other types of layers. The order of the layers can be arbitrarily chosen; typically, the fully connected layer 515 is used as the last layer before the output layer 516.

[0077] Specifically, within the convolutional neural network 500, nodes 520, 522, and 524 of node layers 510, 512, and 514 can be viewed as arranged as a d-dimensional matrix or a d-dimensional image. Specifically, in the two-dimensional case, the values ​​of nodes 520, 522, and 524 indexed by i and j in the nth node layer 510, 512, and 514 can be represented as x(n)[i, j]. However, the arrangement of nodes 520, 522, and 524 in a node layer 510, 512, and 514 has no impact on the computations performed within the convolutional neural network 500, as these are given solely by the structure and weights of the edges.

[0078] Convolutional layer 511 is a connection layer between the preceding node layer 510 (with node value x(n-1)) and the following node layer 512 (with node value x(n)). Specifically, convolutional layer 511 is characterized by the structure and weights of the input edges forming a certain number of kernel convolution operations. Specifically, the structure and weights of the edges in convolutional layer 511 are chosen such that the value x(n) of node 522 in the following node layer 512 is computed as a convolution x(n) = K * x(n-1) based on the value x(n-1) of node 520 in the preceding node layer 510, where convolution * is defined in the two-dimensional case as...

[0079] Here, the kernel K is a d-dimensional matrix (in this embodiment, a two-dimensional matrix), which is typically small compared to the number of nodes 520, 522 (e.g., a 3×3 or 5×5 matrix). Specifically, this means that the weights of the edges in convolutional layer 511 are not independent, but are chosen such that they produce the convolution equation. In particular, for a 3×3 kernel, there are only 9 independent weights (each entry in the kernel matrix corresponds to one independent weight), independent of the number of nodes 520, 522 in the preceding node layer 510 and the following node layer 512.

[0080] Typically, convolutional neural networks 500 use node layers 510, 512, and 514 with multiple channels, especially since multiple kernels are used in convolutional layer 511. In these cases, the node layer can be considered as a (d+1)-dimensional matrix (the first dimension indexes the channels). The action of convolutional layer 511 is a two-dimensional example, defined as follows: in Corresponding to channel a of the previous node layer 510, Corresponding to the b-th channel of the subsequent node layer 512, and K a，b This corresponds to one of the kernels. If convolutional layer 511 acts on the preceding node layer 510 with channel A and outputs the following node layer 512 with channel B, then there exists an A·B independent d-dimensional kernel K. a，b .

[0081] Typically, activation functions are used in convolutional neural networks 500. In this embodiment, ReLU (an abbreviation for "Rectified Linear Unit") is used, where R(z) = max(0, z), making the action of convolutional layer 511 in the two-dimensional example...

[0082] Other activation functions can also be used, such as ELU (an abbreviation for "Exponential Linear Unit"), LeakyReLU, Sigmoid, Tanh, or Softmax.

[0083] In the illustrated embodiment, the input layer 510 comprises 36 nodes 520 arranged in a two-dimensional 6×6 matrix. The first hidden node layer 512 comprises 72 nodes 522 arranged in two two-dimensional 6×6 matrices, each of which is the result of convolution of the input layer values ​​with a 3×3 kernel within the convolutional layer 511. Equivalently, the nodes 522 of the first hidden node layer 512 can be interpreted as arranged in a three-dimensional 2×6×6 matrix, where the first dimension corresponds to the channel dimension.

[0084] The advantage of using convolutional layers 511 is that by implementing local connectivity patterns between nodes in adjacent layers, especially by having each node connect only to a small region of nodes in the previous layer, the spatial local correlation of the input data can be utilized.

[0085] The pooling layer 513 is a connection layer between the previous node layer 512 (with node value x(n-1)) and the next node layer 514 (with node value x(n)). Specifically, the pooling layer 513 can be characterized by the edge structure and weights and activation functions forming a pooling operation based on a nonlinear pooling function f. For example, in the two-dimensional case, the value x(n) of node 524 of the next node layer 514 can be calculated based on the value x(n-1) of node 522 of the previous node layer 512 as follows:

[0086] In other words, by using the pooling layer 513, the number of nodes 522 and 524 can be reduced by repositioning the number of neighboring nodes 522 of d1·d2 in the front node layer 512, where a single node 522 in the back node layer 514 is calculated as a function of the value of that number of neighboring nodes. Specifically, the pooling function f can be a maximum function, an average function, or an L2 norm function. Specifically, for the pooling layer 513, the weights of the incoming edges are fixed and are not modified through training.

[0087] The advantage of using pooling layer 513 is that it reduces the number of nodes 522 and 524 and the number of parameters. This results in a reduction in the computational cost of the network and controls overfitting.

[0088] In the illustrated embodiment, pooling layer 513 is a maximum pooling layer, replacing four adjacent nodes with only one node, which is the maximum value among the four adjacent nodes. Maximum pooling is applied to each d-dimensional matrix of the preceding layer; in this embodiment, maximum pooling is applied to each of the two two-dimensional matrices, thereby reducing the number of nodes from 72 to 18.

[0089] Typically, the last layer of a convolutional neural network 500 is a fully connected layer 515. The fully connected layer 515 is the connection layer between the preceding node layer 514 and the following node layer 516. The fully connected layer 513 is characterized by the fact that most, in particular all, of the edges exist between nodes 514 of the preceding node layer 514 and nodes 516 of the following node layer, and the weights of each of these edges can be adjusted individually.

[0090] In this embodiment, the nodes 524 of the preceding node layer 514 of the fully connected layer 515 are displayed as a two-dimensional matrix and are also displayed as unrelated nodes (indicated as a row of nodes, where the number of nodes is reduced for better presentation). This operation is also called "flattening". In this embodiment, the number of nodes 526 in the following node layer 516 of the fully connected layer 515 is less than the number of nodes 524 in the preceding node layer 514. Alternatively, the number of nodes 526 can be equal to or greater than the number of nodes 524.

[0091] Furthermore, in this embodiment, a Softmax activation function is used within the fully connected layer 515. By applying the Softmax function, the sum of the values ​​of all nodes 526 in the output layer 516 is 1, and all values ​​of all nodes 526 in the output layer 516 are real numbers between 0 and 1. In particular, if the input data is classified using a convolutional neural network 500, the values ​​of the output layer 516 can be interpreted as the probability that the input data falls into one of the different categories.

[0092] Specifically, a convolutional neural network 500 can be trained based on the backpropagation algorithm. To prevent overfitting, regularization methods can be used, such as dropping nodes 520, ..., 524, random pooling, the use of artificial data, and weight decay based on L1 or L2 norm or maximum norm constraints.

[0093] According to one aspect, a machine learning model may include one or more Residual Networks (ResNets). Specifically, a ResNet is an artificial neural network that includes at least one skip connection for skipping at least one layer of the artificial neural network. Specifically, a ResNet may be a convolutional neural network that includes one or more skip connections that skip one or more convolutional layers respectively. According to some examples, a ResNet may be represented as an m-layer ResNet, where m is the number of layers in the corresponding architecture, and according to some examples, it may take values ​​of 34, 50, 101, or 152. According to some examples, such an m-layer ResNet may include (m-2) / 2 skip connections respectively.

[0094] Skipping connections can be viewed as bypassing, which bypasses one or more bypassed layers and feeds the output of a preceding layer directly to one or more layers following the bypassed layers. Instead of directly fitting the desired mapping, the bypassed layer will have to fit a residual mapping that "balances" the directly fed output.

[0095] Fitting residual mappings is computationally easier to optimize than directed mappings. More importantly, this mitigates the vanishing / exploding gradient problem during optimization when training machine learning models: if a bypassed layer encounters such a problem, its contribution can be skipped by regularizing the output directly fed into it. Therefore, one benefit of using ResNets is the ability to train deeper networks.

[0096] Generative adversarial models (GA models) consist of a generator function and a discriminator function. The generator function creates synthetic data, and the discriminator function distinguishes between synthetic and real data. By training the generator function and / or discriminator function, on the one hand, the generator function is configured to create synthetic data that is incorrectly classified as real by the discriminator function; on the other hand, the discriminator function is configured to distinguish between real data and synthetic data generated by the generator function. In game theory, GA models can be interpreted as zero-sum games. The training of the generator function and / or discriminator function is primarily based on minimizing a cost function.

[0097] By using a GA (Generalized Aspect-Oriented) model, synthetic data with the same characteristics as the training dataset can be generated based on the training dataset. The training of a GA model can be based on unannotated data (unsupervised learning), thus requiring minimal effort.

[0098] Figure 6 The diagram illustrates a data flow graph according to one or more embodiments, specifically using a generative adversarial network to create synthetic output data G(x)608 based on input data x 602. This synthetic output data is indistinguishable from the real output data y 604. The synthetic output data G(x)608 has the same structure as the real output data y 604, but its content is not derived from real-world data.

[0099] The Generative Adversarial Network (GAN) consists of a jointly trained generator function G606 and a classifier function C610. The generator function G606 is tasked with providing realistic synthetic output data G(x)608 based on the input data x602, while the classifier function C610 is tasked with distinguishing between the real output data y604 and the synthetic output data G(x)608. Specifically, the output of the classifier function C610 is a real number between 0 and 1, corresponding to the probability that the input value is real data, such that an ideal classifier function would compute an output value C(y)614≈1 for the real data y604 and an output value C(G(x))612≈0 for the synthetic data G(x)608.

[0100] During training, the parameters of the generator function G 606 are adapted so that the synthetic output data G(x) 608 has the same characteristics as the real output data y 604, making the classifier function C 610 unable to distinguish between real and synthetic data. Simultaneously, the parameters of the classifier function C 610 are adjusted so that it distinguishes between real and synthetic data in the best possible way. Here, training relies on pairs including input data x 602 and the corresponding real output data y 604. Within a single training step, the generator function G 606 is applied to the input data x 602 to generate the synthetic output data G(x) 608. Furthermore, the classifier function C 610 is applied to the real output data y 604 to generate the first classification result C(y) 614. Additionally, the classifier function C 610 is applied to the synthetic output data G(x) 608 to generate the second classification result C(G(x)) 612.

[0101] The parameters of the adaptive generation function G 606 and the classifier function C 610 are based on minimizing the cost function using the backpropagation algorithm. In this embodiment, the cost function K of the classifier function C 610 is... C It is K C ∝-BCE(C(y), 1)-BCE(C(Gx)), 0), where BCE represents the binary cross-entropy, defined as BCE(z, z′)=z′·log(z)+(1-z′)·log(1-z). By using this cost function, incorrectly classifying true output data as synthetic (indicated by C(y)≈0) and incorrectly classifying synthetic output data as true (indicated by C(G(x))612≈1) both increase the cost function K to be minimized. C Furthermore, the cost function K of the generator function G 606 G It is K G∝-BCE(C(G(x)), 1)=-log(C(G(x)). Using this cost function, correctly classified synthetic output data (indicated by C(G(x))612≈0) results in a cost function K to be minimized. G The increase.

[0102] Specifically, a recursive machine learning model is a machine learning model whose output depends not only on the input values ​​and the parameters of the machine learning model adapted by the training process, but also on the hidden state vector, which is based on the previous inputs used for the recursive machine learning model. In particular, a recursive machine learning model may include additional stored states or additional structures that contain time delays or include feedback loops.

[0103] Specifically, the underlying structure of a recurrent machine learning model can be a neural network, which can be represented as a recurrent neural network. Such a recurrent neural network can be described as an artificial neural network, where the connections between nodes form a directed graph along a time series. Specifically, a recurrent neural network can be interpreted as a directed acyclic graph. Specifically, the recurrent neural network can be a finite-spiking recurrent neural network or an infinite-spiking recurrent neural network (where a finite-spiking network can be unfolded and replaced with a strictly feedforward neural network, and an infinite-spiking network cannot be unfolded and replaced with a strictly feedforward neural network).

[0104] Specifically, training recurrent neural networks can be based on the BPTT algorithm (an acronym for "backward propagation in time"), the RTRL algorithm (an acronym for "real-time recursive learning"), and / or a genetic algorithm.

[0105] By using a recursive machine learning model, input data including sequences of variable length can be used. In particular, this means that the method cannot be used only for a fixed number of input datasets (and requires different training for each additional number of input datasets used as input), but can be used for any number of input datasets. This means that, independent of the number of input datasets contained in different sequences, the entire training dataset can be used for training, and the training data is not reduced to training data corresponding to a certain number of consecutive input datasets.

[0106] Figure 7 The recursive representation 702 and the expanded representation 704 illustrate the schematic structure of a recursive machine learning model F, which can be used to implement one or more machine learning models described herein. The recursive machine learning model takes several input datasets x, x1, ..., x... N Take 706 as input and create a corresponding output dataset y,y1,...,y N708. Furthermore, the output depends on the so-called hidden vectors h,h1,...,h N 710, which implicitly includes information about the input dataset previously used as input to the recursive machine learning model F712. This is achieved by using these hidden vectors h, h1, ..., h... N 710, the order of the input dataset can be utilized.

[0107] In a single step of processing, the recursive machine learning model F712 will process the hidden vector h created in the previous step. n-1 and input dataset x n As input, the recursive machine learning model F generates updated hidden vectors hn and output dataset y within this step. n As output. In other words, a processing step calculates (y n h n )=F(x n h n-1 Alternatively, by splitting the recursive machine learning model F712 into a part F(y) that computes the output data and a part F(h) that computes the hidden vectors, one processing step computes y. n =F (y) (x n ,h n-1 ) and h n =F (h) (x n h n-1 For the first processing step, h0 can be randomly selected or filled with all entries that are zero. The parameters of the recurrent machine learning model F 712, previously trained on the training dataset, remain unchanged between different processing steps.

[0108] In particular, the output data and hidden vectors of the processing step depend on all the previous input datasets used in the preceding steps. n =F (y) (x n F (h) (x n-1 h n-2 )) and h n =F(h)(x n F (h) (x n-1 h n-2 )).

[0109] The systems, apparatus, and methods described herein can be implemented using digital circuitry or using one or more computers employing well-known computer processors, memory units, storage devices, computer software, and other components. Typically, a computer includes a processor for executing instructions and one or more memories for storing instructions and data. A computer may also include or be coupled to one or more mass storage devices, such as one or more disks, internal hard disks and removable disks, magneto-optical disks, optical disks, etc.

[0110] The systems, apparatus, and methods described herein can be implemented using computers operating in a client-server relationship. Typically, in such a system, the client computer is located remotely from the server computer and interacts via a network. The client-server relationship can be defined and controlled by computer programs running on the respective client and server computers.

[0111] The systems, apparatus, and methods described herein can be implemented within a network-based cloud computing system. In such a system, a server or other processor connected to the network communicates with one or more client computers via the network. For example, a client computer can communicate with the server through a web browser application residing and operating on the client computer. The client computer can store data on the server and access the data over the network. The client computer can send data requests or online service requests to the server over the network. The server can perform the requested service and provide data to the client computers(s). The server can also transmit data suitable for causing the client computer to perform specific functions (e.g., perform calculations, display specific data on a screen, etc.). For example, the server can send requests suitable for causing the client computer to perform one or more steps or functions of the methods and workflows described herein, including... Figure 1-3 One or more steps or functions. Some steps or functions of the methods and workflows described herein include... Figure 1-3 One or more steps or functions may be performed by a server or another processor in a web-based cloud computing system. Some steps or functions of the methods and workflows described herein include... Figure 1-3 One or more steps can be performed by a client computer in a web-based cloud computing system. The steps or functions of the methods and workflows described herein include... Figure 1-3 One or more steps can be performed by servers and / or client computers in a web-based cloud computing system in any combination.

[0112] The systems, apparatus, and methods described herein can be implemented using a computer program product tangibly contained in an information carrier, such as a non-transitory machine-readable storage device, for execution by a programmable processor; and the methods and workflow steps described herein include... Figure 1-3 One or more steps or functions can be implemented using one or more computer programs that can be executed by such a processor. A computer program is a set of computer program instructions that can be used directly or indirectly in a computer to perform a specific activity or produce a specific result. Computer programs can be written in any form of programming language, including compiled or interpreted languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0113] Figure 8 A high-level block diagram of an example computer 802, which can be used to implement the systems, apparatus, and methods described herein, is depicted. Computer 802 includes a processor 804 operatively coupled to a data storage device 812 and a memory 810. Processor 804 controls the overall operation of computer 802 by executing computer program instructions that define such operation. The computer program instructions may be stored in the data storage device 812 or other computer-readable medium and loaded into memory 810 when execution is required. Therefore, Figure 1-3 The methods and workflow steps or functions can be defined by computer program instructions stored in memory 810 and / or data storage device 812, and controlled by processor 804 that executes the computer program instructions. For example, the computer program instructions can be implemented as computer executable code programmed by those skilled in the art to perform... Figure 1-3 The methods and workflow steps or functions. Therefore, by executing computer program instructions, processor 804 performs... Figure 1-3 The computer 802 may include methods and workflow steps or functions. It may also include one or more network interfaces 806 for communicating with other devices over a network. The computer 802 may also include one or more input / output devices 808 that enable users to interact with the computer 802 (e.g., monitor, keyboard, mouse, speakers, buttons, etc.).

[0114] Processor 804 may include both general-purpose and special-purpose microprocessors, and may be the sole processor of computer 802 or one of multiple processors. For example, processor 804 may include one or more central processing units (CPUs). Processor 804, data storage device 812, and / or memory 810 may include one or more application-specific integrated circuits (ASICs) and / or one or more field-programmable gate arrays (FPGAs), supplemented or incorporated therein.

[0115] Both data storage device 812 and memory 810 include tangible, non-transitory computer-readable storage media. Both data storage device 812 and memory 810 may include high-speed random access memory, such as dynamic random access memory (DRAM), static random access memory (SRAM), dual data rate synchronous dynamic random access memory (DDR RAM), or other random access solid-state memory devices, and may include non-volatile memory, such as one or more disk storage devices, such as internal hard disks and removable disks, magneto-optical disk storage devices, optical disk storage devices, flash memory devices, semiconductor memory devices, such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM), digital universal disc read-only memory (DVD-ROM), or other non-volatile solid-state storage devices.

[0116] Input / output device 808 may include peripherals such as printers, scanners, displays, etc. For example, input / output device 808 may include display devices such as cathode ray tube (CRT) or liquid crystal display (LCD) monitors for displaying information to a user, keyboards, and pointing devices such as mice or trackballs, through which the user can provide input to computer 802.

[0117] Image acquisition device 814 can be connected to computer 802 to input image data (e.g., medical images) into computer 802. It is possible to implement image acquisition device 814 and computer 802 as a single device. Image acquisition device 814 and computer 802 can also communicate wirelessly via a network. In a possible embodiment, computer 802 can be remotely located relative to image acquisition device 814.

[0118] Any or all systems, apparatuses, and methods discussed herein may be implemented using one or more computers (such as computer 802).

[0119] Those skilled in the art will recognize that actual computer or computer system implementations may have other structures and may include other components, and Figure 8It is a high-level representation of some components of this computer for illustrative purposes.

[0120] Individuals with male or female gender identities are also included in the term, independent of the use of grammatical terms.

[0121] The foregoing detailed description should be understood as illustrative and exemplary in every respect, not restrictive, and the scope of the invention disclosed herein is not determined by the specific embodiments, but by the claims interpreted in the full breadth permitted by patent law. It should be understood that the embodiments shown and described herein are merely illustrative of the principles of the invention, and various modifications can be made by those skilled in the art without departing from the scope and spirit of the invention. Various other combinations of features can be implemented by those skilled in the art without departing from the scope and spirit of the invention.

[0122] The following is a list of non-limiting illustrative embodiments disclosed herein:

[0123] Illustrative Example 1. A computer-implemented method comprising: receiving one or more input medical images of an anatomical object of a patient; extracting characteristics of the anatomical object from the one or more input medical images; determining, using one or more machine learning-based risk assessment models, a risk associated with a medical procedure to be performed on the anatomical object based on the one or more input medical images and the extracted characteristics of the anatomical object; performing a simulation of the medical procedure on the anatomical object based on the one or more input medical images, the extracted characteristics of the anatomical object, and the risk associated with the medical procedure; automatically making one or more clinical decisions associated with the medical procedure based on the risk associated with the medical procedure and the results of the simulation; and outputting one or more clinical decisions.

[0124] Illustrative Example 2. A computer-implemented method according to Illustrative Example 1, wherein one or more clinical decisions include decisions to perform a medical procedure, the method further comprising: receiving one or more preoperative medical images of an anatomical object of a patient for performing the medical procedure; extracting additional characteristics of the anatomical object from the one or more preoperative medical images; determining, using one or more machine learning-based risk assessment models, an updated risk associated with the medical procedure to be performed on the anatomical object based on the one or more preoperative medical images and the extracted additional characteristics of the anatomical object; performing an additional simulation of the medical procedure on the anatomical object based on the one or more preoperative medical images, the extracted additional characteristics of the anatomical object, and the updated risk associated with the medical procedure; automatically making one or more additional clinical decisions associated with the medical procedure based on the updated risk associated with the medical procedure and the results of the additional simulation; and outputting one or more additional clinical decisions.

[0125] Illustrative Example 3. The computer-implemented method according to Illustrative Example 2 further includes: performing a medical procedure based on one or more additional clinical decisions.

[0126] Illustrative Example 4. The computer-implemented method according to Illustrative Example 3 further includes: determining post-procedure risks associated with the performed medical procedure.

[0127] Illustrative Example 5. The computer-implemented method according to any one of the illustrative Examples 1-4 further includes: iteratively repeating the steps of: 1) determining the risk associated with the medical process based on the results of the simulation, and 2) performing a simulation of the medical process to optimize the cost function.

[0128] Illustrative Example 6. A computer-implemented method according to any one of the illustrative Examples 1-5, wherein the anatomical object includes a stenosis, and extracting characteristics of the anatomical object from one or more input medical images includes: extracting geometric characteristics associated with the stenosis and plaque characteristics associated with the stenosis from one or more input medical images.

[0129] Illustrative Example 7. The computer-implemented method according to any one of the illustrative examples 1-6 further includes receiving characteristics of a medical process, wherein performing a simulation of the medical process on the anatomical object based on one or more input medical images, extracted characteristics of the anatomical object, and risks associated with the medical process includes: further performing a simulation of the medical process on the anatomical object based on the characteristics of the medical process.

[0130] Illustrative Example 8. A computer-implemented method according to any of the illustrative Examples 1-7, wherein the anatomical object includes a stenosis, the medical procedure includes a PCI (percutaneous coronary intervention) procedure to place a stent at the stenosis, and the risks include the risk of insufficient stent expansion.

[0131] Illustrative Example 9. A computer-implemented method according to any one of the illustrative Examples 1-8, wherein the one or more input medical images include at least one of photon-counting computed tomography images, X-ray angiography images, intravascular ultrasound images, or optical coherence tomography images.

[0132] Illustrative Example 10. An apparatus comprising: means for receiving one or more input medical images of an anatomical object of a patient; means for extracting characteristics of the anatomical object from the one or more input medical images; means for determining, using one or more machine learning-based risk assessment models, a risk associated with a medical procedure to be performed on the anatomical object based on the one or more input medical images and the extracted characteristics of the anatomical object; means for performing a simulation of the medical procedure on the anatomical object based on the one or more input medical images, the extracted characteristics of the anatomical object, and the risk associated with the medical procedure; means for automatically making one or more clinical decisions associated with the medical procedure based on the risk associated with the medical procedure and the results of the simulation; and means for outputting one or more clinical decisions.

[0133] Illustrative Example 11. The apparatus according to Illustrative Example 10, wherein one or more clinical decisions include decisions to perform a medical procedure, the apparatus further comprising: means for receiving one or more preoperative medical images of an anatomical object of a patient for performing the medical procedure; means for extracting additional characteristics of the anatomical object from the one or more preoperative medical images; means for determining an updated risk associated with the medical procedure to be performed on the anatomical object based on the one or more preoperative medical images and the extracted additional characteristics of the anatomical object using one or more machine learning-based risk assessment models; means for performing an additional simulation of the medical procedure on the anatomical object based on the one or more preoperative medical images, the extracted additional characteristics of the anatomical object, and the updated risk associated with the medical procedure; means for automatically making one or more additional clinical decisions associated with the medical procedure based on the updated risk associated with the medical procedure and the results of the additional simulation; and means for outputting one or more additional clinical decisions.

[0134] Illustrative Example 12. The device according to any one of the illustrative examples 10-11 further includes: means for performing a medical procedure based on one or more additional clinical decisions.

[0135] Illustrative Example 13. The device according to any one of the illustrative examples 10-12 further includes: means for determining post-procedure risks associated with the performed medical procedure.

[0136] Illustrative Example 14. The apparatus according to any one of the illustrative examples 10-13 further includes means for iteratively repeating the following steps: 1) determining the risk associated with the medical process based on the results of a simulation, and 2) performing a simulation of the medical process to optimize the cost function.

[0137] Illustrative Example 15. A non-transitory computer-readable storage medium including instructions that, when executed by a computer, cause the computer to perform operations including: receiving one or more input medical images of an anatomical object of a patient; extracting characteristics of the anatomical object from the one or more input medical images; determining, using one or more machine learning-based risk assessment models, a risk associated with a medical procedure to be performed on the anatomical object based on the one or more input medical images and the extracted characteristics of the anatomical object; performing a simulation of the medical procedure on the anatomical object based on the one or more input medical images, the extracted characteristics of the anatomical object, and the risk associated with the medical procedure; automatically making one or more clinical decisions associated with the medical procedure based on the risk associated with the medical procedure and the results of the simulation; and outputting one or more clinical decisions.

[0138] Illustrative Example 16. A non-transitory computer-readable storage medium according to Illustrative Example 15, wherein the one or more clinical decisions include decisions to perform a medical procedure, the operation further includes: receiving one or more preoperative medical images of an anatomical object of a patient for performing the medical procedure; extracting additional characteristics of the anatomical object from the one or more preoperative medical images; determining, using one or more machine learning-based risk assessment models, an updated risk associated with the medical procedure to be performed on the anatomical object based on the one or more preoperative medical images and the extracted additional characteristics of the anatomical object; performing an additional simulation of the medical procedure on the anatomical object based on the one or more preoperative medical images, the extracted additional characteristics of the anatomical object, and the updated risk associated with the medical procedure; automatically making one or more additional clinical decisions associated with the medical procedure based on the updated risk associated with the medical procedure and the results of the additional simulation; and outputting one or more additional clinical decisions.

[0139] Illustrative Example 17. A non-transitory computer-readable storage medium according to any of the embodiments of illustrative Examples 15-16, wherein the anatomical object includes a stenosis, and extracting characteristics of the anatomical object from one or more input medical images includes: extracting geometric characteristics associated with the stenosis and plaque characteristics associated with the stenosis from one or more input medical images.

[0140] Illustrative Example 18. The non-transitory computer-readable storage medium according to any of the illustrative examples 15-17 further includes receiving characteristics of a medical procedure, wherein performing a simulation of the medical procedure on the anatomical object based on one or more input medical images, extracted characteristics of the anatomical object, and risks associated with the medical procedure includes: further performing a simulation of the medical procedure on the anatomical object based on the characteristics of the medical procedure.

[0141] Illustrative Example 19. A non-transitory computer-readable storage medium according to any one of Illustrative Examples 15-18, wherein the anatomical object includes a stenosis, the medical procedure includes a PCI (percutaneous coronary intervention) procedure to place a stent at the stenosis, and the risk includes the risk of insufficient stent expansion.

[0142] Illustrative Example 20. A non-transitory computer-readable storage medium according to any one of Illustrative Examples 15-19, wherein the one or more input medical images include at least one of photon-counting computed tomography images, X-ray angiography images, intravascular ultrasound images, or optical coherence tomography images.

Claims

1. A computer-implemented method, comprising: Receive one or more input medical images of the patient's anatomical objects; Extracting features of anatomical objects from one or more input medical images; Using one or more machine learning-based risk assessment models, based on extracted characteristics of one or more input medical images and anatomical objects, the risks associated with the medical procedures to be performed on the anatomical objects are determined. Based on one or more input medical images, extracted characteristics of the anatomical object, and risks associated with the medical procedure, a simulation of the medical procedure is performed on the anatomical object. Based on the risks associated with the medical process and the results of simulations, one or more clinical decisions associated with the medical process are made automatically. as well as Output one or more clinical decisions.

2. The computer-implemented method according to claim 1, wherein, One or more clinical decisions include decisions to perform medical procedures, and the method further includes: Receive one or more preoperative medical images of the anatomical objects of the patient for performing a medical procedure; Extract additional characteristics of anatomical objects from one or more preoperative medical images; Using one or more machine learning-based risk assessment models, based on additional characteristics extracted from one or more preoperative medical images and anatomical objects, the updated risks associated with the medical procedures to be performed on the anatomical objects are determined. Additional simulations of the medical procedure are performed on the anatomical object based on one or more preoperative medical images, extracted additional characteristics of the anatomical object, and updated risks associated with the medical procedure. Based on updated risks and the results of additional simulations associated with the medical process, automatically make one or more additional clinical decisions associated with the medical process; and Output one or more additional clinical decisions.

3. The computer-implemented method according to claim 2 further includes: Medical procedures are performed based on one or more additional clinical decisions.

4. The computer-implemented method according to claim 3 further includes: Identify post-procedure risks associated with the medical procedures performed.

5. The computer-implemented method according to claim 1, further comprising: Repeat the following steps iteratively: 1) Determine the risks associated with the medical process based on the simulation results, and 2) Perform a simulation of the medical process to optimize the cost function.

6. The computer-implemented method of claim 1, wherein the anatomical object includes a stenosis, and extracting characteristics of the anatomical object from one or more input medical images includes: Extract geometric features associated with stenosis and plaque features associated with stenosis from one or more input medical images.

7. The computer-implemented method of claim 1, further comprising receiving characteristics of a medical procedure, wherein performing a simulation of the medical procedure on the anatomical object based on one or more input medical images, extracted characteristics of the anatomical object, and risks associated with the medical procedure comprises: Furthermore, based on the characteristics of medical procedures, simulations of medical procedures are performed on anatomical objects.

8. The computer-implemented method of claim 1, wherein the anatomical object includes a stenosis, the medical procedure includes a PCI (percutaneous coronary intervention) procedure to place a stent at the stenosis, and the risk includes the risk of insufficient stent expansion.

9. The computer-implemented method of claim 1, wherein the one or more input medical images include at least one of photon-counting computed tomography (CT) images, X-ray angiography images, intravascular ultrasound images, or optical coherence tomography (OCT) images.

10. An apparatus comprising: A device for receiving one or more input medical images of a patient's anatomical objects; A device for extracting characteristics of anatomical objects from one or more input medical images; A device for determining the risks associated with a medical procedure to be performed on an anatomical object by using one or more machine learning-based risk assessment models, based on extracted characteristics of one or more input medical images and anatomical objects. A device for simulating a medical procedure on an anatomical object based on one or more input medical images, extracted characteristics of the anatomical object, and risks associated with the medical procedure; Device for automatically making one or more clinical decisions related to a medical process based on risks associated with the process and the results of simulations; as well as A device for outputting one or more clinical decisions.

11. The device of claim 10, wherein the one or more clinical decisions include decisions to perform a medical procedure, the device further comprising: Device for receiving one or more preoperative medical images of a patient's anatomical object for performing a medical procedure; Device for extracting additional characteristics of an anatomical object from one or more preoperative medical images; A device for determining updated risks associated with a medical procedure to be performed on an anatomical object using one or more machine learning-based risk assessment models, based on additional characteristics extracted from one or more preoperative medical images and anatomical objects. A device for performing additional simulations of a medical procedure on an anatomical object based on one or more preoperative medical images, extracted additional characteristics of the anatomical object, and updated risks associated with the medical procedure. Device for automatically making one or more additional clinical decisions associated with a medical process based on updated risks and the results of additional simulations related to the medical process; as well as A device for outputting one or more additional clinical decisions.

12. The device according to claim 11, further comprising: Device for performing medical procedures based on one or more additional clinical decisions.

13. The device according to claim 12, further comprising: Device for determining post-procedure risks associated with a medical procedure being performed.

14. The device according to claim 10, further comprising: An apparatus for iteratively repeating the following steps: 1) determining the risks associated with a medical procedure based on the results of a simulation, and 2) performing a simulation of the medical procedure to optimize the cost function.

15. A non-transitory computer-readable storage medium including instructions that, when executed by a computer, cause the computer to perform operations, the operations including: Receive one or more input medical images of the patient's anatomical objects; Extracting features of anatomical objects from one or more input medical images; Using one or more machine learning-based risk assessment models, based on extracted characteristics of one or more input medical images and anatomical objects, the risks associated with the medical procedures to be performed on the anatomical objects are determined. Based on one or more input medical images, extracted characteristics of the anatomical object, and risks associated with the medical procedure, a simulation of the medical procedure is performed on the anatomical object. Based on the risks associated with the medical process and the results of simulations, one or more clinical decisions associated with the medical process are made automatically. as well as Output one or more clinical decisions.

16. The non-transitory computer-readable storage medium of claim 15, wherein the one or more clinical decisions include decisions to perform a medical procedure, the operation further comprising: Receive one or more preoperative medical images of the anatomical objects of the patient for performing a medical procedure; Extract additional characteristics of anatomical objects from one or more preoperative medical images; Using one or more machine learning-based risk assessment models, based on additional characteristics extracted from one or more preoperative medical images and anatomical objects, the updated risks associated with the medical procedures to be performed on the anatomical objects are determined. Additional simulations of the medical procedure are performed on the anatomical object based on one or more preoperative medical images, extracted additional characteristics of the anatomical object, and updated risks associated with the medical procedure. Based on updated risks and the results of additional simulations associated with the medical process, one or more additional clinical decisions associated with the medical process are made automatically. as well as Output one or more additional clinical decisions.

17. The non-transitory computer-readable storage medium of claim 15, wherein the anatomical object includes a stenosis, and extracting characteristics of the anatomical object from one or more input medical images includes: Extract geometric features associated with stenosis and plaque features associated with stenosis from one or more input medical images.

18. The non-transitory computer-readable storage medium of claim 15, further comprising receiving characteristics of a medical procedure, wherein performing a simulation of the medical procedure on the anatomical object based on one or more input medical images, extracted characteristics of the anatomical object, and risks associated with the medical procedure comprises: Furthermore, based on the characteristics of medical procedures, simulations of medical procedures are performed on anatomical objects.

19. The non-transitory computer-readable storage medium of claim 15, wherein the anatomical object includes a stenosis, the medical procedure includes a PCI (percutaneous coronary intervention) procedure to place a stent at the stenosis, and the risk includes the risk of insufficient stent expansion.

20. The non-transitory computer-readable storage medium of claim 15, wherein the one or more input medical images comprise at least one of photon-counting computed tomography (CT) images, X-ray angiography images, intravascular ultrasound images, or optical coherence tomography (OCT) images.

Images