System and method for joint physiological condition estimation from medical images

Abstract

The present disclosure relates to a system and method for joint abnormality detection and physiological condition estimation based on medical images. The method may include receiving a medical image acquired by an image acquisition device via at least one processor. The medical image includes an anatomical structure. The method may further include applying a joint learning model based on the medical image to jointly determine the abnormal condition of the anatomical structure together with the physiological parameters via the at least one processor. The joint learning model satisfies a predetermined constraint relationship between the abnormal condition and the physiological parameters. In this way, based on a medical image containing blood vessels, the joint learning model can be used to jointly predict vascular abnormalities on the anatomical structure together with functional physiological parameters, and significantly improve the consistency between the prediction results of the vascular abnormality and the prediction results of the physiological parameters of the blood vessels.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is based upon and claims priority from U.S. Provisional Application No. 63 / 133,754, filed on January 4, 2020, which is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure relates to the technical field of medical image processing and analysis, and more particularly, to the technical field of medical image processing and analysis for joint abnormality detection and physiological condition estimation. Background Art

[0004] Coronary artery disease (CAD) is a critical condition characterized by narrowing of the coronary artery lumen, leading to myocardial ischemia. Early and effective assessment of myocardial ischemia is crucial for optimal treatment planning, thereby improving quality of life and reducing healthcare costs. Various imaging techniques provide effective methods for assessing the structure of the coronary arteries, including the extent of vascular lesions. However, anatomical vascular lesions do not necessarily lead to significant functional hemodynamic changes, and accurate diagnosis and optimal treatment based solely on anatomical vascular lesions cannot be achieved.

[0005] Lesions may cause stenosis or occlusion of the patient's coronary arteries, resulting in insufficient blood supply to the myocardium or even myocardial ischemia. On the other hand, the fractional flow reserve (FFR) or instantaneous wave-free rate (iFR) or other measurements can be used to assess physiological conditions. For example, FFR can include the ratio between the average distal lesion and aortic blood pressure under hyperemic conditions. iFR can be measured without the need for hyperemia and is becoming an alternative indicator. These indicators, including FFR and iFR, can be measured invasively in a catheterization laboratory using a pressure guidewire. However, these methods are invasive and cause pain to the patient. Therefore, an image-based coronary artery disease analysis system is desired to assist doctors in diagnosis and treatment during routine surgery.

[0006] Machine learning has become a useful tool for modeling complex systems in various fields. Recent advances in machine learning have enabled its application to CAD analysis, both for anatomical abnormality assessment and physiological parameter estimation. However, existing approaches address these two tasks separately by training separate models for each task, resulting in two different models being used to handle the two tasks (e.g., anatomical abnormality assessment and physiological parameter estimation). As a result, the two models fail to account for the correlation between the two tasks, and the predictions from the two models may be inconsistent. Furthermore, because each model is trained on its own labeled dataset for the corresponding task, the training process may not fully utilize the data annotations. Summary of the Invention

[0007] The present disclosure provides a system and method for joint abnormality detection and physiological condition estimation based on medical images. The system and method can jointly predict abnormalities in anatomical structures (such as vascular abnormalities) and functional physiological parameters of the anatomical structures based on medical images containing anatomical structures (such as blood vessels), and significantly improve the consistency between the predicted results of abnormalities in anatomical structures and the predicted results of functional physiological parameters of the anatomical structures.

[0008] According to a first embodiment of the present disclosure, a method for joint abnormality detection and physiological condition estimation based on medical images is provided. The method may include receiving a medical image acquired by an image acquisition device via at least one processor. The medical image includes an anatomical structure. The method may further include applying a joint learning model based on the medical image to jointly determine the abnormality of the anatomical structure together with the physiological parameter via the at least one processor. The joint learning model satisfies a predetermined constraint relationship between the abnormality and the physiological parameter. The joint learning model may adopt a multi-task learning framework, or a serial model, or a learning framework that integrates the two tasks of abnormality detection and physiological parameter trajectory.

[0009] According to a second aspect of the present disclosure, a system for joint abnormality detection and physiological condition estimation based on medical images is provided. The system may include a communication interface configured to receive the medical image acquired by an image acquisition device. The medical image includes an anatomical structure. The system further includes at least one processor. The at least one processor may be configured to apply a joint learning model based on the medical image to jointly determine abnormalities of the anatomical structure and physiological parameters. The joint learning model satisfies a predetermined constraint relationship between the abnormality and the physiological parameters.

[0010] According to a third aspect of the present disclosure, a non-transitory computer-readable storage medium having computer-executable instructions stored thereon is provided. When the computer-executable instructions are executed by a processor, a method for joint abnormality detection and physiological condition estimation from medical images may be performed. The method may include receiving a medical image acquired by an image acquisition device. The medical image includes an anatomical structure. The method may further include applying a joint learning model based on the medical image to jointly determine an abnormality of the anatomical structure together with a physiological parameter. The joint learning model satisfies a predetermined constraint relationship between the abnormality and the physiological parameter.

[0011] The disclosed systems and methods can jointly predict anatomical abnormalities (vascular abnormalities, such as vascular lesions) and functional physiological parameters of the anatomical structure using a joint learning model based on medical images. By applying this joint learning model, the disclosed systems and methods can significantly improve the consistency between the prediction results of anatomical abnormalities and the prediction results of anatomical physiological parameters.

[0012] The foregoing general description and the following detailed description are exemplary and explanatory only and are not intended to restrict the invention, as claimed. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In the drawings, which are not necessarily drawn to scale, the same reference numerals may describe similar components in different views. Similar reference numerals with letter suffixes or different letter suffixes may represent different examples of similar components. The accompanying drawings generally illustrate various embodiments by way of example and not by way of limitation, and together with the description and claims, serve to illustrate the disclosed embodiments. Such embodiments are illustrative and exemplary and are not intended to be exhaustive or exclusive embodiments of the present method, apparatus, system, or non-transitory computer-readable medium having instructions for implementing the method.

[0014] Figure 1A A flowchart illustrating an exemplary method for performing joint abnormality detection and physiological condition estimation according to an embodiment of the present disclosure is shown.

[0015] Figure 1B A flowchart illustrating another exemplary method for performing joint abnormality detection and physiological condition estimation according to an embodiment of the present disclosure is shown.

[0016] Figure 2 A schematic diagram illustrating an exemplary structure of a joint learning model for joint lesion detection and physiological parameter prediction according to an embodiment of the present disclosure.

[0017] Figure 3 A schematic diagram illustrating another exemplary structure of a joint learning model for joint lesion detection and physiological parameter prediction according to an embodiment of the present disclosure.

[0018] Figure 4 A schematic diagram illustrating a deviation loss term in a loss function for training a joint learning model according to an embodiment of the present disclosure is shown.

[0019] Figure 5 A flowchart illustrating a training process of a joint learning model according to an embodiment of the present disclosure is shown.

[0020] Figure 6 The flowchart shows an analysis process for lesion detection and physiological parameter prediction for a medical image containing a vascular tree according to an embodiment of the present disclosure.

[0021] Figure 7 A schematic block diagram of a system for performing joint abnormality detection and physiological condition estimation according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0022] Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings.

[0023] In the present disclosure, anatomical structures may include blood vessels or any other suitable anatomical structures. Without loss of generality, the following description is performed using blood vessels (such as coronary arteries) as examples of anatomical structures, but the description may also be applied to any other anatomical structures.

[0024] According to the present disclosure, an abnormal vascular condition refers to a condition that differs from the normal tissue structure of a vessel, such as an abnormal vascular condition. For example, an abnormal vascular condition may refer to a condition related to a vascular lesion or sub-health, such as, but not limited to, at least one of vascular plaque, myocardial bridge, and hemangioma formed in the vascular anatomy. In another example, an abnormal vascular condition may also refer to the introduction of a foreign object into the vascular tissue, such as, but not limited to, at least one of a stent, catheter, or guidewire.

[0025] Figure 1A FIG. 1 is a flow chart showing an exemplary method for performing combined abnormality detection and physiological condition estimation according to an embodiment of the present disclosure. Figure 1A As shown, the method may begin at step 101, where a medical image is acquired by an image acquisition device. The medical image may include blood vessels. For example, the medical image may include images generated using various imaging modalities, including but not limited to CT, magnetic resonance imaging (MRI), ultrasound, optical coherence tomography (OCT), and the like.

[0026] At step 102 , a joint learning model may be applied based on the medical image to determine the abnormal condition of the blood vessel together with the physiological parameters. The joint learning model satisfies a predetermined constraint relationship between the abnormal condition of the blood vessel and the physiological parameters of the blood vessel.

[0027] For example, a joint learning model can be constructed for the two tasks of vascular anomaly detection and vascular physiological parameter prediction, enabling both vascular anomaly detection and physiological parameter estimation. This joint learning model can simultaneously detect vascular anomalies and predict physiological parameters from medical images. This joint learning model can adopt a multi-task learning framework, a serial model, or a learning framework that integrates the two tasks.

[0028] This joint learning model can be jointly trained using multiple sets of annotated data, significantly reducing overfitting on a single task and enabling better representations for both tasks. Consequently, the trained joint learning model generalizes better to new test data. Furthermore, through joint training, the task that more readily learns certain useful features can provide guidance to other tasks that struggle to capture these features, enabling them to better learn these features. This offers several advantages over traditional approaches. First, when training a model solely for lesion detection or FFR prediction, there is a risk of overfitting the noise on a single task. On the other hand, training a model for both tasks allows the model to learn better representations that do not overfit to the noise of a single task. As a result, the trained model generalizes better to new test data. Second, training a model on a single task can result in the failure to capture certain features, possibly because the task interacts with these features in complex ways. By training the joint model, the additional task that more readily learns these features can guide the task to better learn these features. Third, as described later, the joint learning model disclosed herein can be explicitly designed to generate consistent lesion and FFR predictions. Other advantages may also be achieved.

[0029] Furthermore, the joint learning model can be trained not only with fully labeled data, but also with partially labeled or even unlabeled data, thereby fully utilizing various data annotations. Taking physiological parameters as an example, a fully labeled example of a physiological parameter includes a pullback curve indicating the physiological parameter at each location along the path. Physiological parameter labels can also be generated using other methods. A partially labeled example of a physiological parameter may include one or more physiological parameter values ​​measured at one or several locations. An unlabeled example of a physiological parameter might be a healthy vessel without lesions (which can be considered normal), while a vessel with severe stenosis (e.g., greater than 90% occlusion) can be considered a vessel with significantly affected vascular function. Taking a lesion as an example of an abnormality, lesion labels may include, but are not limited to, the following: labeled lesion location, bounding box, start and end points, lesion type (calcified / non-calcified / mixed; or fragile / stable lesion), lesion severity (area reduction rate, diameter reduction rate, etc.), and so on.

[0030] In step 102, a predetermined constraint relationship between the abnormal vascular condition and the physiological parameters of the blood vessels can be embedded in various ways. For example, the predetermined constraint relationship between the abnormal vascular condition and the physiological parameters can be explicitly modeled (or embedded) within the model structure of the joint learning model, thereby forcibly avoiding any potential conflicts between the model's abnormality detection results and the physiological parameter prediction results. For another example, during the training process of the joint learning model, a loss function can be used to penalize deviations from the predetermined constraint relationship between the abnormal vascular condition and the physiological parameters, so that the joint learning model learns a better representation that can reduce or even eliminate such deviations (or conflicts).

[0031] Figure 1B FIG. 1 is a flow chart showing a method for performing combined abnormality detection and physiological condition estimation according to an embodiment of the present disclosure. Figure 1B As shown, the method starts at step 101, where a medical image containing a blood vessel is received. At step 102a, the centerline of the blood vessel is extracted from the medical image.

[0032] In step 102b, based on the medical image, a sequence of feature maps is extracted from at least one location along the centerline of the blood vessel as a joint feature. For example, the sequence of feature maps may include at least one feature at at least one location along the centerline of the blood vessel. The feature map may be generated by an external model or provided as input. The feature map may be directly learned in the joint learning model. The corresponding features at each location may include at least one of a manually designed feature, an original image, an autonomously learned feature, or a blood vessel mask. For example, the feature at the location may be a manually designed feature or an autonomously learned feature extracted from a cardiovascular image or a coronary artery mask.

[0033] In step 102c, based on the extracted joint features, the abnormal condition of the blood vessel is determined along with physiological parameters of the blood vessel. In some embodiments, the abnormal condition of the blood vessel may include at least one of an abnormality mask at at least one location along the centerline of the blood vessel, representative coordinates of the abnormality (e.g., the center coordinates of the abnormality), the type of abnormality, the location of the abnormality, and the size / severity of the abnormality. The physiological parameters of the blood vessel include at least one of the following vascular function parameters in the physiological function state at at least one location along the centerline of the blood vessel: blood pressure, blood velocity, blood flow, wall shear stress, fractional flow reserve (FFR), microcirculatory resistance index (IMR), instantaneous waveform-free ratio (iFR), relative FFR change parameter compared to adjacent locations, or relative iFR change parameter.

[0034] The following describes the method for combined abnormality detection and physiological condition estimation disclosed herein using coronary artery CT images as an example of medical images, vascular lesions as an example of vascular abnormalities, and FFR as an example of physiological parameters. However, it should be noted that the present disclosure is not limited to this and can be extended to other situations.

[0035] In some embodiments, the abnormal condition may include at least one of a vascular plaque, a myocardial bridge, a hemangioma, a stent, a catheter, and a guidewire. In some embodiments, in addition to FFR, vascular physiological parameters (also referred to as vascular function parameters) may include at least one of physiological function status, blood flow pressure, blood flow velocity, blood volume, wall shear stress, microcirculatory resistance index (IMR), instantaneous waveform-free ratio (iFR), relative FFR change parameter compared to adjacent locations, or relative iFR change parameter.

[0036] Figure 2 Schematic diagram showing an exemplary structure of a joint learning model for abnormality detection (e.g., lesion detection) and physiological condition estimation (e.g., physiological parameter prediction) according to an embodiment of the present disclosure. Figure 2 As shown, the joint learning model may include a joint feature extraction unit 201 , a lesion detection unit 202 (as an example of an abnormality detection unit) and an FFR prediction unit 203 (as an example of a physiological parameter prediction unit).

[0037] The joint feature extraction unit 201 is configured to extract joint features based on the medical image. For example, the joint features may include a sequence of features F along the centerline of the blood vessel. In some embodiments, the joint features can be derived directly from the original medical image, the converted medical image, or the processed image (mask, image obtained by different intensity conversion, etc.). The joint features can be manually designed or automatically learned through a model such as a neural network. In the latter case, the joint feature extraction unit 201 can learn during the training phase. The extracted joint features will be used for lesion detection and physiological parameter prediction. By sharing the joint feature extraction unit 201, the joint learning model can fully utilize the correlation between the two tasks of lesion detection and physiological parameter prediction.

[0038] When the medical images are different, the joint feature extraction unit 201 may extract the joint features in different ways.

[0039] For example, upon receiving as input a sequence of initial feature information for at least one position along the centerline of the blood vessel, the joint feature extraction unit 201 extracts a feature map for the initial feature information for each of the at least one position. In this way, a sequence of feature maps for at least one position along the centerline of the blood vessel can be obtained as the joint feature, wherein the sequence of feature maps can each include at least one feature map for at least one position. In some embodiments, the initial feature information for each position can include at least one of a (2D / 3D) image block extracted at that position, a blood vessel mask, manually extracted initial features, semi-automatically extracted initial features, or automatically extracted initial features. For example, a CNN unit can be used to extract the corresponding feature map for each position, and the parameters of each CNN unit are optimized together via backpropagation during training.

[0040] As another example, when receiving a medical image containing a blood vessel (e.g., a 2D / 3D full image) as input, a convolution operation can be performed on the area surrounding each position along the centerline of the blood vessel to obtain a feature map for each position. As a result, a sequence of feature maps for at least one position along the centerline of the blood vessel can be obtained as the joint feature.

[0041] In some embodiments, the joint learning of joint features can utilize at least one location on the centerline. However, the at least one location is not limited to a centerline point. In some embodiments, each location can also be selected from adjacent regions, including a lateral region, a longitudinal region, a perturbed centerline point, or a vascular mask. For ease of description, this article uses centerline points as an example of each location, but the following description can be flexibly applied to any other type of location without loss of generality.

[0042] The joint features extracted by the joint feature extraction unit 201 serve as common input to both the lesion detection unit 202 and the FFR prediction unit 203. The lesion detection unit 202 can then determine a lesion detection result p for the vessel based on the extracted joint features, while the FFR prediction unit 203 can determine an FFR prediction result y for the vessel based on the extracted joint features. In some embodiments, the lesion detection result p may include at least one of the following: a lesion mask (yes / no at each centerline point), lesion center coordinates, lesion size, lesion location, bounding box, start and end points, lesion severity, and lesion type (calcified / non-calcified / mixed; or fragile / stable lesion). The FFR prediction result y may include the FFR value at each or a portion of the centerline points, the relative FFR change compared to adjacent points, and other information.

[0043] In some embodiments, the lesion detection unit 202 and the FFR prediction unit 203 can be implemented using any one of CNN, MLP (Multi-layer Perceptron), FCN (Fully Connected Neural Network), RNN (Recurrent Neural Network), and GCN (Graph Convolutional Network).

[0044] In some embodiments, the joint learning model can be jointly trained based on a dataset of anomaly annotations and a dataset of physiological parameter annotations, and the loss function used in the joint training penalizes deviations from a predetermined constraint relationship between the abnormal conditions of the blood vessels and the physiological parameters. Alternatively, or in combination, the joint learning model can be configured to model the predetermined constraint relationship between the abnormal conditions of the blood vessels and the physiological parameters into its model structure, thereby eliminating or reducing deviations from the predetermined constraint relationship between the abnormal conditions of the blood vessels and the physiological parameters.

[0045] Using the degree of lesion and FFR value at various locations on a blood vessel as examples of abnormal conditions and physiological parameters, respectively, a predetermined constraint relationship between the vascular lesion condition and the FFR value can be embedded through the architecture of the joint learning model and / or the combination of loss functions used in joint training. This predetermined constraint relationship can indicate that the FFR value changes according to the degree of lesion, while the FFR value at non-lesioned locations does not change or changes slowly. In this way, the lesion detection results and FFR prediction results obtained by the trained joint learning model can conform to the physiological mechanism of action between the two, be physiologically consistent, and significantly improve the accuracy and robustness of the lesion detection results and FFR prediction results.

[0046] Figure 3 FIG. 1 is a schematic diagram showing another exemplary structure of a joint learning model for lesion detection and physiological parameter prediction according to an embodiment of the present disclosure. Figure 3 As shown, the lesion detection unit 202 can be configured to detect whether each position along the centerline of the blood vessel is abnormal (for example, whether an abnormality occurs at the position). The FFR prediction unit 203 can be configured to: for each position: predict the change value of the FFR at the position compared to the upstream adjacent position via the FFR change prediction unit 203a. , and the change value of the upstream of the position is measured by the FFR aggregation unit 203b Aggregation is performed to determine the FFR at that location. For example, the change value of each location can be a decrease in FFR compared to an adjacent upstream location, and It can be used to express FFR reduction values, as described below.

[0047] In some embodiments, the predetermined constraint relationship between the FFR change of the blood vessel and the lesion may be forcibly embedded in the process of predicting the change value by the FFR change predictor 203a and / or in the configuration of the FFR change predictor 203a.

[0048] Specifically, the initial prediction subunit 203a1 can predict the initial drop value of the FFR at each location compared to the upstream adjacent location based on the joint features ( In some embodiments, the initial prediction subunit 203a1 may be implemented using a linear layer.

[0049] The initial drop value can be adjusted via the first adjustment sub-unit 203a2 Make the first adjustment to get the intermediate drop value , making the middle drop value Falling into the FFR drop value In some embodiments, the first adjustment subunit 203a2 can use a rectified linear unit (RELU) to adjust the initial drop value. Processing is performed as shown in formula (1), retaining positive values ​​and mapping negative values ​​to 0. As a result, the intermediate FFR decreases It has a non-negative value (such as 0 or positive value), which conforms to the prior knowledge that FFR value decreases from upstream to downstream of the blood vessel. Thus, it avoids unreasonable FFR decrease value. And it does not violate this prior knowledge.

[0050] Formula (1)

[0051] Then, the intermediate FFR reduction value obtained after the first adjustment can be adjusted based on the location of the lesion through the second adjustment sub-unit 203a3. A second adjustment is performed to obtain the FFR reduction value , so that the location of the lesion is consistent with the FFR decrease value obtained after the second adjustment Here, FFR is used as an example of a vascular function parameter. The predetermined constraint relationship can be expressed as follows: the FFR value changes according to the degree of lesion, while the FFR value does not change or changes slowly at the non-lesion location. For example, according to formula (2), the FFR at the non-lesion location can be compared with the drop value of the upstream adjacent location. The value is set to zero, or to a number close to zero (eg, a number less than or equal to a predetermined threshold).

[0052] Formula (2)

[0053] like Figure 3 As shown, the gray circle represents The positive value indicates the centerline point where the lesion occurs, while the white circle indicates The centerline point of non-lesion (such as normal) should be zero or close to zero. After the first adjustment sub-unit 203a2 and the second adjustment sub-unit 203a3 work together, the predicted value at each centerline point is Only the center line point predicted as the lesion point may be non-zero (positive to be exact), and the values ​​at the other center line points (that is, normal positions) are are all 0 (or small value). Therefore, after two adjustments, Not only does it satisfy the prior knowledge that the vascular path extending downstream along the vessel does not usually increase, but it also satisfies the prior knowledge that FFR changes mainly due to lesions. , and then the FFR aggregation unit 203b generates the final FFR value at the t-th center line point For example, the final FFR value can be obtained according to the following formula (3): .

[0054] Formula (3)

[0055] in, represents the final FFR value at the t-th centerline point, represents the initial FFR value near the aorta, represents the set of all centerline points upstream of the t-th centerline point, Denotes the FFR change value (e.g., FFR decrease value) predicted for the jth centerline point. Because these operations are differentiable, the entire joint learning model can be learned end-to-end.

[0056] In some embodiments, the loss function used in the joint training may include a deviation loss term, wherein the deviation loss term represents the degree of deviation between the detection result of the lesion and the prediction result of the physiological parameter. Figure 4 As shown, for vascular lesions and FFR values, the deviation penalty term can include the degree of inconsistency between the FFR distribution and the lesion distribution 205. The higher the inconsistency, the higher the penalty. A gray circle indicates that the centerline point belongs to the lesion area and a white circle indicates that the centerline point belongs to the non-lesion area. For each position in the non-lesion area, for example Figure 4 The centerline point corresponding to the white circle shown in the figure shows the cumulative FFR change value of the current position compared to the upstream adjacent position. The larger the cumulative value, the more likely the inconsistency between the FFR distribution and the lesion distribution, and the larger the loss function value. This allows the joint learning model to adjust parameters accordingly to train towards reducing the loss function (including the degree of inconsistency). Consequently, the joint learning model can learn a better representation that reduces or even eliminates this inconsistency (conflict). The predetermined constraint may dictate that the FFR value varies based on the severity of the lesion. In some embodiments, the degree of deviation may also be associated with the severity of the lesion in the lesion area, such that points or locations with a greater degree of lesion receive less penalty, while points or locations with a lesser degree of lesion receive more penalty.

[0057] Figure 5 Flowchart showing the training process of the joint learning model according to the embodiment of the present disclosure. Figure 5 As shown, a training dataset for training a joint learning model can be constructed based on the ground truth values ​​of vascular lesions and FFR values ​​in medical images in step 501. The training dataset can include fully labeled, partially labeled, or even unlabeled data, so that various data labels can be fully utilized in training.

[0058] In step 502, the joint training model can be jointly trained using the training dataset until the loss function converges. Specifically, an optimization method such as stochastic gradient descent (SGD), RMSProp, or Adam can be used for end-to-end training. For example, the loss function can be expressed as formula (4):

[0059] Formula (4)

[0060] in, represents the lesion detection loss, represents the FFR regression loss, represents the weight of FFR regression loss, represents the additional consistency loss enforced between FFR and lesion prediction, is the weight of the consistency loss. For example, the FFR drop value at the centerline point of the non-lesion can be forced to zero. For example, but not limitation, It can be expressed by formula (5):

[0061] Formula (5)

[0062] in, is the set of all centerline points predicted to be non-lesional, is the FFR drop at the jth centerline point relative to its upstream centerline point. Based on prior knowledge of the physiological mechanisms of FFR formation, the FFR drop between adjacent centerline points in non-lesioned areas is expected to be zero or nearly zero. By penalizing the cumulative absolute value of the FFR drop in non-lesioned areas, the joint learning model is guided to acquire model parameters that better align with this prior knowledge.

[0063] Figure 6 FIG. 1 is a flow chart showing an analysis process for lesion detection and physiological parameter prediction for a medical image containing a vascular tree according to an embodiment of the present disclosure. Figure 6 As shown, in step 601, a medical image containing a vascular tree can be received and the centerline of the vascular tree can be extracted. In step 602, a series of image patches can be extracted along the centerline. Based on these extracted image patches, the lesion status of the vascular tree (particularly at the candidate stenosis) and the FFR value can be determined using a trained joint learning model. In step 603, the lesion status and FFR value of the entire vascular tree can be output and displayed.

[0064] In some embodiments, during joint training, the physiological function state downstream of a vascular lesion or on a branch of a vascular lesion can be estimated. This physiological function state can be a continuous variable, either 0 (normal) or 1 (abnormal), or can be graded (grades I, II, III, IV, etc.). If there is no lesion upstream of the vessel or in its branches (e.g., no lesion point), the predicted physiological function state of the vessel should be normal. On the other hand, if there is a lesion upstream of the vessel or in its branches (e.g., a lesion point), the physiological function state of the vessel should be abnormal. Any inconsistency between the physiological function state and the lesion location can be penalized using a penalty function or adjusted through logical processing.

[0065] Figure 7 FIG. 1 is a schematic block diagram of a system for performing combined abnormality detection and physiological condition estimation according to an embodiment of the present disclosure. Figure 7 As shown, the system may include a model training device 700a, an image acquisition device 700b and an image analysis device 700c.

[0066] In some embodiments, the image analysis device 700c may be a dedicated computer or a general-purpose computer. For example, the image analysis device 700c may be a computer customized for performing image acquisition and image processing tasks in a hospital, or a server in the cloud.

[0067] The image analysis device 700c may include at least one processor 703 configured to perform the functions described herein. For example, the at least one processor 703 may be configured to perform the method disclosed herein, particularly a method for performing combined abnormality detection and physiological condition estimation.

[0068] In some embodiments, the processor 703 may be a processing device including one or more general-purpose processing devices, such as a microprocessor, a central processing unit (CPU), a graphics processing unit (GPU), etc. More specifically, the processor 703 may be a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a processor that runs other instruction sets, or a processor that runs a combination of instruction sets. The processor 703 may also be one or more special-purpose processing devices, such as an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), a system on a chip (SoC), etc.

[0069] The image analysis device 700c may also include a memory 701, which may be configured to load or store a trained joint learning model according to any one or more embodiments of the present disclosure, or an image processing / analysis program, which, when executed by the processor 703, may implement the method disclosed herein.

[0070] The memory 701 may be a non-transitory computer-readable medium, such as a read-only memory (ROM), a random access memory (RAM), a phase-change random access memory (PRAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), an electrically erasable programmable read-only memory (EEPROM), other types of random access memory (RAM), flash memory or other forms of flash memory, cache, registers, static memory, compact disc read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic tape cassettes or other magnetic storage devices, or any other possible non-transitory medium for storing information or instructions accessible by a computer device, etc. When the instructions stored on the memory 701 are executed by the processor 703, the methods described herein may be performed.

[0071] Although Figure 7 While the model training device 700a and the image analysis device 700c are shown as separate devices, in some embodiments, the image analysis device 700c can also perform model training functions. Therefore, the memory 701 can be configured to load a training dataset annotated with abnormal blood vessel conditions and physiological parameters. For example, the processor 703 can be configured to jointly train the joint learning model, such as the joint feature extraction unit, the abnormality detection unit, and the physiological parameter prediction unit, based on the loaded training dataset.

[0072] In some embodiments, image analysis device 700c may further include memory 702, configured to load a federated learning model according to any one or more embodiments of the present disclosure from, for example, storage 701, or temporarily store intermediate data generated during processing / analysis using the federated learning model. Processor 703 may be communicatively coupled to memory 702 and configured to execute executable instructions stored thereon to perform the methods disclosed herein.

[0073] In some embodiments, memory 702 can store intermediate information generated during the training phase or prediction phase, such as feature information generated while executing a computer program, abnormal conditions and / or physiological parameters of blood vessels, values ​​of various loss terms, etc. In some embodiments, memory 702 can store computer-executable instructions, such as one or more image processing programs. In some embodiments, the joint learning model, and the various components and subcomponents within the joint learning model, can be implemented as applications stored in memory 701, and these applications can be loaded into memory 702 and then executed by processor 703 to implement corresponding functions.

[0074] In some embodiments, the memory 702 may be a non-transitory computer-readable medium, such as a read-only memory (ROM), a random access memory (RAM), a phase-change random access memory (PRAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), an electrically erasable programmable read-only memory (EEPROM), other types of random access memory (RAM), a flash disk or other form of flash memory, a cache, a register, a static memory, or any other possible medium for storing information or instructions that can be accessed and executed by a computer device, etc.

[0075] In some embodiments, the image analysis device 700c may further include a communication interface 704 for receiving medical images acquired by the image acquisition device 700b. In some embodiments, the communication interface 704 may include any one of a network adapter, a cable connector, a serial connector, a USB connector, a parallel connector, a high-speed data transmission adapter (such as an optical fiber, USB 3.0, or Thunderbolt interface), a wireless network adapter (such as a WiFi adapter), a telecommunications (3G, 4G / LTE, 5G, etc.) adapter, and the like.

[0076] The image analysis device 700c can be connected to the model training device 700a, the image acquisition device 700b, and other components via the communication interface 704. In some embodiments, the communication interface 704 can be configured to receive the trained joint learning model from the model training device 700, and can also be configured to receive a medical image containing a blood vessel from the image acquisition device 700b.

[0077] Specifically, the image acquisition device 700b may include any one of conventional CT, conventional MRI, functional magnetic resonance imaging (such as fMRI, DCE-MRI and diffusion MRI), cone beam computed tomography (CBCT), positron emission tomography (PET), single photon emission computed tomography (SPECT), X-ray imaging, optical tomography (OCT), fluorescence imaging, ultrasound imaging, radiotherapy field imaging, etc.

[0078] In some embodiments, the model training device 700a can be configured to train a joint learning model and transmit the trained joint learning model to the image analysis device 700c, so that the trained joint learning model can be used to determine abnormal conditions and physiological parameters of blood vessels based on acquired medical images containing blood vessels. In some embodiments, the model training device 700a and the image analysis device 700c can be implemented by a single computer or processor.

[0079] In some embodiments, the model training device 700a can be implemented using hardware specifically programmed by software that performs the training process. For example, the model training device 700a may include a processor and a non-transitory computer-readable medium similar to the image analysis device 700c. The processor implements training by executing executable instructions of the training process stored in the computer-readable medium. The model training device 700a may also include input and output interfaces to communicate with a training database, a network, and / or a user interface. The user interface can be used to select a training data set, adjust one or more parameters in the training process, select or modify the framework of the learning model, etc.

[0080] Another aspect of the present disclosure is to provide a non-transitory computer-readable medium storing instructions that, when executed, cause one or more processors to perform the above-described method. The computer-readable medium may include volatile or non-volatile, magnetic, semiconductor-based, tape-based, optical, removable, non-removable, or other types of computer-readable media or computer-readable storage devices. For example, the computer-readable medium may be a storage device or storage module having computer instructions stored therein, as disclosed. In some embodiments, the computer-readable medium may be a disk or flash drive having computer instructions stored thereon.

[0081] Various modifications and variations can be made to the methods, devices, and systems disclosed herein. Based on the description and practice of the disclosed systems and related methods, other embodiments can be derived by those skilled in the art. Each claim of the present disclosure is to be understood as an independent embodiment, and any combination thereof also serves as an embodiment of the present disclosure, and such embodiments are deemed to be included in the present disclosure.

[0082] It is intended that the description and examples be considered as exemplary only, with a true scope being indicated by the following claims and their equivalents.

Claims

1. A method for joint physiological condition estimation from medical images, comprising: receiving, via at least one processor, a medical image acquired by an image acquisition device, wherein the medical image includes an anatomical structure; and applying, via the at least one processor, a joint learning model to jointly determine an abnormality of the anatomical structure together with a physiological parameter based on the medical image, wherein the joint learning model satisfies a predetermined constraint relationship between the abnormality and the physiological parameter; The anatomical structure includes a blood vessel, the abnormality of the blood vessel including at least one of an abnormality mask for at least one location along a centerline of the blood vessel, representative coordinates of the abnormality, a type of the abnormality, a location of the abnormality, and a size of the abnormality; The joint learning model includes: a joint feature extraction unit configured to extract joint features based on the medical image, the joint features comprising a sequence of features at at least one position along the centerline of the blood vessel, the corresponding features at each position comprising at least one of an artificially designed feature, an original image, an autonomously learned feature, or a blood vessel mask; an abnormality detecting unit configured to detect whether an abnormality occurs at each position along the center line of the blood vessel; The physiological parameter prediction unit is configured to predict, for each position, a change value of the vascular function parameter at the position compared with the upstream adjacent position, specifically including: predicting an initial change value of the vascular function parameter at the position compared with the upstream adjacent position based on the joint feature; performing a first adjustment on the initial change value to obtain an intermediate change value, so that the intermediate change value falls within a predetermined range of the vascular function parameter; and performing a second adjustment on the intermediate change value obtained after the first adjustment based on the abnormality occurrence position to obtain the change value, so that the abnormality occurrence position and the change value obtained after the second adjustment satisfy the predetermined constraint relationship; and aggregating the change values ​​upstream of the position to determine the vascular function parameter at the position.

2. The method according to claim 1, characterized in that The joint learning model is trained using joint training based on the abnormality annotated dataset and the physiological parameter annotated dataset, and the loss function of the joint training penalizes deviations from a predetermined constraint relationship between the abnormal condition and the physiological parameter.

3. The method according to claim 1, characterized in that The predetermined constraint relationship between the abnormal condition and the physiological parameter is modeled into the model structure of the joint learning model to reduce the deviation of the predetermined constraint relationship between the abnormal condition and the physiological parameter.

4. The method according to claim 1, wherein The physiological parameters of the blood vessel include at least one of the following vascular function parameters in the physiological function state of at least one position along the centerline of the blood vessel: blood flow pressure, blood flow velocity, blood flow, wall shear force, blood flow reserve fraction parameter, microcirculatory resistance index, instantaneous waveform-free ratio parameter, relative blood flow reserve fraction change parameter compared to adjacent positions, or relative instantaneous waveform-free ratio change parameter.

5. The method according to claim 2, characterized in that The anatomical structure includes a blood vessel, wherein the loss function used in the joint training includes a deviation loss term, and wherein the deviation loss term accumulates the absolute value of the change of the vascular function parameter at each position in the non-abnormal area of ​​the blood vessel centerline compared with the upstream adjacent position.

6. The method according to claim 1, characterized in that The vascular function parameter includes a fractional flow reserve parameter, wherein the first adjustment sets change values ​​predicted to be negative to zero, and wherein the second adjustment sets non-zero change values ​​at non-abnormal locations to zero or a value less than a predetermined threshold.

7. The method according to claim 1, characterized in that The anatomical structure includes a blood vessel, and wherein the joint feature extraction unit is further configured to extract the joint feature by at least the following manner: receiving as input a sequence of initial feature information of at least one position along the centerline of the blood vessel, and extracting a feature map for the initial feature information of each position, thereby obtaining a sequence of feature maps of the at least one position along the centerline of the blood vessel as the joint feature, wherein the initial feature information of each position includes at least one of an image block extracted at the position, a blood vessel mask, a manually extracted initial feature, a semi-automatically extracted initial feature, or an automatically extracted initial feature; or A medical image containing the blood vessel is received as input, and a feature map of each position is obtained for a surrounding area of ​​each position in at least one position along the centerline of the blood vessel, thereby obtaining a sequence of feature maps of at least one position along the centerline of the blood vessel as the joint feature.

8. The method according to claim 1, characterized in that The abnormal condition includes at least one of a condition related to a vascular plaque, a myocardial bridge, a hemangioma, a stent, a catheter, or a guidewire.

9. A system for joint physiological condition estimation from medical images, comprising: a communication interface configured to receive the medical image acquired by an image acquisition device, wherein the medical image includes an anatomical structure; and At least one processor configured to: The method for joint physiological condition estimation from medical images according to any one of claims 1-8 is performed.

10. A non-transitory computer-readable storage medium having computer-executable instructions stored thereon, wherein when the computer-executable instructions are executed by a processor, the method for joint physiological condition estimation based on medical images according to any one of claims 1 to 8 is performed.

Images