Method and system for computer-aided medical image analysis using sequential models

Abstract

Embodiments of the present disclosure provide systems and methods for analyzing medical images containing vascular structures using a sequential model. An example system includes a communication interface configured to receive the medical image and the sequential model. The sequential model includes a vessel extraction sub-model and a lesion analysis sub-model. The vessel extraction sub-model and the lesion analysis sub-model are trained independently or jointly. The example system further includes at least one processor configured to apply the vessel extraction sub-model to the received medical image to extract location information of the vascular structure. The at least one processor also applies the lesion analysis sub-model to the received medical image and the location information extracted by the vessel extraction sub-model to obtain a lesion analysis result of the vascular structure. The at least one processor further outputs the lesion analysis result of the vascular structure.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to U.S. Provisional Application No. 63 / 133,766, filed January 4, 2021, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure relates to systems and methods for medical image analysis, and more specifically, to systems and methods for analyzing medical images containing vascular structures using sequential models. Background Technology

[0004] Medical image analysis systems are typically designed to perform multiple tasks based on medical images. For example, a system might segment blood vessels in a medical image and detect lesions along those vessels. However, existing systems often perform multiple tasks independently, ignoring potential relationships between them. Consequently, the analytical results from these multiple tasks are often inconsistent.

[0005] For example, lesion detection models trained by existing systems typically do not consider vascular information in medical images. Therefore, trained lesion detection models may produce many false positives (e.g., lesions detected in non-vascular areas). This type of error can be significantly reduced if vascular location information is considered during lesion detection (e.g., detecting lesions along the vessel centerline). As another example, consider performing a vessel segmentation task without following the vessel centerline or without considering existing lesions along the vessel. As a result, the system may return a broken vessel segmentation mask due to severe vascular stenosis.

[0006] Furthermore, existing medical image analysis systems often cannot easily incorporate human intervention to adjust the analysis results. For example, existing systems do not include separate human intervention editing units that convert different forms of human intervention (e.g., dragging, dropping, adding graffiti, or erasing operations) into a unified form, such as modifying a mask or a modified vascular centerline.

[0007] Embodiments of this disclosure address the aforementioned problems by introducing systems and methods for analyzing medical images containing vascular structures using sequential modeling. Summary of the Invention

[0008] In one aspect, embodiments of the present disclosure provide a system for analyzing a medical image containing a vascular structure using a sequential model. An example system includes a communication interface configured to receive the medical image and the sequential model. The sequential model includes a vessel extraction sub-model and a lesion analysis sub-model. The vessel extraction sub-model and the lesion analysis sub-model are trained independently or jointly. The example system further includes at least one processor configured to apply the vessel extraction sub-model to the received medical image to extract location information of the vascular structure. The at least one processor is further configured to apply the lesion analysis sub-model to the received medical image and the location information extracted by the vessel extraction sub-model to obtain a lesion analysis result of the vascular structure. The at least one processor is further configured to output the lesion analysis result of the vascular structure.

[0009] In another aspect, embodiments of the present disclosure also provide a method for analyzing a medical image containing a vascular structure using a sequential model. An example method includes receiving, by a communication interface, the medical image and the sequential model. The sequential model includes a vessel extraction sub-model and a lesion analysis sub-model. The vessel extraction sub-model and the lesion analysis sub-model are trained independently or jointly. The method further includes applying, by at least one processor, the vessel extraction sub-model to the received medical image to extract location information of the vascular structure. The method further includes applying, by the at least one processor, the lesion analysis sub-model to the received medical image and the location information extracted by the vessel extraction sub-model to obtain a lesion analysis result of the vascular structure. The method further includes outputting, by the at least one processor, the lesion analysis result of the vascular structure.

[0010] In yet another aspect, embodiments of the present disclosure also provide a non-transitory computer-readable medium having instructions stored thereon that, when implemented by at least one processor, cause the at least one processor to perform a method for analyzing a medical image containing a vascular structure using a sequential model. The method includes receiving the medical image and the sequential model. The sequential model includes a vessel extraction sub-model and a lesion analysis sub-model. The vessel extraction sub-model and the lesion analysis sub-model are trained independently or jointly. The method further includes applying the vessel extraction sub-model to the received medical image to extract location information of the vascular structure. The method further includes applying the lesion analysis sub-model to the received medical image and the location information extracted by the vessel extraction sub-model to obtain a lesion analysis result of the vascular structure. The method further includes outputting the lesion analysis result of the vascular structure.

[0011] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application claimed. Attached Figure Description

[0012] Figure 1 A schematic diagram of an exemplary medical image analysis system for analyzing medical images according to an embodiment of the present disclosure is shown.

[0013] Figure 2 A flowchart of an exemplary sequential model for analyzing medical images according to embodiments of the present disclosure is shown.

[0014] Figure 3 A flowchart of an exemplary sub-model of a sequential model according to an embodiment of the present disclosure is shown.

[0015] Figure 4 A block diagram of an exemplary image processing apparatus for analyzing medical images according to embodiments of the present disclosure is shown.

[0016] Figure 5 A flowchart is shown of an exemplary sequential model for detecting lesions and segmenting blood vessels according to embodiments of the present disclosure.

[0017] Figure 6 This is a flowchart of an exemplary method for analyzing medical images containing vascular structures according to embodiments of the present disclosure. Detailed Implementation

[0018] Reference will now be made in detail to exemplary embodiments, examples of which are shown in the accompanying drawings.

[0019] The disclosed systems and methods utilize sequential models to perform various tasks, such as analyzing medical images to solve lesion detection and vessel segmentation problems. In some embodiments, the sequential model can leverage a divide-and-conquer strategy to break down a large task into more than one simpler / smaller task. For example, an image analysis task can be broken down into more than one related medical image analysis task. Multiple sub-models are then assigned to solve these individual simpler / smaller tasks. The inputs to these sub-models may depend on the outputs of other sub-models, and they are performed sequentially. The results of some sub-tasks can be reused by more than one other sub-task. Therefore, it is easier to train sub-models to solve sub-tasks because training sub-models typically requires less training data than training a single model. Furthermore, consistency between multiple related tasks can be ensured. Additionally, the sequential structure of the sub-models makes it easier to incorporate human interventions as inputs into a particular sub-model in a convenient form / representation.

[0020] An exemplary sequential model for blood vessel image analysis can include three submodels: a vessel extraction submodel configured to extract blood vessel locations from a medical image, a lesion analysis submodel configured to detect lesions and analyze the detected lesions, and a vessel segmentation submodel configured to refine blood vessel boundaries and obtain a vessel segmentation mask. In some embodiments, the vessel extraction submodel can receive a first user edit to adjust blood vessel location information of a blood vessel structure in the medical image. The lesion analysis submodel can receive a second user edit to adjust lesion analysis results of the blood vessel structure in the medical image. The vessel segmentation submodel can receive a third user edit to adjust a segmentation task of the blood vessel structure in the medical image. In some embodiments, the first user edit, the second user edit, and the third user edit are independent of each other. It is contemplated that the tasks processed by the disclosed sequential model can be any tasks other than image analysis tasks. The submodels will be designed accordingly to solve the corresponding subtasks.

[0021] In some embodiments, the sequential model is applied to a medical image and / or user edits in three stages. For example, first, the vessel extraction submodel of the sequential model is applied to a received medical image and / or a first user edit to extract location information of a blood vessel structure in the received medical image. Second, the lesion analysis submodel of the sequential model is applied to the received medical image, the location information extracted by the vessel extraction submodel, and / or a second user edit to obtain lesion analysis results of the blood vessel structure in the received medical image. Third, the vessel segmentation submodel of the sequential model is applied to the received medical image, the location information extracted by the vessel extraction submodel, the lesion analysis results of the blood vessel structure generated by the lesion analysis submodel, and / or a third user edit to segment the blood vessel structure from the received medical image. In some embodiments, the sequential model can receive as inputs the medical image, the first user edit, the second user edit, and the third user edit, and output the lesion analysis results of the blood vessel structure and a segmentation mask of the blood vessel structure.

[0022] The disclosed systems and methods do not separately process multiple related medical image analysis tasks in isolation. Instead, the disclosed systems and methods apply individual submodels of a sequential model to a received medical image in a sequential manner to create a unified solution for related medical image analysis tasks (e.g., nearby anatomical locations, or on the same target anatomical structure). As a result, individual output analysis results (e.g., detected lesions, segmented blood vessels) are consistent with each other.

[0023] The disclosed system and method provide five main advantages over the prior art: (1) interdependent image analysis tasks are addressed in a unified manner rather than in isolation; (2) the interdependencies between related image analysis tasks are enforced by the disclosed system and method to ensure consistent results for related image analysis tasks such as vessel segmentation and lesion quantification; (3) better performance of individual image analysis tasks can be achieved based on intermediate results of upstream image analysis tasks; (4) modeling and training are easier and faster because less training data is required to train individual sub-models; (5) additional information such as human intervention can be easily incorporated into the disclosed system and method in a convenient form / representation (e.g., vessel centerline).

[0024] Figure 1 A schematic diagram of an exemplary medical image analysis system 100 (hereinafter also referred to as system 100) for analyzing medical images according to some embodiments of the present disclosure is shown. Consistent with the present disclosure, the system 100 is configured to analyze medical images acquired by an image acquisition device 105. In some embodiments, the image acquisition device 105 can use more than one imaging modality, including, for example, magnetic resonance imaging (MRI), computed tomography (CT), functional MRI (e.g., fMRI, DCE-MRI, and diffusion MRI), cone beam CT (CBCT), positron emission tomography (PET), single photon emission computed tomography (SPECT), X-ray, optical coherence tomography (OCT), fluorescence imaging, ultrasound imaging, and radiotherapy portal imaging, among others. In some embodiments, the image acquisition device 105 can capture images containing at least one tree-structured object, such as a blood vessel. For example, the image acquisition device 105 can be an MRI scanner or a CT scanner that captures images of coronary blood vessels, or an OCT device that captures images of retinal blood vessels. In some embodiments, the captured medical images can be two-dimensional (2D) or three-dimensional (3D) images. The 3D images can contain multiple 2D image slices.

[0025] As Figure 1 shown, the system 100 can include components for performing two stages, a training stage and a prediction stage. To perform the training stage, the system 100 can include a training database 101 and a model training device 102. To perform the prediction stage, the system 100 can include an image processing device 103 and a medical image database 104. In some embodiments, the system 100 can include more or fewer components than shown in FIG. 1. For example, when a sequential model is pre-trained and provided for analyzing medical images, the system 100 can include only the image processing device 103 and the medical image database 104. Figure 1

[0026] ​The system 100 can optionally include a network 106 to facilitate communication between various components of the system 100, such as the databases 101 and 104, the devices 102, 103, and 105. For example, the network 106 can be a local area network (LAN), a wireless network, a cloud computing environment (e.g., software as a service, platform as a service, infrastructure as a service), client-server, wide area network (WAN), etc. In some embodiments, the network 106 can be replaced by a wired data communication system or device.

[0027] In some embodiments, the various components of the system 100 can be remote from each other or located at different locations and connected through the network 106 as shown. Figure 1 In some alternative embodiments, certain components of the system 100 can be co-located or located within one device. For example, the training database 101 can be co-located with the model training device 102 or be a part of the model training device 102. As another example, the model training device 102 and the image processing device 103 can be within the same computer or processing device.

[0028] The model training device 102 can use the training data received from the training database 101 to train a sequential model for analyzing medical images received from, for example, the medical image database 104. As shown in Figure 1 The model training device 102 can communicate with the training database 101 to receive a set of training data, which can be 2D or 3D images. Each set of training data can include at least one medical image and its corresponding ground truth that provides an analysis result for the respective medical image. In some embodiments, the ground truth analysis result can be a final result of the entire model or an intermediate result of a sub-model. For example, the analysis result can include vessel location information, lesion analysis results, vessel segmentation masks, or other various forms of derived results (e.g., distance transform, feature map, probability map, etc.).

[0029] In some embodiments, the training phase can be performed “online” or “offline”. “Online” training refers to performing the training phase concurrently with the prediction phase, for example, performing the model in real-time before analyzing a new medical image. “Online” training can be beneficial for obtaining the most up-to-date model based on the training data available at the time. However, “online” training can require a large amount of computational cost to perform and can not always be feasible if the training data is large and / or the model is complex. In line with the present disclosure, “offline” training is used, where the training phase is performed separately from the prediction phase. The sequential model of offline training is saved and reused to analyze new medical images.

[0030] The model training device 102 can be implemented using hardware specifically programmed by software that performs training processing. For example, the model training device 102 may include a processor and a non-transitory computer-readable medium (in combination with...). Figure 4 (Detailed discussion follows). The processor can train the model by executing instructions stored in a computer-readable medium. The model training device 102 may additionally include input and output interfaces for communicating with the training database 101, the network 106, and / or a user interface (not shown). The user interface can be used to select a training dataset, adjust more than one parameter of the training process, select or modify the framework of the sequential model, and / or manually or semi-automatically provide predictions associated with the images used for training.

[0031] Consistent with some embodiments, the sequential model trained by the model training device 102 may be a machine learning model (e.g., a deep learning model) comprising at least two sub-models. For example, the two sub-models may be a vascular extraction sub-model configured to extract location information of vascular structures in a received medical image, and a lesion analysis sub-model configured to obtain lesion analysis results based on the location information extracted by the vascular extraction sub-model and the received medical image. In some embodiments, the sequential model may include more than two sub-models. For example, the sequential model may additionally include a third sub-model, such as a vascular segmentation sub-model configured to refine vascular boundaries based on the outputs of two upstream sub-models (e.g., the vascular extraction sub-model and the lesion analysis sub-model).

[0032] In some embodiments, the sequential model can be formulated based on known mathematical relationships (e.g., dynamical systems, statistical models, differential equations, or game theory models). For example, the sequential model can be modeled using explicitly given mathematical functions. In some alternative embodiments, the sequential model can be a hybrid model comprising machine learning sub-models and mathematical sub-models. For example, the machine learning sub-model is trained and tuned using artificial neural networks or other machine learning methods, while the parameters of the mathematical sub-model are determined through curve fitting. However, it is conceivable that the structure of the sequential model is not limited to what has been disclosed, as long as the sub-models in the sequential model encode sequential relationships (e.g., the input of a downstream sub-model depends on at least one output of an upstream sub-model), and the sub-models are executed sequentially to analyze new medical images.

[0033] For example, Figure 2 A flowchart of an exemplary sequential model 200 for analyzing medical images according to embodiments of the present disclosure is shown. Figure 2As shown, sequential model 200 (hereinafter also referred to as model 200) can receive input 211 (e.g., a medical image containing vascular structures from medical image database 104) and generate outputs 213-1 and 214. Model 200 may include three sub-models 201, 202, and 203. The sub-models are applied sequentially to input 211. For example, sub-model 201 receives input 211 to generate output 212 (e.g., location information of vascular structures) to feed downstream sub-models 202 and 203. Sub-model 202 receives input 212-1 (e.g., derived from output 212) to generate output 213, for example, including lesion analysis results of vascular structures. Sub-model 202 can then send output 213-1 (e.g., derived from output 213) out of model 200. Sub-model 203 receives inputs 213-2 (e.g., derived from output 213) and inputs 212-2 (e.g., derived from output 212) to generate output 214 (e.g., a segmentation mask of vascular structures). Sub-model 203 then provides output 214 as part of the output of model 200 (e.g., analysis result 120).

[0034] Figure 3 A flowchart illustrating an exemplary sub-model 310 of a sequential model according to an embodiment of the present disclosure is shown. Figure 3 As shown, sub-model 310 (as an exemplary sub-model in the disclosed sequential model) can receive at least one of raw data / images 301, the output of upstream sub-model 302, or human intervention 303. Consistent with some embodiments, raw data / images 301 can be acquired by image acquisition device 105 and stored in medical image database 104. For example, raw data / images 301 (e.g., Figure 2 The input 211 can be a 2D / 3D medical image or other type of raw data generated by an MRI scanner or CT scanner (as an example of image acquisition device 105).

[0035] In some embodiments, the output of the upstream sub-model 302 may be derived from the upstream sub-model (e.g., Figure 2 The analysis results of medical images (e.g., raw data / image 301) generated by sub-model 201 or sub-model 202 in the upstream model. For example, the medical image analysis results may include vascular location information or lesion analysis information generated by the upstream sub-model in the form of feature maps, probability maps, bounding boxes, etc. In some embodiments, the manual intervention 303 may be an edit input by a user (e.g., a doctor, professional, expert operator) based on their knowledge and experience. In some embodiments, user edits are input in the form of editing masks, drag / drop operations, adding doodles, erasing operations, etc.

[0036] In some embodiments, sub-model 310 may generate at least one output (e.g., output 311). For example, output 311 may be used internally to feed downstream sub-models (e.g., Figure 2 The submodel 310 may take inputs 212-1 and 213-2 as inputs, but not output them from system 100 as part of the analysis results. For example, if submodel 310 is a lesion analysis submodel, output 311 may include lesion analysis results, such as lesion detection bounding boxes, lesion characterization, and quantification information. Downstream submodels used to refine vessel boundaries (e.g., vessel segmentation submodels) may take output 311 as input and generate a pixel-by-pixel vessel segmentation mask. In some embodiments, submodel 310 may generate a second output (e.g., output 312), which may be output from the medical image analysis system. For example, if submodel 310 is a lesion analysis submodel, output 312 may be a lesion detection bounding box, a probability map, a segmentation mask, or other representation of lesion location information. Output 312 may be output from system 100 as part of the analysis result 120. In some embodiments, the inputs and outputs of submodel 310 may be displayed in the form of other transformations, such as an expanded centerline, a Gaussian field heatmap around key points, points in transformation space (e.g., polar coordinates), etc.

[0037] return Figure 1 The model training device 102 can initially train the sequential model by training each sub-model individually. Consistent with some embodiments, to train individual sub-models (e.g., sub-model 310), the model training device 102 can receive a set of training data, which includes medical images (e.g., raw data / images 301) and the output of a previously trained upstream sub-model (e.g., the output of upstream sub-model 302). If the upstream sub-model has not yet been trained or sub-model 310 does not have an upstream sub-model, expert manual annotations (e.g., identification of blood vessels or lesion locations) can be used instead of the output 302 of the upstream sub-model. The model training device 102 can also receive user edits / simulations (e.g., human intervention 303) as part of the training data. Furthermore, each received training dataset can include underlying fact outputs (e.g., outputs 311 and / or 312), which can be obtained using methods similar to those used for the output of upstream sub-model 302 (e.g., from previously trained sub-models or expert manual annotations).

[0038] In some embodiments, the model training device 102 can jointly train the individually trained sub-models. For example, the model training device 102 can jointly optimize the parameters of adjacent sub-models (e.g., sub-model 201 and sub-model 202). The model training device 102 can then jointly optimize the parameters of sub-models 201, 202, and 203. Joint training can achieve better performance in the final output of the disclosed system. Joint training can also guarantee consistent results for the final output (e.g., output 213-1 and output 214). For example, the lesion detection results are consistent with the vessel segmentation mask, and vice versa.

[0039] In some embodiments, when all sub-models are jointly trained, the model training device 102 can integrate the trained sub-models into a sequential model and deploy the trained sequential model (e.g., trained sequential model 110) to the image processing device 103. In some embodiments, the trained sequential model 110 can be used by the image processing device 103 to analyze new medical images. The image processing device 103 can include a processor and a non-transitory computer-readable medium (in combination with Figure 4 discussed in detail). The processor can execute instructions of the image analysis process stored in the medium. The image processing device 103 can additionally include input and output interfaces (in combination with Figure 4 discussed in detail) to communicate with the medical image database 104, the network 106, and / or a user interface (not shown). The user interface can be used to select medical images for analysis, initiate the analysis process, display medical images, and / or analysis results.

[0040] The image processing device 103 can communicate with the medical image database 104 to receive one or more medical images. In some embodiments, the medical images stored in the medical image database 104 can include 2D image slices from a 3D scan. The medical images can be acquired by the image acquisition device 105. The image processing device 103 can use the trained sequential model 110 received from the model training device 102 to analyze the received medical images received from the medical image database 104. For example, the image processing device 103 can first perform vessel extraction to locate a rough outline of the vessel structure (e.g., vessel centerline) in the received medical images. Based on the vessel location information, the image processing device 103 then locates, characterizes, or quantifies lesions associated with (e.g., along) the vessel structure. The image processing device 103 can further refine the boundaries of the vessel structure and generate a pixel-wise vessel segmentation mask based on the lesion detection results and vessel location information. The image processing device 103 can output the analysis results 120 (e.g., segmentation mask and lesion detection results) from the system 100.

[0041] Figure 4A block diagram of an exemplary image processing device 103 for analyzing medical images in accordance with embodiments of the present disclosure is shown. In some embodiments, the image processing device 103 can be a special-purpose computer or a general-purpose computer. For example, the image processing device 103 can be a computer customized for a hospital to perform image acquisition and image processing tasks. As Figure 4 shown, the image processing device 103 can include a communication interface 402, a memory 404, a memory 406, a processor 408, and a bus 410. The communication interface 402, the memory 404, the memory 406, and the processor 408 are connected with and communicate with each other through the bus 410.

[0042] The communication interface 402 can include a network adapter, a cable connector, a serial connector, a USB connector, a parallel connector, a high-speed data transfer adapter (e.g., fiber, USB 3.0, thunderbolt, etc.), a wireless network adapter (e.g., WiFi adapter), a telecommunication (3G, 4G / LTE, etc.) adapter, etc. The image processing device 103 can connect to other components of the system 100 and the network 106 through the communication interface 402. In some embodiments, the communication interface 402 receives medical images from the image acquisition device 105. Consistent with some embodiments, the medical images capture tree structure objects, such as blood vessels. For example, the medical images can be coronary vessel images or retinal vessel images. In some embodiments, the communication interface 402 also receives a sequential model (e.g., the trained sequential model 110) from the modeling training device 102.

[0043] The memory 404 / memory 406 can be a non-transitory computer-readable medium, such as a read-only memory (ROM), a random access memory (RAM), a phase change random access memory (PRAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), an electrically erasable programmable read-only memory (EEPROM), other types of random access memory (RAM), a flash memory, or other forms of flash memory, a cache, a register, a static memory, a compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), or other optical storage, a magnetic cased tape, or other magnetic storage device, or any other non-transitory medium that can be used to store information or instructions that can be accessed by a computer device or the like.

[0044] In some embodiments, memory 404 may store trained sequential models (e.g., trained sequential model 110) and data such as location information of vascular structures (e.g., vascular centerline) generated during the execution of a computer program. In some embodiments, memory 406 may store computer-executable instructions, such as one or more image processing programs. In some embodiments, lesions may be detected based on vascular centerlines, and lesion information (e.g., location, characterization, quantification) may be stored in memory 404. Lesion information may be read from memory 404 and stored in memory 406.

[0045] Processor 408 may be a processing device that includes one or more general-purpose processing devices, such as a microprocessor, a central processing unit (CPU), a graphics processing unit (GPU), etc. More specifically, the processor may be a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a processor that runs other instruction sets, or a processor that runs a combination of instruction sets. The processor may also be one or more special-purpose processing devices, such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), system-on-a-chip (SoCs), etc. Processor 408 may be communicatively coupled to memory 406 and configured to execute computer-executable instructions stored thereon.

[0046] In some embodiments, processor 408 is configured to analyze received medical images. For example, processor 408 uses a trained sequential model to analyze vascular structures in the received medical images and outputs analysis results (e.g., lesion information, vascular segmentation mask). Figure 5 A flowchart is shown of an exemplary sequential model 500 for detecting lesions and segmenting blood vessels according to embodiments of the present disclosure. Figure 5 As shown, the sequential model 500 may include a vessel extraction sub-model 520, a lesion analysis sub-model 530, and a vessel segmentation sub-model 540. Consistent with some embodiments, applying the sequential model 500 to the medical image 510 may include applying the vessel extraction sub-model 520, the lesion analysis sub-model 530, and the vessel segmentation model 540 sequentially to the medical image 510.

[0047] For example, the vessel extractor sub-model 520 is applied to the medical image 510 to extract vessel location information of the vessel structure in the medical image 510. In some embodiments, the vessel location information (e.g., output 521) can be a coarse outline of the vessel structure (e.g., a centerline of the vessel shown in a distance transform form) and is sent to downstream sub-models (e.g., the lesion analysis sub-model 530 and the vessel segmentation sub-model 540) as input. For example, the output 521 can be a distance transform indicating the centerline location of the vessel structure. In some alternative embodiments, the output 521 can be a feature map, a probability map, or other representation of the vessel location information. In some embodiments, the output 521 is intermediate data that feeds only one or more downstream sub-models. In other words, the output 521 is not output from the sequential model.

[0048] In some embodiments, human intervention (e.g., user edits 501) can be sent to the vessel extractor sub-model 520 to influence the vessel extraction result. For example, an expert operator can substantially alter the vessel location information by adding scribbles via points, dragging and dropping to different locations, expanding or erasing existing vessel locations via the user edits 501. In some embodiments, a separate unit (not shown in FIG. 5) is used to convert different forms of human intervention (e.g., drag-and-drop, add-scribble, expand, or erase operations) to a uniform form (e.g., modified centerline), and then apply the uniform form to the vessel extractor sub-model 520 to adjust the output 521. Figure 5

[0049] In some embodiments, the vessel location information (e.g., output 521) can be used to facilitate the lesion detection task. For example, the lesion analysis sub-model 530 can detect lesions along the vessel path based on the output 521. Because the output 521 provides location information of the vessel structure in the medical image, the lesion analysis sub-model 530 can search for lesions along the vessel region without wasting time in non-vessel regions. In some embodiments, if more than one lesion is detected, the lesion analysis sub-model 530 can further quantify and characterize the detected lesions. For example, the lesion analysis sub-model 530 can determine the class of the detected vessel lesions (e.g., calcified or non-calcified). In addition, the lesion analysis sub-model 530 can calculate the degree of stenosis of the detected lesions. In some embodiments, the lesion analysis sub-model 530 generates and outputs a lesion analysis result 531 including the detected lesion information (e.g., lesion location, lesion quantification, and lesion characterization) from the sequential model 500.

[0050] ​In some embodiments, a user can send user edits 502 to the lesion analysis submodel 530 to edit the lesion analysis results 531. The user can be the same expert operator as the input user edits 501 or a different expert operator. The user can use various operations (e.g., add, drag, drop, expand, or erase) to change the lesion locations, lesion characterization (e.g., calcified or non-calcified), and lesion quantification (e.g., stenosis degree). Consistent with some embodiments, an independent unit (not shown in Figure 5 ) can be applied on the user edits 502 to convert the various human operations into a unified form to adjust the lesion analysis results 531.

[0051] In some embodiments, a copy of the lesion analysis results 531 and the output 521 are used to refine the vessel boundaries in the vessel segmentation submodel 540. For example, the vessel segmentation submodel 540 can generate a pixel-wise vessel segmentation mask (e.g., vessel segmentation mask 541) based on the information of the vessel locations (e.g., output 521) and the detected lesions (e.g., lesion analysis results 531). In some embodiments, the vessel segmentation mask can be a pixel-wise or voxel-wise annotation of the received medical image, where each pixel / voxel is classified and labeled, e.g., with a value of 1 if the pixel / voxel belongs to the vessel structure, and a value of 0 otherwise. The vessel segmentation submodel 540 can use the lesion location information (e.g., lesion analysis results 531) to refine the vessel boundaries around the lesion regions. In some embodiments, the vessel segmentation submodel 540 can generate a probability map indicating the probability of each pixel in the medical image belonging to the vessel structure. The vessel segmentation submodel 540 can then perform thresholding to obtain the pixel-wise segmentation mask. For example, the vessel segmentation submodel 540 can set the pixels of the segmentation mask to 1 (i.e., belonging to the vessel structure) if the corresponding probability in the probability map is greater than 0.8, and set the remaining pixels of the segmentation mask to 0 (i.e., not belonging to the vessel structure). The threshold can be set by an operator or automatically selected by the vessel segmentation submodel 540.

[0052] In some embodiments, a user can send human interventions (e.g., user edits 503) to the vessel segmentation submodel 540 to adjust the vessel segmentation mask 541. The user can be the same expert operator as the input user edits 501 or the user edits 502, or a different expert operator. The human interventions can include grid-based dragging of the vessel boundaries, pixel pencil, and eraser tools to refine the vessel boundaries. Consistent with some embodiments, an independent unit (not shown in Figure 5 ) can be applied on the user edits 503 to convert the various human operations into a unified form that can be applied to the vessel segmentation mask 541. Returning to Figure 4The processor 408 can be configured to output the lesion analysis result 531 and the vessel segmentation mask 541 (as an example of the analysis result 120) from the image processing device 103 via the bus 410. The analysis result can be displayed in a display connected to the image processing device 103, printed on a medical report, or stored in an external memory.

[0053] Consistent with the present disclosure, the model training device 102 can have the same or similar structure as the image processing device 103. In some embodiments, the model training device 102 includes a processor and other components configured to train the vessel extraction sub-model, the lesion analysis sub-model, and the vessel segmentation sub-model using training data, individually or jointly.

[0054] Figure 6 is a flowchart of an exemplary method 600 for analyzing a medical image containing a vascular structure according to embodiments of the present disclosure. For example, the method 600 can be implemented by the image processing device 103 in Figure 1 using a trained sequential model such as the model 200 in Figure 2 However, the method 600 is not limited to this exemplary embodiment. The method 600 can include steps S602-S612 as described below. It should be appreciated that some steps can be optional for performing the disclosure provided herein. In addition, some steps can be performed simultaneously, or in a different order than shown in Figure 6 .

[0055] In step S602, the image processing device 103 receives a medical image, for example, from the medical image database 104. The medical image captures a vascular structure object, for example, a blood vessel. The image processing device 103 can additionally receive a medical image analysis model, for example, the sequential model 500. Consistent with some embodiments, the image analysis model can include three sub-models (e.g., the vessel extraction sub-model 520, the lesion analysis sub-model 530, the vessel segmentation sub-model 540). In some embodiments, the image processing device 103 can also receive one or more human interventions (e.g., user edits 501, 502, or 503) input by one or more expert operators.

[0056] In step S604, the image processing device 103 extracts location information of a vascular structure contained in the received medical image. In some embodiments, vessel extraction can be performed to locate a profile (e.g., a vessel centerline) of the vascular structure. For example, the centerline traces a path of the vascular structure. In some embodiments, the image processing device 103 can apply the received human intervention (e.g., user edit 501) to the extracted vessel location to adjust the output vessel extraction result.

[0057] In step S606, the image processing device 103 detects lesions along the vessel path using the extracted vessel location information. In some embodiments, the image processing device 103 can also determine whether the detected lesions are calcified or non-calcified. The image processing device 103 can also determine the degree of stenosis for each detected lesion. In some embodiments, the image processing device 103 can apply the received human intervention (e.g., user edits 502) to the detected lesions to adjust the outputted lesion information, e.g., lesion location, lesion characterization, and lesion quantification. In step S608, the image processing device 103 outputs the lesion information (e.g., lesion analysis results 531). For example, the lesion analysis results can be displayed on a display connected to the external medical instrument or printed in a medical report.

[0058] In step S610, the image processing device 103 refines the vessel boundary based on the vessel location information extracted in step S604 and the lesion analysis results outputted in step S608. For example, the image processing device 103 can accurately segment the vessel boundary around the lesion region based on the lesion analysis results provided in step S608, and generate a pixel-wise vessel segmentation mask. In some embodiments, the image processing device 103 can apply the received human intervention (e.g., user edits 503) to the vessel segmentation mask to adjust the refined vessel boundary. In step S612, the image processing device 103 outputs the refined vessel boundary (e.g., vessel segmentation mask 541). For example, the refined vessel boundary can be displayed on a display connected to the external medical instrument or printed in a medical report.

[0059] Although certain embodiments of the present disclosure are described using medical images containing vessel structures as examples, it is contemplated that the disclosed systems and methods are not limited to analyzing vessel structure images. The sub-models (e.g., sequential models 200) in the sequence network are general-purpose models that can be adjusted and trained to analyze any other image or non-image raw data. Figure 2 The sequential model 200 of the sequence network is a general-purpose model that can be adjusted and trained to analyze any other image or non-image raw data.

[0060] For example, the disclosed systems and methods can be configured to analyze medical images containing an organ (e.g., a heart) and compute a coronary artery calcification score using a sequential model. The coronary artery calcification score is a clinical indicator score of cardiovascular disease risk. In some embodiments, the sequential model can include at least two sub-models: a coronary artery detection sub-model and a calcification detection sub-model. The coronary artery detection sub-model is configured to extract the coronary arteries surrounding the heart. The calcification detection sub-model can then receive the extracted coronary arteries and detect calcification around the extracted coronary arteries. The sequential model can further include other sub-models to compute the coronary artery calcification score based on the detected calcification. Because the sequential model can only detect calcification on the coronary arteries and not on other blood vessels (e.g., the aorta), the sequential model can achieve better performance than a single coronary artery calcification detection model.

[0061] As another example, the disclosed systems and methods can include a sequential model configured to segment an organ (e.g., a pancreas) in a medical image. Because the pancreas is an elongated organ with a specific shape that varies among different people, it is very challenging to train a single segmentation model for segmenting the pancreas to achieve a satisfactory segmentation result. In some embodiments, the sequential model can include two sub-models: a larger organ segmentation sub-model that segments larger organs (e.g., stomach, liver) surrounding the pancreas, and a pancreas segmentation sub-model that segments the pancreas based on the results of the segmented larger organs. Thus, the sequential model can easily exclude non-pancreas regions by segmenting other larger organs and provide an accurate segmentation mask of the pancreas.

[0062] Another aspect of the present disclosure relates to a non-transitory computer-readable medium storing instructions that, when executed, cause one or more processors to perform the above-described methods. The computer-readable medium can include volatile or non-volatile, magnetic, semiconductor, tape, optical, removable, non-removable, or other types of computer- readable media or computer-readable storage devices. For example, as disclosed, the computer-readable medium can be a storage device or a memory module having computer instructions stored thereon. In some embodiments, the computer-readable medium can be a disk or a flash drive having computer instructions stored thereon.

[0063] It will be apparent to those skilled in the art that various modifications and variations can be made to the disclosed systems and related methods. Other embodiments will be apparent to those skilled in the art from consideration of the specification and practice of the disclosed systems and related methods.

[0064] This specification and examples are to be construed as illustrative only and not as restrictive on the true scope of the claims and equivalents thereof.

Claims

1. A system for analyzing medical images containing vascular structures using a sequential model, comprising: A communication interface configured to receive the medical image and the sequential model, wherein the sequential model includes a vessel extraction sub-model and a lesion analysis sub-model, wherein the vessel extraction sub-model and the lesion analysis sub-model are trained independently or jointly; and At least one processor configured to: The blood vessel extraction sub-model is applied to the received medical image to extract the centerline position information of the blood vessel structure, which is used to detect lesions along the centerline of the blood vessel structure, wherein the centerline tracks the path of the blood vessel structure. The lesion analysis submodel is applied to the received medical image and the centerline position information extracted by the vessel extraction submodel to obtain the lesion analysis results of the vascular structure; and Output the lesion analysis results of the vascular structure.

2. The system according to claim 1, wherein, The communication interface is configured to receive edits from a first user and edits from a second user, wherein the first user edit and the second user edit are independent of each other. Wherein, the at least one processor is further configured to: Alternatively, the blood vessel extraction sub-model is applied to the first user editor to extract the centerline position information of the blood vessel structure; or Additionally, the lesion analysis sub-model is applied to the second user editor to obtain lesion analysis results for the vascular structure.

3. The system according to claim 1, wherein, The sequential model also includes a vascular segmentation sub-model trained independently or jointly with respect to the vascular extraction sub-model and the lesion analysis sub-model. Wherein, the at least one processor is further configured to: The blood vessel segmentation submodel is applied to the received medical image, the centerline position information extracted by the blood vessel extraction submodel, and the lesion analysis results of the blood vessel structure obtained by the lesion analysis submodel to segment the blood vessel structure from the medical image.

4. The system according to claim 3, wherein, The communication interface is configured to receive edits from a first user, a second user, and a third user, wherein the first user edit, the second user edit, and the third user edit are independent of each other. Wherein, the at least one processor is further configured to: Additionally, the blood vessel extraction sub-model is applied to the first user editor to extract the centerline position information of the blood vessel structure; Alternatively, the lesion analysis sub-model is applied to the second user editor to obtain the lesion analysis results of the vascular structure; or Additionally, the blood vessel segmentation sub-model is applied to the third user editor to segment the blood vessel structure.

5. The system according to claim 2, wherein, The first user editing or the second user editing includes at least one of the following operations: editing a mask, dragging, dropping, adding graffiti, or erasing.

6. The system according to claim 1, wherein, The lesion analysis sub-model is configured to locate, characterize, or quantify lesions associated with the vascular structure.

7. The system according to claim 4, wherein, The blood vessel segmentation sub-model is configured to refine blood vessel boundaries and generate pixel-by-pixel blood vessel segmentation masks.

8. A method for analyzing medical images containing vascular structures using a sequential model, comprising: The medical image and the sequential model are received via a communication interface, wherein the sequential model includes a vessel extraction sub-model and a lesion analysis sub-model, and wherein the vessel extraction sub-model and the lesion analysis sub-model are trained independently or jointly. The blood vessel extraction sub-model is applied to the received medical image by at least one processor to extract the centerline position information of the blood vessel structure for detecting lesions along the centerline of the blood vessel structure, wherein the centerline tracks the path of the blood vessel structure. The at least one processor applies the lesion analysis sub-model to the received medical image and the centerline position information extracted by the vascular extraction sub-model to obtain the lesion analysis result of the vascular structure; and The at least one processor outputs the lesion analysis results of the vascular structure.

9. The method according to claim 8, further comprising: The communication interface receives first user edits and second user edits, wherein the first user edits and the second user edits are independent of each other; The at least one processor further applies the vascular extraction sub-model to the first user editor to extract the centerline position information of the vascular structure; or The at least one processor further applies the lesion analysis sub-model to the second user to obtain lesion analysis results for the vascular structure.

10. The method of claim 8, wherein the sequential model further comprises a vascular segmentation sub-model trained independently or jointly with respect to the vascular extraction sub-model and the lesion analysis sub-model. in, The method further includes: The at least one processor applies the blood vessel segmentation sub-model to the received medical image, the centerline position information extracted by the blood vessel extraction sub-model, and the lesion analysis results of the blood vessel structure obtained by the lesion analysis sub-model to segment the blood vessel structure from the medical image.

11. The method of claim 8, further comprising: The communication interface receives edits from a first user, a second user, and a third user, wherein the first user edit, the second user edit, and the third user edit are independent of each other. The at least one processor further applies the vascular extraction sub-model to the first user editor to extract the centerline position information of the vascular structure; The at least one processor further applies the lesion analysis sub-model to the second user editor to obtain lesion analysis results for the vascular structure; or The at least one processor further applies a vascular segmentation sub-model to the third user editor to segment the vascular structure.

12. The method according to claim 9, wherein, The first user editing or the second user editing includes at least one of the following operations: editing a mask, dragging, dropping, adding graffiti, or erasing.

13. The method of claim 8, further comprising: The at least one processor uses the lesion analysis sub-model to locate, characterize, or quantify lesions associated with the vascular structure.

14. The method of claim 11, further comprising: The at least one processor uses the blood vessel segmentation sub-model to refine the blood vessel boundaries and generate a pixel-by-pixel blood vessel segmentation mask.

15. A non-transitory computer-readable medium having a computer program stored thereon, wherein, When the computer program is implemented by at least one processor, it performs a method for analyzing medical images containing vascular structures using a sequential model, the method comprising: The medical image and the sequential model are received, wherein the sequential model includes a vessel extraction sub-model and a lesion analysis sub-model, wherein the vessel extraction sub-model and the lesion analysis sub-model are trained independently or jointly; The blood vessel extraction sub-model is applied to the received medical image to extract the centerline position information of the blood vessel structure, which is used to detect lesions along the centerline of the blood vessel structure, wherein the centerline tracks the path of the blood vessel structure. The lesion analysis submodel is applied to the received medical image and the centerline position information extracted by the vessel extraction submodel to obtain the lesion analysis results of the vascular structure; and Output the lesion analysis results of the vascular structure.

16. The non-transitory computer-readable medium according to claim 15, wherein, The method further includes: Receive edits from a first user and edits from a second user, wherein the first user edit and the second user edit are independent of each other; Alternatively, the blood vessel extraction sub-model is applied to the first user editor to extract the centerline position information of the blood vessel structure; or Additionally, the lesion analysis sub-model is applied to the second user editor to obtain lesion analysis results for the vascular structure.

17. The non-transitory computer-readable medium according to claim 15, wherein, The sequential model also includes a vascular segmentation sub-model trained independently or jointly with respect to the vascular extraction sub-model and the lesion analysis sub-model. The method further includes: The blood vessel segmentation submodel is applied to the received medical image, the centerline position information extracted by the blood vessel extraction submodel, and the lesion analysis results of the blood vessel structure obtained by the lesion analysis submodel to segment the blood vessel structure from the medical image.

18. The non-transitory computer-readable medium according to claim 17, wherein, The method further includes: The system receives edits from a first user, a second user, and a third user, wherein the first user edit, the second user edit, and the third user edit are independent of each other. Additionally, the blood vessel extraction sub-model is applied to the first user editor to extract the centerline position information of the blood vessel structure; Alternatively, the lesion analysis sub-model is applied to the second user editor to obtain the lesion analysis results of the vascular structure; or Additionally, the blood vessel segmentation sub-model is applied to the third user editor to segment the blood vessel structure.

Images