Pulmonary embolism grade evaluation method and device, equipment and storage medium

Abstract

The invention relates to the technical field of computer-aided medical treatment, and discloses a pulmonary embolism grade evaluation method and device, equipment and a storage medium. Firstly, the embolus image and the cardiopulmonary structure index are obtained in an automatic mode, the step of manually obtaining data is omitted, time is saved, and errors caused by manual operation can be reduced. Then, performing prediction evaluation based on the embolism sub-image to obtain a preliminary pulmonary embolism level prediction result; combined prediction evaluation is carried out based on the embolism sub-images and the cardiopulmonary structure indexes in different modes, pulmonary embolism information is captured more comprehensively, and a combined pulmonary embolism level prediction result is obtained. And finally, integrating the preliminary pulmonary embolism grade prediction result and the combined pulmonary embolism grade prediction result, mutually supplementing and verifying the two results, and finally obtaining a target pulmonary embolism grade prediction result. And compared with a prediction result obtained by only depending on a single method, the accuracy of the prediction result of the target pulmonary embolism level is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer-aided medical treatment, and in particular to a pulmonary embolism grade evaluation method, device, equipment and storage medium. BACKGROUND

[0002] Pulmonary embolism is a serious pulmonary vascular disease, which can progress rapidly and pose a threat to the life and health of patients.

[0003] In related technologies, clinicians diagnose based on CT pulmonary angiography (CTPA) images and laboratory parameters such as pulmonary arterial hypertension. However, this diagnosis process often relies on the subjective judgment of doctors, and the process is cumbersome and time-consuming, which may delay the best treatment opportunity. Therefore, a new pulmonary embolism grade evaluation method is needed. SUMMARY

[0004] The embodiments of the present specification aim to solve at least one of the technical problems in the related art. To this end, the embodiments of the present specification propose a pulmonary embolism grade evaluation method, device, equipment and storage medium.

[0005] The embodiments of the present specification provide a pulmonary embolism grade evaluation method, which comprises:

[0006] Obtaining embolus images and cardiopulmonary structure indicators;

[0007] Performing prediction evaluation based on the embolus images to obtain a preliminary pulmonary embolism grade prediction result;

[0008] Performing prediction evaluation based on the embolus images and the cardiopulmonary structure indicators to obtain a joint pulmonary embolism grade prediction result;

[0009] Integrating the preliminary pulmonary embolism grade prediction result and the joint pulmonary embolism grade prediction result to obtain a target pulmonary embolism grade prediction result.

[0010] In one of the embodiments, the preliminary pulmonary embolism grade prediction result comprises a preliminary pulmonary embolism prediction grade and a preliminary pulmonary embolism grade prediction probability, and the prediction evaluation based on the embolus images to obtain a preliminary pulmonary embolism grade prediction result comprises:

[0011] Performing feature extraction on the embolus images to obtain preliminary feature data;

[0012] Performing a max-pooling operation on the preliminary feature data to obtain the preliminary pulmonary embolism prediction grade;

[0013] Performing an average-pooling operation on the preliminary feature data to obtain the preliminary pulmonary embolism grade prediction probability.

[0014] In one of the embodiments, the prediction evaluation based on the embolus image and the heart-lung structure index obtains a joint pulmonary embolism grade prediction result, which includes:

[0015] The feature extraction is performed on the embolus image to obtain preliminary feature data;

[0016] The joint feature data is obtained based on the preliminary feature data and the heart-lung structure index;

[0017] The joint pulmonary embolism grade prediction result is obtained based on the joint feature data.

[0018] In one of the embodiments, the joint pulmonary embolism grade prediction result includes a joint pulmonary embolism prediction grade and a joint pulmonary embolism grade prediction probability, and the joint pulmonary embolism grade prediction result is obtained based on the joint feature data, which includes:

[0019] The maximum pooling operation is performed on the joint feature data to obtain the joint pulmonary embolism prediction grade;

[0020] The average pooling operation is performed on the joint feature data to obtain the joint pulmonary embolism grade prediction probability.

[0021] In one of the embodiments, the joint feature data is obtained based on the preliminary feature data and the heart-lung structure index, which includes:

[0022] The fusion feature is obtained based on the fusion of the preliminary feature data and the heart-lung structure index;

[0023] The joint feature data is obtained by performing the ensemble learning processing on the fusion feature.

[0024] In one of the embodiments, the embolus image and the heart-lung structure index are obtained, which includes:

[0025] The pulmonary arteriography image is obtained;

[0026] The segmentation processing and the feature extraction are performed based on the pulmonary arteriography image to obtain the embolus image;

[0027] In the case where the pulmonary arteriography image meets the preset condition, the segmentation processing and the index calculation are performed based on the pulmonary arteriography image to obtain the heart-lung structure index.

[0028] In one of the embodiments, the heart-lung structure index includes at least one of an arterial diameter ratio, a ventricular diameter ratio, and an interventricular septum angle, and the heart-lung structure index is obtained based on the segmentation processing and the index calculation of the pulmonary arteriography image, which includes:

[0029] Segmenting based on the pulmonary artery angiogram image to obtain a segmentation image, wherein the segmentation image includes a pulmonary artery vessel, an ascending aorta, a left ventricle, a right ventricle, a left lung lobe, and a right lung lobe;

[0030] Calculating a diameter ratio based on the pulmonary artery vessel and the ascending aorta to obtain an arterial diameter ratio;

[0031] Calculating a diameter ratio based on the left ventricle and the right ventricle to obtain a ventricular diameter ratio;

[0032] Determining a left-right ventricular centerline based on the left ventricle and the right ventricle;

[0033] Determining a double lung lobe centerline based on the left lung lobe and the right lung lobe;

[0034] Obtaining an interventricular septum angle based on the left-right ventricular centerline and the double lung lobe centerline.

[0035] The present specification embodiment provides a pulmonary embolism grade evaluation device, the device comprising:

[0036] A thrombus image and structure index acquisition module is configured to acquire a thrombus image and a heart-lung structure index;

[0037] A preliminary pulmonary embolism grade prediction module is configured to perform prediction evaluation based on the thrombus image to obtain a preliminary pulmonary embolism grade prediction result;

[0038] A joint pulmonary embolism grade prediction module is configured to perform prediction evaluation based on the thrombus image and the heart-lung structure index to obtain a joint pulmonary embolism grade prediction result;

[0039] A pulmonary embolism grade comprehensive prediction module is configured to comprehensively integrate the preliminary pulmonary embolism grade prediction result and the joint pulmonary embolism grade prediction result to obtain a target pulmonary embolism grade prediction result.

[0040] The present specification embodiment provides a medical imaging device, comprising: a memory and one or more processors in communication connection with the memory; the memory stores instructions executable by the one or more processors, and the instructions are executed by the one or more processors to enable the one or more processors to implement the steps of the method of any one of the above embodiments.

[0041] The present specification embodiment provides a computer device, comprising: a memory and one or more processors in communication connection with the memory; the memory stores instructions executable by the one or more processors, and the instructions are executed by the one or more processors to enable the one or more processors to implement the steps of the method of any one of the above embodiments.

[0042] The embodiments of the present specification provide a computer readable storage medium, which stores a computer program, and the computer program implements the steps of the method described in any one of the above embodiments when executed by a processor.

[0043] The embodiments of the present specification provide a computer program product, which includes instructions, and the instructions make the computer device execute the steps of the method described in any one of the above embodiments when executed by a processor of the computer device.

[0044] In the above embodiments of the present specification, first, the embolus image and the heart-lung structure index are acquired in an automatic manner, reducing the step of manually acquiring data by artificial operation, saving time, and reducing errors caused by artificial operation. Then, the preliminary pulmonary embolism grade prediction result is obtained by predicting and evaluating based on the embolus image. The joint pulmonary embolism grade prediction result is obtained by predicting and evaluating based on the embolus image and the heart-lung structure index under different modalities, more comprehensively capturing the information of pulmonary embolism. Finally, the target pulmonary embolism grade prediction result is obtained by comprehensively combining the preliminary pulmonary embolism grade prediction result and the joint pulmonary embolism grade prediction result, which complement and verify each other. Compared with the prediction result obtained by relying on a single method, the accuracy of the target pulmonary embolism grade prediction result is improved. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 A flowchart of a pulmonary embolism grade evaluation method provided by the embodiments of the present specification is shown;

[0046] Figure 2 A flowchart of obtaining a preliminary pulmonary embolism grade prediction probability provided by the embodiments of the present specification is shown;

[0047] Figure 3 A flowchart of obtaining a joint pulmonary embolism grade prediction result provided by the embodiments of the present specification is shown;

[0048] Figure 4 A flowchart of obtaining a joint pulmonary embolism grade prediction result provided by the embodiments of the present specification is shown;

[0049] Figure 5 A flowchart of obtaining joint feature data provided by the embodiments of the present specification is shown;

[0050] Figure 6a A flowchart of obtaining an embolus image and a heart-lung structure index provided by the embodiments of the present specification is shown;

[0051] Figure 6b A segmentation result image of a pulmonary lobe provided by the embodiments of the present specification is shown;

[0052] Figure 7aA flowchart for obtaining an arterial diameter ratio provided for an embodiment of the present specification;

[0053] Figure 7b A segmented image including a pulmonary artery vessel and an ascending aorta provided for an embodiment of the present specification;

[0054] Figure 7c A schematic diagram of an ascending aorta diameter and a pulmonary artery vessel diameter provided for an embodiment of the present specification;

[0055] Figure 8a A flowchart for obtaining a ventricular diameter ratio provided for an embodiment of the present specification;

[0056] Figure 8b A segmented image including a left ventricle and a right ventricle provided for an embodiment of the present specification;

[0057] Figure 9a A flowchart for obtaining an interventricular septum angle provided for an embodiment of the present specification;

[0058] Figure 9b A schematic diagram of an interventricular septum angle provided for an embodiment of the present specification;

[0059] Figure 10 A flow architecture diagram of a pulmonary embolism grade evaluation method provided for an embodiment of the present specification;

[0060] Figure 11 A schematic diagram of a pulmonary embolism grade evaluation device provided for an embodiment of the present specification;

[0061] Figure 12 An internal structure diagram of a computer device provided for an embodiment of the present specification. DETAILED DESCRIPTION

[0062] Embodiments of the present application are described in detail below with reference to examples thereof shown in the attached drawings, wherein the same or similar components or components having the same or similar functions are denoted throughout by the same or similar reference numerals. The embodiments described below by reference to the drawings are exemplary and are intended to explain the present application, and cannot be understood as limiting the present application.

[0063] Pulmonary embolism is a serious pulmonary vascular disease, which can progress rapidly and pose a threat to the life and health of patients.

[0064] In the related art, a clinician makes a diagnosis based on a CTPA image of a patient and laboratory parameters such as pulmonary arterial hypertension, however, this diagnosis process often relies on the subjective judgment of the doctor, and the process is complicated and time-consuming, which may delay the best treatment opportunity.

[0065] In the related art, the prediction of the severity of pulmonary embolism is based on machine learning or deep learning methods, and the modalities used are mostly a combination of deep learning features and clinical features. Among them, many biomarker features such as the interventricular septum angle, the left and right ventricular ratio, and the ratio of the diameter of the main pulmonary artery to the ascending aorta need to be manually marked and calculated by doctors, which not only consumes time but also is laborious to operate, and has obvious limitations. In addition, based on the different severity of pulmonary embolism in patients, the corresponding changes in blood flow dynamics and flow will occur, which will lead to significant changes in the interventricular septum angle and the ratio of the diameter of the main pulmonary artery to the ascending aorta, and the more severe the embolism, the larger these values will be.

[0066] In the related art, in terms of model construction, first, one of machine learning or deep learning methods is used to organically combine deep features and surface features, resulting in poor generalization performance of the model. Second, the number of features used to construct the model is limited, and only a few features are selected for modeling. However, for accurate assessment of the severity of pulmonary embolism, many features closely related to it such as the interventricular septum angle, the left and right ventricular ratio, and the ratio of the diameter of the main pulmonary artery to the ascending aorta are not fully utilized, which affects the model's assessment of the severity of pulmonary embolism.

[0067] Therefore, the present embodiment provides a pulmonary embolism grade evaluation method. First, the embolus image and the heart-lung structure index are obtained in an automatic manner, reducing the step of manually obtaining data by artificial, not only saving time, but also reducing the error caused by artificial operation. Then, based on the embolus image, a preliminary pulmonary embolism grade prediction result is obtained. Based on the embolus image and the heart-lung structure index under different modalities, a joint prediction evaluation is performed to more comprehensively capture the information of pulmonary embolism, and a joint pulmonary embolism grade prediction result is obtained. Finally, the preliminary pulmonary embolism grade prediction result and the joint pulmonary embolism grade prediction result are comprehensively combined, and the target pulmonary embolism grade prediction result is finally obtained. Compared with the prediction result obtained by relying on a single method, the accuracy of the target pulmonary embolism grade prediction result is improved.

[0068] The present embodiment provides a pulmonary embolism grade evaluation method, which can include the following steps: Figure 1 The pulmonary embolism grade evaluation method can include the following steps:

[0069] S110, obtaining an embolus image and a heart-lung structure index.

[0070] S120, based on the embolus image, a preliminary pulmonary embolism grade prediction result is obtained.

[0071] S130, based on the embolus image and the heart-lung structure index, a joint pulmonary embolism grade prediction result is obtained.

[0072] S140, integrate the preliminary pulmonary embolism grade prediction result and the joint pulmonary embolism grade prediction result to obtain a target pulmonary embolism grade prediction result.

[0073] Specifically, a target part of a user to be detected including a region related to a lung, a heart and a blood vessel is scanned using a medical imaging device to obtain a medical image. After obtaining the medical image, an image processing algorithm is used to perform processing operation and related calculation on the medical image, so that a thrombus image can be extracted and a cardio-pulmonary structure index can be calculated. Next, a prediction model taking the extracted thrombus image as input data is used to analyze and predict the corresponding features of the thrombus to obtain a preliminary pulmonary embolism grade prediction result. A prediction model taking the thrombus image and the cardio-pulmonary structure index as input is used to process multi-modal data to obtain more information, and a joint pulmonary embolism grade prediction result is obtained. Finally, the preliminary pulmonary embolism grade prediction result and the joint pulmonary embolism grade prediction result are comprehensively analyzed, and the two results are mutually complementary and mutually verified to obtain a more accurate target pulmonary embolism grade prediction result.

[0074] Exemplarily, on a test set of 94 cases, the accuracy of the pulmonary embolism grade prediction result determined by the above embodiment can reach 0.9.

[0075] In the above embodiment, first, the thrombus image and the cardio-pulmonary structure index are obtained in an automatic manner, which reduces the step of manually obtaining data, saves time, and reduces errors caused by manual operation. Then, the thrombus image is used for prediction evaluation to obtain a preliminary pulmonary embolism grade prediction result. Joint prediction evaluation is performed based on the thrombus image and the cardio-pulmonary structure index under different modalities to more comprehensively capture information of pulmonary embolism and obtain a joint pulmonary embolism grade prediction result. Finally, the preliminary pulmonary embolism grade prediction result and the joint pulmonary embolism grade prediction result are integrated, and the two results are mutually complementary and mutually verified to ultimately obtain a target pulmonary embolism grade prediction result. Compared with the prediction result obtained by relying on a single method, the accuracy of the target pulmonary embolism grade prediction result is improved.

[0076] In some embodiments, referring to Figure 2 , the preliminary pulmonary embolism grade prediction result includes a preliminary pulmonary embolism prediction grade and a preliminary pulmonary embolism grade prediction probability, and the prediction evaluation based on the thrombus image can include the following steps:

[0077] S210, feature extraction is performed on the thrombus image to obtain preliminary feature data.

[0078] S220, maximum pooling operation is performed on the preliminary feature data to obtain a preliminary pulmonary embolism prediction grade.

[0079] S230, performing an average pooling operation on the preliminary feature data to obtain a preliminary pulmonary embolism level prediction probability.

[0080] Specifically, a deep learning model trained by a large amount of data is used to extract features from the embolus images, capturing multi-dimensional information presented in the embolus images, including but not limited to the shape, size, and density distribution of the embolus, to obtain preliminary feature data corresponding to each embolus image. Then, the preliminary feature data corresponding to each embolus image is collectively input to perform a max-pooling operation. The max-pooling operation selects the maximum value of the preliminary feature data corresponding to each embolus image, achieving the purpose of dimension reduction while retaining the most prominent and representative feature information. Then, further probability prediction is performed to determine the pulmonary embolism level corresponding to the maximum probability, which is taken as the preliminary pulmonary embolism prediction level of the user to be detected.

[0081] The preliminary feature data corresponding to each embolus image is collectively input to perform an average pooling operation. The average pooling operation calculates the average value of the preliminary feature data corresponding to each embolus image to smooth the feature data and obtain more robust statistical features. Further probability prediction can quantitatively express the possibility of the pulmonary embolism level of the user to be detected in a more balanced manner, obtaining the probability corresponding to each pulmonary embolism level, i.e., the preliminary pulmonary embolism level prediction probability of the user to be detected.

[0082] For example, a convolutional neural network (CNN) is used to extract features from the embolus images to obtain preliminary feature data. Then, a GMP (Global Max Pooling) is used to perform a max-pooling operation on the preliminary feature data to obtain a preliminary pulmonary embolism prediction level. A GAP (Global Average Pooling) is used to perform an average pooling operation on the preliminary feature data to obtain a preliminary pulmonary embolism level prediction probability.

[0083] In the above embodiments, features are extracted from the embolus images to obtain preliminary feature data, a max-pooling operation is performed on the preliminary feature data to obtain a preliminary pulmonary embolism prediction level, and an average pooling operation is performed on the preliminary feature data to obtain a preliminary pulmonary embolism level prediction probability, providing a data basis for subsequent target pulmonary embolism level prediction results.

[0084] In some embodiments, referring to Figure 3 , the prediction evaluation based on the embolus images and the cardiopulmonary structure indicators to obtain a joint pulmonary embolism level prediction result can include the following steps:

[0085] S310, performing feature extraction on the embolus images to obtain preliminary feature data.

[0086] S320, obtain joint feature data based on the preliminary feature data and the cardiopulmonary structure index.

[0087] S330, obtain joint pulmonary embolism grade prediction result based on the joint feature data.

[0088] Specifically, a deep learning model trained by a large amount of data is used to extract features of the embolus image, capture multi-dimensional information presented in the embolus image, including but not limited to the shape, size, density distribution of the embolus, etc., and obtain preliminary feature data corresponding to each embolus image. On this basis, the preliminary feature data and the cardiopulmonary structure index are fused and processed to consider the mutual relationship and correlation between the two, and the joint feature data corresponding to each embolus image is obtained. Finally, a machine learning model is used to predict the pulmonary embolism grade of the joint feature data as input data, and a joint pulmonary embolism grade prediction result is obtained.

[0089] In some embodiments, in addition to the preliminary feature data and the cardiopulmonary structure index, clinical data can also be combined to obtain joint feature data, that is, the preliminary feature data, the cardiopulmonary structure index and the clinical data are fused and processed to obtain joint feature data. Exemplarily, the clinical data can include the overall volume of pulmonary embolism, blood test results (such as D-dimer, brain natriuretic peptide), etc. Among them, the overall volume of pulmonary embolism can be obtained by using the minimum bounding box algorithm on the embolus image.

[0090] In the above embodiments, the embolus image is subjected to feature extraction to obtain preliminary feature data, joint feature data is obtained based on the preliminary feature data and the cardiopulmonary structure index, and joint pulmonary embolism grade prediction result is obtained based on the joint feature data, which provides a data basis for subsequent target pulmonary embolism grade prediction result.

[0091] In some embodiments, referring to Figure 4 , the joint pulmonary embolism grade prediction result includes joint pulmonary embolism prediction grade and joint pulmonary embolism grade prediction probability, and obtaining joint pulmonary embolism grade prediction result based on joint feature data can include the following steps:

[0092] S410, perform maximum pooling operation on the joint feature data to obtain joint pulmonary embolism prediction grade.

[0093] S420, perform average pooling operation on the joint feature data to obtain joint pulmonary embolism grade prediction probability.

[0094] Specifically, the joint feature data corresponding to each embolus image is taken as input together to perform a max-pooling operation. The max-pooling operation selects the maximum value of the joint feature data corresponding to each embolus image, thereby achieving the purpose of dimension reduction while retaining the most prominent and representative feature information. Then, probability prediction is further performed to determine the pulmonary embolism grade corresponding to the maximum probability value, which is taken as the joint pulmonary embolism prediction grade of the user to be detected.

[0095] The joint feature data corresponding to each embolus image is taken as input together to perform an average-pooling operation. The average-pooling operation calculates the average value of the joint feature data corresponding to each embolus image to perform smoothing processing on the feature data, thereby obtaining more robust statistical features. Then, probability prediction is further performed to quantitatively express the possibility of the pulmonary embolism grade of the user to be detected in a more balanced manner, thereby obtaining the probability corresponding to each pulmonary embolism grade, i.e., the joint pulmonary embolism grade prediction probability of the user to be detected.

[0096] In the above embodiments, the joint feature data is subjected to max-pooling operation to obtain the joint pulmonary embolism prediction grade, and the joint feature data is subjected to average-pooling operation to obtain the joint pulmonary embolism grade prediction probability, thereby providing a data basis for obtaining the target pulmonary embolism grade prediction result.

[0097] In some embodiments, referring to Figure 5 Based on the preliminary feature data and the cardiopulmonary structure index, the joint feature data can be obtained by the following steps:

[0098] S510, fusion processing is performed based on the preliminary feature data and the cardiopulmonary structure index to obtain fusion features.

[0099] S520, integrated learning processing is performed on the fusion features to obtain joint feature data.

[0100] Specifically, the preliminary feature data and the cardiopulmonary structure index are subjected to fusion processing to obtain fusion features with more dimensions and more comprehensive information. Then, the fusion features are further processed by using an integrated learning method to generate joint feature data with better representation ability. For example, the integrated learning algorithm can use a random forest model. The fusion features are subjected to integrated learning processing by using the random forest model to obtain the joint feature data.

[0101] In the above embodiments, the preliminary feature data and the cardiopulmonary structure index are subjected to fusion processing to obtain fusion features, and the fusion features are subjected to integrated learning processing to obtain joint feature data, thereby providing a data basis for subsequent machine learning of the side pulmonary embolism grade prediction.

[0102] In some embodiments, referring to Figure 6a, obtaining the embolus image and the cardio-pulmonary structure index can comprise the following steps:

[0103] S610, obtaining a pulmonary angiography image.

[0104] S620, performing segmentation processing and feature extraction based on the pulmonary angiography image to obtain the embolus image.

[0105] Specifically, a medical imaging device is used to scan a target site of a user to be detected, which includes a region related to the lung, heart and blood vessels, to obtain a pulmonary angiography image. After obtaining the pulmonary angiography image, a segmentation algorithm is used to perform segmentation processing on the pulmonary angiography image to obtain a segmentation result image including emboli. Then, feature extraction of the emboli is performed on the segmentation result image including the emboli to extract an embolus image corresponding to each embolus from the segmentation result image, so that the embolus image can be multiple.

[0106] Exemplarily, the pulmonary angiography image is segmented by an automatic segmentation algorithm (Unet) to obtain a segmentation result image including emboli. Then, feature extraction of the emboli is performed on the segmentation result image including the emboli by using a minimum bounding box algorithm to obtain an embolus image corresponding to each embolus.

[0107] In some embodiments, the pulmonary angiography image can be adjusted in window width and window level to optimize the contrast and detail presentation of the image. Then, the adjusted image is segmented and features are extracted.

[0108] S630, in the case where the pulmonary angiography image meets the preset condition, performing segmentation processing and index calculation based on the pulmonary angiography image to obtain a cardio-pulmonary structure index.

[0109] Specifically, the quality of the pulmonary angiography image obtained by the medical imaging device varies, some images are clear and have high contrast, and can accurately present the structural details of the pulmonary artery and its surrounding tissues, while some images may have quality problems such as blurring, more noise, and unclear blood vessel edges due to improper device parameter setting, patient physiological motion interference and other factors. Therefore, the quality of the pulmonary angiography image needs to be evaluated. In the case where the quality of the pulmonary angiography image does not meet the preset condition, it indicates that the quality of the pulmonary angiography image is poor, which is not convenient for subsequent local segmentation and subsequent calculation of cardio-pulmonary structure index, and affects the accuracy of subsequent data. Therefore, in the case where the quality of the pulmonary angiography image meets the preset condition, a segmentation algorithm is used to perform segmentation processing on the pulmonary angiography image to obtain a segmentation result image of the lung lobe. Then, index calculation is performed based on the segmentation result image of the lung lobe to obtain a cardio-pulmonary structure index.

[0110] Exemplarily, the pulmonary arteriography image is segmented by an automatic segmentation algorithm (Unet) to obtain a segmentation result of lung lobes. Please refer to Figure 6b , Figure 6b The segmentation result image of the lung lobes is shown.

[0111] In the above embodiment, the pulmonary arteriography image is obtained, segmentation processing and feature extraction are performed based on the pulmonary arteriography image to obtain a embolus image, and in the case that the pulmonary arteriography image meets a preset condition, segmentation processing and index calculation are performed based on the pulmonary arteriography image to obtain a cardiopulmonary structure index, which is the basis for subsequent preliminary and joint pulmonary embolism grade prediction.

[0112] In some embodiments, please refer to Figure 7a The cardiopulmonary structure index includes at least one of an artery diameter ratio, a ventricular diameter ratio, and an interventricular septum angle; and the segmentation processing and index calculation based on the pulmonary arteriography image to obtain the cardiopulmonary structure index can include the following steps:

[0113] S710, segmentation processing is performed based on the pulmonary arteriography image to obtain a segmentation image.

[0114] S720, a diameter ratio calculation is performed based on the pulmonary artery and the ascending aorta to obtain an artery diameter ratio.

[0115] The segmentation image includes the pulmonary artery, the ascending aorta, the left ventricle, the right ventricle, the left lung lobe, and the right lung lobe.

[0116] Specifically, the segmentation algorithm is used to segment the pulmonary arteriography image to obtain a segmentation image including the pulmonary artery, the ascending aorta, the left ventricle, the right ventricle, the left lung lobe, and the right lung lobe. Next, in the segmentation image, the two-dimensional layer with the largest area of the pulmonary artery and the ascending aorta is selected, which can best show the morphological characteristics of the target blood vessels. In the selected two-dimensional layer, two pixel points farthest apart are found in the pulmonary artery along the horizontal coordinate direction and connected, and two pixel points farthest apart are found in the ascending aorta along the horizontal coordinate direction and connected, under the condition that the horizontal coordinates are consistent. Finally, the length ratio of the two connecting lines is calculated to obtain an artery diameter ratio for measuring the size relationship of the blood vessels.

[0117] Exemplarily, please refer to Figure 7b , Figure 7b The segmentation image including the pulmonary artery and the ascending aorta is shown. Please refer to Figure 7c , Figure 7c The blue connecting line is the diameter of the ascending aorta, and the orange connecting line is the diameter of the pulmonary artery.

[0118] Exemplarily, on the 94 test sets, the artery diameter ratio consistency calculated by the above embodiment is 0.75.

[0119] In the above embodiment, the segmentation processing is performed based on the pulmonary angiography image to obtain a segmentation image, and the diameter ratio calculation is performed based on the pulmonary artery and ascending aorta to obtain the artery diameter ratio, which is the basis for subsequent joint pulmonary embolism grade prediction.

[0120] In some embodiments, referring to Figure 8a , the cardio-pulmonary structure indicators include at least one of the artery diameter ratio, the ventricular diameter ratio, and the interventricular septum angle; the segmentation processing and the indicator calculation are performed based on the pulmonary angiography image to obtain the cardio-pulmonary structure indicators, which can include the following steps:

[0121] S810, performing segmentation processing based on the pulmonary angiography image to obtain a segmentation image.

[0122] S820, performing diameter ratio calculation based on the left ventricle and the right ventricle to obtain a ventricular diameter ratio.

[0123] The segmentation image includes the pulmonary artery, the ascending aorta, the left ventricle, the right ventricle, the left lung lobe, and the right lung lobe.

[0124] Specifically, the segmentation algorithm is used to perform segmentation processing on the pulmonary angiography image to obtain a segmentation image including the pulmonary artery, the ascending aorta, the left ventricle, the right ventricle, the left lung lobe, and the right lung lobe. Next, in the segmentation image, the two-dimensional layer with the largest area of the left ventricle and the right ventricle is selected, which can best reveal the spatial relationship. In the selected two-dimensional layer, two pixel points farthest apart are found in the left ventricle along the horizontal coordinate direction and connected, and two pixel points farthest apart are found in the right ventricle along the horizontal coordinate direction and connected, under the condition that the horizontal coordinates are consistent. Finally, the length ratio of the two connecting lines is calculated to obtain the ventricular diameter ratio.

[0125] Exemplarily, referring to Figure 8b , Figure 8b The segmentation image including the left ventricle and the right ventricle is shown.

[0126] Exemplarily, on the 94 test sets, the consistency of the ventricular diameter ratio calculated by the above embodiment is 0.74.

[0127] In the above embodiment, the segmentation processing is performed based on the pulmonary angiography image to obtain a segmentation image, and the diameter ratio calculation is performed based on the left ventricle and the right ventricle to obtain the ventricular diameter ratio, which is the basis for subsequent joint pulmonary embolism grade prediction.

[0128] In some embodiments, referring to Figure 9a , the cardio-pulmonary structure indicators include at least one of the artery diameter ratio, the ventricular diameter ratio, and the interventricular septum angle; the segmentation processing and the indicator calculation are performed based on the pulmonary angiography image to obtain the cardio-pulmonary structure indicators, which can include the following steps:

[0129] S910, segmentation processing is performed based on the pulmonary angiography image to obtain a segmentation image.

[0130] S920, based on the left ventricle and the right ventricle, a left-right ventricular center line is determined.

[0131] S930, based on the left lung lobe and the right lung lobe, a double lung lobe center line is determined.

[0132] S940, based on the left-right ventricular center line and the double lung lobe center line, an intraventricular septum angle is obtained.

[0133] The segmentation image includes pulmonary artery vessels, ascending aorta, left ventricle, right ventricle, left lung lobe, and right lung lobe.

[0134] Specifically, the pulmonary angiography image is segmented by using a segmentation algorithm to obtain a segmentation image including pulmonary artery vessels, ascending aorta, left ventricle, right ventricle, left lung lobe, and right lung lobe. Next, in the segmentation image, the two-dimensional layer with the largest area of the left ventricle and the right ventricle is selected. In the selected two-dimensional layer, first, the highest point and the lowest point of the left ventricle are determined, and the two points are connected, and the midpoint of the connecting line is taken; similarly, for the right ventricle, the highest point and the lowest point are also determined and connected, and the midpoint of the connecting line is taken. The two midpoints taken by the left ventricle and the right ventricle are connected, and the connecting line formed thereby is the left-right ventricular center line. In the selected two-dimensional layer, the highest point and the lowest point of the left lung lobe are determined, and the highest point and the lowest point of the right lung lobe are determined. The highest point and the lowest point of the left lung lobe are connected to take the midpoint, and the same is done for the right lung lobe. The two midpoints taken by the left lung lobe and the right lung lobe are connected, and this connecting line is the double lung lobe center line. Finally, the intersection angle formed by the left-right ventricular center line and the double lung lobe center line is the intraventricular septum angle.

[0135] For example, in FIG. 1, Figure 9b , Figure 9b The blue connecting line in the figure is the left-right ventricular center line, and the orange connecting line is the double lung lobe center line. The intersection angle between the blue connecting line and the orange connecting line is the intraventricular septum angle.

[0136] For example, on the 94 test sets, the intraclass correlation coefficient (ICC) consistency of the intraventricular septum angle calculated by the above embodiment is 0.76.

[0137] In the above embodiment, based on the pulmonary angiography image, segmentation processing is performed to obtain a segmentation image, based on the left ventricle and the right ventricle, a left-right ventricular center line is determined, based on the left lung lobe and the right lung lobe, a double lung lobe center line is determined, and based on the left-right ventricular center line and the double lung lobe center line, an intraventricular septum angle is obtained, which is the basis for subsequent joint pulmonary embolism grade prediction.

[0138] The embodiment of the present specification provides an embodiment of a pulmonary embolism grade evaluation method, which combines deep learning and machine learning and uses a multiple instance learning algorithm. Different from supervised learning, semi-supervised learning and unsupervised learning modes, the unique feature is to use a multiple instance bag as a basic training unit. The method relies on pulmonary angiography images, and combines clinical data for comprehensive analysis to obtain a target pulmonary embolism grade prediction result. The pulmonary embolism grade can be divided into three severity levels: mild (0), moderate (1) and severe (2).

[0139] Specifically, first, a pulmonary angiography image is obtained. Then, based on the pulmonary angiography image, segmentation processing and feature extraction are performed to obtain embolus images Patch_1, …, Patch_K, …, Patch_N. In the case where the pulmonary angiography image meets a preset condition, segmentation processing and index calculation are performed based on the pulmonary angiography image to obtain an interventricular septum angle A, an artery diameter ratio R1 and a ventricular diameter ratio R2.

[0140] Then, referring to Figure 10 , by performing feature extraction on the embolus images Patch_1, …, Patch_K, …, Patch_N, preliminary feature data P1, …, P k , …, P n are obtained. The GMP is used to perform a maximum pooling operation on the preliminary feature data P1, …, P k , …, P n to obtain a preliminary pulmonary embolism prediction grade N. The GAP is used to perform a maximum pooling operation on the preliminary feature data P1, …, P k , …, P n to obtain a preliminary pulmonary embolism grade prediction probability M.

[0141] The preliminary feature data P1, the interventricular septum angle A, the clinical data C, the artery diameter ratio R1 and the ventricular diameter ratio R2 are fused to obtain fusion features, and then the fusion features are processed by a random forest for ensemble learning to obtain joint feature data P 1c , …, the preliminary feature data P k , the interventricular septum angle A, the clinical data C, the artery diameter ratio R1 and the ventricular diameter ratio R2 are fused to obtain fusion features, and then the fusion features are processed by a random forest for ensemble learning to obtain joint feature data P kc , …, the preliminary feature data P n , the interventricular septum angle A, the clinical data C, the artery diameter ratio R1 and the ventricular diameter ratio R2 are fused to obtain fusion features, and then the fusion features are processed by a random forest for ensemble learning to obtain joint feature data Pnc The joint feature data P 1c , …, P kc , …, P nc is subjected to a max-pooling operation to obtain a joint pulmonary embolism prediction level N c The joint feature data P 1c , …, P kc , …, P nc is subjected to a max-pooling operation to obtain a joint pulmonary embolism level prediction probability M c .

[0142] The embodiment of the present specification provides a pulmonary embolism level evaluation device 1100, please refer to Figure 11 , the pulmonary embolism level evaluation device 1100 includes: embolus image and structure index acquisition module 1110, preliminary pulmonary embolism level prediction module 1120, joint pulmonary embolism level prediction module 1130, pulmonary embolism level comprehensive prediction module 1140.

[0143] The embolus image and structure index acquisition module 1110 is configured to acquire embolus images and cardiopulmonary structure indexes.

[0144] The preliminary pulmonary embolism level prediction module 1120 is configured to perform prediction evaluation based on the embolus images to obtain a preliminary pulmonary embolism level prediction result.

[0145] The joint pulmonary embolism level prediction module 1130 is configured to perform prediction evaluation based on the embolus images and the cardiopulmonary structure indexes to obtain a joint pulmonary embolism level prediction result.

[0146] The pulmonary embolism level comprehensive prediction module 1140 is configured to comprehensively predict the preliminary pulmonary embolism level prediction result and the joint pulmonary embolism level prediction result to obtain a target pulmonary embolism level prediction result.

[0147] For specific description of the pulmonary embolism level evaluation device, please refer to the description of the pulmonary embolism level evaluation method in the foregoing, which will not be repeated here.

[0148] The embodiment of the present specification provides a medical imaging device, which includes a memory and a processor, the memory stores a computer program, and the processor executes the computer program to realize the method steps in the above embodiments.

[0149] In some embodiments, a computer device is provided, which can be a terminal, and its internal structure diagram can be as shown in Figure 12As shown in the figure. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The communication interface of the computer device is used to communicate with external terminals in a wired or wireless manner. The wireless manner can be achieved through WIFI, operator network, NFC (near field communication) or other technologies. The computer program is executed by the processor to implement a pulmonary embolism grade evaluation method. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the computer device. It can also be an external keyboard, touchpad or mouse, etc.

[0150] Those skilled in the art can understand that, Figure 12 The skilled in the art can understand that,

[0151] In some embodiments, a computer device is provided, including a memory and a processor, the memory stores a computer program, and the processor executes the computer program to implement the method steps in the above embodiments.

[0152] An embodiment of the present specification provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps of the method in any one of the above embodiments.

[0153] An embodiment of the present specification provides a computer program product, which includes instructions, and the instructions are executed by a processor of a computer device to enable the computer device to perform the steps of the method of any one of the above embodiments.

[0154] It is to be appreciated that the logical and / or steps represented in the flow diagrams, or otherwise described herein, can be considered as a sequence of executable instructions for implementing the logical function, and can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor- containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions, or a combination thereof. For purposes of this specification, a "computer-readable medium" can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The computer-readable medium can be a product of taping and / or packaging the computer program. The computer-readable medium can be, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electrical connection having one or more wires (electrical connections), a portable computer diskette (magnetic), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium can even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, for example, via an optical scanner, then compiled, interpreted, or otherwise processed in a suitable manner, if necessary, and then stored in a computer memory.

Claims

1. A method for assessing the grade of pulmonary embolism, characterized in that, The method includes: Obtain images of emboli and cardiopulmonary structural parameters; Based on the embolism images, a preliminary prediction of the pulmonary embolism level is obtained; Based on the embolism image and the cardiopulmonary structural indicators, a combined pulmonary embolism level prediction result is obtained; By combining the preliminary pulmonary embolism level prediction results and the combined pulmonary embolism level prediction results, the target pulmonary embolism level prediction result is obtained.

2. The method according to claim 1, characterized in that, The preliminary pulmonary embolism level prediction result includes a preliminary pulmonary embolism prediction level and a preliminary pulmonary embolism level prediction probability. The prediction and evaluation based on the embolus image to obtain the preliminary pulmonary embolism level prediction result includes: Feature extraction is performed on the embolus image to obtain preliminary feature data; Max pooling is performed on the preliminary feature data to obtain the preliminary pulmonary embolism prediction level; The preliminary feature data is subjected to average pooling to obtain the preliminary pulmonary embolism level prediction probability.

3. The method according to claim 1, characterized in that, The prediction and evaluation based on the embolism image and the cardiopulmonary structural indicators to obtain a combined pulmonary embolism grade prediction result includes: Feature extraction is performed on the embolus image to obtain preliminary feature data; Based on the preliminary feature data and the cardiopulmonary structure indicators, joint feature data is obtained; Based on the combined feature data, the combined pulmonary embolism level prediction result is obtained.

4. The method according to claim 3, characterized in that, The combined pulmonary embolism level prediction result includes a combined pulmonary embolism prediction level and a combined pulmonary embolism level prediction probability. The process of obtaining the combined pulmonary embolism level prediction result based on the combined feature data includes: Max pooling is performed on the combined feature data to obtain the combined pulmonary embolism prediction level; The combined feature data is subjected to average pooling to obtain the combined pulmonary embolism level prediction probability.

5. The method according to claim 3 or 4, characterized in that, The process of obtaining joint feature data based on the preliminary feature data and the cardiopulmonary structural indicators includes: Based on the preliminary feature data and the cardiopulmonary structure indicators, a fusion process is performed to obtain the fused features; The fused features are subjected to ensemble learning processing to obtain the joint feature data.

6. The method according to claim 1, characterized in that, The acquisition of embolus images and cardiopulmonary structural parameters includes: Obtain pulmonary angiography images; The embolus image is obtained by segmentation and feature extraction based on the pulmonary angiography image; If the pulmonary angiography image meets the preset conditions, the cardiopulmonary structural indicators are obtained by segmenting and calculating the indicators based on the pulmonary angiography image.

7. The method according to claim 6, wherein the cardiopulmonary structural indicators include at least one of arterial diameter ratio, ventricular diameter ratio, and interventricular septal angle; the step of segmenting and calculating the cardiopulmonary structural indicators based on the pulmonary angiography image to obtain the cardiopulmonary structural indicators includes: The pulmonary angiography image is segmented to obtain a segmented image, wherein the segmented image includes the pulmonary artery, ascending aorta, left ventricle, right ventricle, left lobe and right lobe; The arterial diameter ratio is obtained by calculating the diameter ratio of the pulmonary artery and the ascending aorta. The ventricular diameter ratio is obtained by calculating the diameter ratio of the left ventricle and the right ventricle. Based on the left ventricle and the right ventricle, determine the center lines of the left and right ventricles; Determine the center line of both lung lobes based on the left and right lung lobes; The interventricular septal angle is obtained based on the central lines of the left and right ventricles and the central lines of the two lung lobes.

8. A device for assessing the level of pulmonary embolism, characterized in that, The device includes: Embolism image and structural index acquisition module, used to acquire embolism images and cardiopulmonary structural indexes; The preliminary pulmonary embolism level prediction module is used to perform prediction and evaluation based on the embolus image to obtain a preliminary pulmonary embolism level prediction result; The combined pulmonary embolism level prediction module is used to perform prediction and evaluation based on the embolus image and the cardiopulmonary structural indicators to obtain the combined pulmonary embolism level prediction result. The pulmonary embolism level prediction module is used to combine the preliminary pulmonary embolism level prediction results and the combined pulmonary embolism level prediction results to obtain the target pulmonary embolism level prediction result.

9. A medical imaging device, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images