Tumor classification method and device based on echocardiogram video

Abstract

The invention provides a tumor classification method based on an echocardiogram video, and the method comprises the steps: S1, obtaining echocardiogram video data, extracting and obtaining video key frame data, and constructing an image data set; s2, traditional feature extraction is carried out on the echocardiogram video data, and traditional feature and depth feature extraction is carried out on the image data set; s3, a single-mode data machine learning classification model and a multi-mode data machine learning classification model are constructed, the single-mode data machine learning classification model processes the traditional features and the depth features to obtain a single-mode data classification result, and the multi-mode data machine learning classification model processes the depth features to obtain a multi-mode data classification result; a multi-modal data classification result is obtained; and S4, performing analysis and comparison processing based on the single-modal data classification result and the multi-modal data classification result to obtain a final classification result. According to the method, the accuracy can be improved based on the constructed classification model, and the method is of great significance to clinical diagnosis and cardiac function evaluation.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a tumor classification method and device based on echocardiography video. Background Art

[0002] Cardiac tumors are an important part of oncology practice. Tumors include a range of lesions or masses that can be classified as neoplastic or non-neoplastic. Neoplastic lesions can be further divided into primary and secondary tumors. Up to 90% of primary cardiac tumors are benign. The incidence of primary cardiac tumors in clinical diagnosis is approximately 1380 cases per 100 million people, and the incidence of secondary cardiac tumors is 22 to 132 times higher than that of primary cardiac tumors. The correct classification of these lesions is an important predictor of prognosis.

[0003] There is great value in classifying the benign and malignant nature of cardiac tumors through ultrasound videos or images. At present, the benign and malignant nature of cardiac tumors can generally be determined through surgical biopsy. Although surgical biopsy has the advantages of high accuracy and detailed information, it has the disadvantages of greater trauma, difficult postoperative recovery and high risk. For patients with mild symptoms, surgical conservative treatment can be exempted. If the benign and malignant nature of the tumor can be predicted in a non-invasive way based on echocardiographic videos and images, patients with mild benign cardiac tumors can be spared from surgical trauma and have a better medical experience, while also reducing the medical costs of the disease.

[0004] In recent years, many scholars have conducted in-depth research on medical videos and images. In terms of echocardiography, traditional image processing methods can extract the geometric features, grayscale features, texture features, etc. of tumors, and clinically judge whether there are imaging features such as calcification and necrosis. Omics research based on these traditional features can achieve benign and malignant tumor classification, but the accuracy rate is around 60%, and there is still a lot of room for improvement, which is not enough to fully and effectively support clinical judgment.

[0005] In view of the above problems existing in the prior art, it is urgent to design a tumor classification method based on echocardiography video to solve the above problems. Summary of the invention

[0006] In order to overcome the technical problems existing in the prior art, the present application provides a tumor classification method based on echocardiography video, thereby being able to solve the problems raised in the above-mentioned background technology.

[0007] The present application provides a tumor classification method based on echocardiography video, comprising the following steps:

[0008] Step S1: Acquire and preprocess echocardiography video data to extract video key frame data, and construct an image data set based on the video key frame data;

[0009] Step S2: performing traditional feature extraction on the echocardiography video data, and performing traditional feature and deep feature extraction on the image data set;

[0010] Step S3: constructing a unimodal data machine learning classification model and a multimodal data machine learning classification model, wherein the unimodal data machine learning classification model processes the traditional features and the deep features to obtain a unimodal data classification result, and the multimodal data machine learning classification model processes the deep features to obtain a multimodal data classification result;

[0011] Step S4: Analyze and compare the unimodal data classification result and the multimodal data classification result to obtain a final classification result.

[0012] In some embodiments, the single modality includes traditional features of the image dataset, traditional features of the echocardiography video data, deep features of the echocardiography video data, and clinical features of the patient;

[0013] The multimodality includes combined features of at least two of traditional features of the image dataset, traditional features of the echocardiography video data, deep features of the echocardiography video data, and clinical features of the patient.

[0014] By extracting and screening traditional features and deep features from image datasets and echocardiography video data to remove these redundant features, the model can more centrally learn the features that are truly important for tumor classification and diagnosis, thereby improving the training efficiency and accuracy of the model.

[0015] In some embodiments, the unimodal data machine learning classification model includes a machine learning model based on traditional features of the image dataset, a machine learning model based on traditional features of the echocardiography video data, a machine learning model based on patient clinical features, and a machine learning model based on deep features of the image dataset;

[0016] The multimodal data machine learning classification model includes a machine learning model based on traditional features and clinical features of the image dataset, a machine learning model based on traditional features and clinical features of the image dataset, a machine learning model based on traditional features, deep features and clinical features of the image dataset, and a machine learning model based on traditional features, deep features, traditional features and clinical features of the image dataset.

[0017] The unimodal data machine learning classification model is mainly constructed based on traditional features and deep features separately, so as to obtain effective and reliable classification results simply and intuitively. The multimodal model can make use of the complementarity between different modal data to make up for the shortcomings of unimodal data, thereby improving the accuracy of classification.

[0018] In some embodiments, the unimodal data machine learning classification model is constructed based on any one of a logistic regression model, a decision tree model, a support vector machine model, a random forest model, and an extreme gradient boosting model;

[0019] The multimodal data machine learning classification model is constructed by integrating any of a logistic regression model, a decision tree model, a support vector machine model, a random forest model and an extreme gradient boosting model.

[0020] By setting a variety of different types of learning models, we can enrich the types and functional diversification of training models, so that single-modal data machine learning classification models and multi-modal data machine learning classification models can complement each other's advantages and adapt to different data distributions, thereby achieving the purpose of enhancing the accuracy, stability and reliability of single-modal data machine learning classification models and multi-modal data machine learning classification models.

[0021] In some embodiments, the step S1 comprises:

[0022] Step S11: using a programming language to uniformly align the frame number and frame rate of the echocardiography video data to obtain an echocardiography video frame;

[0023] Step S12: converting the echocardiogram video frame into a grayscale image;

[0024] Step S13: performing optical flow analysis on the grayscale image to obtain optical flow information, and analyzing the optical flow information to obtain an average amount of motion and a standard deviation;

[0025] Step S14: setting a preset threshold based on the average exercise amount and the standard deviation;

[0026] Step S15: obtaining a frame of the echocardiography video data as the current frame image, comparing the motion speed information of the current frame image with the preset threshold, and determining that the current frame image whose motion speed information is greater than the preset threshold is a key frame image;

[0027] Step S16: Save and extract the key frame images to construct the image dataset.

[0028] By processing the echocardiography video data, we can avoid the adverse effects on the operation of the subsequent machine learning classification model due to inconsistent frame numbers or frame rates, thereby ensuring the accuracy of the subsequent classification results. In addition, it also makes different videos directly comparable, so that different video images can be accurately and objectively compared and analyzed.

[0029] In some embodiments, step S2 comprises:

[0030] Step S21: extracting the traditional features of the tumor in the image dataset and the echocardiography video data based on tumor annotation using a programming language through the Pyradiomics toolkit;

[0031] Step S22: extracting the deep features of the tumor in the image dataset based on the Resnet-50 deep neural network model.

[0032] By using professional programming languages ​​and toolkits to screen features, redundant features can be effectively filtered out, reducing the risk of overfitting and facilitating the training of machine learning models.

[0033] In some implementations, the step S4 specifically includes: comparing the unimodal data classification result and the multimodal data classification result by cross-validation, and taking the average value as the final classification result.

[0034] By comparing the classification results of single-modal and multi-modal data, we can comprehensively consider the advantages of different modes, reduce the deviation that may be caused by a single mode, and use multi-modal complementary information to provide a more comprehensive evaluation perspective. The stability of each model is compared by cross-validation to help select the appropriate model and model parameters. When calculating the average value, it is necessary to perform reasonable statistical processing on the multiple results obtained by cross-validation, remove outliers, and then perform average calculations to effectively improve the accuracy and stability of the final classification results.

[0035] In some embodiments, the cross-validation is k-fold cross-validation.

[0036] By setting up k-fold cross validation, the learning model can be trained and tested on different combinations of data subsets, thereby more comprehensively evaluating its performance and adaptability.

[0037] In some embodiments, the unimodal data machine learning classification model is constructed based on transthoracic echocardiography.

[0038] The accurate classification model of benign and malignant tumors constructed based on transthoracic echocardiography has greater clinical value and is more suitable for clinical diagnosis of patients with mild symptoms.

[0039] In some embodiments, the present application further provides a tumor classification device based on echocardiography video, which is used to implement the tumor classification method as described in any of the above embodiments, including:

[0040] An acquisition module, used for processing and preprocessing the echocardiography video data;

[0041] Image extraction module, used to extract video key frames to construct image dataset;

[0042] A first feature extraction module, configured to perform traditional feature extraction on the image data set and the echocardiography video data;

[0043] A second feature extraction module, used for performing deep feature extraction on the image data set;

[0044] The first model training module is used to build a single-modal data machine learning classification model;

[0045] The second model training module is used to build a multimodal data machine learning classification model.

[0046] The present invention has at least the following beneficial effects:

[0047] Compared with the prior art, the deep features of the echocardiogram images are extracted through the deep neural network model, and the deep features of the echocardiogram images are deeply explored, and it is found that the deep features have a promoting effect on the classification of benign and malignant cardiac tumors, thereby achieving the effect of effectively improving the accuracy rate, constructing a single-modal data machine learning classification model, and training the single-modal model, realizing the non-invasive classification of benign and malignant tumors in echocardiograms based on single-modal features. In addition, the present invention combines the data of three modes of patient clinical information (text), echocardiogram video key frame data (image) and echocardiogram video data (video) to form a multimodal data set, correspondingly constructs a multimodal model, and trains the multimodal model, and then can realize the non-invasive classification of benign and malignant tumors in echocardiograms based on multimodal features, so as to realize a more accurate classification prediction of the benign and malignant cardiac tumors.

[0048] Other features and corresponding beneficial effects of the present invention are described in the latter part of the specification, and it should be understood that at least some of the beneficial effects become obvious from the description in the specification of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A schematic diagram of the process of the tumor classification method based on echocardiography video of the present application;

[0050] Figure 2This is a specific flow chart of step S1 in the tumor classification method based on echocardiography video of the present application;

[0051] Figure 3 This is a specific flow chart of step S2 in the tumor classification method based on echocardiography video of the present application;

[0052] Figure 4 This is a schematic diagram of the key frame extraction in the tumor classification method based on echocardiography video of the present application;

[0053] Figure 5 A schematic diagram of a traditional feature extraction process in the tumor classification method based on echocardiography video of the present application;

[0054] Figure 6 A schematic diagram of deep feature extraction of image data in the tumor classification method based on echocardiography video of the present application;

[0055] Figure 7 This is a comparison curve diagram of the evaluation results of each single-modality model in the tumor classification method based on echocardiography video of the present application;

[0056] Figure 8 This is a comparison curve chart of the evaluation results of each multimodal model in the tumor classification method based on echocardiography video of this application. DETAILED DESCRIPTION

[0057] The following specific embodiments illustrate the implementation of the present application, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. Although the description of the present application will be introduced in conjunction with some embodiments, this does not mean that the features of this application are limited to the implementation. On the contrary, the purpose of introducing the application in conjunction with the implementation is to cover other options or modifications that may be extended based on the claims of the present application. In order to provide a deep understanding of the present application, the following description will include many specific details. The present application can also be implemented without using these details. In addition, in order to avoid confusion or blurring the focus of the present application, some specific details will be omitted in the description. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0058] It should be noted that in this specification, similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0059] This application provides a tumor classification method based on echocardiographic video, see Figures 1 to 3 , including the following steps:

[0060] Step S1: Acquire echocardiography video data and perform preprocessing to extract video key frame data, and construct an image data set based on the video key frame data;

[0061] Step S2: performing traditional feature extraction on the echocardiography video data, and performing traditional feature and deep feature extraction on the image data set;

[0062] Step S3: constructing a unimodal data machine learning classification model and a multimodal data machine learning classification model, wherein the unimodal data machine learning classification model processes traditional features and deep features to obtain unimodal data classification results, and the multimodal data machine learning classification model processes deep features to obtain multimodal data classification results;

[0063] Step S4: Analyze and compare the unimodal data classification results and the multimodal data classification results to obtain the final classification results.

[0064] Specifically, the echocardiography video data mainly includes images of the patient's atria, ventricles, myocardium and heart valves, and records the patient's relevant clinical information, such as age, gender, symptoms, medical history, etc., for reference and use in subsequent analysis. The specific acquisition method is mainly to use professional echocardiography equipment for ultrasound probe examination, usually transthoracic echocardiography and transesophageal echocardiography. The collected video data is saved in a standardized format, and then the echocardiography video data is professionally screened and processed to obtain video key frame data, and then the video key frame data is constructed into a video key frame dataset, and the video key frame dataset is annotated.

[0065] Programming tools were used to extract traditional features from echocardiography video data and traditional features and deep features from image datasets for subsequent classification processing. Traditional features mainly capture the geometry, morphology, location and other information of the tumor in the patient's body, which can describe the lesion characteristics in a fine-grained manner and help improve classification accuracy. Deep features are high-order, abstract features extracted from deep neural networks, which can effectively reduce human bias and reliance on prior knowledge, and provide the model with more dimensional feature information.

[0066] A unimodal data machine learning classification model and a multimodal data machine learning classification model were constructed. Then, the processed traditional features and deep features were used as input and the tumor category (such as benign or malignant) was used as the output label to train the two models. By optimizing the parameters of the two models, the two models achieved better classification performance on the training data, thereby obtaining two classification results: unimodal data classification results and multimodal data classification results.

[0067] A detailed comparative analysis is conducted on the unimodal data classification results and the multimodal data classification results, such as comparing the accuracy, recall rate and other indicators of the two, and comparing the performance differences of the two classification results on different types of tumor samples, in order to obtain the advantages, stability and accuracy of different classification models in dealing with different types of tumors, and then adopt appropriate decision-making strategies based on the results of the comparative analysis to determine the final classification results.

[0068] This patent's method of extracting key frames based on motion information has the following advantages over traditional classification methods based on echocardiography videos: 1. Capturing physiologically meaningful periodic motions, such as key moments in the heart's contraction and relaxation processes; 2. Identifying tiny tissue movements, especially subtle changes in areas such as heart valves and myocardium; 3. Reducing redundant frames and extracting only key frames that truly reflect motion changes; 4. Effectively reducing the interference of noise and artifacts: filtering out unimportant interference through motion analysis; 5. Capturing global and local motions simultaneously to be able to more comprehensively reflect cardiac function. Therefore, in echocardiography videos, the use of key frame extraction algorithms based on motion information has significant advantages and can more accurately extract frames that reflect the important physiological state of the heart, which is of great significance for clinical diagnosis and cardiac function assessment.

[0069] In some embodiments, the single modality includes traditional features of the image dataset, traditional features of the echocardiography video data, deep features of the echocardiography video data, and clinical features of the patient;

[0070] The multimodality includes combined features of at least two of traditional features of the image dataset, traditional features of the echocardiography video data, deep features of the echocardiography video data, and clinical features of the patient.

[0071] Specifically, clinical features include information such as the patient's systemic symptoms, heart-related symptoms, and past medical history. They are based on the doctor's observation and diagnosis, have strong interpretability, provide background information about the patient, and help improve the accuracy of the final classification results. The traditional features of the image dataset can capture information such as the geometry, shape, position, and texture of the tumor, and can describe the characteristics of the lesion in a fine-grained manner, which helps improve the classification accuracy; the traditional features of the echocardiography video data have time series information, which can obtain and monitor information such as the dynamic size changes, dynamic changes in shape, and cardiac movement patterns of the tumor, and perform correlation analysis on the movement patterns and abnormalities of the heart and lesions, thereby enabling the model to be trained to learn more complex patterns; the deep features of the echocardiography video data are feature extraction based on convolutional neural networks. By extracting and screening traditional features and deep features of the image dataset and echocardiography video data to remove these redundant features, the model can more centrally learn the features that are truly important for tumor classification and diagnosis, thereby improving the training efficiency and accuracy of the model.

[0072] In some embodiments, the unimodal data machine learning classification model includes a machine learning model based on traditional features of an image dataset, a machine learning model based on traditional features of echocardiography video data, a machine learning model based on clinical features of a patient, and a machine learning model based on deep features of an image dataset;

[0073] Multimodal data machine learning classification models include machine learning models based on traditional features and clinical features of image datasets, machine learning models based on traditional features and clinical features of image datasets, machine learning models based on traditional features, deep features and clinical features of image datasets, and machine learning models based on traditional features, deep features, traditional features and clinical features of image datasets.

[0074] Specifically, see Figure 5 , Figure 7 and Figure 8 It can be seen that among them, Figure 7 and Figure 8 They are the comparison curves of the evaluation results of each model of the single modality and the comparison curves of the evaluation results of each model of the multimodality. The comparison curve includes the receiver operating characteristic curve, and the curve graph includes the area under the curve value. The single modality data machine learning classification model is mainly constructed based on traditional features and deep features separately, so that effective and reliable classification results can be obtained simply and intuitively. At the same time, it can focus on a single feature for targeted in-depth research, so as to obtain results quickly. For example, in the diagnosis of cardiac tumors, the single modality model can carefully study the relationship between symptoms such as fever, palpitations, and dyspnea and the nature of the tumor, and find the most representative symptom combination for classification, which helps to discover valuable feature patterns hidden in the single modality data. The multimodal data machine learning classification model is mainly constructed based on the combination of traditional features and deep features. The multimodal model can use the complementarity between different modality data to make up for the shortcomings of single modality data, thereby improving the accuracy of classification. For example, in some cases, echocardiographic images may not accurately judge the nature of the tumor due to reasons such as imaging quality, but combined with the patient's clinical symptoms (such as rapid weight loss, severe dyspnea, etc.), it can more accurately judge whether the tumor is benign or malignant. This method of integrating multiple information can reduce the misjudgment that may be caused by single-modal data and provide more reliable classification results. At the same time, the multimodal model can automatically explore the potential relationships between different modal data during training, providing medical researchers with a deeper and more comprehensive understanding of the disease.

[0075] In some embodiments, the unimodal data machine learning classification model is constructed based on any one of a logistic regression model, a decision tree model, a support vector machine model, a random forest model, and an extreme gradient boosting model;

[0076] The multimodal data machine learning classification model is constructed by integrating any of the following models: logistic regression model, decision tree model, support vector machine model, random forest model and extreme gradient boosting model.

[0077] Specifically, the unimodal data machine learning classification model is mainly any one of the logistic regression model, decision tree model, support vector machine model, random forest model and extreme gradient boosting model, and the multimodal data classification machine learning model can preferably be based on the fusion integration of any at least two of the logistic regression model, decision tree model, support vector machine model, random forest model and extreme gradient boosting model. By setting a variety of different types of learning models, it is possible to enrich the types and functional diversification of training models, so that the unimodal data machine learning classification model and the multimodal data machine learning classification model can complement each other's advantages and adapt to different data distributions, thereby achieving the purpose of enhancing the accuracy, stability and reliability of the unimodal data machine learning classification model and the multimodal data machine learning classification model.

[0078] It should be pointed out that it was found that in the actual training process of the machine learning classification model for single-modal data, the random forest classification model based on deep features can achieve the best classification results in transthoracic echocardiography data, and the random forest classification model based on traditional features can achieve the best classification results in transesophageal echocardiography data; it was found that in the actual training process of the machine learning classification model for multimodal data, the support vector integral model based on the multimodal feature data of transthoracic echocardiography achieved the best classification results; based on the multimodal feature data of transesophageal echocardiography, the random forest classification model achieved the best classification results, and the classification model constructed based on the multimodal feature data of transthoracic echocardiography had a slightly higher performance than the classification model constructed based on the multimodal feature data of transesophageal echocardiography. Finally, this application integrates transthoracic echocardiography and transesophageal echocardiography multimodal feature data to construct a classification model, and found that both the random forest classification model and the support vector integral model achieved the best classification results. However, due to the scarcity of cardiac tumor patients and the small amount of data, the receiver operating characteristic curve values ​​of the classification results all reached 1.000.

[0079] In some embodiments, see Figure 1 and Figure 2 , step S1 comprises:

[0080] Step S11: using a programming language to uniformly align the number of frames and frame rates of the echocardiography video data to obtain echocardiography video frames;

[0081] Step S12: converting the echocardiogram video frame into a grayscale image;

[0082] Step S13: performing optical flow analysis on the grayscale image to obtain optical flow information, and analyzing the optical flow information to obtain the average amount of motion and the standard deviation;

[0083] Step S14: setting a preset threshold based on the average amount of exercise and the standard deviation;

[0084] Step S15: obtaining a frame of image of the echocardiogram video data as the current frame image, comparing the motion speed information of the current frame image with a preset threshold, and determining that the current frame image whose motion speed information is greater than the preset threshold is a key frame image;

[0085] Step S16: Save and extract key frame images to construct an image dataset.

[0086] Specifically, multimedia video processing tools are used in the Python programming language to adjust and align the frame rate and frame number, so that the video data can be unified to the same frame number and frame rate, so as to ensure that subsequent image feature extraction, motion analysis or other complex algorithm processing can be carried out on a standard and consistent data basis, so as to avoid the adverse effects of inconsistent frame number or frame rate on the operation of subsequent machine learning classification models, thereby ensuring the accuracy of subsequent classification results, and also making different videos directly comparable, so as to enable accurate and objective comparative analysis of different video images, for example, see Figure 4 As shown, the original video data contains 92 frames. After processing in step S1, 14 key frames are extracted and saved as 14 two-dimensional image data. Each of the image data after screening is different and clear and representative.

[0087] It should be pointed out that the median is usually used as the frame rate and frame number of the adjusted video data to better reflect the central trend of this set of video data sets, making the unified frame rate and frame number more representative, stable and adaptable.

[0088] In some embodiments, see Figure 1 and Figure 3 , step S2 comprises:

[0089] Step S21: extracting traditional features of tumors in the image dataset and echocardiography video data based on tumor annotation using a programming language through the Pyradiomics toolkit;

[0090] Step S22: Extracting deep features of the tumor in the image dataset based on the Resnet-50 deep neural network model.

[0091] Specifically, the Pyradiomics toolkit is an open source Python toolkit for medical image feature extraction. It extracts a large number of quantitative image features from medical images (such as CT, MRI, ultrasound, etc.). These features can provide valuable information for disease diagnosis, prognosis assessment, and treatment response monitoring. See Figure 5 It can be seen that the actual process of step S21 is: based on the Python programming language, the Pyradiomics toolkit is used to complete the traditional feature extraction, and then the extracted features are screened. For example, the specific process is:

[0092] a) Feature normalization, using the z-score algorithm, subtracting the mean and dividing by the standard deviation to normalize the differences in feature data;

[0093] b) Use the variance method to preliminarily filter the features and delete the features with variance less than 0.0001;

[0094] c) The Pearson method was used to further screen the features, and the threshold was set to 0.8;

[0095] d) The least absolute shrinkage and selection operator regression methods are used for the final filtering of features.

[0096] After feature screening, redundant features can be effectively filtered out, reducing the risk of overfitting, which is conducive to the training of machine learning models. The fixed seed point is set to 10 to ensure the repeatability of the random process, and the maximum number of iterations is set to 300, which ensures that the model has enough time to learn and converge while avoiding overtraining and unnecessary waste of computing resources.

[0097] The actual process of step S22 is as follows: the image data is passed through the Resnet-50 network model as the backbone, and the feature vector of the last fully connected layer of the network is exported as the deep feature, totaling 2048. Usually, the image data is adjusted to 224x 224 size for batch processing and calculation. The tumor area is cropped based on the existing determined outline annotations, and only the deep feature extraction is performed on the tumor. The extracted tumor deep feature is stored in the format of DL_serial number, and all deep features are saved as CSV files with corresponding names. Figure 6 It is a schematic diagram of deep feature extraction of image data.

[0098] In some embodiments, step S4 specifically includes: comparing the unimodal data classification result and the multimodal data classification result by cross-validation, and taking the average value as the final classification result.

[0099] Specifically, when classifying based on single-modal data, it may be limited by the characteristics of its own modality and produce deviations. For example, echocardiography video data may be misclassified or even wrong due to factors such as imaging quality, tumor location and morphological complexity. Clinical symptom descriptions may also be misjudged due to individual differences and symptom similarity (such as overlapping symptoms of cardiac tumors and other heart disease symptoms). By comparing the classification results of single-modal and multi-modal data, the advantages of different modalities can be comprehensively considered, the deviations that may be caused by a single modality can be reduced, and multi-modal complementary information can be used to provide a more comprehensive evaluation perspective. The stability of each model is compared in a cross-validation manner to help select appropriate models and model parameters. When calculating the average value, it is necessary to perform reasonable statistical processing on the multiple results obtained by cross-validation, remove outliers, and then perform average calculations to effectively improve the accuracy and stability of the final classification results.

[0100] In some embodiments, the cross validation is k-fold cross validation.

[0101] Specifically, according to the experience accumulated from actual operation, the 5-fold cross-validation method used in this application can obtain the optimal segmentation model parameters, and evenly divide the data set into 5 mutually exclusive subsets of similar size. In each cross-validation process, one of the subsets is selected in turn as an independent test set, and the remaining 4 subsets are combined as a training set. In this way, the model can be trained and tested on different combinations of data subsets, so as to more comprehensively evaluate its performance and adaptability. In addition, by setting a fixed seed point, the data division position is guaranteed to be the same during each training, thereby eliminating the impact of data division differences caused by random factors on the final classification results.

[0102] In some embodiments, a unimodal data machine learning classification model is constructed based on transthoracic echocardiography.

[0103] Specifically, according to the analysis of actual experimental data, the performance of the unimodal feature machine learning classification model based on transthoracic echocardiography is generally higher than that of the unimodal feature machine learning classification model based on transesophageal echocardiography. In addition, transthoracic echocardiography is non-invasive, can reduce patient discomfort, and is easy to operate. Therefore, the accurate classification model for benign and malignant tumors based on transthoracic echocardiography has more clinical value and is more suitable for clinical diagnosis of patients with mild symptoms.

[0104] In some embodiments, the present application further provides a tumor classification device based on echocardiography video, which is used to implement the tumor classification method in any of the above embodiments, including:

[0105] An acquisition module, used for processing and preprocessing the echocardiography video data;

[0106] Image extraction module, used to extract video key frames to construct image dataset;

[0107] A first feature extraction module, for performing traditional feature extraction on the image data set and the echocardiography video data;

[0108] A second feature extraction module is used to perform deep feature extraction on the image data set;

[0109] The first model training module is used to build a single-modal data machine learning classification model;

[0110] The second model training module is used to build a multimodal data machine learning classification model.

[0111] A tumor classification device that can implement the tumor classification method as in any of the above embodiments can achieve non-invasive classification of benign and malignant tumors in echocardiograms based on multimodal features with higher accuracy, which is of great significance for clinical diagnosis and cardiac function assessment.

[0112] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and modifications of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and modifications.

Claims

1. A tumor classification method based on echocardiographic video, characterized in that: The steps include: Step S1: Acquire and preprocess echocardiography video data to extract video key frame data, and construct an image data set based on the video key frame data; Step S2: performing traditional feature extraction on the echocardiography video data, and performing traditional feature and deep feature extraction on the image data set; Step S3: constructing a unimodal data machine learning classification model and a multimodal data machine learning classification model, wherein the unimodal data machine learning classification model processes the traditional features and the deep features to obtain a unimodal data classification result, and the multimodal data machine learning classification model processes the deep features to obtain a multimodal data classification result; Step S4: Analyze and compare the unimodal data classification result and the multimodal data classification result to obtain a final classification result.

2. The method for tumor classification based on echocardiographic video according to claim 1, characterized in that: The single modality includes traditional features of the image dataset, traditional features of the echocardiography video data, deep features of the echocardiography video data, and clinical features of the patient; The multimodality includes combined features of at least two of traditional features of the image dataset, traditional features of the echocardiography video data, deep features of the echocardiography video data, and clinical features of the patient.

3. The method for tumor classification based on echocardiographic video according to claim 2, characterized in that: The single-modal data machine learning classification model includes a machine learning model based on traditional features of the image dataset, a machine learning model based on traditional features of the echocardiography video data, a machine learning model based on patient clinical features, and a machine learning model based on deep features of the image dataset; The multimodal data machine learning classification model includes a machine learning model based on traditional features and clinical features of the image dataset, a machine learning model based on traditional features and clinical features of the image dataset, a machine learning model based on traditional features, deep features and clinical features of the image dataset, and a machine learning model based on traditional features, deep features, traditional features and clinical features of the image dataset.

4. The method for tumor classification based on echocardiographic video according to claim 3, characterized in that: The unimodal data machine learning classification model is constructed based on any one of a logistic regression model, a decision tree model, a support vector machine model, a random forest model, and an extreme gradient boosting model; The multimodal data machine learning classification model is constructed by integrating any of a logistic regression model, a decision tree model, a support vector machine model, a random forest model and an extreme gradient boosting model.

5. The method for tumor classification based on echocardiographic video according to any one of claims 1 to 4, characterized in that: The step S1 comprises: Step S11: using a programming language to uniformly align the frame number and frame rate of the echocardiography video data to obtain an echocardiography video frame; Step S12: converting the echocardiogram video frame into a grayscale image; Step S13: performing optical flow analysis on the grayscale image to obtain optical flow information, and analyzing the optical flow information to obtain an average amount of motion and a standard deviation; Step S14: setting a preset threshold based on the average exercise amount and the standard deviation; Step S15: obtaining a frame of the echocardiography video data as the current frame image, comparing the motion speed information of the current frame image with the preset threshold, and determining that the current frame image whose motion speed information is greater than the preset threshold is a key frame image; Step S16: Save and extract the key frame images to construct the image dataset.

6. The method for tumor classification based on echocardiographic video according to claim 5, characterized in that: The step S2 comprises: Step S21: extracting the traditional features of the tumor in the image dataset and the echocardiography video data based on tumor annotation using a programming language through the Pyradiomics toolkit; Step S22: extracting the deep features of the tumor in the image dataset based on the Resnet-50 deep neural network model.

7. The method for tumor classification based on echocardiographic video according to claim 6, characterized in that: The step S4 specifically includes: comparing the unimodal data classification result and the multimodal data classification result by cross-validation, and taking the average value as the final classification result.

8. The method for tumor classification based on echocardiographic video according to claim 7, characterized in that: The cross validation is k-fold cross validation.

9. The method for tumor classification based on echocardiographic video according to any one of claims 6 to 8, characterized in that: The unimodal data machine learning classification model is built based on transthoracic echocardiography.

10. A tumor classification device based on echocardiography video, used to implement the tumor classification method according to any one of claims 1 to 9, characterized in that: include: An acquisition module, used for processing and preprocessing the echocardiography video data; Image extraction module, used to extract video key frames to construct image dataset; A first feature extraction module, configured to perform traditional feature extraction on the image data set and the echocardiography video data; A second feature extraction module, used for performing deep feature extraction on the image data set; The first model training module is used to build a single-modal data machine learning classification model; The second model training module is used to build a multimodal data machine learning classification model.

Images