Construction method and equipment based on multi-modal ultrasonic image knowledge graph, and storage medium

Abstract

The invention relates to a multi-modal ultrasonic image-based knowledge graph construction method and device, and a storage medium, and the method comprises the steps: obtaining a tumor gray-scale ultrasonic image, obtaining a tumor edge image and a tumor internal image through target segmentation, respectively extracting corresponding multi-modal candidate image omics features, and carrying out the target segmentation; forming a sample set comprising a plurality of samples in combination with a tumor type corresponding to the tumor gray-scale ultrasonic image; performing synthetic minority oversampling processing on a sample set, performing Z scoring processing on each candidate radiomics feature in each sample, screening radiomics features with strong correlation through principal component analysis, and performing recursive feature elimination to obtain finally reserved radiomics features corresponding to each tumor type; and establishing a knowledge graph and / or a machine learning model based on the finally reserved radiomics features corresponding to each tumor type. The method has the advantages that the association between the tumor type and the selected iconography features is fully established, and the abundant information of the tumor edge is fully considered.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical image processing, in particular to a multi-modal ultrasound image knowledge graph construction method and device and storage medium. BACKGROUND

[0002] With the advent of the artificial intelligence big data era and the rapid development of emerging imaging technologies, accurately identifying tumors in pictures and predicting and evaluating their disease outcomes have greatly improved the efficiency of doctors.

[0003] Chinese patent application publication No. CN113436150A discloses a method for constructing an ultrasound image-based model for predicting lymph node metastasis. By constructing an ultrasound image-based model, the image-based features and clinical information are used to solve the problem of accuracy in predicting cervical lymph node metastasis in patients with papillary thyroid cancer, and to achieve non-invasive and non-invasive efficient prediction, providing more comprehensive diagnostic information. The application realizes feature selection through Lasso analysis, however, tumors have high heterogeneity, and selecting key features from the regression method can easily lead to overfitting problems.

[0004] In summary, the existing technology still has the following problems to be solved: ① Previous studies were mainly based on traditional ultrasound images or single-mode gray-scale ultrasound image-based features, and there was no multi-modal multi-dimensional data prediction method that fused multi-modal ultrasound image-based features and clinical text information; ② Previous studies focused on the heterogeneity of tumor internal structure, while ignoring the rich tissue information contained in the tumor edge structure; ③ How to efficiently and accurately fuse multi-dimensional information such as gray-scale image, elastography, ultrasound contrast, and text information, and realize the knowledge graph representation form with explainability, further in-depth research is needed. SUMMARY

[0005] The present application is to overcome the defects of the prior art and provide a multi-modal ultrasound image knowledge graph construction method, device and storage medium to solve or partially solve the problem of poor performance of downstream tumor pictures and knowledge graph construction due to the lack of correlation between tumor types and selected imaging features.

[0006] The object of the present application can be achieved by the following technical solutions:

[0007] In one aspect of the present application, a multi-modal ultrasound image knowledge graph construction method is provided, comprising the following steps:

[0008] Acquire a tumor gray-scale ultrasound image, obtain a tumor edge image and a tumor internal image through target segmentation, respectively extract corresponding multi-modal candidate image features, and combine the tumor gray-scale ultrasound image corresponding to the tumor type to form a sample set including multiple samples;

[0009] Perform synthetic minority over-sampling processing on the sample set, perform Z-score processing on each candidate image feature in each sample, filter image features with strong correlation through principal component analysis, and obtain final retained image features corresponding to each tumor type through recursive feature elimination;

[0010] Based on the final retained image features corresponding to each tumor type, a knowledge graph and / or a machine learning model is established.

[0011] As a preferred technical solution, based on the final retained image features corresponding to each tumor type, the process of establishing or machine learning model includes:

[0012] Based on the final retained image features corresponding to each tumor type, label information is obtained to form a training set, and multiple types of tumor recognition and prognosis prediction models are constructed and trained, and the effectiveness of each prediction model is evaluated and compared.

[0013] As a preferred technical solution, the multiple types of tumor recognition and prognosis prediction models include at least one of a support vector machine model, a decision tree model, a random forest model, an adaptive boosting model, a logistic regression model, and a linear discriminant analysis model.

[0014] As a preferred technical solution, the performance evaluation parameters of the tumor recognition and prognosis prediction model prediction include at least one of prediction sensitivity, specificity, accuracy, positive predictive value, negative predictive value, and area under the receiver-operator characteristic curve.

[0015] As a preferred technical solution, the process of obtaining a tumor edge image and a tumor internal image through target segmentation includes the following steps:

[0016] For the tumor gray-scale ultrasound image, the region of interest with the largest cross-sectional area is segmented, the intratumoral part of the tumor is segmented, and a 3mm thick edge structure outside the tumor region is obtained.

[0017] As a preferred technical solution, the process of respectively extracting corresponding multi-modal candidate image features includes:

[0018] Extract shape features, first-order features, second-order features, high-order features, and wavelet-related features.

[0019] As a preferred technical solution, the second-order features include a gray-level co-occurrence matrix and a gray-level dependence matrix.

[0020] As a preferred technical solution, the high-order features include a gray scale run length matrix, a gray scale size region matrix, and a neighborhood gray scale difference matrix.

[0021] In another aspect of the present application, an electronic device is provided, comprising one or more processors and a memory having stored therein one or more programs, the one or more programs including instructions for performing the aforementioned method for constructing a multi-modal ultrasound image knowledge graph.

[0022] In another aspect of the present application, a computer-readable storage medium is provided, comprising one or more programs for execution by one or more processors of an electronic device, the one or more programs including instructions for performing the aforementioned method for constructing a multi-modal ultrasound image knowledge graph.

[0023] Compared with the prior art, the present application has at least one of the following beneficial effects:

[0024] (1) Fully establish the association between tumor types and selected imaging features: The present application takes into account the high heterogeneity of tumors, forms a sample set including multiple samples by combining the extracted candidate image features with the tumor types corresponding to the gray scale ultrasound images of the tumors, then performs a synthetic minority over-sampling processing, performs Z-score processing for each candidate image feature in each sample, filters the image features with strong correlation through principal component analysis, and obtains the final retained image features corresponding to each tumor type through recursive feature elimination. Finally, the heterogeneity of image features of different tumor types is fully considered, the feature combination that best represents the characteristics of each tumor is found, and the association between tumor types and selected imaging features is fully established.

[0025] (2) Fully consider the rich information of tumor edges: The present application obtains tumor edge images and tumor internal images through target segmentation, extracts corresponding multi-modal candidate image features, fully utilizes the rich information contained in the tumor edge structure, explores the role of the tumor edge structure based on ultrasound image features, and fuses the tumor internal structure and clinical pathological prediction factors to realize the extraction of the rich information of the tumor edges.

[0026] (3) Wide application: Based on the final retained image features corresponding to each tumor type, the present application can establish an interpretable knowledge graph and can be used for downstream applications such as tumor identification and prognosis prediction. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 Flowchart of the method for constructing a multi-modal ultrasound image knowledge graph in the embodiments;

[0028] Figure 2A schematic diagram of an electronic device according to an embodiment. DETAILED DESCRIPTION

[0029] The technical solutions in the embodiments of the present application will be clearly and completely described with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative effort should fall within the protection scope of the present application.

[0030] Embodiment 1

[0031] In view of the problems in the prior art, the embodiment provides a construction method of a multi-modal ultrasound image knowledge graph, relates to a knowledge graph of tumor identification and prognosis based on multi-modal ultrasound imageomics, and includes the following steps: inputting clinical information, collecting multi-modal ultrasound image features such as conventional ultrasound, elasticity, and contrast, extracting omics features of each grayscale ultrasound image, performing big data analysis, and establishing a pixel-level correspondence relationship between the knowledge graph and a clinical problem gold standard; in addition to analyzing the heterogeneity of the internal structure of the tumor, the tumor edge structure composed of the surrounding area containing tumor cells, inflammatory cells, and immune reaction mixtures is further studied, and a new, structured, intuitive, and comprehensive knowledge graph is formed.

[0032] With reference to Figure 1 The method includes the following steps:

[0033] In step S1, a tumor grayscale ultrasound image is acquired, a tumor edge image and a tumor internal image are obtained through target segmentation, corresponding multi-modal candidate imageomics features are extracted, and a sample set including multiple samples is formed in combination with a tumor type corresponding to the tumor grayscale ultrasound image.

[0034] In this step, grayscale ultrasound imageomics research is performed, mainly including acquisition of standardized images, segmentation of target regions, and extraction of tumor edge and internal structure imageomics features.

[0035] Ultrasound images of a target tumor are collected for imageomics feature extraction. Different from previous imageomics research, the tumor edge and internal structure are extracted in this research, and the method segments a region of interest with the maximum cross-sectional area through a 3D Slicer software package.

[0036] Firstly, the intratumoral part of the tumor was segmented, and then a 3mm thick margin structure outside the tumor area was obtained using the "hollow" and "edge" editing tools. Pyradiomics was used to extract the image feature set of the lesion, a total of 851 tumor edge and intratumoral ultrasound image features were extracted, including 14 shape features, 18 first-order features, 38 second-order features including 24 gray level co-occurrence matrix (GLCM) and 14 gray level dependence matrix (GLDM), 37 high-order features including 16 gray level run length matrix (GLRLM), 16 gray level size zone matrix (GLSZM), and 5 neighboring gray tone difference matrix (NGTDM), and 744 wavelet-related features.

[0037] Shape features describe the shape of the lesion region of interest and its geometric properties. First-order statistical features describe the distribution of individual voxel values, including voxel intensity values and kurtosis. Second-order statistical features refer to texture features, which reflect the spatial arrangement of voxel intensity by calculating the relationship between adjacent voxels, and suggest the heterogeneity of the lesion. High-order statistical features are designed to identify repeated or non-repeated patterns, suppress noise or highlight details after applying filters or transformators (such as wavelets) to the image, and can fully suggest the characteristics of the tumor.

[0038] Step S2, perform synthetic minority over-sampling for the sample set, perform Z-score processing for each candidate image feature in each sample, filter the image features with strong correlation through principal component analysis, and obtain the final retained image features corresponding to each tumor type through recursive feature elimination.

[0039] Big data analysis and selection, in the feature selection process, unlike previous studies that use variance analysis to filter data and use least absolute shrinkage and selection operator (LASSO) to screen features, in order to prevent potential invalid calculations and overfitting of the model, the first step uses synthetic minority over-sampling technique (SMOTE) to remove imbalanced samples in the training and validation set. Considering that random oversampling uses a simple replication strategy to increase the number of minority class samples, it is easy to cause model overfitting, that is, the information learned by the model is too special and not generalizable, the SMOTE algorithm analyzes the minority class samples and artificially synthesizes new samples according to the minority class samples and adds them to the dataset, the algorithm is as follows:

[0040] x new = x + rand(0, 1) * |x - x n |

[0041] where x new is the newly synthesized sample, x is the sample in the minority data class, and x n is its nearest neighbor. It should be noted that the class here refers to different categories of data, and in this embodiment, different types of tumors.

[0042] Thereafter, the Z-score and mean standardization method are used to standardize each corresponding feature, and the calculation formula of the Z-score is:

[0043] Z = (X - μ) / σ

[0044] where X is a specific value, μ is the population mean, and σ is the population standard deviation.

[0045] The Z-score can represent the relative position between a value and the population mean in standard deviation units.

[0046] Thereafter, principal component analysis (PCA) is used to reduce the feature dimension and increase data analysis. PCA is to simplify the data set by retaining the most important features (i.e., principal components) in the data set while removing noise and redundant information. By calculating the covariance matrix between each dimension in the data set, the direction with the largest variance is found. These principal components are arranged in order of variance from large to small, and the principal component with larger variance contains more information. This helps to improve the visualization, storage and computational efficiency of the data.

[0047] Finally, recursive feature elimination (RFE) is used to detect prediction features with important relevance. Recursive feature elimination is a commonly used feature selection method that recursively removes features through continuous iteration until the pre-set number of features is reached. RFE evaluates the importance of each feature by recursively removing features and retraining the model. This process will continue until the specified number of features is reached or no more features can be removed.

[0048] Step S3, based on the final retained image features corresponding to each tumor type, a knowledge graph and / or a machine learning model is established.

[0049] Machine learning can use algorithms to analyze data, build models, map from input quantitative features to target values, and make decisions or predictions. The samples in the training set in machine learning all have labels. These labeled samples are used to adjust the modeling, and new data is used to verify, so that the model produces efficient prediction and inference functions.

[0050] The present embodiment adopts various machine learning classifiers including support vector machine (SVM), decision tree (DT), random forest (RF), adaptive boosting (Adaboost), logistic regression (LR) and linear discriminant analysis (LDA) and the like.

[0051] The support vector machine is a generalized linear classifier for binary classification of data, and an optimization algorithm for solving a quadratic programming. It is a linear classifier with the largest margin in the feature space. Compared with other machine learning algorithms, the support vector machine can better solve the small sample problem, and can be combined with other algorithms, and has greater generalization ability.

[0052] The decision tree is a data mining algorithm, which is a tree-like pattern, and the construction process includes three steps of feature selection, node splitting and pruning. The feature selection refers to selecting the best feature at each node, which can better classify the data.

[0053] The node splitting refers to dividing the data into different subsets according to the value of the feature and corresponding branches. The pruning refers to pruning and selecting the constructed model tree, removing redundant data, so as to improve the model efficiency.

[0054] The random forest randomly extracts multiple samples from the original training samples to generate a new training set, and then generates multiple classification trees according to the sample set, and combines a large number of randomly generated decision trees to form a random forest. The random sampling process in the random forest algorithm makes the overall model have anti-overfitting ability, and the prediction performance is more stable and accurate. The logistic regression is a generalized linear regression analysis model, which is mainly used to solve the binary classification problem.

[0055] The logistic regression trains the model through the training set, and classifies the validation set after the training is completed. It is commonly used in data mining, disease diagnosis and prognosis prediction, such as exploring the risk factors of causing diseases, and predicting the probability of disease occurrence and the risk of disease recurrence according to the risk factors.

[0056] The AdaBoost algorithm is to improve the weak classifier to a high-precision strong classifier, so as to reduce the training error. Compared with other machine learning algorithms, the AdaBoost algorithm is more sensitive to abnormal values and noise data, and can overcome the overfitting problem.

[0057] The linear discriminant analysis can perform a linear combination on the data features, and the obtained combination can be used as a linear classifier for subsequent classification and dimensionality reduction processing. The main purpose is to project the data in the high-dimensional space to a lower-dimensional space, so that the samples have the largest inter-class distance and the smallest intra-class variance in the new subspace, that is, the same class is gathered together, and the different classes are far apart.

[0058] The embodiments of the present application use a variety of machine learning algorithms to input the imageomic data selected by the above steps, construct and verify tumor recognition and prognosis prediction models, and evaluate and compare the performance of each prediction model. The performance evaluation parameters of the prediction model include prediction sensitivity (SEN), specificity (SPE), accuracy (ACC), positive predictive value (PPV), negative predictive value (NPV), and area under the receiver-operator characteristic curve (AUC).

[0059] Referring to Tables 1 and 2, the features extracted in the study of ultrasound imageomic tumor margin structure (PURS) and internal structure (IURS) features based on machine learning classifier in predicting the response of breast cancer lesions after neoadjuvant chemotherapy.

[0060] Table 1. 10 imageomic features and their coefficients selected by AdaBoost classifier in tumor margin structure prediction

[0061]

[0062] Table 2. 11 imageomic features and their coefficients selected by SVM classifier in tumor internal structure prediction

[0063]

[0064]

[0065] As can be seen from Tables 1 and 2, the main features selected in the ultrasound imageomic prediction model based on tumor margin or tumor internal structure are wavelet-related features. After wavelet transform, GLSZM and GLRLM are the most selected features in the tumor margin structure model, while GLCM and GLDM are the main features in the tumor internal structure model. GLRLM is a matrix containing gray level run length, which can provide spatial distribution information of continuous pixels in one or more directions. GLSZM can be calculated for different pixel or region distances in the neighborhood. These features can indicate the overall change information of the image in the adjacent region. GLCM is a symmetric matrix representing the probability distribution of pixel pairs. GLDM is a gray level dependence matrix, which has finer resolution for heterogeneous and homogeneous tissue structures. Wavelet features have higher details and complexity than original images through reorganization and transformation of texture features, and can provide more valuable information about the tumor microenvironment.

[0066] The construction process of the knowledge graph includes: firstly, determining the application direction and range of the knowledge graph. The knowledge graph technology applied in the medical field can solve the current data processing and actual needs in the medical field. The mode layer of the knowledge graph can effectively represent the expert knowledge in the medical field, and can be used as prior knowledge to guide information extraction and subsequent other tasks. Secondly, the data source is obtained, including the clinical characteristics information of tumor patients, laboratory examination indexes, image data and the like; thirdly, information extraction including entity recognition, relation extraction, attribute and event extraction and the like. In the medical field, the entity refers to the name, age, place of origin, nationality and gender of the patient; the relation extraction refers to the past medical history, operation history, medication history and family history of the patient; the attribute extraction refers to the clinical symptoms, signs and laboratory indexes of the patient; and the event extraction refers to the disease risk that has occurred or may occur in the future. The knowledge graph can systematically extract structured knowledge, and the non-structured imageomics big data is realized through a machine prediction model. The knowledge graph modeling is beneficial to systematically, comprehensively and comprehensively analyzing the relationship between the origin, development, treatment and prognosis of medical problems, and is beneficial to fast reading, searching, sharing and correlation analysis of medical knowledge for the general public; finally, since the influencing factors of diseases are often isolated, the potential correlation between the influencing factors can be further mined through knowledge graph reasoning and other technologies.

[0067] Embodiment 2

[0068] Based on the embodiment 1, the electronic device provided in the embodiment includes one or more processors and a memory, the memory stores one or more programs, and the one or more programs include instructions for executing the method for constructing the knowledge graph based on multi-modal ultrasound image as described in embodiment 1.

[0069] As Figure 2 described, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory and a non-volatile memory, and of course, other hardware required by the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs to implement the above Figure 1 described method. Of course, in addition to the software implementation, the present application does not exclude other implementation manners, such as logic devices or a combination of software and hardware, etc. That is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0070] The memory can include a non-permanent memory in a computer readable medium, a random access memory (RAM) and / or a non-volatile memory such as a read-only memory (ROM) or a flash memory (flash RAM). The memory is an example of a computer readable medium.

[0071] Embodiment 3

[0072] The embodiment provides a computer readable storage medium including one or more programs for execution by one or more processors of an electronic device, the one or more programs including instructions for performing the foregoing method for constructing a multi-modal ultrasound image knowledge graph.

[0073] Computer readable media includes permanent and non-permanent, removable and non-removable media, which can be implemented by any method or technology to store information. The information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by a computing device. According to the definition herein, computer readable media does not include transitory media such as modulated data signals and carriers.

[0074] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present application, and these modifications or replacements should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for constructing a multi-modal ultrasound image knowledge graph, characterized in that, The method comprises the following steps: Obtaining a tumor gray-scale ultrasound image, obtaining a tumor edge image and a tumor internal image through target segmentation, extracting corresponding multi-modal candidate image features, and combining the tumor gray-scale ultrasound image and the tumor type to form a sample set comprising multiple samples; Performing synthetic minority over-sampling processing on the sample set, performing Z-score processing on each candidate image feature in each sample, screening image features with strong correlation through principal component analysis, and obtaining final retained image features corresponding to each tumor type through recursive feature elimination; Establishing a knowledge graph and / or a machine learning model based on the final retained image features corresponding to each tumor type. 2.The method of claim 1, wherein, The process of establishing a machine learning model based on the final retained image features corresponding to each tumor type comprises: Based on the final retained image features corresponding to each tumor type, obtaining label information to form a training set, constructing and training multiple types of tumor recognition and prognosis prediction models, and evaluating and comparing the effectiveness of each prediction model. 3.The method of claim 2, wherein, The multiple types of tumor recognition and prognosis prediction models comprise at least one of a support vector machine model, a decision tree model, a random forest model, an adaptive boosting model, a logistic regression model, and a linear discriminant analysis model. 4.The method of claim 2, wherein, The performance evaluation parameters of the tumor recognition and prognosis prediction model prediction comprise at least one of prediction sensitivity, specificity, accuracy, positive predictive value, negative predictive value, and area under the receiver-operator characteristic curve. 5.The method of claim 1, wherein, The process of obtaining a tumor edge image and a tumor internal image through target segmentation comprises the following steps: For the tumor gray-scale ultrasound image, segmenting a region of interest with the largest cross-sectional area, segmenting an intratumoral part of the tumor, and obtaining a 3mm-thick edge structure outside the tumor region. 6.The method of claim 1, wherein, The process of extracting corresponding multi-modal candidate image features comprises: Extracting shape features, first-order features, second-order features, high-order features, and wavelet-related features.

7. The method of claim 6, wherein the method further comprises: The second-order features comprise a gray-level co-occurrence matrix and a gray-level dependence matrix. 8.The method of claim 6, wherein, The high-order features comprise a gray-level run-length matrix, a gray-level size zone matrix, and a neighborhood gray-level difference matrix.

9. An electronic device, comprising: The method comprises: One or more processors and a memory, the memory storing one or more programs, the one or more programs comprising instructions for performing the method for constructing a knowledge graph based on multi-modal ultrasound images according to any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, One or more programs for execution by one or more processors of an electronic device, the one or more programs comprising instructions for performing the method for constructing a knowledge graph based on multi-modal ultrasound images according to any one of claims 1-8.

Images