A deep learning medical image processing classification system combined with multi-modal big data

Abstract

The application discloses a kind of deep learning medical image processing classification systems of combined multimodal big data, comprising: image acquisition module, for collecting historical medical image;Image processing module is connected with image acquisition module, for processing historical medical image, obtains target medical image;Model construction module is connected with image processing module, for constructing time-dependent Deep-Cox model, and time-dependent Deep-Cox model is trained based on target medical image, obtains target time-dependent Deep-Cox model;Image classification module is connected with model construction module, for processing by target time-dependent Deep-Cox model to image to be classified, obtains classification processing result.The classification system of the application is suitable for medical image feature extraction and early risk identification analysis of biomarker data, aims at accurately distinguishing medical image into disease high-risk and low-risk image by model improvement and integration.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of medical image processing and biological information data analysis, and particularly relates to a deep learning medical image processing classification system combined with multi-modal big data. BACKGROUND

[0002] Early identification of disease is of great significance in reducing the risk of morbidity and all-cause mortality in patients. Fundus images serve as a window to detect microvascular lesions and are crucial for predicting future disease risk. In current medical practice, deep learning models have become an important tool for medical image processing and biological information analysis. Advanced deep learning methods have been developed to diagnose and classify based on cross-sectional fundus images, but there is a lack of early risk identification and classification based on cohort fundus images, especially the comprehensive use of medical image texture features and biomarker data in predictive models. Through the application of deep learning models, doctors and researchers can more accurately analyze medical images, identify disease signs and abnormalities, and help with early diagnosis and treatment planning.

[0003] Existing image texture extraction techniques:

[0004] (1) Gray level co-occurrence matrix: In previous related studies, the gray level co-occurrence matrix method was used to extract fewer texture feature parameters, which was far from enough for diagnosing early lung cancer based on lung CT image texture features.

[0005] (2) Contourlet transform

[0006] The basic idea of Contourlet transform is to first capture edge singular points with a multi-scale decomposition similar to wavelet, and then collect singular points with similar positions into contour segments according to direction information. The basis functions of Contourlet transform are distributed in multiple scales and multiple directions, and a small number of coefficients can effectively capture the edge contours in the image, but it has the defect of producing artificial texture.

[0007] (3) Non-subsampled dual-tree complex contourlet transform (NSDTCT)

[0008] NSDTCT transform is composed of two parts: dual-tree complex wavelet (DTCWT) and non-subsampled directional filter (NSDFB). The specific operation is to first perform DTCWT decomposition on the image to obtain a dual-tree structure subband, and then use two-dimensional NSDFB operation on each layer of the six different direction high-frequency subbands, which can obtain 2 integer power direction subbands.

[0009] The essence of NSDTCT is to replace the non-subsampled pyramid filter structure in NSCT (non-subsampled contourlet transform) with the dual-tree decomposition structure in DTCWT (dual-tree complex wavelet transform). The six directional coefficients of the two-dimensional DTCWT are calculated in parallel from the sum and difference of four separable two-dimensional wavelet transform coefficients, which makes the two-dimensional DTCWT have both shift invariance and direction selectivity, i.e., when an image is decomposed by the two-dimensional DTCWT, two low-frequency subbands and six high-frequency subbands are generated. Thus, the original one high-frequency component is replaced by six high-frequency components with different directions, which can capture the rich details and texture information of the original image. NSDTCT can more sparsely represent the original image. For the transform domain image fusion algorithm, the sparser the transform is, the better the energy function can represent the clarity of the image, and thus the higher the quality of the fused image. This process can describe the image in multiple scales and multiple directions, has the advantages of high direction selectivity, shift invariance and anisotropy, can extract high-dimensional fine image texture features, and can solve the problem of image unstructuredness.

[0010] The existing commonly used classification prediction models are:

[0011] (1) Model based on convolutional neural network (CNN): CNN is widely used in medical image analysis for image classification, object recognition, target detection and other tasks. It simulates the human visual system to record and learn graphical features, so as to achieve the most effective recognition effect of graphics. The basic structure of CNN mainly includes convolution layer, pooling layer and full connection layer. The convolution layer performs convolution calculation on the image through the convolution kernel, the pooling layer performs down-sampling on the feature map through the pooling operation, and finally the full connection layer is used for classification or regression. CNN has strong image feature extraction capability, can effectively extract spatial information of images, and also has high computational efficiency and model universality, and can be applied to different image classification tasks. Although these models show effectiveness in image classification tasks, they mainly focus on local features, sometimes ignoring global information of images, limiting the performance in processing complex medical images, and lacking consideration of other biomarker indicators.

[0012] (2) Cox proportional risk deep neural network (Deep-Cox)

[0013] Deep neural network learning algorithm provides a new choice for precise prediction of disease risk. Faraggi-Simon et al. combined survival analysis and neural network to construct a prediction model, which considered the non-linear relationship and could visualize the prediction tool. A single hidden layer network was developed to predict the risk of individual outcome, the adjusted Cox partial likelihood function was optimized, and the network instead of

[0014]

[0015] where θ is the network weights, x is the set of covariates, is the single node output probability, N E＝1 is the number of patients with observable events, and λ is the regularization parameter.

[0016] Deep-Cox model is a multilayer perceptron based on Cox proportional hazards, the traditional Cox proportional hazards model contains a log-linear regression of the relative hazard function, which links covariates to patient survival. Deep-Cox model replaces the CPH log-linear regression with a multilayer perceptron to estimate the non-linear properties of the hazard function, so it is possible to achieve better performance in survival prediction. Deep-Cox is a configurable feedforward neural network, the input of the network is the baseline predictor. The network propagates the input data through several hidden layers with specified weights. The hidden layers include batch normalization, nonlinear rectified linear unit (ReLU) activation, fully connected layers, and dropout layers. The last layer is a single node that performs linear combination to generate the final output. These hyperparameters affect the performance of the Deep-Cox model in terms of training time, model convergence speed, and prediction accuracy. Therefore, the optimization of hyperparameters is crucial for model training. The grid search method is used to determine the hyperparameters of the network (including the number of hidden layers, the number of nodes per layer, the initial learning rate, the learning rate decay, and the dropout rate), and the settings of the hyperparameters are determined according to the prediction performance and training time cost. Finally, the proposed Deep-Cox perceptron is composed of an input layer (the number of nodes is equal to the number of selected features), a hidden layer (including batch normalization, ReLU activation, 32-node fully connected, and dropout layer), and an output layer.

[0017] The Cox model based on deep learning (Deep-Cox) achieves an AUC of more than 0.9 in the test set. However, the current Deep-Cox model only uses baseline individual data to predict the outcome risk, without considering the dynamic changes of individual covariates during follow-up.

[0018] Therefore, it is necessary to consider the advantages of the above-mentioned prior art and overcome its shortcomings, improve the deep learning modeling method, and improve the accuracy of medical image classification. SUMMARY

[0019] To solve the above technical problems, the present application develops a time-dependent medical image analysis system based on deep learning, which comprehensively analyzes and classifies medical images by combining biomarker information and its dynamic changes. Specifically, the present application provides a deep learning medical image processing classification system combining multi-modal big data, which comprises:

[0020] An image acquisition module for collecting historical medical images;

[0021] An image processing module, connected with the image acquisition module, configured to process the historical medical images to obtain target medical images;

[0022] A model construction module, connected with the image processing module, configured to construct a time-dependent Deep-Cox model, and train the time-dependent Deep-Cox model based on the target medical images to obtain a target time-dependent Deep-Cox model;

[0023] An image classification module, connected with the model construction module, configured to process a to-be-classified image by the target time-dependent Deep-Cox model to obtain a classification processing result.

[0024] Preferably, the image processing module comprises a pre-processing unit and a feature extraction unit.

[0025] The pre-processing unit is configured to perform image region of interest acquisition, image standardization, denoising, and contrast adjustment processing on the historical medical images.

[0026] The feature extraction unit is configured to perform feature extraction based on a non-subsampled dual-tree complex contourlet transform to obtain image features.

[0027] Preferably, the image features include but are not limited to shape features, texture features, and edge features.

[0028] Preferably, the system further comprises a multi-modal big data dimension reduction module connected with the image processing module and the image classification module, respectively.

[0029] The multi-modal big data dimension reduction module is configured to integrate the multi-modal big data obtained through classification, perform ultra-high-dimensional big data dimension reduction by using an adaptive group weighted elastic net, and screen to obtain early-occurring and comprehensive risk factors.

[0030] Preferably, the process of performing ultra-high-dimensional big data dimension reduction by using the adaptive group weighted elastic net comprises:

[0031] Grouping features according to biomarker characteristics, considering inter-group and intra-group characteristics, and applying different penalties to features for feature screening;

[0032] Standardizing covariates based on the adaptive group weighted elastic net, and the formula expression is:

[0033]

[0034] Suppose that the features are divided into G groups, each feature belongs to one group, let be the set of all feature indices of the gth group, where g = 1, …, G; and let For the in-group penalty weight factor, the model parameters are estimated by a log-likelihood function with a penalty, expressed as:

[0035]

[0036] wherein, L(x, t) is a likelihood function of the observed data, is a global penalty parameter; λ' g is regarded as a penalty factor.

[0037] Preferably, the process of constructing the time-dependent Deep-Cox model comprises

[0038] Based on the extracted image features and multi-modal biomarker information, the existing Deep-Cox risk prediction model is improved, a multi-hidden layer neural network structure is considered, and a time-dependent network module is embedded to obtain the time-dependent Deep-Cox model.

[0039] Preferably, the target time-dependent Deep-Cox model adds a nonlinear Cox proportional risk loss function to several hidden layers, considers the change of repeated measurement covariates over time on the basis of multi-layer Deep-Cox model input and parameter optimization, adds a time t factor in the covariate set and X i , increases a time-dependent function module, and considers the time-dependent covariance structure Cov(x t ) i of individual repeated measurement data through a network instead of

[0040] Compared with the prior art, the present application has the following advantages and technical effects:

[0041] The present application utilizes deep learning technology, realizes efficient processing and analysis of medical image and biomarker data by combining texture feature extraction and deep learning model.

[0042] The present application constructs a deep learning model combined with images and multi-modal biomarkers, extracts fine image features by using NADTCT in the training process, reduces the dimension of high-dimensional data by using adaptive group elastic net, and finally incorporates a time-dependent Deep-Cox model to construct a medical image processing and classification system combined with images and multi-modal biomarkers, which can not only consider the dynamic change of covariates, but also accurately identify the occurrence time and risk of a specific outcome event in early stage, and improve the prediction accuracy of the model. BRIEF DESCRIPTION OF DRAWINGS

[0043] The accompanying drawings, which form a part of this application, are included to provide a further understanding of the application and are incorporated in and constitute a part of this application. The embodiments illustrated in the drawings, and their specification, serve to explain the application without imposing on its scope. In the drawings:

[0044] Figure 1 A schematic diagram of a time-dependent Deep-Cox model network architecture for an embodiment of the present application;

[0045] Figure 2 A schematic diagram of a time-dependent Deep-Cox modeling overall structure for an embodiment of the present application;

[0046] Figure 3 A technical roadmap for constructing a deep learning model for precise identification of early and middle-stage diseases of medical images and multi-modal biomarkers. DETAILED DESCRIPTION

[0047] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in combination with embodiments.

[0048] It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown herein.

[0049] As shown in Figures 1-3 The present application integrates the basic framework of neural networks in the prior art, and proposes a feature extraction and classification model in combination with biomarker information and dynamic changes. In the training process, the NSDTCT is used to extract fine texture features of images, the image features are combined with multi-modal biomarkers for dimension reduction, and a time-dependent deep learning model of image combined with biomarkers is constructed. Through the following four main modules, the accuracy of processing and analysis of medical images is optimized: a deep learning medical image processing classification system combining multi-modal big data is provided in the present embodiment, which includes:

[0050] An image acquisition module for collecting historical medical images;

[0051] An image processing module connected with the image acquisition module, for processing the historical medical images to obtain target medical images;

[0052] A model construction module connected with the image processing module, for constructing a time-dependent Deep-Cox model, and training the time-dependent Deep-Cox model based on the target medical images to obtain a target time-dependent Deep-Cox model;

[0053] An image classification module connected with the model construction module, for processing the images to be classified by the target time-dependent Deep-Cox model to obtain a classification processing result.

[0054] Further, the image processing module comprises a preprocessing unit and a feature extraction unit;

[0055] The preprocessing unit is responsible for the preliminary processing of medical image data, including image region of interest acquisition, image standardization, denoising, and contrast adjustment on historical medical images, to ensure the quality and consistency of the input data. This module extracts the region of interest through the Hough circle transformation method, and uses image enhancement methods such as normalization and standardization to preprocess the image, providing more accurate and reliable input data for subsequent feature extraction and classification.

[0056] The feature extraction unit is used to extract features based on the non-subsampled dual-tree complex contourlet transform to obtain image features.

[0057] Further, the image features include but are not limited to shape features, texture features, and edge features, laying a foundation for further analysis and classification of medical images.

[0058] Further, the system further comprises a multi-modal big data dimension reduction module connected with the image processing module and the image classification module respectively;

[0059] The multi-modal big data dimension reduction module is used to integrate the multi-modal big data obtained by classification, such as imageomic markers, glycosyl and gene risk factors, and clinical indicators, and to perform ultra-high-dimensional big data dimension reduction using adaptive group weighted elastic net to screen for early occurring and comprehensive risk factors.

[0060] Further, the process of using adaptive group weighted elastic net for ultra-high-dimensional big data dimension reduction comprises:

[0061] Grouping features according to biomarker characteristics, considering inter-group and intra-group characteristics, and applying different penalties to features for feature selection;

[0062] Standardizing covariates based on adaptive group weighted elastic net, with the formula expression being:

[0063]

[0064] Assuming that the features are divided into G groups, each feature belongs to one group, let be the set of all feature indices of the gth group, where g = 1, …, G; and let be the intra-group penalty weight factor, and the model parameters are estimated by a penalized log-likelihood function, with the formula expression being:

[0065]

[0066] In the formula, is the likelihood function of the observed data, is the global penalty parameter; λ'g is considered as a penalty factor. When λ' g = 1, the equation is the general elastic net. By adjusting the value of λ' g in the penalty function, the problem of collinearity between features can be better handled, and sparsity and stability are achieved.

[0067] Further, the process of constructing the time-dependent Deep-Cox model comprises

[0068] Based on the extracted image features and multi-modal biomarker information, the existing Deep-Cox risk prediction model is improved, the multi-hidden layer neural network structure is considered, and the time-dependent network module is embedded to obtain the time-dependent Deep-Cox model. This method considers the time-dependent change of covariates, can construct an early accurate prediction model, and realizes medical image classification.

[0069] Further, neither CNN nor Deep-Cox model considers the repeated measurement characteristics of covariates in actual follow-up cohort data, such as the changes of blood biochemical indicators and life behavior factors in the follow-up process. The above models do not consider the change of covariates over time, and the outcome risk prediction needs to be further optimized. The target time-dependent Deep-Cox model adds a nonlinear Cox proportional risk loss function to several hidden layers, considers the change of repeated measurement covariates over time, adds the time t factor in the covariate set and X i , increases the time-dependent function module, considers the time-dependent covariance structure Cov(x t ) i , and replaces

[0070] In order to adapt to the heterogeneity appearing in the data, the survival distribution of each individual is modeled as a mixture of survival distributions of a fixed size. At the time of testing, the survival function corresponding to this individual is described as a weighted mixture of survival distributions. Here, the weight is the maximum value of the deep neural network output. At the time of training, the network parameters and the survival distribution will be learned together. Considering the maximum likelihood estimation of non-censored data, its expression form is as follows:

[0071]

[0072] For censored data, its expression form is:

[0073]

[0074] Where x i ​is the input covariate of the ith observation unit, f(t) is the probability density function of the original distribution, S(t) is the initial distribution of the survival function, ELBO U (θ) and ELBO C (θ) represent the competition risk of the mth observation unit when other events are censored. Let β ki and η ki is the parameterized expression of the ith observation, and the calculation process is as follows:

[0075]

[0076]

[0077] If the original distribution is Weibull distribution, act(.) represents the SELU activation function, and if the original distribution is lognormal distribution, act(.) represents the Tanh activation function, Φ θ (.) represents a multi-layer perception network.

[0078] The survival distribution with positive support usually has a long tail, which is a complex case that increases bias when performing maximum likelihood estimation. The embodiment reduces the bias generated by long tail data by directly multiplying the censored loss by the coefficient α [0, 1]. The calculation formula is as follows:

[0079]

[0080] The structure diagram of the Deep-Cox modeling network is shown in Figure 1 . The input feature vector x passes through a deep multi-layer perceptron, and then a mixed K-layer Softmax activation function. The conditional distribution of x is described as a mixture of K original distributions, which is obtained in the prior distribution.

[0081] After establishing the prediction model as shown in Figure 2 :

[0082] 1) Collect color fundus images and multi-modal biomarkers of patients, such as glycosyl, genetic risk factors and clinical indicators, and establish a disease patient information database;

[0083] 2) Use the image preprocessing module to preprocess the original image;

[0084] 3) Divide the processed image into a training set and a test set, and repeat the following steps:

[0085] ① Use the NADTCT-based medical image feature extraction module to extract fine feature indicators of the image and screen specific imageomic markers;

[0086] ②Adopting a multi-modal big data dimension reduction module based on adaptive group weighted elastic net, integrating multi-modal big data of outcome occurrence for ultra-high dimensional data dimension reduction, and screening out early and comprehensive risk features;

[0087] ③Using a medical image classification module to output the final classification result of the image;

[0088] ④Evaluating the accuracy of the classification result.

[0089] 4) Adjust the settings of the model training hyperparameters on the validation set to correct the prediction model and obtain a result more accurate ideal model. Finally, evaluate the performance of the model on the test set.

[0090] Embodiment

[0091] The following embodiment introduces the processing and classification system of the present application based on color fundus images and biomarkers.

[0092] Image source: fundus images collected by Beijing Health Management Cohort and UK Biobank, images in JPG and PNG formats respectively;

[0093] Environmental behavior factors, laboratory indicators, and other data sources: basic information, physical examination indicators, environmental behavior factors, and blood samples collected by Beijing Health Management Cohort and UK Biobank;

[0094] Omics data source: genomic markers obtained from blood samples collected by Beijing Health Management Cohort and UK Biobank

[0095] Method: Matlab software programming is used to extract fine texture features of fundus images using dual-tree complex contourlet transform algorithm, and to screen image-specific textures with good screening effect and strong interpretability. After completing the screening of omics biomarkers, adaptive group weighted elastic net is used for multi-modal big data dimension reduction. Python software programming is used to complete the DR incidence prediction model construction using Deep-Cox,

[0096] The following is an example of constructing a medical image processing and classification system based on color fundus images and biomarkers, with the following steps:

[0097] Collect 1514 original fundus images of diabetic patients.

[0098] Use the image preprocessing module to process the original images.

[0099] Randomly allocate the processed images into training set and validation set in the ratio of 8:2.

[0100] The texture features are extracted by using the dual-tree complex wavelet transform, and the program is written to determine the key parameters, such as the number of layers, the number of subbands of the dual-tree complex wavelet transform, the number of final directional subbands, and the number of textures extracted from each image. Directly run.

[0101] The spatio-temporal-multiple migration-adversarial neural network method is used, and the program is written to screen the image-specific texture with good effect and strong explainability for early prediction of DR based on fundus images.

[0102] GWAS analysis and glycosylation comparison are performed on DR patients and matched control population to screen significant SNP sites and glycosylation markers.

[0103] Adaptive group weighted elastic net is used, and the program is written to perform multi-class multi-modal big data dimension reduction.

[0104] The time-dependent Deep-Cox model is used to consider the multi-modal data variables of different follow-up times of repeated measurements, to build a time-dependent dynamic risk prediction innovation model, and to evaluate the accuracy of the classification results.

[0105] The model training hyperparameter settings are adjusted on the validation set to correct the prediction model and obtain a more accurate ideal model. Finally, the performance of the model is evaluated on the test set.

[0106] The images include but are not limited to fundus images, and the multi-modal big data for model construction includes but is not limited to physical examination information, genomics, glycomics, etc.

[0107] The above is only the preferred specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or replacements within the technical scope disclosed in the present application can be easily thought of by those skilled in the art, and should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A deep learning-based medical image processing and classification system combining multimodal big data, characterized in that, include: The image acquisition module is used to collect historical medical images; An image processing module, connected to the image acquisition module, is used to process the historical medical images to obtain the target medical image; The model building module, connected to the image processing module, is used to build a time-dependent Deep-Cox model and train the time-dependent Deep-Cox model based on the target medical image to obtain the target time-dependent Deep-Cox model. The target time-dependent Deep-Cox model incorporates a nonlinear Cox proportional hazards loss function into several hidden layers. Based on the optimization of inputs and parameters of the multi-layer Deep-Cox model, it considers the time-varying changes of repeated measures covariates within the covariate set. By incorporating the time factor (t) and adding a time-dependent function module, the time-dependent covariance structure of individual repeated measures data can be considered. via the network replace ; The image classification module, connected to the model building module, is used to process the image to be classified using the target-time Deep-Cox model to obtain the classification result.

2. The deep learning medical image processing and classification system combining multimodal big data as described in claim 1, characterized in that, The image processing module includes a preprocessing unit and a feature extraction unit; The preprocessing unit is used to perform image region of interest acquisition, image standardization, noise reduction, and contrast adjustment on historical medical images. The feature extraction unit is used to extract features based on non-subsampled dual-tree complex contour transformation to obtain image features.

3. The deep learning medical image processing and classification system combining multimodal big data as described in claim 2, characterized in that, The image features include shape features, texture features, and edge features.

4. The deep learning medical image processing and classification system combining multimodal big data as described in claim 1, characterized in that, The system also includes a multimodal big data dimensionality reduction module that is connected to the image processing module and the image classification module respectively; The multimodal big data dimensionality reduction module is used to integrate the multimodal big data obtained from classification, and uses an adaptive group weighted elastic network to perform dimensionality reduction on ultra-high-dimensional big data, thereby screening out early-occurring and comprehensive risk factors.

5. The deep learning medical image processing and classification system combining multimodal big data according to claim 4, characterized in that, The process of using the adaptive group-weighted elastic network for dimensionality reduction of ultra-high-dimensional big data includes: Features are grouped according to their biomarker characteristics, and features are selected by considering both inter-group and intra-group characteristics and applying different penalties to each feature. Based on the adaptive group-weighted elastic network, the covariates are standardized using the following formula: ； Assuming the features are divided into Groups, where each feature belongs to one of the groups, let For the first The set of all feature indices of the group, where At the same time, let The within-group penalty weighting factor is used, and the model parameters are estimated using a penalized log-likelihood function, expressed as follows: ； In the formula, The likelihood function of the observed data. , This is a global penalty parameter; It is considered a punitive factor.

6. The deep learning medical image processing and classification system combining multimodal big data as described in claim 1, characterized in that, The process of constructing the time-dependent Deep-Cox model includes: Based on the extracted image features and multimodal biomarker information, the existing Deep-Cox risk prediction model is improved by considering a neural network structure with multiple hidden layers and embedding a time-dependent network module to obtain the time-dependent Deep-Cox model.

Images