A method and device for predicting neural function based on imaging genomics, an electronic device and a storage medium

Abstract

The application discloses a method and device for predicting nerve function based on imageomics, electronic equipment and storage medium. The method comprises the following steps: acquiring magnetic resonance imaging; constructing a first neural network model using a two-dimensional convolution architecture, a three-dimensional convolution architecture, an encoder and a decoder, and segmenting the magnetic resonance imaging using the first neural network model to obtain an image of a region of interest; extracting features of the image of the region of interest to obtain feature data of the magnetic resonance imaging; screening the feature data by combining a filtering method and an embedding method; constructing a second neural network model using a one-dimensional convolution architecture, a gated recurrent module and a full connection architecture, and predicting nerve function according to the screened feature data using the second neural network model. The application is a brand-new method for predicting nerve function based on imageomics, and realizes the function of predicting the nerve function retention after surgery before surgery.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence research, and more particularly, to a method and device for predicting neural function based on radiomics, an electronic device and a storage medium. BACKGROUND

[0002] In the treatment of some central nervous system tumors, surgical operation is still the preferred treatment method for most doctors. However, whether the corresponding central nervous function can be preserved during tumor resection in surgery is difficult to grasp. If the postoperative facial nerve function preservation can be predicted before surgery, individualized treatment plans can be developed in advance to avoid irreversible nerve function damage during surgery.

[0003] At present, deep learning and machine learning methods are increasingly applied to postoperative prognosis research. However, the existing deep learning and machine learning have deficiencies in processing related images, including large workload and subjective errors in manual image segmentation, low accuracy of segmentation method, difficulty in feature extraction and selection, and overfitting of postoperative prognosis prediction model under small sample data and poor classification accuracy.

[0004] Therefore, how to improve the existing deep learning and machine learning methods to make them more suitable for postoperative prognosis prediction and improve prediction efficiency and accuracy has become a problem that needs to be solved in the field. SUMMARY

[0005] Therefore, in a first aspect, the present application proposes a method for predicting neural function based on radiomics, comprising:

[0006] Obtaining magnetic resonance imaging;

[0007] Using a two-dimensional convolution architecture, a three-dimensional convolution architecture, an encoder and a decoder to construct a first neural network model, and using the first neural network model to segment the magnetic resonance imaging to obtain an image of a region of interest;

[0008] Performing feature extraction on the image of the region of interest to obtain feature data of the magnetic resonance imaging;

[0009] Combining a filtering method and an embedding method to perform feature selection on the feature data; wherein the filtering method includes one or more of Spearman coefficient test, independent sample t-test and Levene test; and the embedding method includes Lasso regression verification;

[0010] Using a one-dimensional convolution architecture, a gated recurrent module and a fully connected architecture to construct a second neural network model, and using the second neural network model to predict neural function according to the selected feature data.

[0011] Preferably, the first neural network model is used to segment the magnetic resonance imaging to obtain an image of a region of interest, comprising:

[0012] Two two-dimensional convolution layers and two three-dimensional convolution layers are used in sequence to perform convolution operation on the magnetic resonance imaging to obtain a feature map;

[0013] The feature map is linearly mapped to obtain a vector sequence of the feature map;

[0014] The encoder is used to perform global semantic extraction on the vector sequence to obtain a global semantic code;

[0015] The decoder is used to perform deconvolution operation on the global semantic code to obtain the image of the region of interest.

[0016] Further preferably:

[0017] The two two-dimensional convolution layers are two two-dimensional convolution layers with residual connection, so that the in-plane resolution and the inter-plane resolution of the feature map are consistent.

[0018] Further preferably:

[0019] The encoder is a Transformer encoder;

[0020] The decoder is a mirror deconvolution layer of the two two-dimensional convolution layers and the two three-dimensional convolution layers.

[0021] Preferably, the Levene test comprises:

[0022] The W value of the Levene test of the feature data is calculated;

[0023] The F distribution table of the W value of the feature data is searched to obtain the p value corresponding to the W value of the feature data;

[0024] If the p value corresponding to the W value of the feature data is not greater than a first preset value, the feature data is excluded.

[0025] Preferably, the independent sample t test comprises:

[0026] The t value of the independent sample t test of the feature data is calculated;

[0027] The t value of the feature data is compared with a theoretical t value distribution to obtain the p value corresponding to the t value of the feature data;

[0028] If the p value corresponding to the t value of the feature data is less than a second preset value, the feature data is excluded.

[0029] Preferably, the Spearman coefficient test comprises:

[0030] calculating a Spearman coefficient p of the feature data;

[0031] if the Spearman coefficient p of the feature data is not greater than a third preset value, excluding the feature data.

[0032] Preferably, the combined filtering method and embedding method perform feature screening on the feature map, including:

[0033] sequentially performing Levene test, independent sample t test, Spearman coefficient test and Lasso regression verification on the feature data.

[0034] Preferably, the second neural network model is used to perform neural function prediction according to the screened feature data, including:

[0035] three one-dimensional convolution layers are used to extract spatial features of the screened feature data;

[0036] the gating cycle module is used to learn context information of the spatial features to obtain sequence features of the feature data;

[0037] three fully connected layers are used to map the sequence features to a probability interval to obtain classification probability values;

[0038] neural function prediction is performed according to the classification probability values.

[0039] In a second aspect, the embodiments of the present application also provide a neural function prediction device based on imageomics, including:

[0040] an acquisition module configured to acquire magnetic resonance imaging;

[0041] a first network module configured to use a two-dimensional convolution architecture, a three-dimensional convolution architecture, an encoder and a decoder to construct a first neural network model, and use the first neural network model to segment the magnetic resonance imaging to obtain an image of a region of interest;

[0042] a feature extraction module configured to perform feature extraction on the image of the region of interest to obtain feature data of the magnetic resonance imaging;

[0043] a feature screening module configured to combine a filtering method and an embedding method to perform feature screening on the feature data; wherein the filtering method includes one or more of a Spearman coefficient test, an independent sample t test and a Levene test; and the embedding method includes a Lasso regression verification;

[0044] A second network module is configured to build a second neural network model using a one-dimensional convolution architecture, a gated recurrent module, and a fully connected architecture, and to perform neural function prediction according to the screened feature data by using the second neural network model.

[0045] In a third aspect, the embodiments of the present disclosure further provide an electronic device, which comprises a processor, a memory for storing executable instructions of the processor, and the processor is configured to read the executable instructions from the memory and execute the instructions to implement the method disclosed in the first aspect.

[0046] In a fourth aspect, the embodiments of the present disclosure further provide a computer readable storage medium, which stores a computer program for executing the method disclosed in the first aspect.

[0047] The neural function prediction method provided in the present application uses a two-dimensional convolution architecture, a three-dimensional convolution architecture, an encoder, and a decoder to build a first neural network model to perform region of interest segmentation on magnetic resonance imaging, then performs feature extraction and feature screening on the image of the region of interest, and finally uses a one-dimensional convolution architecture, a gated recurrent module, and a fully connected architecture to build a second neural network model to perform neural function prediction. By improving the region of interest labeling, feature extraction, and feature screening, and the prediction model establishment three steps respectively, the function of predicting the postoperative neural function preservation before operation is realized, which can assist doctors to formulate individualized treatment plan in advance, avoid irreversible neural function damage in the treatment process, and better implement individualized diagnosis and treatment for patients.

[0048] Other features and advantages of the present application will be described in detail in the following specific embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0049] The accompanying drawings, which form a part of this application, are included to provide a further understanding of the application and are incorporated in and constitute a part of this application. In the drawings:

[0050] Figure 1 Flow chart of the neural function prediction method based on imageomics of the preferred embodiment of the application;

[0051] Figure 2 Structure schematic diagram of the first neural network model of the preferred embodiment of the application;

[0052] Figure 3 Structure schematic diagram of the encoder of the preferred embodiment of the application;

[0053] Figure 4 Training Loss curve of the first neural network model of the preferred embodiment of the application;

[0054] Figure 5 A Mean Dice plot for the first neural network model of the preferred embodiment of the application;

[0055] Figure 6 A distribution histogram of the Dice coefficient for the first neural network model of the preferred embodiment of the application;

[0056] Figure 7 A segmentation result illustration using the first neural network model of the preferred embodiment of the application;

[0057] Figure 8 A structure illustration of the second neural network model of the preferred embodiment of the application;

[0058] Figure 9 An iteration plot of the second neural network model of the preferred embodiment of the application during the training process;

[0059] Figure 10 A confusion matrix of the second neural network model of the preferred embodiment of the application and other deep learning models on the test set;

[0060] Figure 11 A ROC curve result comparison plot of the second neural network model of the preferred embodiment of the application and other models;

[0061] Figure 12 A schematic diagram of a neural function prediction device based on radiomics of the preferred embodiment of the application;

[0062] Figure 13 A structure schematic diagram of an electronic device of the preferred embodiment of the application. DETAILED DESCRIPTION

[0063] The technical solutions of the present application will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

[0064] Firstly, the present application proposes a neural function prediction method based on radiomics, as shown in Figure 1 The steps are as follows: 110-150:

[0065] Step 110, acquiring magnetic resonance imaging;

[0066] Specifically, magnetic resonance imaging (MRI) is a kind of magnetic resonance phenomenon that uses hydrogen atoms in the human body to be excited by a strong magnetic field, and through spatial encoding technology, the electromagnetic waves emitted during the magnetic resonance process and various information parameters related thereto are processed by a computer, and finally an image is formed.

[0067] Step 120, a first neural network model is constructed using a two-dimensional convolution architecture, a three-dimensional convolution architecture, an encoder and a decoder, and the magnetic resonance imaging is segmented by using the first neural network model to obtain an image of a region of interest;

[0068] Specifically, for the artificial segmentation of magnetic resonance imaging, there are problems of large workload and different segmentation standards for different doctors, and the first neural network model of the application automatically segments each group of magnetic resonance imaging, and takes the segmentation label as the region of interest (ROI) in feature extraction in the following steps, that is, an image of a region of interest is obtained. In the field of machine vision and image processing, the region of interest can be understood as a selected image region from the image, which is the focus of subsequent image analysis, that is, the focus of feature extraction. Therefore, the application needs to first segment the image of the region of interest from the magnetic resonance imaging, and then perform feature extraction in the segmented image of the region of interest, thereby reducing the processing time and increasing the processing accuracy.

[0069] The first neural network model in the application is a self-designed neural network model with deep learning function, which is constructed using a two-dimensional convolution architecture, a three-dimensional convolution architecture, an encoder and a decoder. The two-dimensional convolution architecture specifically includes two two-dimensional convolution layers, and the three-dimensional convolution architecture includes two three-dimensional convolution layers. The four convolution layers can form a 2.5D UNet (U-shaped Network) architecture. The 2.5D combines the processing method of two-dimensional (2D) and three-dimensional (3D) information, and the UNet is a "U" type deep learning network model. The encoder in the application preferably adopts the Transformer encoder in the Transformer model. The Transformer model is a deep learning model for processing sequence data, which has excellent ability to capture global semantic features, so the encoder module of the Transformer is integrated into the network design, and the first neural network model is a new 2.5D Trans-UNet model.

[0070] Further specifically, Figure 2 The 2.5D Trans-UNet model structure provided by the application is also the first neural network model architecture diagram, in which the left side is the magnetic resonance imaging, and the right side is the output image of the region of interest, and the two are combined Figure 2 As shown in the figure, the first neural network model is used to segment the magnetic resonance imaging to obtain an image of a region of interest, which includes steps 121-124:

[0071] Step 121: sequentially using two two-dimensional convolution layers and two three-dimensional convolution layers, performing convolution operation on the magnetic resonance imaging to obtain a feature map;

[0072] The first and second layers of the first neural network model in the present application are convolution layers, and a two-dimensional convolution layer with a residual connection is used. The reason is that the in-plane resolution of magnetic resonance imaging is about four times the inter-plane resolution, so that the in-plane resolution of the feature map can be reduced by two-step 2 and a convolution kernel size of 3, so that the in-plane and inter-plane resolutions of the feature map are consistent, so that the three-dimensional convolution layer can perform convolution operation and extract in-plane features. The in-plane resolution, also known as the lateral resolution or radial resolution, refers to the minimum distance or minimum interval at which the imaging system can clearly distinguish two adjacent objects in the imaging plane; the inter-plane resolution, also known as the longitudinal resolution or axial resolution, refers to the ability of the imaging system to clearly distinguish two adjacent objects between imaging planes (i.e., in the depth direction).

[0073] The third and fourth layers of the first neural network model are still convolution layers, but use three-dimensional convolution, which is used to further convolve the feature map output by the two-dimensional convolution layer, extract inter-plane and in-plane features, and obtain the final feature map.

[0074] In some specific embodiments, batch normalization (BatchNorm) is used in the four convolution layers to keep the input of each layer of the neural network the same distribution. Batch normalization is a commonly used technique in deep learning, which normalizes the values of each feature to make the data distribution more stable and make the training simpler and faster. In terms of activation function, the parametric rectified linear unit (pReLu) is used as the activation function. At the same time, maximum pooling is performed after each convolution layer to extract important information. In addition, in order to prevent overfitting, the Dropout coefficient of the two two-dimensional convolution layers with lower dimensions is 0.1, and the three-dimensional convolution layer with higher dimensions is added to 0.2. Dropout refers to randomly deleting part of the nodes in the input layer and hidden layer with a certain probability in the neural network layer, and the Dropout coefficient of 0.1 or 0.2 can be understood as that during the training process, each neuron of the layer has a 10% or 20% probability of being randomly discarded.

[0075] Step 122: performing linear mapping on the feature map to obtain a vector sequence of the feature map;

[0076] The transformer model architecture can be mainly divided into an encoder and a decoder. The encoder is responsible for converting the input sequence into a series of hidden representations that can contain semantic and contextual information of the input sequence, providing the necessary information basis for the decoder to generate the output sequence. In order to adapt to the one-dimensional input form of the transformer encoder, a linear mapping method is used to change the dimension between the UNet architecture and the transformer encoder after the fourth convolutional layer, so that the feature map is converted into a vector sequence format that the transformer encoder can read.

[0077] Step 123: using the encoder, performing global semantic extraction on the vector sequence to obtain global semantic encoding;

[0078] The encoder of the first neural network model is a 12-layer transformer encoder module, which further integrates various features to obtain global semantic encoding, and then outputs the global semantic encoding to the decoder part of the network.

[0079] In some specific embodiments, in combination with Figure 3 As shown in the encoder structure diagram, when using the transformer encoder to perform global semantic extraction, first, the input feature dimension of the magnetic resonance imaging of the transformer encoder module is formula (1):

[0080] x∈R C×W×H×D (1)

[0081] Wherein, the feature x is a four-dimensional tensor, R represents a real set, C represents a modal number or a channel number of the magnetic resonance imaging, W represents a pixel number of the magnetic resonance imaging in a horizontal direction, H represents a pixel number of the magnetic resonance imaging in a vertical direction, and D represents a segmentation number of the magnetic resonance imaging.

[0082] Then, the vector sequence of the input feature map is divided into N non-overlapping small blocks x p As shown in formula (2) and (3):

[0083]

[0084] Wherein, x p represents a small block with a voxel size of P 3 , a modal number of C, N is a segmentation number of the magnetic resonance imaging, P is a unit length of each dimension of the voxel, and i is a counting unit between 1 and N.

[0085] Then, the small block is sent to a linear mapping layer to obtain embedded projection information E that retains the content information of the small block. The dimension M of this layer is equal to P 3 multiplied by C.

[0086] In order to retain the position information, the position embedding information E is added to the basis E p , to obtain the initial input embedding sequence a0 of the Transformer layer, as shown in formula (4):

[0087]

[0088] wherein x p is the segmented image block, E is the embedding projection of each small block, E p is the position embedding information, N is the number of segmentation of magnetic resonance imaging, and P is the unit length of each dimension of the voxel.

[0089] Meanwhile, the Transformer module also includes a multilayer perceptron (MLP). The multilayer perceptron includes two linear layers, and uses GELU (Gaussian Error Linear Unit) as the activation function. The multi-headed self-attention (MSA) of the Transformer module is composed of 12 self-attention heads, which are used to extract information in the input embedding sequence. In these self-attention heads, each element a i in the embedding sequence contains three different vectors, namely: a query (Query) vector q, a key (Key) vector k, and a value (Value) vector v.

[0090] The output of the self-attention head is shown in formula (5):

[0091]

[0092] wherein d n is the dimension of the input data, is a scaling factor of the feature map, which is used to prevent the inner product of the vector q and the vector k T from being too large, SA() represents the self-attention head function, softmax() is a normalized exponential function, N is the number of segmentation of magnetic resonance imaging, M is the dimension of the current layer, R represents the real number set, T is the transpose operator, q is the query vector, k is the key vector, and v is the value vector.

[0093] The expression of the multi-headed self-attention is shown in formula (6):

[0094] MSA(a)=[SA1(a);SA2(a);…;SA 12 (a);]W msa (6)

[0095] where MSA(a) represents the multi-head self-attention of the input embedded sequence a, W msa For the weight matrix used for training, the mean and variance of the input of each layer of neurons can be converted into the same value to accelerate convergence.

[0096] In summary, the expressions of the multi-head self-attention and the multi-layer perceptron are shown in equations (7) and (8):

[0097]

[0098] where, represents the multi-head self-attention output result of the feature map a l , represents the multi-layer perceptron output result of the feature map a l , norm() represents a normalization function, a l+1 and a l―1 represent the previous and next elements of the element a l in the current embedded sequence, respectively, and the dimension of the element a l is The meanings of N, M, P, H, W, and D are the same as those in other expressions above.

[0099] Step 124: Using the decoder, the global semantic encoding is deconvoluted to obtain the image of the region of interest.

[0100] The decoder of the first neural network model is a mirror deconvolution layer of the first four convolution layers, and the parameter settings are consistent with the corresponding convolution layers. Between each corresponding convolution layer and deconvolution layer, a skip connection is used to directly connect the input to the output to solve the gradient vanishing problem and accelerate the training process.

[0101] The following shows the effect of the first neural network model, i.e., the 2.5D U-Net model, in a specific embodiment of a vestibular schwannoma (VS).

[0102] The 2.5D U-Net model is trained, validated, and predicted using a private dataset, with a total data amount of 98 patients, each with two sequences of MRI data. According to k-fold cross-validation, the allocation of the training set, validation set, and test set is 7:1:2, and the saved model with the highest Dice coefficient on the validation set is used to predict the test set. K-fold cross-validation is a form of cross-validation. The Dice coefficient is a set similarity measure function used to calculate the similarity of two sets. The model converges at about 300 epochs, and the training loss curve of the model is as follows:Figure 4 All Dice coefficients and average Dice coefficients are shown in Table 2.

[0103] All Dice coefficients and average Dice coefficients are shown in Table 2. Figure 5 The highest Dice coefficient is 0.9626, the lowest is 0.7909, and the average Dice coefficient is 0.9066±0.0389. From the results, it can be seen that the highest Dice coefficient and the average Dice coefficient are higher than the segmentation accuracy required by the task 0.8, and the lowest Dice coefficient is also close to 0.8, so it can be considered that the model in this paper can better complete the VS segmentation task.

[0104] The distribution histogram of the Dice coefficients of the test set is shown in Figure 2. Figure 6 As can be seen from the figure, most of the Dice coefficients fall within the range of 0.86-0.97, and the number of samples with a Dice coefficient near 0.93 is the largest. Although the Dice coefficients of individual segmentation results are low, most of them are above 0.86, so overall the segmentation accuracy of the model in this paper is good.

[0105] The lowest Hausdorff distance of the test set prediction result is 1.41, the highest is 12.84, and the average value is 4.67±3.19. From the results, it can be seen that the average Hausdorff distance value is low, indicating that the boundaries of the true value and the prediction result are more coincident. The overall sensitivity and average sensitivity of the results during the prediction of the test set are close to 1, indicating that the proportion of correctly predicted areas to the true value is high, and the results have good reliability.

[0106] An example of the segmentation result is shown in Figure 3. Figure 7 From left to right are the original MRI image, the segmentation true value label, and the prediction output of the model of the application, and the corresponding Dice coefficient and Hausdorff distance are shown above the output image.

[0107] Comparing the segmentation results obtained by the model of the application with the original MRI and the true value label, it can be seen that the position and size of VS in the brain are not fixed, and the shape is irregular, making the segmentation task more difficult. However, the results show that the tumor area in the segmentation label of the model in this study is basically consistent with the true value label, and has a good segmentation result.

[0108] In addition, the segmentation results of the method of the application and other methods are compared. Table 1 compares the segmentation results of several classical methods on the data set of this study. As can be seen, the model proposed in this application performs well on the self-collected data set, and achieves the best results in both Hausdorff distance and Dice score, which is significantly improved compared with traditional 3D U-Net and 2.5D U-Net, especially in the accuracy of contour segmentation.

[0109]

[0110] Table 1 Comparison of results of other methods on self-collected data set

[0111] Meanwhile, in order to further verify the effectiveness of the model proposed in the present research, the present application uses the data set disclosed by Sharpey et al. on The Cancer Imaging Archive (TCIA) to further verify the model. Table 2 compares the results of the segmentation of the present method and several different methods on the TCIA public data set. It can be seen that the segmentation model proposed in the present research has obtained the best segmentation results in the Hausdorff distance and the Dice score, surpassing the SOTA (State-of-the-Art) method. In terms of the Hausdorff distance, the present method has obtained a result similar to that of the SOTA method.

[0112]

[0113]

[0114] Table 2 Comparison of results of other literatures on TCIA data set

[0115] In summary, the method proposed in the present application effectively combines the Transformer and the 2.5D U-Net network structure, and has obtained competitive tumor segmentation performance.

[0116] In step 130, feature extraction is performed on the image of the region of interest to obtain feature data of the magnetic resonance imaging;

[0117] Specifically, Pyradiomics is an open-source python software package that can extract imaging features from medical images. The feature extraction is performed on the region of interest obtained by the 2.5D Trans-UNet automatic segmentation by using the Pyradiomics tool to obtain the feature data of the magnetic resonance imaging.

[0118] In one specific embodiment, the extracted radiomics features of the magnetic resonance imaging include one or more of the following features: First-Order, Gray Level Co-occurrence Matrix (GLCM), Gray Level Dependence Matrix (GLDM), Gray Level Run Length Matrix (GLRLM), Gray Level Size Zone Matrix (GLSZM), Neighboring Gray Tone Difference Matrix (NGTDM), three-dimensional shape, two-dimensional shape, and Wavelet-based.

[0119] Step 140, feature screening of the feature data is performed by combining the filtering method and the embedding method.

[0120] Specifically, the radiomics features extracted in the above steps may have a high number of feature types and feature quantities, which may easily result in a large number of redundant features and noise features, thereby affecting the performance of the subsequent prediction model and increasing the calculation cost. Therefore, screening the radiomics features helps to improve the performance of the subsequent prediction model, reduce the risk of overfitting, and reduce the calculation cost of model training and inference. At the same time, screening the key radiomics features helps to enhance the accuracy and interpretability of the prediction results.

[0121] In existing feature screening methods, the interaction between features is usually ignored, or the training complexity is sharply increased in order to take into account the feature correlation. The present application combines the filtering method and the embedding method, which has the advantages of high calculation efficiency, good adaptation to learning algorithms, and easy avoidance of overfitting, and can well meet the research goal of the small sample deep learning algorithm.

[0122] In some specific examples, the filtering method includes one or more of a Spearman coefficient test, an independent sample t-test, and a Levene test. The embedding method includes using a L1 norm least absolute shrinkage and selection operator (LASSO) regression test. The feature data set is divided into two groups of samples according to the postoperative neurological function injury level of the patients: mild and moderate to severe, the mild sample group corresponds to a label of 0, and the moderate to severe sample group corresponds to a label of 1, and each patient includes a large number of original image features. In a more preferred embodiment, the feature screening of the original image features in combination with the filtering method and the embedding method includes the above four test methods, that is, the Levene test, the independent sample t-test, the Spearman coefficient test, and the Lasso regression test are sequentially performed on the feature data. The above three test methods in the filtering method remove abnormal samples, samples without significance, and samples with low stability, and more comprehensively preliminarily screen the extracted image features. Then, the Lasso regression test is used to select features that have an important influence on subsequent classification to suppress model overfitting and improve model performance. Among them:

[0123] Levene's test is a statistical method for testing whether the variances of multiple groups are equal, and is usually used for variance homogeneity test, that is, to test whether the variances of different groups are equal. The null hypothesis of Levene's test is that the variances of different groups are equal, and the alternative hypothesis is that at least one group has a different variance. The basic idea is to test the homogeneity of variance by comparing the differences between the observed values and the mean values within each group. If the variances of different groups are small, the differences should be small, and if the variances of different groups are large, the differences may be large.

[0124] Levene's test includes calculating the W value of the Levene test of the feature data, and the W value is a test statistic that directly measures the difference in variances of different groups. The F distribution table of the W value of the feature data is looked up to obtain the p value corresponding to the W value of the feature data. If the p value corresponding to the W value of the feature data is not greater than a first preset value, the current feature data is excluded. Here, the p value corresponding to the W value indicates the probability of obtaining the current W value or a more extreme value under the assumption that the variances are equal, and is used to determine whether the variance homogeneity hypothesis is established. The first preset value is preferably 0.05.

[0125] Specifically, after the Levene test of the two groups of original image feature samples, the features with significant variance difference can be selected to remove the features with variance inequality caused by abnormal samples. For the features that pass the Levene test, that is, the variances of the features in the two groups of samples are equal, the independent sample t-test can be used for further feature screening.

[0126] Independent Samples t-test is a statistical test method for comparing whether the means of two independent samples are significantly different. It is used to determine whether there is a significant difference in the mean between two independent groups. The basic assumption of Independent Samples t-test is that the two samples are independent samples from the same population, and the distribution of the two populations is normal distribution. The null hypothesis of this test is that the means of the two populations are equal, and the alternative hypothesis is that the means of the two populations are not equal.

[0127] The Independent Samples t-test includes: calculating the t value of the Independent Samples t-test of the feature data, comparing the t value of the feature data with the theoretical t value distribution, obtaining the p value corresponding to the t value of the feature data, and if the p value corresponding to the t value of the feature data is less than a second preset value, the current feature data is excluded. Here, the p value corresponding to the t value represents the probability of the tail area corresponding to the t value in the t distribution. The second preset value is preferably 0.05.

[0128] Specifically, the Independent Samples t-test is performed on each feature in the two groups of feature samples after the Levene test. Each time, the t value is calculated by taking the feature values corresponding to the same feature in the two groups of samples, and the p value of the feature is obtained. If the p value is less than 0.05, it indicates that the mean difference of the feature between the two groups is significant, and the feature is retained; otherwise, the feature is removed.

[0129] Spearman's rank correlation coefficient is a statistical measure of the correlation between two variables. Spearman's rank correlation coefficient is a non-parametric method suitable for non-linear relationships. Spearman's rank correlation coefficient is usually used to measure the rank relationship between variables, especially when the data does not meet the normal distribution assumption or there are extreme values. If two features have a high correlation, one of them can be considered redundant and thus removed. Such redundant features are often unstable, especially when the data dimension is high. Therefore, using Spearman's rank correlation coefficient can filter out features with high redundancy and low stability.

[0130] The Spearman's rank correlation coefficient test includes: calculating the Spearman's rank correlation coefficient p of the feature data, and if the Spearman's rank correlation coefficient p of the feature is not greater than a third preset value, the current feature data is excluded. The third preset value is preferably 0.8.

[0131] Specifically, the two groups of feature samples after Levene test and independent sample t test are combined and subjected to Spearman coefficient test together, wherein the target label from the mild sample group is 0, and the target label from the moderate and severe sample group is 1. The Spearman correlation coefficient p of each feature and the target label is calculated. If the Spearman correlation coefficient p is not greater than 0.8, it indicates that the correlation between the feature and the target label is low, and the feature is excluded; otherwise, the feature is retained.

[0132] Lasso regression verification (Least absolute shrinkage and selection operator), a linear regression method, automatically selects those features that contribute less to the model and excludes them by making the coefficient 0, so as to achieve the purpose of feature selection and dimension reduction, and better solve the problem of multicollinearity in regression analysis.

[0133] Specifically, the Lasso regression test is performed on the entire feature set containing two groups of feature samples after the feature selection by the above three filtering methods, wherein the target label from the mild sample group is 0, and the target label from the moderate and severe sample group is 1. The Lasso regression model is trained using the feature data and the corresponding target label, and the regularization coefficient of each feature is automatically determined by cross-validation during the training process. After the model is trained, the regularization coefficient of each feature is obtained, and if the coefficient of a certain feature is 0, it means that the feature is considered unimportant and can be excluded. The features with non-zero coefficients are retained as the important imageomic features obtained after the final screening.

[0134] Step 150, a second neural network model is constructed using a one-dimensional convolution architecture, a gated recurrent module and a full connection architecture, and neural function prediction is performed according to the screened feature data using the second neural network model;

[0135] Specifically, the one-dimensional convolution architecture can be used as the backbone of the serial hybrid neural network, and the gated recurrent module (Gated Recurrent Unit, GRU) is used as an improved recurrent neural network to control the flow of information through update gate and reset gate. The update gate determines the mixing ratio of the current hidden state and the previous state, and the reset gate controls how much past information is discarded. By fusing past and current state information, the long-term dependence problem in the sequence can be effectively addressed, the structure of the long short-term memory (Long Short-Term Memory, LSTM) is simplified, and the model has the characteristics of easy training and easy implementation. Therefore, the one-dimensional convolution architecture is used as the backbone network, the gated recurrent module is introduced, and a 1DCNN-GRU model is constructed to build a prognosis prediction model, i.e. the second neural network model, which is used to classify the previously extracted and screened imageomic features and make predictions based on the classification results.

[0136] Further specifically, Figure 8 is a 1DCNN-GRU model structure diagram, that is, a second neural network model architecture diagram, combined with Figure 8 As shown in the figure, the second neural network model is used to perform neural function prediction according to the screened feature data, including steps 151-153:

[0137] Step 151, three one-dimensional convolution layers are used to extract spatial features of the screened feature data.

[0138] The spatial features can be understood as spatial entity positions, topological relationships and geometric features of the feature data.

[0139] In some specific embodiments, a maximum pooling is also performed after each one-dimensional convolution layer here to extract important information.

[0140] Step 152, a gated recurrent module is used to learn context information of the spatial features to obtain sequence features of the feature data.

[0141] The gated recurrent module is used to capture sequence features of the input data, learn context information of the spatial features, and obtain sequence features of the feature data.

[0142] Step 153, three fully connected layers are used to map neuron data to a probability interval to obtain classification probability values, and classification is performed according to the classification probability values, so as to perform neural function prediction according to the classification results.

[0143] The fully connected layer connects all nodes of the previous layers to each node of the following layers, realizing global mapping of the features. This fully connected manner can make the network learn complex features in the data and can establish a nonlinear relationship between the input and the output. As the last layer of the present application, the role of the fully connected layer is to integrate the features extracted by the previous convolution layers or pooling layers and generate the final output result.

[0144] In some specific embodiments, the convolution kernel size of the three one-dimensional convolution layers in the one-dimensional convolution architecture is 3, the filter number of the first convolution layer is 32, the filter number of the second convolution layer is 64, and the filter number of the third convolution layer is 128. A maximum pooling layer (MaxPooling1D) with a dimension of 1 is added after each convolution layer in the network, followed by a batch normalization (Batch Normalization, BN) operation.

[0145] The gated recurrent module is a GRU layer with 128 units, which is used to learn sequence context information.

[0146] The full connection architecture is three full connection layers (Dense), and finally uses a normalized exponential function (Softmax) as the classification network activation function, and a coefficient of 0.3 is added to the three full connection layers. Dropout, with a probability of 30%, discards the output of the neurons to reduce the risk of network overfitting.

[0147] In terms of the output of the results, the model is binary classified using a normalized exponential function (Softmax) to convert the output of the neural network into a vector representing the probability of different categories.

[0148] So far, the second neural network model can obtain a probability for prediction, classify the prognosis of the patient's nerve damage according to the probability value, and predict the neural function according to the neural function prediction.

[0149] The neural function prediction method provided by the application uses a two-dimensional convolution architecture, a three-dimensional convolution architecture, an encoder and a decoder to construct a first neural network model to segment the region of interest of magnetic resonance imaging, then extracts and screens features from the image of the region of interest, and finally uses a one-dimensional convolution architecture, a gated recurrent module and a full connection architecture to construct a second neural network model to predict the neural function. By improving the region of interest labeling, feature extraction and feature screening, and the prediction model establishment, the function of predicting the postoperative neural function preservation before surgery is realized, which can assist doctors to formulate individualized treatment plans in advance, avoid irreversible neural function damage during treatment, and better implement individualized diagnosis and treatment for patients.

[0150] Next, in a specific embodiment of a vestibular schwannoma (VS), the effect brought by the second neural network model in the application is demonstrated.

[0151] The application performs experiments on a self-collected data set to verify the effectiveness of the proposed imageomics feature screening method combining filtering and embedding methods and the 1DCNN-GRU hybrid network model.

[0152] Figure 9 The iteration curve of the 1DCNN-GRU network model proposed by the application in the training process. The two curves respectively represent the loss curve of the training and the accuracy curve of the training set.

[0153] Figure 10 The confusion matrix of the 1DCNN-GRU network model proposed by the application and other deep learning models on the test set, the input features of the prediction model come from the automatic segmentation results obtained by using 2.5D Trans-UNet and the tumor regions segmented by artificial segmentation. Among them, Figure 10(A), 10 (B) is the confusion matrix of 1D CNN-LSTM model on the test set, Figure 10 (C), 10 (D) is the confusion matrix of 1D CNN model on the test set; Figure 10 (E), 10 (F) is the confusion matrix of 1D CNN-GRU model proposed in this study on the test set. Figure 10 (A), 10 (C), 10 (E) are the confusion matrices of the features of the automatic segmentation results obtained using 2.5D Trans-UNet for prediction. Figure 10 (B), 10 (D), 10 (F) are the confusion matrices of the features of the manual segmentation results for prediction.

[0154] In order to further evaluate the performance of the proposed 1D CNN-GRU network model, this paper conducts a number of comparative experiments on the self-collected data set. The comparative models used are 1D CNN, 1D CNN-LSTM and several representative machine learning algorithms, including Support Vector Machine (SVM), Logistic regression, Bootstrap aggregating (Bagging), decision tree. The comparison results are shown in Figure 11 Figure 11 (A) is the ROC (Receiver Operating Characteristic curve) curve of prognosis prediction using automatic segmentation results under different networks, Figure 11 (B) is the ROC curve of prognosis prediction using manual segmentation results under different networks.

[0155] From the above figure, it can be seen that in the two experiments, the prediction effect of the 1D CNN-GRU hybrid network (black line) proposed in this study is better than that of the other four traditional neural networks, and the optimal value of the model can be reached faster.

[0156]

[0157]

[0158] Table 3 Comparison of prediction results of different networks using automatic segmentation results

[0159]

[0160] Table 4 Comparison of prediction results of different networks using manual segmentation results

[0161] ​Table 3 and Table 4 are respectively the evaluation indexes of the results of using 2.5D Trans-UNet for automatic tumor segmentation and using artificial segmentation of the tumor area, and then performing feature extraction and screening, and then inputting each deep neural network model for prognosis prediction task on the self-collected data set.

[0162] As can be seen from Table 3 and Table 4, the model proposed in the present application almost achieves the best classification effect in the three indexes of accuracy, ROC curve and recall rate. And except for SVM, the evaluation indexes of other machine learning and deep learning models in the automatic segmentation group are better than those in the artificial segmentation group, proving that the features extracted from the automatic segmentation results have advantages in improving the performance of the model.

[0163] In summary, the method proposed in the present application has achieved the best effect in many indexes, surpassing the machine learning models such as SVM and the two deep learning models of 1DCNN and 1DCNN-LSTM, achieving the optimal performance, proving that the 2.5D Trans-UNet automatic segmentation model designed in the present application, the imageomics feature screening method combining the filtering method and the embedding method, and the 1DCNN-GRU classification model have strong performance in the VS postoperative prognosis prediction task.

[0164] In addition, the present application also provides a neural function prediction device based on imageomics, for realizing the neural function prediction method based on imageomics, as shown in the following Figure 12 The device comprises:

[0165] The acquisition module 210 is configured to acquire magnetic resonance imaging;

[0166] The first network module 220 is configured to use a two-dimensional convolution architecture, a three-dimensional convolution architecture, an encoder and a decoder to construct a first neural network model, and use the first neural network model to segment the magnetic resonance imaging to obtain an image of a region of interest;

[0167] The feature extraction module 230 is configured to perform feature extraction on the image of the region of interest to obtain feature data of the magnetic resonance imaging;

[0168] The feature screening module 240 is configured to perform feature screening on the feature data by combining a filtering method and an embedding method; wherein the filtering method comprises one or more of Spearman coefficient test, independent sample t test and Levene test; and the embedding method comprises Lasso regression verification;

[0169] The second network module 250 is configured to use a one-dimensional convolution architecture, a gated recurrent module and a fully connected architecture to construct a second neural network model, and use the second neural network model to perform neural function prediction according to the screened feature data.

[0170] The effect of the neural function prediction device based on the radiomics disclosed in the present application is the same as the neural function prediction method based on the radiomics described above, and will not be repeated here.

[0171] Below, the electronic device 11 according to the embodiments of the present application will be described with reference to Figure 13 Figure 13 The block diagram of the electronic device according to the embodiments of the present application is shown.

[0172] As shown in Figure 13 , the electronic device 11 includes one or more processors 111 and a memory 112.

[0173] The processor 111 can be a central processing unit (CPU) or other form of processing unit having data processing and / or instruction execution capabilities, and can control other components in the electronic device 11 to perform desired functions.

[0174] The memory 112 can include one or more computer program products, which can include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include random access memory (RAM), cache memory, and / or the like. The non-volatile memory may, for example, include read-only memory (ROM), hard disk, flash memory, and / or the like. One or more computer program instructions can be stored on the computer-readable storage medium, and the processor 111 can execute the program instructions to implement the test method of the embodiments of the present application described above and / or other desired functions. Various contents such as input signals, signal components, noise components, and the like can also be stored in the computer-readable storage medium.

[0175] In one example, the electronic device 11 can further include an input device 113 and an output device 114, which are interconnected by a bus system and / or other forms of connection mechanism (not shown).

[0176] The input device 113 can include, for example, a keyboard, a mouse, and the like.

[0177] The output device 114 can output various information to the outside, including the determined distance information, direction information, and the like. The output device 114 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0178] Of course, in order to simplify, Figure 13 ​Only some of the components of the electronic device 11 related to the present application are shown, and components such as a bus, an input / output interface, and the like are omitted. In addition to these, the electronic device 11 can include any other appropriate components according to the specific application.

[0179] In addition to the above method and device, an embodiment of the present application can also be a computer program product, which includes computer program instructions, which, when executed by a processor, cause the processor to perform the steps of the test method according to various embodiments of the present application described in the above "Exemplary Method" section of the present specification.

[0180] The computer program product can be written in any combination of one or more programming languages, including an object-oriented programming language such as Java, C++, and the like, and conventional procedural programming languages, such as the "C" programming language, or the like. The program code can execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server.

[0181] In addition, an embodiment of the present application can also be a computer readable storage medium, which stores computer program instructions, which, when executed by a processor, cause the processor to perform the steps of the test method according to various embodiments of the present application described in the above "Exemplary Method" section of the present specification.

[0182] The computer readable storage medium can take any combination of one or more of the following readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium, for example, can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the above. More specific examples (a non-exhaustive list) of the readable storage medium include an electrical connection having one or more wires, a portable disc, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0183] The above describes the preferred embodiments of the present application, but the present application is not limited to the specific details of the above-described embodiments. Within the technical concept of the present application, various simple modifications can be made to the technical solution of the present application, and these simple modifications all belong to the protection scope of the present application.

[0184] It should be further noted that various specific technical features described in the above detailed description are not to be interpreted as essential to the application, unless explicitly stated as such. In other words, the various technical features described in the above detailed description are not to be interpreted as essential to the application, unless explicitly stated as such.

[0185] Furthermore, the various embodiments of the application can also be combined with each other, as long as it does not violate the spirit of the application, it should also be considered as disclosed in the present application.

Claims

1. A method for predicting neurological function based on radiomics, characterized in that, The method comprises: acquiring magnetic resonance imaging; constructing a first neural network model using a two-dimensional convolution architecture, a three-dimensional convolution architecture, an encoder, and a decoder; in the first neural network model, two two-dimensional convolution layers and two three-dimensional convolution layers are used in sequence to perform convolution operation on the magnetic resonance imaging to obtain a feature map, wherein the two two-dimensional convolution layers are two two-dimensional convolution layers with residual connections, and the in-plane resolution and the inter-plane resolution of the feature map are consistent; performing linear mapping on the feature map to obtain a vector sequence of the feature map; performing global semantic extraction on the vector sequence by using the encoder to obtain global semantic encoding, wherein the encoder is a Transformer encoder; performing inverse convolution operation on the global semantic encoding by using the decoder to obtain an image of a region of interest, wherein the decoder is a mirror inverse convolution layer of the two two-dimensional convolution layers and the two three-dimensional convolution layers; performing feature extraction on the image of the region of interest to obtain feature data of the magnetic resonance imaging; performing feature screening on the feature data in sequence by using Levene test method, independent sample t test method, Spearman coefficient test method, and Lasso regression verification method; constructing a second neural network model using a one-dimensional convolution architecture, a gated recurrent module, and a fully connected architecture; in the second neural network model, three one-dimensional convolution layers are used to extract spatial features of the screened feature data; the context information of the spatial features is learned by using the gated recurrent module to obtain sequence features of the feature data; the sequence features are mapped to a probability interval by using three fully connected layers to obtain classification probability values; neural function prediction is performed according to the classification probability values.

2. The method of claim 1, wherein, The Levene test comprises: calculating the W value of the Levene test of the feature data; finding the F distribution table of the W value of the feature data to obtain the p value corresponding to the W value of the feature data; if the p value corresponding to the W value of the feature data is not greater than a first preset value, the feature data is excluded.

3. The method of claim 1, wherein, The independent sample t test comprises: an independent samples t-test of the feature data values; Compare the feature data Values ​​and Theories Value distribution, to obtain the feature data The value corresponds to the p-value; If the feature data If the p-value corresponding to the value is less than the second preset value, then the feature data is excluded.

4. The method of claim 1, wherein, The Spearman coefficient test comprises: calculating a spearman coefficient for the feature data ; If the Spearman coefficient of the feature data If the value is not greater than the third preset value, then the feature data will be excluded.

5. A neural function prediction device based on radiomics, characterized in that, The device comprises: an acquisition module configured to acquire magnetic resonance imaging; a first network module configured to construct a first neural network model using a two-dimensional convolution architecture, a three-dimensional convolution architecture, an encoder, and a decoder; in the first neural network model, two two-dimensional convolution layers and two three-dimensional convolution layers are used in sequence to perform convolution operation on the magnetic resonance imaging to obtain a feature map, wherein the two two-dimensional convolution layers are two two-dimensional convolution layers with residual connections, and the in-plane resolution and the inter-plane resolution of the feature map are consistent; performing linear mapping on the feature map to obtain a vector sequence of the feature map; performing global semantic extraction on the vector sequence by using the encoder to obtain global semantic encoding, wherein the encoder is a Transformer encoder; The decoder is used to perform inverse convolution operation on the global semantic code to obtain an image of the region of interest, wherein the decoder is a mirror inverse convolution layer of the two two-dimensional convolution layers and the two three-dimensional convolution layers; a feature extraction module configured to perform feature extraction on the image of the region of interest to obtain feature data of the magnetic resonance imaging; a feature screening module configured to perform feature screening on the feature data by using Levene test method, independent sample t test method, Spearman coefficient test method and Lasso regression verification method in sequence; a second network module configured to construct a second neural network model by using one-dimensional convolution architecture, a gated recurrent module and full connection architecture; in the second neural network model, three one-dimensional convolution layers are used to extract spatial features of the screened feature data; the gated recurrent module is used to learn context information of the spatial features to obtain sequence features of the feature data; three full connection layers are used to map the sequence features to a probability interval to obtain classification probability values; and neural function prediction is performed according to the classification probability values. 6.An electronic device, comprising: a processor; a memory for storing executable instructions of the processor; the processor is used to read the executable instructions from the memory and execute the instructions to realize the method of any one of claims 1-4. 7.A computer readable storage medium, the storage medium stores a computer program, the computer program is used to execute the method of any one of claims 1-4.

Images