Method for constructing a magnetic resonance imaging prediction model

Abstract

The application discloses a kind of nuclear magnetic resonance image prediction model construction methods, main design concept is in, from the information between using multi-stage liver cancer nuclear magnetic image data start to improve the prediction performance of the prediction model constructed, specifically, propose by multiple multilayer convolutional neural network as the extraction part of image feature, in this link, introduce multi-sequence image difference extraction pyramid module, difference information is mined between the input two two image stages and is integrated into image feature extraction operation, then the feature map containing difference information extracted by convolutional neural network is sent into multi-sequence image correlation attention module, learn the correlation between the input different stage nuclear magnetic image, and then provide rich reference information for subsequent accurate prediction classification.The present application can make full use of the difference information and correlation information between the images of different stages in the enhanced nuclear magnetic image, so as to effectively improve the prediction performance of the model based on image information.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to a method for constructing a nuclear magnetic resonance image prediction model. BACKGROUND

[0002] Hepatocellular carcinoma (HCC) is the most common primary liver cancer, accounting for more than 80% of primary liver cancer worldwide, and HCC is the third leading cause of cancer-related death in the world. For patients diagnosed at an early stage, liver resection and liver transplantation both have certain therapeutic effects and are currently the first-line treatment for early liver cancer patients. Among them, liver transplantation has superior therapeutic effect on hepatocellular carcinoma, but is expensive, has high requirements and is in short supply of liver sources, so liver resection is still the main treatment for most early hepatocellular carcinoma patients in clinical practice. However, due to the extremely high risk of recurrence, the overall survival rate after liver resection has not been satisfactory, and the 5-year recurrence rate after surgery is as high as 70%. The main risk factors for liver cancer recurrence include tumor vascular invasion, incomplete tumor resection, high tumor malignancy, and poor patient condition, and many other factors. In recent years, the blood-borne metastasis theory caused by tumor vascular invasion has received increasing attention. Microvascular invasion (MVI) usually occurs in cancer cells within the vascular invasion within 1 cm around the cancer lesion and beyond, and is considered an early sign of cancer cell distant metastasis and colonization, and is an important risk factor for early recurrence and metastasis after tumor resection, with a highest incidence rate of nearly 60%. Studies have shown that MVI increases the recurrence rate of patients by 1 time and shortens the survival period, and is the main risk factor for early recurrence within the first 2 years after curative treatment. Therefore, preoperative prediction of MVI has very important value for the development of surgical strategies and prognosis evaluation, and reduces the recurrence rate while minimizing unnecessary liver damage.

[0003] Since MVI is a nest of cancer cells within the endothelial cell-lined vascular lumen visible under a microscope, it usually occurs in small branches of the portal vein, and clinicians cannot accurately judge MVI before surgery, whether through imaging alone or through biopsy.

[0004] Currently, the preoperative prediction method for MVI of liver cancer is mostly to preprocess data, then extract features from the data (such as image features based on deep learning), and then input the extracted features into a prediction model (machine learning model or deep learning model). For example, in which the method of using multiple stages of images and simultaneously extracting image features to predict microvascular invasion in hepatocellular carcinoma is mostly used. Although this image feature-based prediction scheme has greatly improved the prediction of microvascular invasion in hepatocellular carcinoma compared to a single sequence, the current image processing scheme simply concatenates the image feature vectors extracted from different stages, and does not specifically use the difference information and correlation information between different stages of images, resulting in poor prediction accuracy of the current image processing method. SUMMARY

[0005] In view of the above, the present application aims to provide a magnetic resonance imaging prediction model construction method to solve the aforementioned technical problems.

[0006] The technical scheme adopted by the present application is as follows:

[0007] The present application provides a magnetic resonance imaging prediction model construction method, which comprises:

[0008] An enhanced magnetic resonance image sample of hepatocellular carcinoma is obtained, and an enhanced magnetic resonance image database of hepatocellular carcinoma is constructed;

[0009] The magnetic resonance images of different stages of hepatocellular carcinoma are taken as input, and the binary classification results of the preset image indicators are taken as output to construct a prediction model; wherein the prediction model comprises: a plurality of convolutional neural networks for extracting image features, each convolutional neural network corresponding to different stage magnetic resonance image input; a multi-sequence image difference extraction pyramid module for obtaining difference information between different stages of magnetic resonance images during image feature extraction; a multi-sequence image correlation attention module for obtaining correlation information between different stages of magnetic resonance images based on the extracted image features; and a classification prediction module for outputting results;

[0010] The prediction model is trained and tested using the enhanced magnetic resonance image database of hepatocellular carcinoma.

[0011] In at least one possible implementation, the construction method comprises: setting one of the multi-sequence image difference extraction pyramid modules between two of the convolutional neural networks, and the input of each of the multi-sequence image difference extraction pyramid modules is the difference between the magnetic resonance images of two stages; the multi-sequence image difference extraction pyramid module delivers the difference information of the two-stage magnetic resonance images obtained to the corresponding two convolutional neural networks respectively, and inputs the sum of the outputs of the corresponding convolutional layers in the convolutional neural networks to the next convolutional layer of the convolutional neural network for subsequent image feature extraction operations.

[0012] In at least one possible implementation, the size and the number of channels of the feature images output by each layer of the multi-sequence image difference extraction pyramid module are consistent with and aligned with the size and the number of channels of the corresponding convolutional layer in the corresponding convolutional neural network.

[0013] In at least one possible implementation, the input of one of the multi-sequence image correlation attention modules is the feature maps output by all the convolutional neural networks respectively, wherein the feature map input by the convolutional neural network corresponding to the current multi-sequence image correlation attention module is a main feature map, and the feature maps input by the other convolutional neural networks are secondary feature maps, so as to learn the correlation information between the magnetic resonance images of different stages.

[0014] In at least one possible implementation, the processing process of the multi-sequence image correlation attention module comprises:

[0015] The main feature map is used to obtain a key matrix, a value matrix, and a query matrix respectively, and the query matrix is obtained by using the secondary feature map;

[0016] The query matrix of each feature map and the key matrix are used to obtain a corresponding attention map by using the similarity calculation of the attention mechanism;

[0017] The total attention map is obtained by weighted summation of all the attention maps;

[0018] The output of the multi-sequence image correlation attention module is obtained by combining the total attention map, the value matrix, and the main feature map.

[0019] In at least one possible implementation, the process of constructing the hepatocellular carcinoma enhanced magnetic resonance image database comprises: pre-processing and segmenting and labeling the obtained enhanced magnetic resonance image samples, and generating a format suitable for the input of the prediction model based on the segmentation result, wherein the pre-processing comprises calibrating and standardizing the magnetic resonance images of different stages.

[0020] In at least one possible implementation, the generating, based on the segmentation result, an adapted format of the prediction model input comprises: extracting a cube with a preset scaling size according to the segmentation annotation of the nuclear magnetic image of different stages.

[0021] The main design concept of the present application is to improve the prediction performance of the constructed prediction model by utilizing the information between the multi-stage liver cancer nuclear magnetic image data. Specifically, a plurality of multi-layer convolutional neural networks are proposed as the image feature extraction part. In this link, a multi-sequence image difference extraction pyramid module is introduced to mine the difference information between the input two images of each stage and integrate it into the image feature extraction operation. Then, the feature map containing the difference information extracted by the convolutional neural network is sent to the multi-sequence image correlation attention module to learn the correlation between the input nuclear magnetic images of different stages, thereby providing rich reference information for subsequent accurate prediction classification. The present application can fully utilize the difference information and correlation information between the images of different stages in the enhanced nuclear magnetic image, thereby effectively improving the prediction performance of the model based on image information. BRIEF DESCRIPTION OF DRAWINGS

[0022] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described below with reference to the drawings, in which:

[0023] Figure 1 The flowchart of the nuclear magnetic resonance image prediction model construction method provided by the embodiment of the present application is shown in

[0024] Figure 2 The overall architecture schematic diagram of the prediction model provided by the embodiment of the present application is shown in

[0025] Figure 3 The processing flowchart of the multi-sequence image correlation attention module provided by the embodiment of the present application is shown in DETAILED DESCRIPTION

[0026] The embodiments of the present application will be described in detail below, and the examples of the embodiments are shown in the drawings, in which the same or similar reference signs represent the same or similar elements or elements with the same or similar functions throughout. The embodiments described below by referring to the drawings are exemplary and are only used to explain the present application, and cannot be interpreted as a limitation on the present application.

[0027] The present application provides an embodiment of a nuclear magnetic resonance image prediction model construction method, specifically as shown in Figure 1 , which includes:

[0028] Step S1, acquiring an enhanced nuclear magnetic image sample of hepatocellular carcinoma and constructing an enhanced nuclear magnetic image database of hepatocellular carcinoma;

[0029] Specifically, the dynamic contrast-enhanced MRI of patients with hepatocellular carcinoma with or without microvascular invasion confirmed by pathology before liver resection surgery within a certain time range can be collected as samples to form the hepatocellular carcinoma enhanced magnetic resonance imaging database. More preferably, the MRI image samples can refer to the source data of the tumor lesion Volume of Interest (VOI) in the arterial late phase, portal vein phase, and HBP phase, that is, the example provides three-stage magnetic resonance images of hepatocellular carcinoma.

[0030] It can be understood that, in order to match the model input mentioned below, preferably, the process of constructing the database further includes: pre-processing and segmentation labeling of the obtained enhanced magnetic resonance image samples, and generating a format suitable for model input based on the segmentation results.

[0031] Specifically, the dynamic contrast-enhanced MRI images in different stages can be calibrated and standardized, and then a cube is uniformly extracted according to the tumor segmentation labeling of MRI in different stages and scaled to a uniform size, which is convenient for input of subsequent models. Thus, the task of constructing the dynamic contrast-enhanced MRI image database is completed. For example, the target region (tumor region) in the dynamic contrast-enhanced magnetic resonance image can be segmented and labeled, and the data of the segmented single target is retained. In addition, the pre-processing includes normalizing and calibrating the magnetic resonance images in different stages. Then, the segmented images in different stages are merged, and a cube is cropped from the target region according to multiple segmentation results and scaled to, for example, 60x60x60, which is used as the input of the model.

[0032] Step S2, taking the magnetic resonance images of hepatocellular carcinoma in different stages as input and taking the binary classification results of the preset image indicators as output, constructing a prediction model; wherein the prediction model comprises: a plurality of convolutional neural networks for extracting image features, each convolutional neural network corresponding to a magnetic resonance image input in a different stage; a multi-sequence image difference extraction pyramid module for obtaining difference information between magnetic resonance images in different stages during image feature extraction; a multi-sequence image correlation attention module for obtaining correlation information between magnetic resonance images in different stages based on the extracted image features; and a classification prediction module for outputting results.

[0033] For the four parts of the prediction model mentioned above, combined with Figure 2 For example, specific descriptions are as follows:

[0034] (1) In actual operation, in combination with the three phase stages mentioned in the foregoing, three convolutional neural networks can be used as image feature extraction networks for the three stages of MRI input, each of which can have a three-layer structure, use a 3x3x3 convolution kernel, set the moving step to 1, the padding to 1, and have 32 channels, 64 channels, and 128 channels, respectively. More specifically, a batch normalization layer, a ReLU activation function, and a max pooling layer are provided after each convolutional layer.

[0035] (2) In the process of extracting multi-stage image features by multiple convolutional neural networks, a multi-sequence image difference extraction pyramid module is further used to extract difference information in the image, so that different sequence characteristics can be extracted at different scales and fed back to the convolutional neural network, which can help the model learn the differences between different sequences to improve the prediction performance of the model.

[0036] The specific implementation can be: a multi-sequence image difference extraction pyramid module is arranged between each two convolutional neural networks, and the input of each multi-sequence image difference extraction pyramid module is the difference between the two stages of nuclear magnetic images (referring to the difference related to the physical properties of the image itself, such as but not limited to pixels, colors, positions, etc.). Taking the foregoing convolutional neural network (including three convolutional layers) used for extracting image features as an example, the multi-sequence image difference extraction pyramid module in this example is composed of two layers. Specifically, each layer of the multi-sequence image difference extraction pyramid module is down-sampled by the three linear interpolation method, and after each down-sampling, the size of the feature image corresponds to the size of the corresponding convolutional layer (i.e., the image size obtained by the current layer in a pyramid module is consistent with the corresponding layer of the convolutional neural network). Then, the same layer uses a 1x1x1 convolution operation to align the channel number of the feature image with the channel number of the corresponding convolutional layer (i.e., each layer in a pyramid module is subjected to two convolutional layers with a 1x1x1 convolution kernel and a step of 1, to adjust the channel number to be the same as the channel number of the corresponding convolutional layer). Then, the output of the current layer of the multi-sequence image difference extraction pyramid module is added to the output of the corresponding convolutional layer element by element, and then input to the next convolutional layer of the convolutional neural network for feature extraction.

[0037] (3) After that, the image features extracted by the last convolutional neural network (containing the difference information extracted by the pyramid module) are input into the multi-sequence image correlation attention module. The purpose of the multi-sequence image correlation attention module is to learn the correlation information between the input magnetic resonance images of different stages by using the similarity calculation in the image attention mechanism. In the foregoing example, the input of the attention module is three feature maps output by the three convolutional neural networks for feature extraction respectively (one vector as the main feature map of the input, and the other vectors as the secondary feature maps of the input). Among them, the main feature map respectively passes through three 1x1x1 convolution operations and deformation to obtain three matrices: key matrix, value matrix, and query matrix; two secondary feature maps each pass through a 1x1x1 convolution operation and deformation to obtain a query matrix respectively. After that, each query matrix and value matrix can obtain an attention map through the similarity calculation in the attention mechanism, and then the three attention maps are weighted and added after passing through the softmax layer to obtain a total attention map. The transpose of the total attention map is multiplied by the value matrix, and then added to the main feature map to obtain the output of the multi-sequence image correlation attention module.

[0038] For this purpose, in combination with Figure 3 For specific description, the magnetic resonance images of each stage correspond to a multi-sequence image correlation attention module, and the input of each multi-sequence image correlation attention module is the feature map finally output by all convolutional neural networks (in the example, the feature map is input into three multi-sequence image correlation attention modules respectively by the last layer of each convolutional neural network). Among them, the feature map corresponding to the magnetic resonance images of the current stage is the main feature map A of the corresponding multi-sequence image correlation attention module, and the feature maps corresponding to the magnetic resonance images of other stages are the secondary feature maps B, C, which are expressed as follows: A, B, C ∈ R CH ×D×W×H , CH, D, W, H represent the number of channels, depth, width, and height respectively.

[0039] After that, the processing process in the multi-sequence image correlation attention module can be referred to as follows: the main feature map passes through three convolution layers with a convolution kernel of 1x1x1 to obtain Q A , K A , V A ∈ R CH×D×W×H , which represent the query, key, and value in the attention mechanism; two secondary feature maps also pass through two convolution layers with a convolution kernel of 1x1x1 to obtain Q B , Q C ∈ R CH ×D×W×H , which represent the query; then Q A , Q B , Q C , KA , V A transformed into R CH×N , where N=DxWxH is the number of voxels per layer; then, the transpose of each stage Q is multiplied by K A and passed through a softmax layer to obtain three spatial attention maps S A , S B , S C ∈R N ×N where the element of each spatial position is calculated as follows:

[0040]

[0041] where s ij represents the degree to which position i affects position j. That is, the closer the feature vectors of the two positions are, the stronger the correlation between the two positions is, and the greater the proportion of the attention map of the corresponding position is.

[0042] Following the foregoing, the three spatial attention maps are weighted and summed: S=α A S A +α B S B +α C S C , where α A , α B , and α C can be initialized to 1 and gradually adjusted during the training process and obtained through a softmax layer; finally, multiplication is performed between V A and the transpose of S and a preset scaling parameter β, and the main feature map A is added to obtain the final output 0 of the multi-sequence image correlation attention module:

[0043]

[0044] (4) The classification prediction module can be composed of a neural network (such as a classifier constructed by a three-layer neural network). Specifically, the feature maps (containing correlation information) output by each multi-sequence image correlation attention module are converted into feature vectors through a global average pooling layer, and then the feature vectors corresponding to the three stages in the foregoing example can be changed from 128 to 50 through the respective corresponding fully connected layers, and then the three feature vectors are concatenated and changed to 50, 2 through two fully connected layers in turn, and finally the output of the prediction model is obtained through a softmax layer.

[0045] Step S3, training and testing the prediction model using the liver cancer-enhanced magnetic resonance image database.

[0046] Specifically, as described above, the image samples in the database can be divided into a training set and a verification set in a ratio of 7:3, wherein the training set can be enhanced to twice the original data amount by means of random rotation, random translation, random affine transformation, etc.

[0047] With the training set, the three-stage PUMEX MRI (the size can be 60x60x60) of the patient is input into the prediction model, and the training target of the prediction model is to output a two-dimensional vector for representing the classification possibility of the image features of the microvessel infiltration (1 or 0), for example, after being processed by the last softmax layer of the model, the classification result is determined by using a probability threshold. Here, the hyperparameter settings can refer to the following examples: the batch size during model training is 16, the iteration number is 50, and the learning rate is 0.001.

[0048] When the training is completed, the accuracy, precision and recall of the model can be tested by using the aforementioned verification set, and the present application will not be described in detail.

[0049] In summary, the main design concept of the present application is to improve the prediction performance of the constructed prediction model by using the information between the multi-stage liver cancer nuclear magnetic image data. Specifically, a plurality of multi-layer convolutional neural networks are proposed as the image feature extraction part, a multi-sequence image difference extraction pyramid module is introduced in this link, the difference information between the input two images is mined and integrated into the image feature extraction operation, and then the feature map containing the difference information extracted by the convolutional neural network is sent to the multi-sequence image correlation attention module, the correlation between the input nuclear magnetic images of different stages is learned, and then rich reference information is provided for subsequent accurate prediction and classification. The present application can fully utilize the difference information and correlation information between the images of different stages in the enhanced nuclear magnetic image, thereby effectively improving the prediction performance of the model based on image information.

[0050] In the embodiments of the present application, "at least one" means one or more, and "a plurality of" means two or more. "And / or" describes the association relationship of the associated objects, which means that there can be three kinds of relationships, for example, A and / or B, which can represent the cases of A alone, A and B together, and B alone. Wherein A and B can be singular or plural. The character " / " generally represents an "or" relationship between the front and rear associated objects. "At least one of the following" and the like means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b and c can represent: a, b, c, a and b, a and c, b and c, or a and b and c, wherein a, b, and c can be single or multiple.

[0051] The above detailed description of the embodiments shown in the drawings illustrates the structure, features and effects of the present application, but the above is only a preferred embodiment of the present application, and it should be noted that the technical features involved in the above embodiments and preferred modes can be reasonably combined and matched into various equivalent schemes by those skilled in the art without departing from or changing the design idea and technical effects of the present application; therefore, the present application is not limited to the implementation range shown in the drawings, and any changes or modifications made in accordance with the concept of the present application, or equivalent embodiments with equivalent changes, shall be within the scope of protection of the present application.

Claims

1. A method for constructing a prediction model for nuclear magnetic resonance imaging, characterized in that, include: To obtain enhanced MRI images of hepatocellular carcinoma and construct a database of enhanced MRI images of hepatocellular carcinoma; Using MRI images of different stages of hepatocellular carcinoma as input and binary classification results of preset image indicators as output, a prediction model is constructed. The prediction model includes: multiple convolutional neural networks (CNNs) for extracting image features, each corresponding to an MRI image input at a different stage; a multi-sequence image difference extraction pyramid module for acquiring difference information between MRI images at different stages during image feature extraction; a multi-sequence image correlation attention module for acquiring correlation information between MRI images at different stages based on extracted image features; and a classification prediction module for outputting the results. In this module, each layer of the multi-sequence image difference extraction pyramid module is downsampled using trilinear interpolation. After each downsampling, the size of the feature image corresponds to the size of the corresponding convolutional layer. Furthermore, the same layer uses a 1×1×1 convolution operation to align the number of channels in the feature image with the number of channels in the corresponding convolutional layer. The output of the current layer of the multi-sequence image difference extraction pyramid module is added element-wise with the output of the corresponding convolutional layer, and then input into the next convolutional layer of the convolutional neural network for feature extraction. The processing steps of the multi-sequence image correlation attention module include: the main feature map is passed through three convolutional layers with 1×1×1 kernels to obtain the query, key, and value in the attention mechanism; the two secondary feature maps are passed through two convolutional layers with 1×1×1 kernels to obtain the query; the query, key, and value are transformed; then, the transpose of the query and the key at each stage are multiplied and passed through a softmax layer to obtain three spatial attention maps; the three spatial attention maps are then weighted and summed; finally, a multiplication is performed between the value and the transpose of the summation result, multiplied by a preset scaling parameter, and then added to the main feature map to obtain the final output of the multi-sequence image correlation attention module. The prediction model was trained and tested using the hepatocellular carcinoma enhanced MRI image database.

2. The method for constructing a nuclear magnetic resonance imaging prediction model according to claim 1, characterized in that, The construction method includes: setting a multi-sequence image difference extraction pyramid module between each pair of convolutional neural networks, and the input of each multi-sequence image difference extraction pyramid module is the difference between two stages of MRI images; the multi-sequence image difference extraction pyramid module sends the difference information of the two stages of MRI images to the corresponding two convolutional neural networks respectively, and after adding it with the output of the corresponding convolutional layer in the convolutional neural network, it inputs it to the next convolutional layer of the convolutional neural network for subsequent image feature extraction operations.

3. The method for constructing a nuclear magnetic resonance imaging prediction model according to claim 2, characterized in that, The size and number of channels of the feature image output by each layer of the multi-sequence image difference extraction pyramid module are consistent with and aligned with the size and number of channels of the corresponding convolutional layer in the convolutional neural network.

4. The method for constructing a nuclear magnetic resonance imaging prediction model according to claim 1, characterized in that, The input to one of the multi-sequence image correlation attention modules is the feature map output by each of the convolutional neural networks, wherein the feature map input by the convolutional neural network corresponding to the current multi-sequence image correlation attention module is the primary feature map, and the feature maps input by the other convolutional neural networks are secondary feature maps, thereby learning the correlation information between MRI images at different stages of input.

5. The method for constructing a nuclear magnetic resonance imaging prediction model according to claim 4, characterized in that, The processing procedure of the multi-sequence image correlation attention module includes: The key matrix, value matrix, and query matrix are obtained from the primary feature map, and the query matrix is ​​obtained from the secondary feature map. By using the query matrix of each feature map and the key matrix, and combining the similarity calculation of the attention mechanism, the corresponding attention map is obtained; The total attention map is obtained by weighted summation of all attention maps. The output of the multi-sequence image correlation attention module is obtained by combining the total attention map, the value matrix, and the main feature map.

6. The method for constructing a nuclear magnetic resonance imaging prediction model according to any one of claims 1 to 5, characterized in that, The process of constructing the hepatocellular carcinoma enhanced MRI image database includes: preprocessing and segmenting the acquired enhanced MRI image samples, and generating a format adapted to the input of the prediction model based on the segmentation results. The preprocessing includes calibrating and standardizing MRI images at different stages.

7. The method for constructing a nuclear magnetic resonance imaging prediction model according to claim 6, characterized in that, The method of generating a format adapted to the prediction model input based on the segmentation results includes: extracting a cube with a preset scaling size based on the segmentation annotations of MRI images at different stages.

Images