Liver cancer neoadjuvant therapy curative effect prediction method and system based on multi-sequence MRI images

Abstract

The invention discloses a liver cancer neoadjuvant therapy curative effect prediction method and system based on a multi-sequence MRI (Magnetic Resonance Imaging) image. The method comprises the following steps: acquiring multi-sequence MRI images of a target liver before and after treatment; performing cutting and size normalization processing on the multi-sequence MRI image to obtain a preprocessed MRI image; and inputting the preprocessed MRI image into a trained image classification model to obtain a new adjuvant therapy curative effect prediction result. According to the method, the problems of small sample size, unbalanced categories and the like in medical images are solved, and the classification capability of the model on small sample liver cancer MRI image data is effectively improved.

Description

Technical Field

[0001] This invention relates to the field of medical image analysis technology, and more specifically, to a method and system for predicting the efficacy of neoadjuvant therapy for liver cancer based on multi-sequence MRI images. Background Technology

[0002] In recent years, chemotherapy has also been used as a form of "neoadjuvant therapy" to help shrink tumors before surgical removal. The main goal of neoadjuvant therapy for hepatocellular carcinoma is to reduce tumor size, decrease surgical difficulty, and reduce postoperative recurrence, but it carries potential risks such as tumor progression and loss of surgical opportunity.

[0003] Current neoadjuvant therapy suffers from several challenges: some patients lose the opportunity for surgical treatment due to adverse drug reactions, tumor progression, or severe treatment-related adverse events; and the treatment effect is often unclear, delaying initial treatment. Furthermore, existing machine learning methods often rely on model ensembles when dealing with small, imbalanced datasets, resulting in high computational resource consumption or complex hierarchical training, making efficient deployment on edge devices or mobile terminals difficult. Therefore, further improvements are needed to more accurately and efficiently predict the efficacy of neoadjuvant therapy for liver cancer. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method and system for predicting the efficacy of neoadjuvant therapy for liver cancer based on multi-sequence MRI images.

[0005] According to a first aspect of the present invention, a method for predicting the efficacy of neoadjuvant therapy for liver cancer based on multi-sequence MRI images is provided. The method includes the following steps:

[0006] Acquire multi-sequence MRI images of the liver before and after targeted treatment;

[0007] The multi-sequence MRI images are cropped and normalized to obtain preprocessed MRI images;

[0008] The preprocessed MRI images are input into a trained image classification model to obtain prediction results of neoadjuvant therapy efficacy.

[0009] According to a second aspect of the present invention, a system for predicting the efficacy of neoadjuvant therapy for liver cancer based on multi-sequence MRI images is provided. The system includes:

[0010] Image acquisition unit: used to acquire multi-sequence MRI images of the liver before and after the target treatment;

[0011] Preprocessing unit: used to crop and normalize the size of the multi-sequence MRI images to obtain preprocessed MRI images;

[0012] Prediction unit: used to input the preprocessed MRI images into a trained image classification model to obtain prediction results of neoadjuvant therapy efficacy.

[0013] Compared with existing technologies, the advantages of this invention lie in its provision of a multi-sequence MRI image-based neoadjuvant therapy efficacy prediction scheme for liver cancer. This scheme utilizes multi-sequence MRI images before and after treatment to predict the efficacy of neoadjuvant therapy for liver cancer. Given several MRI images of a patient before or during neoadjuvant therapy, and based on the patient's neoadjuvant therapy efficacy data labeled by the physician (e.g., label 0 indicating ineffective treatment and label 1 indicating effective treatment), a time-dimensional feature fusion method is introduced for model training. The model is then tested and validated using data collected from the same hospital. This invention allows for more accurate and automated prediction of the efficacy of neoadjuvant therapy for liver cancer using small-sample training.

[0014] Other features and advantages of the invention will become clear from the following detailed description of exemplary embodiments of the invention with reference to the accompanying drawings. Attached Figure Description

[0015] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments of the invention and, together with their description, serve to explain the principles of the invention.

[0016] Figure 1 This is a flowchart of a method for predicting the efficacy of neoadjuvant therapy for liver cancer based on multi-sequence MRI images according to an embodiment of the present invention;

[0017] Figure 2 This is a schematic diagram of the structure of a multi-sequence MRI image classification model according to an embodiment of the present invention;

[0018] Figure 3 This is a schematic diagram illustrating the process of training and testing a multi-sequence MRI image classification model according to an embodiment of the present invention.

[0019] Figure 4 This is a schematic diagram of the results of a five-fold cross-validation experiment using a comparison method according to an embodiment of the present invention. Detailed Implementation

[0020] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the invention.

[0021] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the invention or its application or use.

[0022] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0023] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0024] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0025] In summary, this invention proposes a deep learning method for classifying multi-sequence MRI images before and after neoadjuvant therapy for hepatocellular carcinoma. Several improvements address challenges in medical image data, such as limited sample size, imbalanced class distribution, and complex image quality, aiming to enhance the model's practicality and accuracy in predicting the efficacy of neoadjuvant therapy. For example, the RegNetY network can be used, with a regularization strategy combining random dropout and dropblock in the backbone to mitigate overfitting. For data augmentation, methods such as expanding the types of transformation operations and enhancing the probability space can be employed to further improve image diversity and model generalization ability. Furthermore, deep convolutional networks can be used to extract features, and pre-treatment and post-treatment features can be fused within the network. These designs significantly improve prediction accuracy.

[0026] Specifically, see Figure 1 As shown, the method for predicting the efficacy of neoadjuvant therapy for liver cancer based on multi-sequence MRI images includes the following steps:

[0027] S1. Acquire and preprocess multiple sequence MRI images of the patient's liver to obtain a sample dataset.

[0028] For example, cropping raw multi-sequence MRI images to highlight the size of the lesion area. First, locate the tumor in the image and mark it using a minimum bounding rectangle. Then, crop the image, preserving a portion of the tumor region and its boundaries, and finally normalize the image size to a predetermined value.

[0029] In one embodiment, the preprocessing of multi-sequence MRI images specifically includes the following steps:

[0030] S11, Acquire multiple sequence liver MRI images of the patient before (start) and after (end) treatment.

[0031] S12, under the guidance of the doctor, marks the tumor area and locates the minimum bounding rectangle.

[0032] S13, crop the image to the rectangular area.

[0033] S15, normalize the size of the cropped image to the same set size, for example, set the normalized size to 224*224.

[0034] Using the above steps, a sample dataset can be obtained for subsequent model training. This sample dataset reflects the location and size of lesions in multi-sequence MRI images before and after treatment.

[0035] Step S2: Construct a multi-sequence MRI image classification model

[0036] Image classification models can be built using various types of deep learning models, such as convolutional neural networks. An image classification model generally consists of a feature extraction network and a classification module. The feature extraction network extracts feature maps from the input images, which include multiple sequences of MRI images before and after treatment. The classification module uses the extracted feature maps to predict and classify the efficacy of neoadjuvant therapy.

[0037] In one embodiment, a DCNN (Deep Convolutional Neural Network) is used as the feature extraction network. After experiments, the RegNetY-1.6G model, which performed best, was selected as the feature extraction backbone. This model has been pre-trained on the ImageNet dataset. Each input is an image pair [I start I end ], where I start Indicates the image before treatment, I end This represents the image after treatment. The feature extraction network contains multiple convolutional layers, activation functions, batch normalization, and a Squeeze-and-Excitation (SE) attention module, ultimately yielding image features.

[0038] See Figure 2 As shown, the constructed feature extraction network contains multiple convolutional layers and a regularized DropBlock module to alleviate overfitting. Specifically, the feature extraction network extracts input image features according to the following steps:

[0039] S21, for a given image (which can be a pre-treatment image or a post-treatment image), find the corresponding image to form an image pair [I start I end The processed image pair is obtained after image enhancement and cropping.

[0040] S22, the processed image pairs are passed through an improved deep convolutional neural network (DCNN), which consists of an input layer (Stem Layer), four stages, and two DropBlock modules. Each stage contains multiple convolutional modules, and the two DropBlock modules are placed after the third and fourth stages, respectively, to alleviate overfitting. The image pairs enter the network from the input layer, and after passing through the first three stages and the first DropBlock module, feature pairs [F] are obtained. start F end ].

[0041] S23, feature pair [F start F end Before entering the fourth stage, the images are fused to obtain the fused image features F, which is represented as:

[0042] F = F start ⊙F end (1)

[0043] Where F represents the fused image features, F start F represents the features of the pre-treatment image. end Represents the features of the post-treatment image, and ⊙ represents the element-wise multiplication of corresponding matrix positions. After fusion, the feature pairs [F] are used to represent the features. start F end The image features are transformed into fused image features F, which are then processed through the fourth stage and the second DropBlock module to obtain the final multi-sequence image features.

[0044] Furthermore, the obtained multi-sequence image features are used as input, passed through a global average pooling layer and a DropOut module, and finally through a fully connected layer to obtain the classification result. For example, if the classification result is that the treatment is effective, the output is 1; if the classification result is that the treatment is ineffective, the output is 0. The global average pooling layer, the DropOut module, and the fully connected layer can be regarded as the classification module in the image classification model.

[0045] In summary, the constructed image classification model consists of two stages. In the first stage, the model extracts image features through depthwise convolution, uses the DropBlock module to alleviate overfitting, and multiplies the resulting feature pairs element-wise to fuse the features of images before and after treatment. The final fused feature map F has a size of 888*7*7. In the second stage, the obtained feature map F is pooled in the spatial dimension to obtain a feature vector P, which is then regularized using the DropOut module, and finally, a fully connected layer is used to obtain the classification result.

[0046] Step S3: Train the image classification model using the sample dataset until the set loss function criteria are met.

[0047] For example, an image classification model is trained with the optimization objective of minimizing a predetermined overall loss function. In one embodiment, a multi-weighted loss function is designed to supervise the training of the image classification model. The final loss function is expressed as:

[0048]

[0049] in:

[0050]

[0051] Among them, C y These are coefficients for the weights of category y, N y α is the number of samples of category y, α is a hyperparameter, C represents the number of categories, and Loss is... i Let G represent the loss for the i-th category. T is the hyperparameter threshold that limits the outlier loss; experimental results show that T = 0.1 yields the best results. * =(1-T) r log(T) is a constant determined by T. The sign-adjusted class prediction probability is set as follows:

[0052]

[0053] Where t is the category sign adjustment marker, used to distinguish between true and non-true categories. i This represents the predicted value for category i, when category i is the true category. Take z i Otherwise, take -z i .

[0054] S4. For the actual acquired multi-sequence MRI images, the trained image classification model is used to obtain the prediction results of neoadjuvant therapy efficacy.

[0055] After the image classification model is trained, optimized parameters such as weights and biases can be obtained. To evaluate the performance of the image classification model, commonly used classification metrics such as accuracy, precision, recall, F1 score, sensitivity, and specificity, as well as commonly used medical evaluation metrics, can be used to evaluate and compare the obtained hierarchical classification models. Higher values ​​for these metrics generally indicate better model performance.

[0056] For actual acquired multi-sequence MRI images, the image classification model, which has been trained and tested, can be used to obtain prediction results of neoadjuvant therapy efficacy.

[0057] Accordingly, the present invention also provides a system for predicting the efficacy of neoadjuvant therapy for liver cancer based on multi-sequence MRI images, used to implement one or more aspects of the above-mentioned method. For example, the system includes: an image acquisition unit for acquiring multi-sequence MRI images of the liver before and after treatment; a preprocessing unit for cropping and size normalizing the multi-sequence MRI images to obtain preprocessed MRI images; and a prediction unit for inputting the preprocessed MRI images into a trained image classification model to obtain a prediction result for the efficacy of neoadjuvant therapy. Each unit can employ a general-purpose processor, a dedicated processor, an FPGA, etc., to implement its corresponding function.

[0058] To further verify the effectiveness of this invention, image classification model training and experimental verification were conducted. Taking the actual collected dataset as an example, a total of 6695 images were obtained, including MRI images of patients before and after treatment in six modalities. Only the T1V modality images were actually used, totaling 1210 images. During the experiment, the training and test sets were divided in a 4:1 ratio, and the experimental results presented are all from five-fold cross-validation. All experiments can use PyTorch or other tools as algorithm building tools. The experiments were conducted using a 4080 graphics card, and images during training were recorded using TensorBoard. The training batch size was set to 32, using the Adam optimizer with an initial learning rate of 1e-4 and an optimizer weight-decay of 1e-4. The learning rate update strategy used was multistep. See Table 1 for comparative experimental results. Figure 4 As shown, the baseline method uses only the results of the experiment based on images after treatment. Figure 4 In the diagram, the left graph represents accuracy, and the right graph represents the F1 score. The blue line represents the baseline, and the orange line represents the method proposed in this invention. The experimentally obtained accuracy was 0.8540, precision was 0.7435, recall was 0.6553, and F1 score was 0.6817. Furthermore, validation was performed on the baseline method, which showed an accuracy of 0.8401 and an F1 score of 0.5990, nearly 10% lower than the proposed method. Comparative validation demonstrates that this invention can improve the classification performance of MRI images.

[0059] Table 1: Comparative experimental results.

[0060] method accuracy Accuracy Recall rate F1 Baseline method 0.8401 0.6679 0.5680 0.5990 This invention 0.8540 0.7435 0.6553 0.6817

[0061] In summary, existing methods often rely on model ensembles when dealing with small, imbalanced datasets, resulting in high computational resource consumption or complex hierarchical training, making efficient deployment on edge devices or mobile terminals difficult. However, the neoadjuvant therapy efficacy prediction method for liver cancer based on multi-sequence MRI images designed in this invention solves the problems of small sample size and class imbalance in medical images. This invention improves the single-model deep learning strategy used in skin lesion classification tasks, effectively enhancing the model's classification ability on small-sample liver cancer MRI image data through structure optimization, data augmentation, and feature fusion.

[0062] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0063] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0064] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0065] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0066] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0067] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It will be known to those skilled in the art that implementation in hardware, implementation in software, and implementation using a combination of software and hardware are equivalent.

[0068] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of the invention is defined by the appended claims.

Claims

1. A method for predicting the efficacy of neoadjuvant therapy for liver cancer based on multi-sequence MRI images, comprising the following steps: Acquire multi-sequence MRI images of the liver before and after targeted treatment; The multi-sequence MRI images are cropped and normalized to obtain preprocessed MRI images; The preprocessed MRI images are input into a trained image classification model to obtain prediction results of neoadjuvant therapy efficacy.

2. The method according to claim 1, characterized in that, The image classification model is trained according to the following steps: Multiple sequence MRI images of the patient's liver were acquired and preprocessed to obtain the patient's preprocessed MRI images; A sample dataset was constructed using the preprocessed MRI images; The image classification model is trained using the sample dataset with the goal of minimizing the set overall loss function.

3. The method according to claim 2, characterized in that, The patient's pre-processed MRI images were obtained according to the following steps: Acquired multi-sequence liver MRI images of the patient before and after treatment; The tumor region is labeled, and the minimum bounding rectangle is located; Cropping is performed on the located rectangular region; Normalize the size of the cropped image to the same set size.

4. The method according to claim 1, characterized in that, The image classification model includes a feature extraction network, a pooling layer, a regularization layer, and a fully connected layer. The feature extraction network is used to extract feature maps from the input multi-sequence MRI images. The pooling layer pools the feature maps in the spatial dimension to obtain feature vectors. Then, the regularization layer performs regularization processing, and the fully connected layer obtains the treatment efficacy prediction results.

5. The method according to claim 4, characterized in that, The feature extraction network extracts feature maps according to the following steps: For the image pairs composed of the set images [I] start I end After image enhancement and cropping, the processed image pair is obtained, where I start This image shows the image before treatment. end Images showing the results of treatment; The processed image pairs are passed through a deep convolutional neural network, which includes an input layer, four stages, and two block dropout modules. Each stage contains multiple convolutional modules, and the two block dropout modules are respectively placed after the third and fourth stages. The processed image pairs enter from the input layer, and after passing through the first three stages and the first block dropout module, feature pairs [F] are obtained. start F end ]; The feature pair [F start F end] Before proceeding to the fourth stage, the images are fused to obtain the fused image features F, denoted as: F=F start ⊙F end Among them, F start F represents the features of the pre-treatment image. end Features representing the image after treatment; The fused image features F are then processed through the fourth stage and the second block discarding module to obtain a feature map.

6. The method according to claim 2, characterized in that, The overall loss function is set as follows: in: Among them, L total It is the total loss value, C y These are the weighting coefficients for category y, N y The number of samples in category y, Loss i Let G represent the loss for the i-th category, α be a hyperparameter, T be the hyperparameter threshold that limits the loss for outliers, and G be the loss for the i-th category. * =(1-T) r log(T) is a constant determined by T, z i represents the predicted value for category i, and t is the category sign adjustment flag when category i is the true category. Take z i Otherwise, take -z i .

7. The method according to claim 6, characterized in that, Set the hyperparameter threshold T for limiting outlier loss to 0.

1.

8. A system for predicting the efficacy of neoadjuvant therapy for liver cancer based on multi-sequence MRI images, comprising: Image acquisition unit: used to acquire multi-sequence MRI images of the liver before and after the target treatment; Preprocessing unit: used to crop and normalize the size of the multi-sequence MRI images to obtain preprocessed MRI images; Prediction unit: used to input the preprocessed MRI images into a trained image classification model to obtain prediction results of neoadjuvant therapy efficacy.

9. A computer-readable storage medium having a computer program stored thereon, wherein, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer device comprising a memory and a processor, wherein a computer program capable of running on the processor is stored in the memory, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

Images