Colorectal cancer chemoradiotherapy feedback curative effect evaluation method based on mixed feature library

Abstract

The invention provides a colorectal cancer chemoradiotherapy feedback curative effect evaluation method based on a mixed feature library. Firstly, an MRI image of a patient after new adjuvant therapy is input into a segmentation module, and a tumor lesion area mask is extracted; secondly, according to the tumor lesion area mask, lesion areas and clinical information are input into a feature library construction module, and depth features, radiomics features and clinical features are extracted to construct patient multi-modal mixed features; and further inputting the multi-modal mixed features of the patient into a treatment evaluation module to obtain a neoadjuvant chemotherapy stage staging result of the patient. And finally, for data from a special individual, the doctor inputs the modified mask and score into the man-machine feedback module according to the mask of the tumor lesion area and the score of the diagnosis result, and updates the encoder weight of the segmentation module. The device is effectively suitable for different parts, diseases and special patients. According to the method, a targeted deep learning framework and comprehensive patient characteristics are utilized, and the result of neoadjuvant chemoradiotherapy can be accurately and effectively evaluated. And targeted improvement can be provided after man-machine feedback so as to adapt to special individuals and images.

Description

Technical Field

[0001] The present invention relates to the field of medical image processing, and in particular to a method for evaluating the feedback efficacy of radiotherapy and chemotherapy for colorectal cancer based on a mixed feature library. Background Art

[0002] Colorectal cancer (CRC) is the third most common cancer in the world, accounting for about 10% of all cancer cases and the second leading cause of cancer death worldwide. Among them, 60% to 70% of patients with rectal cancer are locally advanced rectal cancer (LARC) patients, which are characterized by large differences between individual patients, including tumor invasion depth, lymph node metastasis and risk of recurrence. Currently, neoadjuvant chemoradiotherapy (nCRT) followed by total mesorectal excision has become the standard treatment for LARC, which can significantly reduce tumor volume and reduce stage. In addition, some studies have found that the prognosis of patients who respond well to nCRT and receive close observation is not significantly different from that of patients who receive radical surgery, but surgery-related complications can be avoided and the quality of life of patients can be improved. Therefore, the treatment of LARC patients has become more personalized, and identifying predictors of nCRT efficacy is crucial for personalized treatment, which can help patients avoid harm caused by overtreatment.

[0003] Recent studies have shown that combining artificial intelligence technology with MRI to develop prognostic image biomarker models has achieved results in predicting risk factors such as extramural venous invasion and chemoradiotherapy response. However, the prediction of rectal cancer relies solely on MRI images, without integrating clinical pathology and molecular biology information, and cannot fully reflect and predict the biological behavior and development trend of the tumor. Due to the lack of a large amount of data interpretation, deep learning also requires a large amount of data for training and fitting. Therefore, the previously proposed model did not achieve the best results, so the prediction is more likely to be biased due to different data distribution.

[0004] To overcome the above problems, the present invention proposes a method for evaluating the efficacy of colorectal cancer chemoradiotherapy feedback based on a hybrid feature library. This method combines leading deep learning methods, integrates features from multiple dimensions such as deep learning, clinical features, and radiomics, has a more powerful and comprehensive feature representation capability, and provides accurate analysis of chemoradiotherapy results. At the same time, a method of human-computer feedback is provided for individuals and images with large differences, and undesirable results can be modified to affect the model. This method can effectively evaluate the results of neoadjuvant chemoradiotherapy, and provide targeted improvements after human-computer feedback to adapt to special situations. Summary of the invention

[0005] The purpose of this application is to propose a method for evaluating the efficacy of colorectal cancer chemoradiotherapy feedback based on a hybrid feature library. This method is used to overcome the problems of low accuracy and huge differences between individuals in the existing technology. This improves the recognition accuracy, reliability, and generalization ability in the evaluation process of neoadjuvant chemoradiotherapy.

[0006] To achieve the above purpose, the technical solution of this application is as follows:

[0007] A method for evaluating the effect of colorectal cancer chemoradiotherapy feedback based on a hybrid feature library, characterized in that the method for evaluating the effect of neoadjuvant chemotherapy human-machine feedback based on a hybrid feature library comprises:

[0008] MRI images of patients after neoadjuvant therapy input Input segmentation module M seg , extract the tumor lesion area mask Mask tumor .

[0009] According to the tumor lesion area mask tumor The lesion area I tumor , Clinical Information clinical Input feature library construction module M feature , extract deep features F deep , radiomics feature F ra , Clinical features F clinical Construct patient multimodal mixed features F mix .

[0010] The patient's multimodal mixed features F mix Enter the Treatment Assessment Module M evaluate , get the patient's neoadjuvant chemotherapy stage staging results R stage .

[0011] For data from a special individual, the doctor uses the mask of the tumor lesion area tumor The diagnosis result F stage Score, the modified mask Mask revise The score S is input into the human-machine feedback module M feedback , Update the encoder weights of the segmentation module. Effectively adapt to different parts, diseases, and special patients.

[0012] Furthermore, the method for evaluating the effect of colorectal cancer chemoradiotherapy feedback based on a hybrid feature library is characterized in that the segmentation module M seg ,include:

[0013] Step 1.1: Take the patient's MRI image I input Input the pre-trained ResNet-34 in ImageNet and encode it to obtain the image encoding feature F encode .

[0014] Step 1.2: Encode the image features F encode Input the self-attention layer to calculate the self-attention weight and add it to the original image feature to get the attention feature F attention .

[0015] Step 1.3: Input the feature into the feedforward network layer, expand the channel dimension by four times, calculate the depth convolution and return to the original channel dimension to obtain the feature F ffn .

[0016] Step 1.4: Set feature F ffn Input pyramid pooling layer, use different sized pooling kernels to obtain different receptive field features and then merge them to get pooling feature F pool .

[0017] Step 1.5: Input the features into the feature decoder and return the tumor lesion area mask of the original image size tumor .

[0018] Furthermore, the segmentation module is characterized in that the pyramid pooling layer comprises:

[0019] Step 2.1: Set feature F ffn Use the pooling calculation with a pooling kernel of 2 and a step size of 2, and the convolution calculation with a convolution kernel of 1, and then use bilinear interpolation to restore the original dimension size to obtain the pooling feature F pool1 .

[0020] Step 2.2: Feature F ffn Use the pooling calculation with a pooling kernel of 3 and a step size of 3, and the convolution calculation with a convolution kernel of 1, and then use bilinear interpolation to restore the original dimension size to obtain the pooling feature F pool2 .

[0021] Step 2.3: Feature F ffn Use the pooling calculation with a pooling kernel of 5 and a step size of 5, and the convolution calculation with a convolution kernel of 1, and then use bilinear interpolation to restore the original dimension size to obtain the pooling feature F pool3 .

[0022] Step 2.4: Feature F ffn Use the pooling calculation with a pooling kernel of 6 and a step size of 6, and the convolution calculation with a convolution kernel of 1, and then use bilinear interpolation to restore the original dimension size to obtain the pooling feature F pool4 .

[0023] Step 2.5: Pooling feature F pool1 , F pool2 , F pool3 , F pool4Merge along the channel dimension to obtain the pooled feature F pool .

[0024] Furthermore, the method for evaluating the effect of colorectal cancer chemoradiotherapy feedback based on a hybrid feature library is characterized in that the feature library module M feature ,include:

[0025] Step 3.1: Mask the lesion area tumor The lesion area I tumor After inputting the lesion feature encoder, the deep feature F is returned deep , where the lesion feature decoder and the segmentation module M seg The feature encoder is also ResNet-34.

[0026] Step 3.2: Mask the lesion area tumor The lesion area I tumor Input the omics feature extraction module to extract the imaging omics feature F ra .

[0027] Step 3.3: Clinical information clinical According to the categories of different information, clinical characteristics F are obtained by coding clinical .

[0028] Furthermore, the feature library module M feature , characterized in that the omics feature extraction module comprises:

[0029] Step 4.1: From the lesion area I tumor Extract shape feature F S .

[0030] Step 4.2: From the lesion area I tumor Extract texture features F T .

[0031] Step 4.3: From the lesion area I tumor Extract intensity feature F I .

[0032] Step 4.4: Shape feature F S , texture feature F T , intensity characteristic F I Combined to obtain radiomics feature F ra .

[0033] Furthermore, the colorectal cancer chemoradiotherapy feedback efficacy evaluation method based on a mixed feature library is characterized in that the treatment evaluation module M evaluate ,include:

[0034] Step 5.1: Multimodal mixed feature F mix Input convolution kernel 3 to the convolution layer operation, and get feature F conv .

[0035] Step 5.2: Feature F conv The input pooling kernel is 2, and the maximum pooling layer with a step size of 2 is calculated to obtain the feature F gp .

[0036] Step 5.3: Set feature F gp Input fully connected layer, output feature F cls , the number of channels is consistent with the number of installments.

[0037] Step 5.4: After the output features are normalized by softmax, the patient's neoadjuvant chemotherapy stage staging result R is obtained according to each class probability stage .

[0038] Furthermore, the method for evaluating the effect of colorectal cancer radiotherapy and chemotherapy based on a hybrid feature library is characterized in that the human-machine feedback module M feedback ,include:

[0039] Step 6.1: The doctor masks the lesion area tumor And the staging results R stage Get a score of S.

[0040] Step 6.2: Doctors segment the mask tumor Make manual modifications and update the mask to Mask′ tumor .

[0041] Step 6.3: According to the updated mask Mask′ tumor Mask extracted by the segmentation module tumor Calculate Loss dice .

[0042] Step 6.4: Update the low-rank learning module M based on the loss back propagation lora Weight parameter.

[0043] Compared with the prior art, this method for evaluating the efficacy of colorectal cancer chemoradiotherapy feedback based on a hybrid feature library inputs the MRI images of patients after neoadjuvant treatment into the segmentation module to extract the tumor lesion area mask. According to the tumor lesion area mask, the lesion area and clinical information are input into the feature library construction module to extract deep features, imaging genomics features, and clinical features to construct the patient's multimodal mixed features. The patient's multimodal mixed features are input into the treatment evaluation module to obtain the patient's neoadjuvant chemotherapy stage staging results. For data from special individuals, doctors score the tumor lesion area mask and the diagnosis results, input the modified mask and score into the human-computer feedback module, and update the segmentation module encoder weights. Effectively adapt to different parts, diseases, and special patients. This method can accurately and effectively evaluate the results of neoadjuvant chemoradiotherapy, and provide targeted improvements after human-computer feedback to adapt to special situations. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a flow chart of a method for evaluating the feedback efficacy of chemoradiotherapy for colorectal cancer based on a hybrid feature library;

[0045] Figure 2 This is a method architecture diagram of a method for evaluating the feedback efficacy of radiotherapy and chemotherapy for colorectal cancer based on a mixed feature library;

[0046] Figure 3 This is a schematic diagram of the pyramid pooling layer in the segmentation module of a method for evaluating the feedback effect of radiotherapy and chemotherapy for colorectal cancer based on a mixed feature library; DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0048] In one embodiment, Figure 1 As shown in the figure, a method for evaluating the effect of radiotherapy and chemotherapy feedback for colorectal cancer based on a hybrid feature library is provided. The architecture diagram of each module is shown in Figure 2 As shown, including:

[0049] Step S1: MRI images of patients after neoadjuvant therapy input Input segmentation module M seg , extract the tumor lesion area mask Mask tumor .

[0050] Specifically, the segmentation module M seg ,include:

[0051] Step 1.1: Take the patient's MRI image I inputInput the pre-trained ResNet-34 in ImageNet and encode it to obtain the image encoding feature F encode .

[0052] Step 1.2: Encode the image features F encode Input the self-attention layer to calculate the self-attention weight and add it to the original image feature to get the attention feature F attention .

[0053] Specifically, the self-attention calculation can be expressed by the formula:

[0054]

[0055] Among them, QKV comes from the image coding feature F encode .

[0056] Step 1.3: Input the feature into the feedforward network layer, expand the channel dimension by four times, calculate the depth convolution and return to the original channel dimension to obtain the feature F ffn .

[0057] Step 1.4: Set feature F ffn Input pyramid pooling layer, use different sized pooling kernels to obtain different receptive field features and then merge them to get pooling feature F popl .

[0058] Step 1.5: Input the features into the feature decoder and return the tumor lesion area mask of the original image size tumor .

[0059] Specifically, the segmentation module is characterized in that the pyramid pooling layer is as follows: Figure 3 As shown, including:

[0060] Step 2.1: Set feature F ffn Use the pooling calculation with a pooling kernel of 2 and a step size of 2, and the convolution calculation with a convolution kernel of 1, and then use bilinear interpolation to restore the original dimension size to obtain the pooling feature F pool1 .

[0061] Step 2.2: Feature F ffn Use the pooling calculation with a pooling kernel of 3 and a step size of 3, and the convolution calculation with a convolution kernel of 1, and then use bilinear interpolation to restore the original dimension size to obtain the pooling feature F pool2 .

[0062] Step 2.3: Feature F ffn Use the pooling calculation with a pooling kernel of 5 and a step size of 5, and the convolution calculation with a convolution kernel of 1, and then use bilinear interpolation to restore the original dimension size to obtain the pooling feature F pool3 .

[0063] Step 2.4: Feature F ffn Use the pooling calculation with a pooling kernel of 6 and a step size of 6, and the convolution calculation with a convolution kernel of 1, and then use bilinear interpolation to restore the original dimension size to obtain the pooling feature F pool4 .

[0064] Step 2.5: Pooling feature F pool1 , F pool2 , F pool3 , F pool4 Merge along the channel dimension to obtain the pooled feature F pool .

[0065] Step S2: Mask the tumor lesion area tumor The lesion area I tumor , Clinical Information clinical Input feature library construction module M feature , extract deep features F dgep , radiomics feature F ra , Clinical features F clinical Construct patient multimodal mixed features F mix .

[0066] Specifically, the feature library module M feature ,include:

[0067] Step 3.1: Mask the lesion area tumor The lesion area I tumor After inputting the lesion feature encoder, the deep feature F is returned deep , where the lesion feature decoder and the segmentation module M seg The feature encoder is also ResNet-34.

[0068] Step 3.2: Mask the lesion area tumor The lesion area I tumor Input the omics feature extraction module to extract the imaging omics feature F ra .

[0069] Step 3.3: Clinical information clinical According to the categories of different information, clinical characteristics F are obtained by coding clinical .

[0070] Specifically, the feature library module M feature , characterized in that the omics feature extraction module comprises:

[0071] Step 4.1: From the lesion area I tumor Extract shape feature F S .

[0072] Step 4.2: From the lesion area I tumor Extract texture features F T .

[0073] Step 4.3: From the lesion area I tumor Extract intensity feature F I .

[0074] Step 4.4: Shape feature F S , texture feature F T , intensity characteristic F I Combined to obtain radiomics feature F ra .

[0075] Specifically, the shape features include area features, perimeter features, aspect ratio features, compactness features, and circular features; the texture features include grayscale contrast difference features, energy features, uniformity features, randomness features, length dependence features, grayscale non-uniformity features, low-frequency texture features, and high-frequency texture features; the intensity features include: mean, maximum, minimum, standard deviation, median, etc. of grayscale values.

[0076] Step S3: The patient's multimodal mixed feature F mix Enter the Treatment Assessment Module M evaluate , get the patient's neoadjuvant chemotherapy stage staging results R stage .

[0077] Specifically, the treatment evaluation module M evaluate ,include:

[0078] Step 5.1: Multimodal mixed feature F mix Input convolution kernel 3 to the convolution layer operation, and get feature F conv .

[0079] Step 5.2: Feature F conv The input pooling kernel is 2, and the maximum pooling layer with a step size of 2 is calculated to obtain the feature F gp .

[0080] Step 5.3: Set feature F gp Input fully connected layer, output feature F cls , the number of channels is consistent with the number of installments.

[0081] Step 5.4: After the output features are normalized by softmax, the patient's neoadjuvant chemotherapy stage staging result R is obtained according to each class probability stage .

[0082] Step S4: For data from a special individual, the doctor uses the mask of the tumor lesion areatumor The diagnosis result F stage Score, the modified mask Mask revise The score S is input into the human-machine feedback module M feedback , Update the encoder weights of the segmentation module. Effectively adapt to different parts, diseases, and special patients.

[0083] Specifically, the rating is divided into 5 points:

[0084] 0 points means no correlation with the actual tumor area, 1 point means the position of the segmented area is too different, 2 points means the position of the segmented area is roughly correct but the shape is completely different, 3 points means the position of the segmented area is basically correct but the shape has some error, 4 points means the position of the segmented area is correct but the shape has some error which is acceptable, and 5 points means the position and shape of the segmented area are basically consistent and the classification is correct.

[0085] Specifically, the human-machine feedback module M feedback ,include:

[0086] Step 6.1: The doctor masks the lesion area tumor And the staging results R stage Get a score of S.

[0087] Step 6.2: Doctors segment the mask tumor Make manual modifications and update the mask to Mask′ tumor .

[0088] Step 6.3: According to the updated mask Mask′ tumor Mask extracted by the segmentation module tumor Calculate Loss dice .

[0089] Step 6.4: Update the low-rank learning module M based on the loss back propagation lora Weight parameter.

[0090] The descriptions of several implementation methods of the application are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the application, which all belong to the protection scope of the application. Therefore, the protection scope of the patent of this application shall be based on the attached claims.

Claims

1. A method for evaluating the feedback effect of radiotherapy and chemotherapy for colorectal cancer based on a mixed feature library, characterized in that: The human-computer feedback efficacy evaluation method for neoadjuvant chemotherapy based on a hybrid feature library includes: MRI images of patients after neoadjuvant therapy input Input segmentation module M seg , extract the tumor lesion area mask Mask tumor ; According to the tumor lesion area mask tumor The lesion area I tumor , Clinical Information clinical Input feature library construction module M feature , extract deep features F deep , radiomics feature F ra , Clinical features F clinical Construct patient multimodal mixed features F mix ; The patient's multimodal mixed features F mix Enter the Treatment Assessment Module M evaluate , get the patient's neoadjuvant chemotherapy stage staging results R stage ; For data from a special individual, the doctor uses the mask of the tumor lesion area tumor The diagnosis result F stage Score, the modified mask Mask revise The score S is input into the human-machine feedback module M feedback , Update the encoder weights of the segmentation module. Effectively adapt to different parts, diseases, and special patients.

2. A method for evaluating the feedback efficacy of colorectal cancer chemoradiotherapy based on a mixed feature library as claimed in claim 1, characterized in that: The segmentation module M seg ,include: Step 1.1: Take the patient's MRI image I input Input the pre-trained ResNet-34 in ImageNet and encode it to obtain the image encoding feature F encode ; Step 1.2: Encode the image features F encode Input the self-attention layer to calculate the self-attention weight and add it to the original image feature to get the attention feature F attention ; Step 1.3: Input the feature into the feedforward network layer, expand the channel dimension by four times, calculate the depth convolution and return to the original channel dimension to obtain the feature F ffn ; Step 1.4: Set feature F ffn Input pyramid pooling layer, use different sized pooling kernels to obtain different receptive field features and then merge them to get pooling feature F pool ; Step 1.5: Input the features into the feature decoder and return the tumor lesion area mask of the original image size tumor .

3. The segmentation module according to claim 2, characterized in that: The pyramid pooling layer comprises: Step 2.1: Set feature F ffn Use the pooling calculation with a pooling kernel of 2 and a step size of 2, and the convolution calculation with a convolution kernel of 1, and then use bilinear interpolation to restore the original dimension size to obtain the pooling feature F pool1 ; Step 2.2: Feature F ffn Use the pooling calculation with a pooling kernel of 3 and a step size of 3, and the convolution calculation with a convolution kernel of 1, and then use bilinear interpolation to restore the original dimension size to obtain the pooling feature F pool2 ; Step 2.3: Feature F ffn Use the pooling calculation with a pooling kernel of 5 and a step size of 5, and the convolution calculation with a convolution kernel of 1, and then use bilinear interpolation to restore the original dimension size to obtain the pooling feature F pool3 ; Step 2.4: Feature F ffn Use the pooling calculation with a pooling kernel of 6 and a step size of 6, and the convolution calculation with a convolution kernel of 1, and then use bilinear interpolation to restore the original dimension size to obtain the pooling feature F pool4 ; Step 2.5: Pooling feature F pool1 , F pool2 , F pool3 , F pool4 Merge along the channel dimension to obtain the pooled feature F pool .

4. A method for evaluating the feedback efficacy of colorectal cancer chemoradiotherapy based on a mixed feature library as claimed in claim 1, characterized in that: The feature library module M feature ,include: Step 3.1: Mask the lesion area tumor The lesion area I tumor After inputting the lesion feature encoder, the deep feature F is returned deep , where the lesion feature decoder and the segmentation module M seg The feature encoder is the same as ResNet-34; Step 3.2: Mask the lesion area tumor The lesion area I tumor Input the omics feature extraction module to extract the imaging omics feature F ra ; Step 3.3: Clinical information clinical According to the categories of different information, clinical characteristics F are obtained by coding clinical .

5. The feature library module M according to claim 4 feature , characterized in that, The omics feature extraction module comprises: Step 4.1: From the lesion area I tumor Extract shape feature F S ; Step 4.2: From the lesion area I tumor Extract texture features F T ; Step 4.3: From the lesion area I tumor Extract intensity feature F I ; Step 4.4: Shape feature F S , texture feature F T , intensity characteristic F I Combined to obtain radiomics feature F ra .

6. A method for evaluating the feedback efficacy of radiotherapy and chemotherapy for colorectal cancer based on a mixed feature library as claimed in claim 1, characterized in that: The treatment assessment module M evaluate ,include: Step 5.1: Multimodal mixed feature F mix Input convolution kernel 3 to the convolution layer operation, and get feature F conv ; Step 5.2: Feature F conv The input pooling kernel is 2, and the maximum pooling layer with a step size of 2 is calculated to obtain the feature F gp ; Step 5.3: Set feature F gp Input fully connected layer, output feature F cls , the number of channels is consistent with the number of installments; Step 5.4: After the output features are normalized by softmax, the patient's neoadjuvant chemotherapy stage staging result R is obtained according to each class probability stage .

7. A method for evaluating the feedback efficacy of colorectal cancer chemoradiotherapy based on a mixed feature library as claimed in claim 1, characterized in that: The human-machine feedback module M feedback ,include: Step 6.1: The doctor masks the lesion area tumor And the staging results R stage Get a score S; Step 6.2: Doctors segment the mask tumor Make manual changes and update the mask to Mask' tumor ; Step 6.3: According to the updated mask Mask′ tumor Mask extracted by the segmentation module tumor Calculate Loss dice ; Step 6.4: Update the low-rank learning module M based on the loss back propagation lora Weight parameter.

Images