A Transformer-based PCR prediction method and device for neoadjuvant chemotherapy of breast cancer

Abstract

The present invention relates to the field of medical image analysis, and in particular to a Transformer-based method and device for predicting pCR of neoadjuvant chemotherapy for breast cancer, the method comprising: obtaining image data of breast cancer patients before neoadjuvant chemotherapy; inputting the image data into a constructed breast cancer tumor segmentation model to obtain a mask image of the tumor lesion area; obtaining a region of interest image according to the mask image, preprocessing the region of interest image to obtain preprocessed image data; obtaining a pathological complete remission pCR prediction result of neoadjuvant chemotherapy for breast cancer patients according to the preprocessed image data and a constructed Transformer deep learning image classification model. The present invention provides a new pCR prediction deep learning model, which can make full use of breast cancer information in DCE-MRI sequence images before treatment and improve the non-invasive prediction performance of pCR.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image analysis, and in particular to a Transformer-based breast cancer neoadjuvant chemotherapy PCR prediction method and device. Background Art

[0002] Neoadjuvant chemotherapy plays an important role in the current comprehensive treatment of breast cancer. Its goals include downstaging surgery, breast-conserving therapy, treatment of potential metastatic or micrometastatic lesions, and providing precise guidance for subsequent treatment. Achieving pCR (pathological Complete Response) with neoadjuvant chemotherapy for breast cancer can significantly improve the patient's prognosis and is an important indicator for evaluating the efficacy of neoadjuvant chemotherapy. However, less than 50% of breast cancer patients can achieve pCR after neoadjuvant chemotherapy. Therefore, accurately predicting the pCR of neoadjuvant chemotherapy for breast cancer patients is of great clinical significance for formulating individualized treatment plans and improving the patient's prognosis.

[0003] Postoperative pathological evaluation is still the gold standard for accurately evaluating the efficacy of neoadjuvant chemotherapy for breast cancer, but it has a significant time lag and is invasive. In comparison, imaging evaluation has a time advantage and is non-invasive. Previous studies have shown that DCE-MRI (Dynamic Contrast Enhanced-Magnetic Resonance Imaging) can more comprehensively judge and predict the efficacy of neoadjuvant chemotherapy compared to CT (Computed Tomography) and ultrasound imaging.

[0004] At present, based on DCE-MRI images, the existing breast cancer neoadjuvant chemotherapy pCR prediction technology mainly uses traditional imaging omics methods and deep learning models based on convolutional neural networks for prediction. Among them, research reports point out that the prediction performance of traditional imaging omics methods is not as good as that of deep learning models based on convolutional neural networks. In addition, the existing deep learning models based on convolutional neural networks fail to effectively utilize the global information of images and the spatial position relationship between information, which limits the prediction performance of the model. Summary of the invention

[0005] In order to solve the technical problem that the existing deep learning model based on convolutional neural network in the prior art fails to effectively utilize the global information of the image and the spatial position relationship between the information, which leads to the limitation of the prediction performance of the model, the embodiment of the present invention provides a Transformer-based breast cancer neoadjuvant chemotherapy PCR prediction method and device. The technical solution is as follows:

[0006] On the one hand, a Transformer-based breast cancer neoadjuvant chemotherapy PCR prediction method is provided, the method is implemented by a breast cancer neoadjuvant chemotherapy PCR prediction device, and the method comprises:

[0007] S1. Obtain image data of breast cancer patients before neoadjuvant chemotherapy.

[0008] S2. Input the image data into the constructed breast cancer tumor segmentation model to obtain a mask image of the tumor lesion area.

[0009] S3. Obtain an image of the region of interest according to the mask image, and preprocess the image of the region of interest to obtain preprocessed image data.

[0010] S4. Based on the preprocessed image data and the constructed Transformer deep learning image classification model, the pathological complete remission (pCR) prediction results of neoadjuvant chemotherapy for breast cancer patients were obtained.

[0011] Optionally, the image data before neoadjuvant chemotherapy in S1 includes: a scanning magnetic resonance imaging MRI image before contrast agent injection, a first scanning MRI image after contrast agent injection, and a second scanning MRI image after contrast agent injection.

[0012] Optionally, preprocessing the image of the region of interest in S3 to obtain preprocessed image data includes:

[0013] S31, resampling the image of the region of interest using an image interpolation method to obtain a resampled image.

[0014] Among them, the pixel size of the resampled image is .

[0015] S32, obtaining a middle layer image of the vertical axis of the resampled image.

[0016] S33, normalizing the middle layer image using a maximum-minimum value normalization method to obtain a normalized middle layer image.

[0017] S34, splicing the scanning MRI image before contrast agent injection, the first scanning MRI image after contrast agent injection, the second scanning MRI image after contrast agent injection, and the normalized middle layer image to obtain 3-channel pre-processed image data.

[0018] Optionally, a Transformer deep learning image classification model includes: an encoder, a multi-stage feature extractor, and a pCR classification head.

[0019] Among them, the multiple stage feature extractors include: a first stage feature extractor, a second stage feature extractor, a third stage feature extractor and a fourth stage feature extractor.

[0020] The first-stage feature extractor includes the Transformer block.

[0021] The second stage feature extractor, the third stage feature extractor and the fourth stage feature extractor include downsampling layers and Transformer blocks.

[0022] Transformer blocks for feature extraction using self-attention, multi-layer perceptrons, and residual connections.

[0023] The downsampling layer is used to reduce the input feature scale to 0.25 times by feature recombination and linear mapping.

[0024] Optionally, in S4, based on the preprocessed image data and the constructed Transformer deep learning image classification model, a pathological complete remission (pCR) prediction result of neoadjuvant chemotherapy for breast cancer patients is obtained, including:

[0025] S41. Input the preprocessed image data into the encoder to obtain encoding features.

[0026] S42, input the encoded features into multiple stage feature extractors to obtain image features.

[0027] S43. Input the image features into the pCR classification head to obtain the pathological complete remission (pCR) prediction results of neoadjuvant chemotherapy for breast cancer patients.

[0028] Optionally, the step of inputting the preprocessed image data into an encoder in S41 to obtain encoding features includes:

[0029] S411, segmenting the preprocessed image data to obtain segmented image data.

[0030] Among them, the image data after segmentation is 256 non-intersecting image blocks, and each image block is expanded into Vector.

[0031] S412 . Map each image block through a linear mapping layer of the encoder to obtain a mapped vector of each image block.

[0032] Among them, the size of the mapped vector is .

[0033] S413: Add a learnable position coding vector of the same size to the mapped vector of each image block to obtain coding features of the preprocessed image data.

[0034] Optionally, the step of inputting the image features into a pCR classification head in S43 to obtain a pCR prediction result of pathological complete remission of breast cancer patients after neoadjuvant chemotherapy includes:

[0035] The image features are input into the pCR classification head, and the predefined one-dimensional vector is mapped to the output value through the fully connected layer of the pCR classification head. The output value is normalized using the Sigmoid activation function to obtain the pathological complete remission (pCR) prediction results of neoadjuvant chemotherapy for breast cancer patients.

[0036] On the other hand, a Transformer-based breast cancer neoadjuvant chemotherapy PCR prediction device is provided, which is applied to the Transformer-based breast cancer neoadjuvant chemotherapy PCR prediction method, and the device includes:

[0037] The acquisition module is used to acquire image data of breast cancer patients before neoadjuvant chemotherapy.

[0038] The input module is used to input the image data into the constructed breast cancer tumor segmentation model to obtain a mask image of the tumor lesion area.

[0039] The preprocessing module is used to obtain the region of interest image according to the mask image, and preprocess the region of interest image to obtain preprocessed image data.

[0040] The output module is used to obtain the pathological complete remission (pCR) prediction results of neoadjuvant chemotherapy for breast cancer patients based on the preprocessed image data and the constructed Transformer deep learning image classification model.

[0041] Optionally, the image data before neoadjuvant chemotherapy includes: a scanning magnetic resonance imaging (MRI) image before contrast agent injection, a first scanning MRI image after contrast agent injection, and a second scanning MRI image after contrast agent injection.

[0042] Optionally, the preprocessing module is further used to:

[0043] S31, resampling the image of the region of interest using an image interpolation method to obtain a resampled image.

[0044] Among them, the pixel size of the resampled image is .

[0045] S32, obtaining a middle layer image of the vertical axis of the resampled image.

[0046] S33, normalizing the middle layer image using a maximum-minimum value normalization method to obtain a normalized middle layer image.

[0047] S34, splicing the scanning MRI image before contrast agent injection, the first scanning MRI image after contrast agent injection, the second scanning MRI image after contrast agent injection, and the normalized middle layer image to obtain 3-channel pre-processed image data.

[0048] Optionally, a Transformer deep learning image classification model includes: an encoder, a multi-stage feature extractor, and a pCR classification head.

[0049] Among them, the multiple stage feature extractors include: a first stage feature extractor, a second stage feature extractor, a third stage feature extractor and a fourth stage feature extractor.

[0050] The first-stage feature extractor includes the Transformer block.

[0051] The second stage feature extractor, the third stage feature extractor and the fourth stage feature extractor include downsampling layers and Transformer blocks.

[0052] Transformer blocks for feature extraction using self-attention, multi-layer perceptrons, and residual connections.

[0053] The downsampling layer is used to reduce the input feature scale to 0.25 times by feature recombination and linear mapping.

[0054] Optionally, the output module is further configured to:

[0055] S41. Input the preprocessed image data into the encoder to obtain encoding features.

[0056] S42, input the encoded features into multiple stage feature extractors to obtain image features.

[0057] S43. Input the image features into the pCR classification head to obtain the pathological complete remission (pCR) prediction results of neoadjuvant chemotherapy for breast cancer patients.

[0058] Optionally, the output module is further configured to:

[0059] S411, segmenting the preprocessed image data to obtain segmented image data.

[0060] Among them, the image data after segmentation is 256 non-intersecting image blocks, and each image block is expanded into Vector.

[0061] S412 . Map each image block through a linear mapping layer of the encoder to obtain a mapped vector of each image block.

[0062] Among them, the size of the mapped vector is .

[0063] S413: Add a learnable position coding vector of the same size to the mapped vector of each image block to obtain coding features of the preprocessed image data.

[0064] Optionally, the output module is further configured to:

[0065] The image features are input into the pCR classification head, and the predefined one-dimensional vector is mapped to the output value through the fully connected layer of the pCR classification head. The output value is normalized using the Sigmoid activation function to obtain the pathological complete remission (pCR) prediction results of neoadjuvant chemotherapy for breast cancer patients.

[0066] On the other hand, a breast cancer neoadjuvant chemotherapy PCR prediction device is provided, and the breast cancer neoadjuvant chemotherapy PCR prediction device comprises: a processor; a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, any one of the above-mentioned Transformer-based breast cancer neoadjuvant chemotherapy PCR prediction methods is implemented.

[0067] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement any of the above-mentioned Transformer-based breast cancer neoadjuvant chemotherapy PCR prediction methods.

[0068] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0069] In the embodiment of the present invention, based on DCE-MRI images, the multi-sequence multi-channel input of the model is constructed by using the MRI image before contrast agent injection and two contrast-enhanced images after injection, which can provide the prediction model with rich tumor information such as morphology, size, and distribution. Compared with CT and ultrasound images, DCE-MRI images are more accurate in predicting pCR performance; secondly, the present invention provides an automatic tumor region segmentation method to replace the manual label tumor region, which provides reliable input data for the pCR prediction of neoadjuvant chemotherapy, thereby improving the timeliness of pCR prediction, specifying treatment plans more quickly, and providing patients with timely feedback and adjustments; in addition, the present invention constructs a Transformer-based deep learning pCR prediction model, which can make full use of the global information of the input image and the spatial position relationship between the information, and effectively improve the prediction accuracy of pCR after neoadjuvant chemotherapy; finally, the present invention provides a Transformer-based breast cancer neoadjuvant chemotherapy pCR prediction system, which realizes end-to-end prediction from input to output. Compared with the prior art, there is no need to manually draw the tumor region as input data for neoadjuvant chemotherapy pCR prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0071] Figure 1 This is a flow chart of a Transformer-based PCR prediction method for neoadjuvant chemotherapy of breast cancer provided by an embodiment of the present invention;

[0072] Figure 2 It is a block diagram of a Transformer-based breast cancer neoadjuvant chemotherapy PCR prediction device provided by an embodiment of the present invention;

[0073] Figure 3 It is a structural schematic diagram of a PCR prediction device for neoadjuvant chemotherapy of breast cancer provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0074] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0075] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0076] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same.

[0077] In the embodiments of the present invention, sometimes the subscripts such as W 1 It may be mistakenly written as a non-subscript form such as W1. When the difference is not emphasized, the meanings they express are the same.

[0078] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0079] The embodiment of the present invention provides a Transformer-based breast cancer neoadjuvant chemotherapy PCR prediction method, which can be implemented by a breast cancer neoadjuvant chemotherapy PCR prediction device, which can be a terminal or a server. Figure 1 The flowchart of the Transformer-based breast cancer neoadjuvant chemotherapy PCR prediction method is shown. The processing flow of the method may include the following steps:

[0080] S1. Obtain image data of breast cancer patients before neoadjuvant chemotherapy.

[0081] Optionally, the image data before neoadjuvant chemotherapy in S1 may be DCE-MRI images, which may specifically include: scanning magnetic resonance imaging MRI images before contrast agent injection , the first scan MRI image after contrast agent injection And the second scan MRI image after contrast agent injection .

[0082] In one feasible implementation, DCE-MRI images can be acquired 4 weeks before the patient is treated. Secondly, the first scan is a contrast-enhanced MRI image. The second contrast-enhanced MRI scan was performed between 2 minutes 15 seconds and 2 minutes 30 seconds after contrast injection. It was performed between 7 minutes 15 seconds and 7 minutes 45 seconds after contrast injection.

[0083] S2. Input the image data into the constructed breast cancer tumor segmentation model to obtain a mask image of the tumor lesion area.

[0084] In a feasible implementation, a breast cancer tumor segmentation model is obtained by training a 3D semantic segmentation model.

[0085] Specifically, the input of the 3D semantic segmentation model training includes: a training data set and a 3D semantic segmentation model. In this embodiment, DCE-MRI images of breast cancer patients in the I-SPY1 TRIAL public data set and corresponding tumor mask labels are used as the training data set, and the 3D semantic segmentation model uses U-Net.

[0086] Furthermore, the process of 3D semantic segmentation model training includes: objective function construction and optimization. In this example, Multi-class Dice Loss is used as the objective function of model training, and the optimization of the objective function uses the commonly used gradient descent method to iteratively update the model parameters.

[0087] The output of the 3D semantic segmentation model training is a trained breast cancer tumor segmentation model, and the model can effectively extract the tumor lesion area mask image corresponding to the DCE-MRI image input data of breast cancer patients.

[0088] S3. Obtain an image of the region of interest according to the mask image, and preprocess the image of the region of interest to obtain preprocessed image data.

[0089] In a feasible implementation manner, the region of interest image refers to an image contained in the circumscribed cube of the tumor after the tumor is expanded by 10 mm, and the expansion of 10 mm is achieved by using the mask image.

[0090] Optionally, preprocessing the image of the region of interest in S3 to obtain preprocessed image data may include the following steps S31-S34:

[0091] S31, resampling the image of the region of interest using an image interpolation method to obtain a resampled image .

[0092] Among them, the pixel size of the resampled image is .

[0093] S32, obtain the middle layer image of the vertical axis (first dimension) of the resampled image .

[0094] S33, using the maximum-minimum normalization method to normalize the middle layer image Normalization is performed to obtain a normalized middle layer image.

[0095] S34, scanning MRI image before contrast agent injection , the first scan MRI image after contrast agent injection , The second scan MRI image after injection of contrast agent And the normalized middle layer image Stitching is performed to obtain the preprocessed image data of the three channels, namely the input image .

[0096] S4. Based on the preprocessed image data and the constructed Transformer deep learning image classification model, the pathological complete remission (pCR) prediction results of neoadjuvant chemotherapy for breast cancer patients are obtained. The existing pCR prediction model based on convolutional neural network has a limited convolution receptive field, mainly focusing on the association between local features, and lacks global spatial and positional information. Compared with the traditional convolutional neural network model, the constructed Transform classification model serializes the input image, captures the dependency of long sequences through the attention mechanism, and then obtains the global information of the input image and the spatial position relationship between the information.

[0097] In a feasible implementation, the Transformer deep learning image classification model is designed based on VisionTransformer.

[0098] Specifically, the input of the prediction model training includes training data and a deep learning image classification model. In this embodiment, DCE-MRI images of 80% of breast cancer patients in the I-SPY1 TRIAL public data set and the corresponding neoadjuvant chemotherapy pCR labels are used as the training data set; a network model built based on Transformer is used as a deep learning image classification model, and the model structure specifically includes: an encoder, a multi-stage feature extractor, and a pCR classification head.

[0099] Optionally, the above step S4 may include the following steps S41-S43:

[0100] S41. Input the preprocessed image data into the encoder to obtain encoding features.

[0101] Optionally, the above step S41 may include the following steps S411-S413:

[0102] S411, preprocessing the image data The segmentation is performed to obtain segmented image data.

[0103] Among them, the image data after segmentation is 256 non-intersecting image blocks, and each image block is expanded into Vector.

[0104] S412, through the linear mapping layer The vector of size is mapped to Vector.

[0105] S413, each The vectors of are added with the same size and learnable position encoding vector to get the input image The encoded features are used as input to the feature extractor .

[0106] S42, input the encoded features into multiple stage feature extractors to obtain image features.

[0107] Among them, the multiple stage feature extractors include: a first stage feature extractor, a second stage feature extractor, a third stage feature extractor and a fourth stage feature extractor.

[0108] The first-stage feature extractor includes the Transformer block.

[0109] The second stage feature extractor, the third stage feature extractor and the fourth stage feature extractor include downsampling layers and Transformer blocks.

[0110] Transformer block, which is used to extract features from the input using self-attention, multi-layer perceptrons, and residual connections.

[0111] The downsampling layer is used to reduce the input feature scale to 0.25 times of the original by feature reorganization and linear mapping.

[0112] Furthermore, the image feature scales of the four stages are: , , and .

[0113] S43. The image features are input into the pCR classification head, the predefined one-dimensional vector is mapped to the output value through the fully connected layer of the pCR classification head, and the output value is normalized using the Sigmoid activation function to obtain the pathological complete remission pCR prediction result of neoadjuvant chemotherapy for breast cancer patients.

[0114] In one feasible implementation, the pCR classification head transforms the “ClassToken” vector ( ) is mapped to an output value and normalized using the Sigmoid activation function to obtain the pCR prediction result, where It is a learnable one-dimensional vector defined to facilitate the aggregation of image features and the final pCR prediction. With the image features of each stage ( , , and ) are spliced ​​into new feature representations ( , , and ), the new features of each stage are represented through the self-attention mechanism ( , , , ) to extract features to update image features and aggregate image features to , and then the aggregated features Used for final pCR prediction.

[0115] The process of predictive model training includes: objective function construction and optimization. In this example, the cross entropy loss function is used as the objective function of predictive model training, and the AdamW optimizer is used to iteratively update the model parameters for objective function optimization.

[0116] Furthermore, the output of the prediction model training is a trained prediction model for pCR of neoadjuvant chemotherapy for breast cancer.

[0117] Prediction model validation: Model validation uses a certain amount of data sets to evaluate the performance of the trained prediction model. In this example, the DCE-MRI images of the remaining 20% ​​of breast cancer patients in the I-SPY1 TRIAL public data set and the corresponding neoadjuvant chemotherapy pCR labels were used as the validation data set, and the area under the curve, accuracy, sensitivity, specificity, positive predictive value and negative predictive value were used to evaluate the model. In this embodiment, the results of the area under the curve, accuracy, sensitivity, specificity, positive predictive value and negative predictive value of the validation data set were: 0.8030, 0.8710, 0.6667, 0.9545, 0.8572, and 0.8750, respectively. Compared with the performance of other existing traditional imaging omics methods and convolutional network models, this method can make full use of the global information of the input image and the spatial position relationship between the information, and effectively improve the pCR prediction performance of neoadjuvant chemotherapy.

[0118] In the embodiment of the present invention, based on DCE-MRI images, the multi-sequence multi-channel input of the model is constructed by using the MRI image before contrast agent injection and two contrast-enhanced images after injection, which can provide the prediction model with rich tumor information such as morphology, size, and distribution. Compared with CT and ultrasound images, DCE-MRI images are more accurate in predicting pCR performance; secondly, the present invention provides an automatic tumor region segmentation method to replace the manual label tumor region, which provides reliable input data for the pCR prediction of neoadjuvant chemotherapy, thereby improving the timeliness of pCR prediction, specifying treatment plans more quickly, and providing patients with timely feedback and adjustments; in addition, the present invention constructs a Transformer-based deep learning pCR prediction model, which can make full use of the global information of the input image and the spatial position relationship between the information, and effectively improve the prediction accuracy of pCR after neoadjuvant chemotherapy; finally, the present invention provides a Transformer-based breast cancer neoadjuvant chemotherapy pCR prediction system, which realizes end-to-end prediction from input to output. Compared with the prior art, there is no need to manually draw the tumor region as input data for neoadjuvant chemotherapy pCR prediction.

[0119] Figure 2 This is a block diagram of a Transformer-based breast cancer neoadjuvant chemotherapy PCR prediction device according to an exemplary embodiment. The device is used in a Transformer-based breast cancer neoadjuvant chemotherapy PCR prediction method. Figure 2 The device includes an acquisition module 210, an input module 220, a preprocessing module 230, and an output module 240. Among them:

[0120] The acquisition module 210 is used to acquire image data of breast cancer patients before neoadjuvant chemotherapy.

[0121] The input module 220 is used to input the image data into the constructed breast cancer tumor segmentation model to obtain a mask image of the tumor lesion area.

[0122] The preprocessing module 230 is used to obtain the region of interest image according to the mask image, and preprocess the region of interest image to obtain preprocessed image data.

[0123] The output module 240 is used to obtain the pathological complete remission (pCR) prediction result of neoadjuvant chemotherapy for breast cancer patients based on the preprocessed image data and the constructed Transformer deep learning image classification model.

[0124] Optionally, the image data before neoadjuvant chemotherapy includes: a scanning magnetic resonance imaging (MRI) image before contrast agent injection, a first scanning MRI image after contrast agent injection, and a second scanning MRI image after contrast agent injection.

[0125] Optionally, the preprocessing module 230 is further configured to:

[0126] S31, resampling the image of the region of interest using an image interpolation method to obtain a resampled image.

[0127] Among them, the pixel size of the resampled image is .

[0128] S32, obtaining a middle layer image of the vertical axis of the resampled image.

[0129] S33, normalizing the middle layer image using a maximum-minimum value normalization method to obtain a normalized middle layer image.

[0130] S34, splicing the scanning MRI image before contrast agent injection, the first scanning MRI image after contrast agent injection, the second scanning MRI image after contrast agent injection, and the normalized middle layer image to obtain 3-channel pre-processed image data.

[0131] Optionally, a Transformer deep learning image classification model includes: an encoder, a multi-stage feature extractor, and a pCR classification head.

[0132] Among them, the multiple stage feature extractors include: a first stage feature extractor, a second stage feature extractor, a third stage feature extractor and a fourth stage feature extractor.

[0133] The first-stage feature extractor includes the Transformer block.

[0134] The second stage feature extractor, the third stage feature extractor and the fourth stage feature extractor include downsampling layers and Transformer blocks.

[0135] Transformer blocks for feature extraction using self-attention, multi-layer perceptrons, and residual connections.

[0136] The downsampling layer is used to reduce the input feature scale to 0.25 times by feature recombination and linear mapping.

[0137] Optionally, the output module 240 is further configured to:

[0138] S41. Input the preprocessed image data into the encoder to obtain encoding features.

[0139] S42, input the encoded features into multiple stage feature extractors to obtain image features.

[0140] S43. Input the image features into the pCR classification head to obtain the pathological complete remission (pCR) prediction results of neoadjuvant chemotherapy for breast cancer patients.

[0141] Optionally, the output module 240 is further configured to:

[0142] S411, segmenting the preprocessed image data to obtain segmented image data.

[0143] Among them, the image data after segmentation is 256 non-intersecting image blocks, and each image block is expanded into Vector.

[0144] S412 . Map each image block through a linear mapping layer of the encoder to obtain a mapped vector of each image block.

[0145] Among them, the size of the mapped vector is .

[0146] S413: Add a learnable position coding vector of the same size to the mapped vector of each image block to obtain coding features of the preprocessed image data.

[0147] Optionally, the output module 240 is further configured to:

[0148] The image features are input into the pCR classification head, and the predefined one-dimensional vector is mapped to the output value through the fully connected layer of the pCR classification head. The output value is normalized using the Sigmoid activation function to obtain the pathological complete remission (pCR) prediction results of neoadjuvant chemotherapy for breast cancer patients.

[0149] In the embodiment of the present invention, based on DCE-MRI images, the multi-sequence multi-channel input of the model is constructed by using the MRI image before contrast agent injection and two contrast-enhanced images after injection, which can provide the prediction model with rich tumor information such as morphology, size, and distribution. Compared with CT and ultrasound images, DCE-MRI images are more accurate in predicting pCR performance; secondly, the present invention provides an automatic tumor region segmentation method to replace the manual label tumor region, which provides reliable input data for the pCR prediction of neoadjuvant chemotherapy, thereby improving the timeliness of pCR prediction, specifying treatment plans more quickly, and providing patients with timely feedback and adjustments; in addition, the present invention constructs a Transformer-based deep learning pCR prediction model, which can make full use of the global information of the input image and the spatial position relationship between the information, and effectively improve the prediction accuracy of pCR after neoadjuvant chemotherapy; finally, the present invention provides a Transformer-based breast cancer neoadjuvant chemotherapy pCR prediction system, which realizes end-to-end prediction from input to output. Compared with the prior art, there is no need to manually draw the tumor region as input data for neoadjuvant chemotherapy pCR prediction.

[0150] Figure 3 Schematic diagram of a PCR prediction device for neoadjuvant chemotherapy of breast cancer provided by an embodiment of the present invention. Figure 3 As shown, the PCR prediction device for neoadjuvant chemotherapy of breast cancer may include the above Figure 2 The Transformer-based breast cancer neoadjuvant chemotherapy PCR prediction device is shown. Optionally, the breast cancer neoadjuvant chemotherapy PCR prediction device 310 may include a first processor 2001 .

[0151] Optionally, the breast cancer neoadjuvant chemotherapy PCR prediction device 310 may further include a memory 2002 and a transceiver 2003 .

[0152] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0153] Combine the following Figure 3 The components of the breast cancer neoadjuvant chemotherapy PCR prediction device 310 are specifically introduced:

[0154] The first processor 2001 is the control center of the breast cancer neoadjuvant chemotherapy PCR prediction device 310, which can be a processor or a general term for multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiment of the present invention, such as one or more microprocessors (digital signal processor, DSP), or one or more field programmable gate arrays (field programmable gate array, FPGA).

[0155] Optionally, the first processor 2001 can perform various functions of the breast cancer neoadjuvant chemotherapy PCR prediction device 310 by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002 .

[0156] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 3 CPU0 and CPU1 are shown in FIG.

[0157] In a specific implementation, as an embodiment, the breast cancer neoadjuvant chemotherapy PCR prediction device 310 may also include multiple processors, such as Figure 3 The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0158] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled to be executed by the first processor 2001. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

[0159] Optionally, the memory 2002 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the first processor 2001, or may exist independently, and may be accessed through the interface circuit ( Figure 3 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0160] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0161] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 3 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0162] Optionally, the transceiver 2003 may be integrated with the first processor 2001, or may exist independently, and may be connected to the first processor 201 through the interface circuit ( Figure 3 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0163] It should be noted that Figure 3 The structure of the breast cancer neoadjuvant chemotherapy PCR prediction device 310 shown in the figure does not constitute a limitation on the router, and the actual knowledge structure recognition device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0164] In addition, the technical effects of the breast cancer neoadjuvant chemotherapy PCR prediction device 310 can refer to the technical effects of the breast cancer neoadjuvant chemotherapy PCR prediction method based on Transformer described in the above method embodiment, which will not be repeated here.

[0165] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0166] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0167] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0168] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0169] In the present invention, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0170] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0171] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0172] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0173] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0174] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0175] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0176] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0177] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A Transformer-based method for predicting pCR of neoadjuvant chemotherapy for breast cancer, characterized in that: The method comprises: S1. Obtain three sets of dynamic enhanced magnetic resonance imaging data of breast cancer patients before neoadjuvant chemotherapy; S2, inputting the image data into the constructed breast cancer tumor segmentation model to intelligently obtain a mask image of the tumor lesion area; S3, obtaining an expanded voxel region of interest image according to the mask image, performing dynamic enhanced magnetic resonance image fusion preprocessing on the expanded voxel region of interest image to obtain preprocessed 3-channel multi-sequence magnetic resonance image data; S4. According to the preprocessed 3-channel multi-sequence magnetic resonance image data and the constructed multi-scale Transformer deep learning image classification model, the pathological complete remission (pCR) prediction result of neoadjuvant chemotherapy for breast cancer patients is obtained before chemotherapy; The three sets of dynamic enhanced magnetic resonance image data before neoadjuvant chemotherapy in S1 include: a scanning magnetic resonance imaging MRI image before contrast agent injection, a first scanning MRI image after contrast agent injection, and a second scanning MRI image after contrast agent injection; The breast cancer tumor segmentation model in S2 is obtained by training a 3D semantic segmentation model, wherein the 3D semantic segmentation model adopts U-Net, and the breast cancer tumor segmentation model is used to extract a tumor lesion area mask image corresponding to the DCE-MRI image input data of a breast cancer patient; The multi-scale Transformer deep learning image classification model includes: an encoder, a multi-scale multi-stage feature extractor, and a pCR classification head; The multi-scale multi-stage feature extractor comprises: a first-scale first-stage feature extractor, a second-scale second-stage feature extractor, a third-scale third-stage feature extractor, and a fourth-scale fourth-stage feature extractor; The feature extractor of the first stage includes a Transformer block; The second stage feature extractor, the third stage feature extractor and the fourth stage feature extractor include a downsampling layer and a Transformer block; The Transformer block is used to implement feature extraction using a self-attention mechanism, a multi-layer perceptron, and a residual connection; The downsampling layer is used to reduce the input feature scale to 0.25 times by using feature recombination and linear mapping; The multi-scale image features of the four stages are: F1∈R 256×256 , F2∈R 64×256 , F3∈R 16×256 With F4∈R 4×256 ; The step S4 obtains the pathological complete remission (pCR) prediction result of neoadjuvant chemotherapy for breast cancer patients based on the preprocessed image data and the constructed Transformer deep learning image classification model, including: S41, inputting the preprocessed image data into the encoder to obtain encoding features; S42, inputting the encoded features into the multi-scale multi-stage feature extractor to obtain image features; S43, inputting the image features into the pCR classification head to obtain the pathological complete remission (pCR) prediction result of neoadjuvant chemotherapy for breast cancer patients; The step of inputting the image features into the pCR classification head in S43 to obtain the pathological complete remission (pCR) prediction result of neoadjuvant chemotherapy for breast cancer patients includes: The image features are input into the pCR classification head, and the predefined one-dimensional vector T is transformed into class The output value is mapped to the output value, and the output value is normalized using the Sigmoid activation function to obtain the pathological complete remission (pCR) prediction result of neoadjuvant chemotherapy for breast cancer patients; wherein the T class The image features of each stage are spliced ​​into a new image feature representation, and the new image feature representation of each stage is extracted through the self-attention mechanism to update the image features and aggregate the image features to T class , and then aggregate the features into T class Used for final pCR prediction.

2. The Transformer-based PCR prediction method for neoadjuvant chemotherapy of breast cancer according to claim 1, characterized in that: The step S3 performs dynamic enhancement fusion preprocessing of the expanded voxel region of interest image to obtain preprocessed 3-channel multi-sequence magnetic resonance image data, including: S31, resampling the expanded voxel region of interest image by using an image interpolation method to obtain a resampled image; Wherein, the pixel size of the resampled image is 64×64×64; S32, obtaining a middle layer image of the longitudinal axis of the resampled image; S33, normalizing the middle layer image by using a maximum-minimum normalization method to obtain a normalized middle layer image; S34, splicing and fusing the scanning MRI image before contrast agent injection, the first scanning MRI image after contrast agent injection, the second scanning MRI image after contrast agent injection, and the normalized middle layer image to obtain preprocessed 3-channel multi-sequence magnetic resonance image data.

3. The Transformer-based PCR prediction method for neoadjuvant chemotherapy of breast cancer according to claim 1, characterized in that: The step of inputting the preprocessed image data into the encoder to obtain encoding features in S41 includes: S411, segmenting the preprocessed image data to obtain segmented image data; The segmented image data is 256 non-overlapping 4×4 image blocks, and each image block is expanded into a 1×48 vector; S412, mapping each image block through a linear mapping layer of an encoder to obtain a mapped vector of each image block; Wherein, the size of the mapped vector is 1×256; S413: Add a learnable position coding vector of the same size to the mapped vector of each image block to obtain coding features of the preprocessed image data.

4. A Transformer-based breast cancer neoadjuvant chemotherapy PCR prediction device, the Transformer-based breast cancer neoadjuvant chemotherapy PCR prediction device is used to implement the Transformer-based breast cancer neoadjuvant chemotherapy PCR prediction method according to any one of claims 1 to 3, characterized in that: The device comprises: An acquisition module, used to acquire three sets of dynamic enhanced magnetic resonance image data of breast cancer patients before neoadjuvant chemotherapy; An input module, used to input the image data into the constructed breast cancer tumor segmentation model to intelligently obtain a mask image of the tumor lesion area; A preprocessing module, used for obtaining an outward-expanded voxel region of interest image according to the mask image, performing a fusion preprocessing of a dynamically enhanced magnetic resonance image on the outward-expanded voxel region of interest image, and obtaining preprocessed 3-channel multi-sequence magnetic resonance image data; The output module is used to obtain the pathological complete remission (pCR) prediction result of neoadjuvant chemotherapy for breast cancer patients before chemotherapy based on the preprocessed image data and the constructed multi-scale Transformer deep learning image classification model.

5. A PCR prediction device for neoadjuvant chemotherapy of breast cancer, characterized in that: The breast cancer neoadjuvant chemotherapy PCR prediction device comprises: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 3 is implemented.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 3.

Images