Triple-negative breast cancer immune phenotype prediction method and system based on multi-scale features

Abstract

The present invention discloses a method and system for predicting the immunophenotype of triple-negative breast cancer based on multi-scale features, including: performing feature learning on the received multi-parameter breast MRI image to obtain a first multi-scale feature; respectively inputting two first image sequences obtained by preprocessing the received full-slice image into a multi-scale feature extraction model, so that the model extracts and fuses the local receptive field features, global information and multi-scale information of the pathological image to output a second multi-scale feature; fusing the two multi-scale features to obtain an image pathology feature map, and classifying the image pathology feature map through a multi-layer perceptron structure to output a prediction result of the immunophenotype of triple-negative breast cancer. By adopting the embodiment of the present invention, the features of the two modalities of multi-parameter breast MRI image and full-slice image are fully integrated, and then classification is performed based on the fusion result, thereby improving the prediction accuracy of the immunophenotype of triple-negative breast cancer.

Description

Technical Field

[0001] The present invention relates to the technical field of triple-negative breast cancer immune phenotype prediction, and in particular to a method and system for predicting triple-negative breast cancer immune phenotype based on multi-scale features. Background Art

[0002] Triple negative breast cancer (TNBC) refers to breast cancer that lacks the expression of estrogen receptor (ER), progesterone receptor (PR), and human epidermal growth factor receptor 2 (HER-2). It accounts for 15-20% of all breast cancer patients, and has a young age of onset, high heterogeneity, strong invasiveness, and poor prognosis. The risk of recurrence and metastasis within 3-5 years can be as high as 20%, and once recurrence and metastasis occur, the overall survival is significantly shortened. TNBC shows strong heterogeneity in tissue morphology, molecular phenotype, clinical characteristics, and treatment response, and requires further stratification and classification of treatment. Tumor heterogeneity and lack of biomarkers are considered to be the most difficult challenges driving treatment resistance and recurrence. In recent years, the emergence of targeted therapeutic drugs targeting different targets and immunotherapy drugs targeting immune checkpoints are changing the clinical practice of TNBC. In particular, the progress of immunotherapy has opened a new chapter in the treatment of TNBC breast cancer, but there are still many problems that need to be further solved and improved. Currently, PD-(L)1 inhibitors have been approved for immunotherapy of TNBC, but due to the high heterogeneity of TNBC, not all TNBC patients are effective for immunotherapy, which urgently requires screening of the advantaged population before treatment. Fudan University Cancer Hospital proposed the "Fudan classification" using multi-omics technologies such as whole exome sequencing, copy number variation analysis, and RNA sequencing: androgen receptor luminal subtype, immunoregulatory subtype, basal-like immunosuppressive subtype, and mesenchymal-like subtype, which clarified the molecular characteristics of TNBC in the Chinese population and refined the molecular classification of TNBC.

[0003] At present, the mainstream technology is to use multi-omics technologies such as whole exome sequencing, copy number variation analysis and RNA sequencing to propose the "Fudan classification", but this multi-omics detection method requires a large sample size, high equipment requirements and strong professionalism, which makes it difficult to carry out and apply routinely in clinical practice, and there is still a lot of room for improvement in accuracy. In addition, the proposed immunohistochemical PD-L1 index, in addition to PD-L1 positive and PD-L1 negative patients, others can also benefit from immunotherapy, so the PD-L1 index is not the most ideal screening index for PD-1 / PD-L1 inhibitors. In addition, MRI examinations are usually used in clinical practice to monitor the changes in tumor size of TNBC patients after neoadjuvant therapy (NAT), which can predict the efficacy of NAT in the early stage, but MRI has a low sensitivity in predicting neoadjuvant chemotherapy. Summary of the invention

[0004] The embodiment of the present invention provides a method and system for predicting the immune phenotype of triple-negative breast cancer based on multi-scale features, which fully integrates the features of the two modalities of multi-parameter breast MRI images and whole-slice images, and then improves the prediction accuracy of the immune phenotype of triple-negative breast cancer based on the fused image pathology feature map.

[0005] In order to solve the above technical problems, the embodiment of the present invention provides a method for predicting the immune phenotype of triple-negative breast cancer based on multi-scale features, comprising:

[0006] Performing feature learning on the received multi-parameter breast MRI image to obtain a first multi-scale feature of the multi-parameter breast MRI image;

[0007] Preprocessing the received full-slice image to obtain two first image sequences;

[0008] respectively inputting the two first image sequences into a multi-scale feature extraction model, so that the multi-scale feature extraction model outputs a second multi-scale feature of the full-slice image;

[0009] The first multi-scale feature and the second multi-scale feature are fused to obtain an image pathology feature map, and the image pathology feature map is classified through a multi-layer perceptron structure to output a triple-negative breast cancer immunophenotype prediction result of the multi-parameter breast MRI image and the full-slice image;

[0010] Among them, the first image sequence includes multiple pathological images of different scales, and the multi-scale feature extraction model includes several local Transformer encoders for extracting local receptive field features of the pathological images, several global Transformer encoders for capturing global information of the pathological images, and several multi-scale Transformer encoders for exchanging image information between two of the first image sequences to obtain multi-scale information of the pathological images.

[0011] In the implementation of the embodiment of the present invention, feature learning is performed on the received multi-parameter breast MRI image to obtain the first multi-scale feature of the multi-parameter breast MRI image, and the received full-slice image is preprocessed to obtain two first image sequences, and then the two first image sequences are respectively input into the multi-scale feature extraction model so that the multi-scale feature extraction model outputs the second multi-scale feature of the full-slice image, and by fusing the first multi-scale feature and the second multi-scale feature, the advantages of the multi-parameter breast MRI image and the full-slice image are combined to obtain an image pathology feature map that can fully characterize the characteristics of the two modal images of the multi-parameter breast MRI image and the full-slice image, so that the image pathology feature map can be used to improve the prediction accuracy of the immune phenotype of triple-negative breast cancer. In addition, in the feature extraction stage, several global Transformer encoders and several local Transformer encoders are used to combine pathological images of different scales to generate more relevant image features, alleviate the feature extraction problem of targets of different scales, and use several multi-scale Transformer encoders to exchange information to further improve the extraction accuracy of feature information.

[0012] As a preferred solution, the first multi-scale feature and the second multi-scale feature are fused to obtain an image pathology feature map, and the image pathology feature map is classified through a multi-layer perceptron structure to output the triple-negative breast cancer immunophenotype prediction results of the multi-parameter breast MRI image and the full-slice image, specifically:

[0013] Generate a first input tensor according to the first multi-scale feature, and generate a second input tensor according to the second multi-scale feature, then iteratively process the first input tensor and the second input tensor, input the current first input tensor and the current second input tensor into the Transformer structure in each iterative process, and convert the current output of the Transformer structure into a feature map, and then update the current first input tensor and the current second input tensor according to the current feature map, until the number of iterations meets the preset number, then splice the current first input tensor and the current second input tensor to obtain an image pathology feature map;

[0014] The image pathology feature map is classified by the multi-layer perceptron structure to output the triple-negative breast cancer immunophenotype prediction results of the multi-parameter breast MRI image and the full-slice image.

[0015] A preferred scheme for implementing an embodiment of the present invention utilizes a Transformer structure to perform multiple processing on the first multi-scale features of the multi-parameter breast MRI image and the second multi-scale features of the whole slice image, so as to better and gradually fuse the features of the multi-parameter breast MRI image and the whole slice image, thereby improving the accuracy of the image pathology feature map.

[0016] As a preferred solution, the two first image sequences are respectively input into a multi-scale feature extraction model so that the multi-scale feature extraction model outputs a second multi-scale feature of the full-slice image, specifically:

[0017] Inputting two first image sequences into a linear projection layer respectively to obtain a plurality of patch tokens corresponding to each of the first image sequences, and then inputting the plurality of patch tokens corresponding to one of the first image sequences into a local Transformer module so that the local Transformer module extracts local receptive field features of all the pathological images in the first image sequence, and inputting the plurality of patch tokens corresponding to another first image sequence into a global Transformer module so that the global Transformer module captures global information of all the pathological images in the first image sequence;

[0018] Using the Class token of the first branch of each of the multi-scale Transformer encoders, information is exchanged with a plurality of patch tokens corresponding to one of the first image sequences, and the exchanged information is back-projected to the first branch; while using the Class token of the second branch of the multi-scale Transformer encoder, information is exchanged with a plurality of patch tokens corresponding to another of the first image sequences, and the exchanged information is back-projected to the second branch, and then multi-scale information of all the pathological images is generated according to the back-projection results of the first branch and the second branch;

[0019] Concatenate and globally average pool the local receptive field feature, the global information, and the multi-scale information, and input the processing result into a fully connected layer to obtain the second multi-scale feature of the full slice image;

[0020] The patch token corresponds to the pathological image one by one, the local Transformer module includes several local Transformer encoders, and the global Transformer module includes several global Transformer encoders.

[0021] A preferred scheme for implementing the embodiment of the present invention is to set a global Transformer module and a local Transformer module to retain more fine-grained features of a specific scale, so that the local receptive field features, global information and multi-scale information can be fully utilized to distinguish between tumor cells and tumor microenvironment, so as to realize the classification of tumor cells in the whole slice image.

[0022] As a preferred solution, the received full-slice image is preprocessed to obtain two first image sequences, specifically:

[0023] According to the preset processing order and size requirements, the received full-slice image is cropped to obtain a plurality of pathological slices, and then all the pathological slices are randomly expanded and image enhanced to obtain the pathological images of multiple different scales;

[0024] According to the magnification of each of the pathological images, all the pathological images are divided into two first image sequences.

[0025] According to a preferred solution of an embodiment of the present invention, before feature extraction is performed on the image, the received full-slice image is cropped according to a preset processing order and size requirements to obtain a number of pathological slices of uniform size and different positions, and then all the pathological slices are randomly expanded and image enhanced to obtain a plurality of pathological images of different scales to improve the clarity of the pathological images. Subsequently, all the pathological images are divided into two first image sequences according to the magnification of each pathological image, so that global information extraction can be performed based on one first image sequence, and local information capture can be performed based on the other first image sequence.

[0026] As a preferred solution, the feature learning is performed on the received multi-parameter breast MRI image to obtain the first multi-scale feature of the multi-parameter breast MRI image, specifically:

[0027] Performing radiomics feature extraction on the received multi-parameter breast MRI image to obtain a corresponding radiomics feature extraction result, and performing deep learning feature extraction on the received multi-parameter breast MRI image to obtain a corresponding deep learning feature extraction result;

[0028] Fusing the radiomics feature extraction result and the deep learning feature extraction result to obtain the first multi-scale feature of the multi-parameter breast MRI image;

[0029] The multi-parameter breast MRI image is obtained by analyzing and processing breast MRI images with different parameters.

[0030] A preferred scheme for implementing the embodiment of the present invention is to perform imaging omics feature extraction and deep learning feature extraction on the received multi-parameter breast MRI images, respectively, and fuse the two types of feature extraction results to fully learn the correlation between breast MRI images with different parameters, enhance the effectiveness of the first multi-scale feature, thereby improving the prediction accuracy of the immune phenotype of triple-negative breast cancer, and reducing the occupation of computer resources and the overall computing time.

[0031] As a preferred embodiment, the method for predicting triple-negative breast cancer immune phenotype based on multi-scale features further includes:

[0032] Using a text encoder, extracting features from the received clinical text information to obtain corresponding clinical information features;

[0033] The combined results of the clinical information features and the image pathology feature map are analyzed and processed through a multi-layer perceptron structure to output a triple-negative breast cancer probability map corresponding to the multi-parameter breast MRI image, the full slice image and the clinical text information.

[0034] A preferred scheme for implementing the embodiment of the present invention is to combine the clinical information features corresponding to the clinical text information with the image pathology feature maps corresponding to the multi-parameter breast MRI images and the whole slice images, and analyze and process the combination results through a multi-layer perceptron structure to output triple-negative breast cancer probability maps corresponding to the three types of images / information: multi-parameter breast MRI images, whole slice images and clinical text information, so as to further improve the prediction and recognition of triple-negative breast cancer.

[0035] In order to solve the same technical problem, the embodiment of the present invention also provides a triple-negative breast cancer immune phenotype prediction system based on multi-scale features, including:

[0036] A feature learning module, used for performing feature learning on the received multi-parameter breast MRI image to obtain a first multi-scale feature of the multi-parameter breast MRI image;

[0037] A preprocessing module, used for preprocessing the received full-slice image to obtain two first image sequences; wherein the first image sequences include a plurality of pathological images of different scales;

[0038] A feature extraction module, used for inputting the two first image sequences into a multi-scale feature extraction model respectively, so that the multi-scale feature extraction model outputs a second multi-scale feature of the full-slice image; wherein the multi-scale feature extraction model includes a plurality of local Transformer encoders for extracting local receptive field features of the pathological image, a plurality of global Transformer encoders for capturing global information of the pathological image, and a plurality of multi-scale Transformer encoders for exchanging image information between the two first image sequences to obtain multi-scale information of the pathological image;

[0039] A prediction module is used to fuse the first multi-scale features and the second multi-scale features to obtain an image pathology feature map, and classify the image pathology feature map through a multi-layer perceptron structure to output a triple-negative breast cancer immunophenotype prediction result for the multi-parameter breast MRI image and the full-slice image.

[0040] As a preferred solution, the prediction module specifically includes:

[0041] A feature fusion unit, used to generate a first input tensor according to the first multi-scale feature, and generate a second input tensor according to the second multi-scale feature, and then iteratively process the first input tensor and the second input tensor, input the current first input tensor and the current second input tensor into the Transformer structure in each iterative process, and convert the current output of the Transformer structure into a feature map, and then update the current first input tensor and the current second input tensor according to the current feature map, until the number of iterations meets the preset number, then splice the current first input tensor and the current second input tensor to obtain an image pathology feature map;

[0042] A classification unit is used to classify the image pathology feature map through the multi-layer perceptron structure to output the triple-negative breast cancer immunophenotype prediction results of the multi-parameter breast MRI image and the full-slice image.

[0043] As a preferred solution, the feature extraction module specifically includes:

[0044] A feature extraction unit is used to input the two first image sequences into the linear projection layer respectively to obtain multiple patch tokens corresponding to each of the first image sequences, and then input the multiple patch tokens corresponding to one of the first image sequences into the local Transformer module so that the local Transformer module extracts the local receptive field features of all the pathological images in the first image sequence, and input the multiple patch tokens corresponding to another of the first image sequences into the global Transformer module so that the global Transformer module captures the global information of all the pathological images in the first image sequence; using the Class token of the first branch of each of the multi-scale Transformer encoders to exchange information with the multiple patch tokens corresponding to one of the first image sequences, and back-projecting the exchanged information to the first branch; and using the Class token of the second branch of the multi-scale Transformer encoder to exchange information with the multiple patch tokens corresponding to another of the first image sequences, and back-projecting the exchanged information to the second branch, and then generating multi-scale information of all the pathological images according to the back-projection results of the first branch and the second branch; wherein, the patch The token corresponds to the pathological image one by one, the local Transformer module includes a plurality of the local Transformer encoders, and the global Transformer module includes a plurality of the global Transformer encoders;

[0045] A splicing processing unit is used to splice and globally average pool the local receptive field features, the global information and the multi-scale information, and input the processing results to a fully connected layer to obtain the second multi-scale features of the full slice image.

[0046] As a preferred solution, the preprocessing module specifically includes:

[0047] An image processing unit, used to perform cropping processing on the received full-slice image according to a preset processing order and size requirements to obtain a plurality of pathological slices, and then perform random expansion and image enhancement on all the pathological slices to obtain the pathological images of a plurality of different scales;

[0048] A division unit is used to divide all the pathological images into two first image sequences according to the magnification of each pathological image. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1: A schematic diagram of a process for predicting the immune phenotype of triple-negative breast cancer based on multi-scale features provided in Example 1 of the present invention;

[0050] Figure 2 : A schematic diagram of the structure of a multi-scale feature extraction model provided in Example 1 of the present invention;

[0051] Figure 3 : A schematic diagram of the structure of a local Transformer encoder provided in Embodiment 1 of the present invention;

[0052] Figure 4 : A schematic diagram of the structure of a global Transformer encoder provided in Embodiment 1 of the present invention;

[0053] Figure 5 : A schematic diagram of the structure of a multi-scale Transformer encoder provided in Example 1 of the present invention;

[0054] Figure 6 : A schematic diagram of a multimodal feature fusion structure provided in Embodiment 1 of the present invention;

[0055] Figure 7 : A structural schematic diagram of a triple-negative breast cancer immune phenotype prediction system based on multi-scale features provided in Example 1 of the present invention. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] Embodiment one:

[0058] Please refer to Figure 1 , a method for predicting triple-negative breast cancer immune phenotype based on multi-scale features provided by an embodiment of the present invention, the method comprises steps S1 to S3, each step is specifically as follows:

[0059] Step S1, performing feature learning on the received multi-parameter breast MRI image to obtain a first multi-scale feature of the multi-parameter breast MRI image.

[0060] As a preferred solution, step S1 includes step S11 to step S12, and each step is specifically as follows:

[0061] Step S11, performing radiomics feature extraction on the received multi-parameter breast MRI image to obtain corresponding radiomics feature extraction results, and performing deep learning feature extraction on the received multi-parameter breast MRI image to obtain corresponding deep learning feature extraction results.

[0062] The multi-parameter breast MRI image is obtained by analyzing and processing breast MRI images with different parameters. In this embodiment, the breast MRI images with different parameters refer to breast MRI images of T1WI+C, T2WI and DWI sequences.

[0063] Step S12: fusing the radiomics feature extraction result and the deep learning feature extraction result to obtain a first multi-scale feature of the multi-parameter breast MRI image.

[0064] Step S2: preprocess the received full-slice image to obtain two first image sequences.

[0065] The first image sequence includes a plurality of pathological images of different scales.

[0066] As a preferred solution, step S2 includes step S21 to step S22, and each step is specifically as follows:

[0067] Step S21, according to the preset processing order and size requirements, the received full-slice image is cropped to obtain a number of pathological slices, and then all the pathological slices are randomly expanded and image enhanced to obtain a plurality of pathological images of different scales.

[0068] Step S22: Divide all pathological images into two first image sequences according to the magnification of each pathological image.

[0069] The scale of the pathological image includes but is not limited to H / 2×W / 2×3 and 2H×2W×3, wherein H and W represent the height and width of the pathological image. The same first image sequence includes multiple pathological images of the same position and different scales on the whole slice image.

[0070] As an example, a whole slide image (WSI) can be an HE-stained slice image, which will be referred to as a WSI slice in the following. Each WSI slice is uniformly processed into a patch (pathological slice) of 256×256 in size from left to right and from top to bottom with a step size of 53, and then each patch is randomly expanded using elastic transformation, and image enhancement is performed on it to obtain multiple pathological images of different scales, and then all pathological images are divided into two first image sequences according to the magnification of each pathological image (wherein there are four pathological images with a magnification of 10× and four pathological images with a magnification of 20×), and the preprocessing of the WSI slice is completed. Among them, the image enhancement methods include but are not limited to flipping, random rotation, random contrast brightness and blur.

[0071] Step S3: input the two first image sequences into the multi-scale feature extraction model respectively, so that the multi-scale feature extraction model outputs the second multi-scale features of the full-slice image.

[0072] Among them, the multi-scale feature extraction model includes several local Transformer encoders for extracting local receptive field features of pathological images, several global Transformer encoders for capturing global information of pathological images, and several multi-scale Transformer encoders for exchanging image information between two first image sequences to obtain multi-scale information of pathological images.

[0073] As a preferred solution, step S3 includes step S31 to step S33, and each step is specifically as follows:

[0074] Step S31, please refer to Figure 2 , the two first image sequences are respectively input into the linear projection layer (Linearprojection) to obtain multiple patch tokens corresponding to each first image sequence, and then the multiple patch tokens corresponding to one first image sequence are input into the local Transformer module, so that the local Transformer module extracts the local receptive field features of all pathological images in the first image sequence, and at the same time, the multiple patch tokens corresponding to another first image sequence are input into the global Transformer module, so that the global Transformer module captures the global information of all pathological images in the first image sequence.

[0075] Among them, the patch token corresponds to the pathological image one by one, please refer to Figure 2, the local Transformer module includes N local Transformer encoders, the global Transformer module includes M global Transformer encoders, and the multi-scale Transformer module includes K multi-scale Transformer encoders. For the specific structure of the local Transformer encoder, please refer to Figure 3 The encoder replaces MSA with window-based multi-head self-attention and uses two consecutive Swintransformer blocks to achieve cross-window connections; please refer to the specific structure of the global Transformer encoder Figure 4 , which contains a multi-head self-attention (MSA), a multi-layer perception (MLP) and two layers of normalization (LN) structure; please refer to the specific structure of the multi-scale Transformer encoder Figure 5 .

[0076] It should be noted that two Transformer modules (one is a local Transformer module responsible for processing all pathological images with a magnification of 20×, and the other is a global Transformer module responsible for processing all pathological images with a magnification of 10×) are used to combine patch tokens of different scales and retain more fine-grained features of specific scales.

[0077] Step S32, please refer to Figure 2 , the Class token of the first branch of each multi-scale Transformer encoder in the multi-scale Transformer is used to exchange information with multiple patch tokens corresponding to a first image sequence, and the exchanged information is back-projected to the first branch. At the same time, the Class token of the second branch of the multi-scale Transformer encoder is used as a proxy to exchange information with multiple patch tokens corresponding to another first image sequence, and the exchanged information is back-projected to the second branch. Then, refer to formula (1), according to the back-projection results of the first branch and the second branch, the multi-scale information f of all pathological images is generated. o .

[0078] f o =PS(reshape(f in )) (1)

[0079] In the formula, f inrepresents the input features of a single set of modules, and PS(·) represents the patch-merging layer.

[0080] In this embodiment, please refer to Figure 5 For branch A, first collect the patch token from branch B, and refer to formula (2) to connect branch A’s own CLS token to the patch token to get x′ A Then, see equations (3)(4)(5)(6)(7), in x′ A and Cross Attention (CA) is performed between them; among them, the CLS token is the only query because the information of the patch token is fused into the CLS token.

[0081]

[0082]

[0083] k=x′ A W k (4)

[0084] v = x′ A W v (5)

[0085]

[0086] CA(x′ A )=Ov (7)

[0087] In the formula, f A (·) represents the size-aligned projection function, represents a learnable parameter, C and h represent the dimension of embedding and the number of attention heads, respectively.

[0088] Step S33, please refer to Figure 2 , the local receptive field features, global information and multi-scale information are spliced ​​and globally averaged pooled, and the processing results are input into the fully connected layer to distinguish tumor cells from the tumor microenvironment and classify the tumor cells in the whole slice image, and obtain the second multi-scale features of the whole slice image.

[0089] Step S4, fusing the first multi-scale feature and the second multi-scale feature to obtain an image pathology feature map, and classifying the image pathology feature map through a multi-layer perceptron structure to output the triple-negative breast cancer immunophenotype prediction results of multi-parameter breast MRI images and whole-slice images.

[0090] As a preferred solution, step S4 includes step S41 to step S42, and each step is specifically as follows:

[0091] Step S41, generating a first input tensor according to the first multi-scale feature, and generating a second input tensor according to the second multi-scale feature, and then referring to Figure 6 , the first input tensor and the second input tensor are iteratively processed, and the current first input tensor and the current second input tensor are input into the Transformer structure in each iterative processing, and the current output of the Transformer structure is converted into a feature map, and then the current first input tensor and the current second input tensor are updated according to the current feature map until the number of iterations meets the preset number, then the current first input tensor and the current second input tensor are spliced ​​to obtain the image pathology feature map.

[0092] Step S42, classifying the image pathology feature map through a multilayer perceptron (MLP) structure to output the triple-negative breast cancer immunophenotype prediction results of the multi-parameter breast MRI image and the whole slice image.

[0093] As a preferred solution, a method for predicting triple-negative breast cancer immune phenotype based on multi-scale features provided in an embodiment of the present invention further includes steps S51 to S52, each of which is specifically as follows:

[0094] Step S51, please refer to Figure 6 , use the text encoder to extract features from the received clinical text information and obtain the corresponding clinical information features.

[0095] Step S52, please refer to Figure 6 ,Through MLP, the combined results of clinical information features and ,image pathology feature maps are analyzed and processed to output ,triplet-negative breast cancer probability maps corresponding to multi-parameter breast MRI images, whole slice images and ,clinical text information, and achieve TNBC prognosis prediction.

[0096] Please refer to Figure 7 , is a schematic diagram of the structure of a triple-negative breast cancer immune phenotype prediction system based on multi-scale features provided by an embodiment of the present invention, the system includes a feature learning module M1, a preprocessing module M2, a feature extraction module M3 and a prediction module M4, and the specific modules are as follows:

[0097] A feature learning module M1 is used to perform feature learning on the received multi-parameter breast MRI image to obtain a first multi-scale feature of the multi-parameter breast MRI image;

[0098] A preprocessing module M2 is used to preprocess the received full-slice image to obtain two first image sequences; wherein the first image sequence includes a plurality of pathological images of different scales;

[0099] A feature extraction module M3 is used to input the two first image sequences into a multi-scale feature extraction model respectively, so that the multi-scale feature extraction model outputs a second multi-scale feature of the full-slice image; wherein the multi-scale feature extraction model includes a plurality of local Transformer encoders for extracting local receptive field features of the pathological image, a plurality of global Transformer encoders for capturing global information of the pathological image, and a plurality of multi-scale Transformer encoders for exchanging image information between the two first image sequences to obtain multi-scale information of the pathological image;

[0100] The prediction module M4 is used to fuse the first multi-scale feature and the second multi-scale feature to obtain an image pathology feature map, and classify the image pathology feature map through a multi-layer perceptron structure to output the triple-negative breast cancer immunophenotype prediction results of multi-parameter breast MRI images and full-slice images.

[0101] As a preferred solution, the prediction module M4 specifically includes a feature fusion unit 41 and a classification unit 42, and the details of each unit are as follows:

[0102] A feature fusion unit 41 is used to generate a first input tensor according to the first multi-scale feature, and to generate a second input tensor according to the second multi-scale feature, and then iteratively process the first input tensor and the second input tensor, input the current first input tensor and the current second input tensor into the Transformer structure in each iterative process, and convert the current output of the Transformer structure into a feature map, and then update the current first input tensor and the current second input tensor according to the current feature map, until the number of iterations meets the preset number, then splice the current first input tensor and the current second input tensor to obtain an image pathology feature map;

[0103] The classification unit 42 is used to classify the image pathology feature map through a multi-layer perceptron structure to output the triple-negative breast cancer immunophenotype prediction results of the multi-parameter breast MRI image and the whole slice image.

[0104] As a preferred solution, the feature extraction module M3 specifically includes a feature extraction unit 31 and a splicing processing unit 32, and the details of each unit are as follows:

[0105] The feature extraction unit 31 is used to input the two first image sequences into the linear projection layer respectively to obtain multiple patch tokens corresponding to each first image sequence, and then input the multiple patch tokens corresponding to one first image sequence into the local Transformer module, so that the local Transformer module extracts the local receptive field features of all pathological images in the first image sequence, and input the multiple patch tokens corresponding to another first image sequence into the global Transformer module, so that the global Transformer module captures the global information of all pathological images in the first image sequence; using the Class token of the first branch of each multi-scale Transformer encoder, information is exchanged with multiple patch tokens corresponding to one first image sequence, and the exchanged information is back-projected to the first branch, and at the same time, using the Class token of the second branch of the multi-scale Transformer encoder, information is exchanged with multiple patchtokens corresponding to another first image sequence, and the exchanged information is back-projected to the second branch, and then according to the back-projection results of the first branch and the second branch, multi-scale information of all pathological images is generated; wherein the patch token corresponds to the pathological image one by one, the local Transformer module includes a plurality of local Transformer encoders, and the global Transformer module includes a plurality of global Transformer encoders;

[0106] The splicing processing unit 32 is used to splice and globally average pool the local receptive field features, global information and multi-scale information, and input the processing results to the fully connected layer to obtain the second multi-scale features of the full slice image.

[0107] As a preferred solution, the preprocessing module M2 specifically includes an image processing unit 21 and a division unit 22, and the specific details of each unit are as follows:

[0108] The image processing unit 21 is used to perform cropping processing on the received full-slice image according to a preset processing order and size requirements to obtain a plurality of pathological slices, and then perform random expansion and image enhancement on all pathological slices to obtain a plurality of pathological images of different scales;

[0109] The dividing unit 22 is used to divide all pathological images into two first image sequences according to the magnification of each pathological image.

[0110] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0111] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0112] The present invention provides a method and system for predicting the immunophenotype of triple-negative breast cancer based on multi-scale features, performs feature learning on a received multi-parameter breast MRI image, obtains a first multi-scale feature of the multi-parameter breast MRI image, and pre-processes the received full-slice image to obtain two first image sequences, and then respectively inputs the two first image sequences into a multi-scale feature extraction model, so that the multi-scale feature extraction model outputs a second multi-scale feature of the full-slice image, and combines the advantages of the multi-parameter breast MRI image and the full-slice image by fusing the first multi-scale feature and the second multi-scale feature, and obtains an image pathology feature map that can fully characterize the characteristics of the images of the two modalities of the multi-parameter breast MRI image and the full-slice image, so that the image pathology feature map can be used to improve the prediction accuracy of the immunophenotype of triple-negative breast cancer. In addition, in the feature extraction stage, several global Transformer encoders and several local Transformer encoders are used to combine pathological images of different scales to generate more relevant image features, alleviate the feature extraction problem of targets of different scales, and use several multi-scale Transformer encoders for information exchange to further improve the extraction accuracy of feature information.

[0113] The specific embodiments described above further illustrate the purpose, technical solutions and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. It is particularly pointed out that for those skilled in the art, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for predicting the immune phenotype of triple-negative breast cancer based on multi-scale features, characterized in that: include: Performing feature learning on the received multi-parameter breast MRI image to obtain a first multi-scale feature of the multi-parameter breast MRI image; Preprocessing the received full-slice image to obtain two first image sequences; respectively inputting the two first image sequences into a multi-scale feature extraction model, so that the multi-scale feature extraction model outputs a second multi-scale feature of the full-slice image; The first multi-scale feature and the second multi-scale feature are fused to obtain an image pathology feature map, and the image pathology feature map is classified through a multi-layer perceptron structure to output a triple-negative breast cancer immunophenotype prediction result of the multi-parameter breast MRI image and the full-slice image; The first image sequence includes a plurality of pathological images of different scales, and the multi-scale feature extraction model includes a plurality of local Transformer encoders for extracting local receptive field features of the pathological images, a plurality of global Transformer encoders for capturing global information of the pathological images, and a plurality of multi-scale Transformer encoders for exchanging image information between two first image sequences to obtain multi-scale information of the pathological images; The received full-slice image is preprocessed to obtain two first image sequences, specifically: According to the preset processing order and size requirements, the received full-slice image is cropped to obtain a plurality of pathological slices, and then all the pathological slices are randomly expanded and image enhanced to obtain the pathological images of multiple different scales; Dividing all the pathological images into two first image sequences according to the magnification of each pathological image; The step of respectively inputting the two first image sequences into a multi-scale feature extraction model so that the multi-scale feature extraction model outputs the second multi-scale features of the full-slice image is specifically as follows: Inputting two first image sequences into a linear projection layer respectively to obtain a plurality of patch tokens corresponding to each of the first image sequences, and then inputting the plurality of patch tokens corresponding to one of the first image sequences into a local Transformer module so that the local Transformer module extracts local receptive field features of all the pathological images in the first image sequence, and inputting the plurality of patch tokens corresponding to another first image sequence into a global Transformer module so that the global Transformer module captures global information of all the pathological images in the first image sequence; Using the Class token of the first branch of each of the multi-scale Transformer encoders, information is exchanged with a plurality of patch tokens corresponding to one of the first image sequences, and the exchanged information is back-projected to the first branch; while using the Class token of the second branch of the multi-scale Transformer encoder, information is exchanged with a plurality of patch tokens corresponding to another of the first image sequences, and the exchanged information is back-projected to the second branch, and then multi-scale information of all the pathological images is generated according to the back-projection results of the first branch and the second branch; Concatenate and globally average pool the local receptive field feature, the global information, and the multi-scale information, and input the processing result into a fully connected layer to obtain the second multi-scale feature of the full slice image; The patch token corresponds to the pathological image one by one, the local Transformer module includes several local Transformer encoders, and the global Transformer module includes several global Transformer encoders.

2. A method for predicting triple-negative breast cancer immune phenotype based on multi-scale features according to claim 1, characterized in that: The first multi-scale feature and the second multi-scale feature are fused to obtain an image pathology feature map, and the image pathology feature map is classified through a multi-layer perceptron structure to output the triple-negative breast cancer immunophenotype prediction results of the multi-parameter breast MRI image and the full-slice image, specifically: Generate a first input tensor according to the first multi-scale feature, and generate a second input tensor according to the second multi-scale feature, then iteratively process the first input tensor and the second input tensor, input the current first input tensor and the current second input tensor into the Transformer structure in each iterative process, and convert the current output of the Transformer structure into a feature map, and then update the current first input tensor and the current second input tensor according to the current feature map, until the number of iterations meets the preset number, then splice the current first input tensor and the current second input tensor to obtain an image pathology feature map; The image pathology feature map is classified by the multi-layer perceptron structure to output the triple-negative breast cancer immunophenotype prediction results of the multi-parameter breast MRI image and the full-slice image.

3. A method for predicting triple-negative breast cancer immune phenotype based on multi-scale features according to claim 1, characterized in that: The feature learning is performed on the received multi-parameter breast MRI image to obtain the first multi-scale feature of the multi-parameter breast MRI image, specifically: Performing radiomics feature extraction on the received multi-parameter breast MRI image to obtain a corresponding radiomics feature extraction result, and performing deep learning feature extraction on the received multi-parameter breast MRI image to obtain a corresponding deep learning feature extraction result; Fusing the radiomics feature extraction result and the deep learning feature extraction result to obtain the first multi-scale feature of the multi-parameter breast MRI image; The multi-parameter breast MRI image is obtained by analyzing and processing breast MRI images with different parameters.

4. The method for predicting triple-negative breast cancer immune phenotype based on multi-scale features according to claim 1, characterized in that: Also includes: Using a text encoder, extracting features from the received clinical text information to obtain corresponding clinical information features; The combined results of the clinical information features and the image pathology feature map are analyzed and processed through a multi-layer perceptron structure to output a triple-negative breast cancer probability map corresponding to the multi-parameter breast MRI image, the full slice image and the clinical text information.

5. A triple-negative breast cancer immune phenotype prediction system based on multi-scale features, characterized in that: include: A feature learning module, used for performing feature learning on the received multi-parameter breast MRI image to obtain a first multi-scale feature of the multi-parameter breast MRI image; A preprocessing module, used for preprocessing the received full-slice image to obtain two first image sequences; wherein the first image sequences include a plurality of pathological images of different scales; A feature extraction module, used for inputting the two first image sequences into a multi-scale feature extraction model respectively, so that the multi-scale feature extraction model outputs a second multi-scale feature of the full-slice image; wherein the multi-scale feature extraction model includes a plurality of local Transformer encoders for extracting local receptive field features of the pathological image, a plurality of global Transformer encoders for capturing global information of the pathological image, and a plurality of multi-scale Transformer encoders for exchanging image information between the two first image sequences to obtain multi-scale information of the pathological image; A prediction module, configured to fuse the first multi-scale feature and the second multi-scale feature to obtain an image pathology feature map, and classify the image pathology feature map through a multi-layer perceptron structure to output a prediction result of the triple-negative breast cancer immunophenotype of the multi-parameter breast MRI image and the full-slice image; Wherein, the preprocessing module specifically includes: An image processing unit, used to perform cropping processing on the received full-slice image according to a preset processing order and size requirements to obtain a plurality of pathological slices, and then perform random expansion and image enhancement on all the pathological slices to obtain the pathological images of a plurality of different scales; a dividing unit, configured to divide all the pathological images into two first image sequences according to the magnification of each of the pathological images; Wherein, the feature extraction module specifically includes: A feature extraction unit is used to input the two first image sequences into the linear projection layer respectively to obtain multiple patch tokens corresponding to each of the first image sequences, and then input the multiple patch tokens corresponding to one of the first image sequences into the local Transformer module so that the local Transformer module extracts the local receptive field features of all the pathological images in the first image sequence, and input the multiple patch tokens corresponding to another of the first image sequences into the global Transformer module so that the global Transformer module captures the global information of all the pathological images in the first image sequence; using the Class token of the first branch of each of the multi-scale Transformer encoders to exchange information with the multiple patch tokens corresponding to one of the first image sequences, and back-projecting the exchanged information to the first branch; and using the Class token of the second branch of the multi-scale Transformer encoder to exchange information with the multiple patch tokens corresponding to another of the first image sequences, and back-projecting the exchanged information to the second branch, and then generating multi-scale information of all the pathological images according to the back-projection results of the first branch and the second branch; wherein, the patch The token corresponds to the pathological image one by one, the local Transformer module includes a plurality of the local Transformer encoders, and the global Transformer module includes a plurality of the global Transformer encoders; A splicing processing unit is used to splice and globally average pool the local receptive field features, the global information and the multi-scale information, and input the processing results to a fully connected layer to obtain the second multi-scale features of the full slice image.

6. A triple-negative breast cancer immune phenotype prediction system based on multi-scale features according to claim 5, characterized in that: The prediction module specifically includes: A feature fusion unit, used to generate a first input tensor according to the first multi-scale feature, and generate a second input tensor according to the second multi-scale feature, and then iteratively process the first input tensor and the second input tensor, input the current first input tensor and the current second input tensor into the Transformer structure in each iterative process, and convert the current output of the Transformer structure into a feature map, and then update the current first input tensor and the current second input tensor according to the current feature map, until the number of iterations meets the preset number, then splice the current first input tensor and the current second input tensor to obtain an image pathology feature map; A classification unit is used to classify the image pathology feature map through the multi-layer perceptron structure to output the triple-negative breast cancer immunophenotype prediction results of the multi-parameter breast MRI image and the full-slice image.

Images