A method for multiplexed immunohistochemistry image generation

Abstract

The present application relates to a kind of multiplex immunohistochemistry image generation method, comprising: obtaining H&E dataset and paired mIHC dataset and pre-processing;mIHC generation model is constructed;mIHC generation model is trained;H&E image in test set is input to the mIHC generation model after training and obtains the generated mIHC image result, and the generation result of mIHC image is visualized.The present application can quickly and accurately obtain mIHC image by mIHC generation model using easily obtained H&E image, and avoids traditional chemical dyeing process, reduces the cost of obtaining mIHC image, provides convenient and accurate tool for the acquisition of mIHC image, to provide pathological information such as cancer, and can shorten the diagnosis and analysis cycle of disease, is conducive to the timely diagnosis and treatment of patient;The mIHC image generated by mIHC generation model using H&E image and H&E image are strictly registered, so H&E image and mIHC image can be accurately combined to diagnose disease, to improve the accuracy of disease diagnosis analysis.

Description

Technical Field

[0001] The present invention relates to the technical field of image generation, in particular to a multiplex immunohistochemistry image generation method. Background Art

[0002] The tumor microenvironment (TME) is the ecological environment in which tumors grow and develop, playing a crucial role in tumor evolution and progression. Analyzing information about the TME is crucial for cancer treatment and assessment. With the development of multiplex imaging technologies, such as mIHC (multiplex immunohistochemical) staining, pathologists can gain a more comprehensive understanding of the TME.

[0003] mIHC images can display the expression of multiple biomarkers on a WSI (Whole Slide Image) while maintaining the spatial positional relationship in the WSI, providing pathologists with sufficient pathological information and facilitating cancer analysis and diagnosis. However, there are certain difficulties in the acquisition and application of mIHC images. First, the preparation cost of mIHC images is high, requiring a large amount of time and resources, and the long preparation cycle may also delay the treatment of the disease. Furthermore, the tissue sections will be damaged during the staining process, so the same tissue section cannot be repeatedly stained. Therefore, the commonly used H&E (Hematoxylin and Eosin) images are difficult to align with the mIHC images, making it impossible to fully utilize the pathological section information. In addition, problems such as molecular degradation and autolysis in tissue sections can cause inaccurate staining results.

[0004] With the development of deep learning technology, artificial neural networks have been widely applied in pathology research and analysis. Artificial neural networks can automatically extract and encode the characteristic information of pathology images and use this pathological information for efficient downstream learning tasks. Generative deep learning plays an important role in the acquisition of pathology images. It can generate images from other domains based on existing input data. For example, it can utilize readily available H&E images and encode them through artificial neural networks to obtain their pathological features. Based on the pathological information contained in these features, mIHC images can be generated. This avoids the complex chemical staining process and addresses the difficulties of acquisition and application, as well as the inaccurate staining. However, the application of generative deep learning also faces challenges, including the accuracy and generalization of pathology image feature encoding, as well as the consumption of computational resources and time. Therefore, when using generative deep learning for mIHC image generation, it is necessary to select appropriate technical methods and strategies to improve the effectiveness and efficiency of mIHC image generation. Summary of the Invention

[0005] To address the difficulties in acquiring mIHC images and aligning them with H&E images, the present invention aims to provide a multiple immunohistochemistry image generation method that can quickly and accurately acquire mIHC images and generate strictly aligned mIHC and H&E images.

[0006] To achieve the above object, the present invention adopts the following technical solution: a method for generating multiple immunohistochemical images, the method comprising the following steps in order:

[0007] (1) Obtain the H&E dataset and the paired mIHC dataset to form a dataset and preprocess it, then divide the preprocessed dataset into a training set and a test set;

[0008] (2) constructing an mIHC generative model, the mIHC generative model comprising an mIHC image generator network and an mIHC image discriminator network; the mIHC image generator network comprising a pathology basis model encoder, a local feature encoder, and a decoder with an attention gate short connection; the first, second, third, fourth, and fifth output layers of the local feature encoder are connected to the third, fourth, fifth, sixth, and seventh input layers of the decoder with an attention gate short connection, respectively;

[0009] (3) training the mIHC generation model to obtain a trained mIHC generation model;

[0010] (4) The H&E images in the test set are input into the trained mIHC generation model to obtain the generated mIHC image results, and the generated results of the mIHC images are visualized.

[0011] In step (1), the preprocessing specifically refers to: registering the WSI data of the H&E images in the H&E dataset and the WSI data of the mIHC images in the mIHC dataset using the Valis method, then downsampling the WSI data of the H&E images and the WSI data of the mIHC images by a factor of four, and then cutting them into image pairs of size 128×128.

[0012] In step (2), the pathology basic model encoder adds a reorganization layer and a projection layer on the basis of the PLIP pathology basic model image encoder; after feature extraction and encoding by the multi-head self-attention module, the encoder enters the reorganization layer, and the reorganization layer first removes the first feature representing the category code in the first dimension of the feature matrix obtained by the multi-head self-attention module to obtain a feature matrix with a scale of (49, 768), and then reorganizes the 49 features of the first dimension according to the spatial position to obtain a feature matrix with a size of (7, 7, 768), and then adjusts the third dimension to the front of the first dimension to obtain a feature matrix with a size of (768, 7, 7); enters the projection layer, and the projection layer obtains a feature matrix with a scale of (512, 7, 7) by convolution, the convolution kernel size is 3×3, the step size is 1, and the output channel is 512; the output is the feature matrix obtained by the projection layer.

[0013] In step (2), the local feature encoder includes:

[0014] The first input layer uses a feature matrix with a scale of (3, 128, 128).

[0015] The first convolutional layer convolves the input feature matrix to obtain a feature matrix with a scale of (64, 128, 128);

[0016] The first activation layer uses the nonlinear ReLU to transform the output feature matrix of the first convolutional layer to obtain a feature matrix with a scale of (64, 128, 128);

[0017] The first downsampling layer downsamples the feature matrix obtained by the first activation layer through maximum pooling to obtain a feature matrix of scale (64, 64, 64);

[0018] The second convolutional layer convolves the feature matrix obtained by the first downsampling layer to obtain a feature matrix with a scale of (128, 64, 64);

[0019] The second activation layer uses the nonlinear ReLU to transform the output feature matrix of the second convolutional layer to obtain a feature matrix with a scale of (128, 64, 64);

[0020] The second downsampling layer downsamples the feature matrix obtained by the second activation layer through maximum pooling to obtain a feature matrix with a scale of (128, 32, 32);

[0021] The third convolutional layer performs a convolution operation on the feature matrix obtained by the second downsampling layer to obtain a feature matrix with a scale of (256, 32, 32);

[0022] The third activation layer uses the nonlinear ReLU to transform the output feature matrix of the third convolutional layer to obtain a feature matrix with a scale of (256, 32, 32);

[0023] The third downsampling layer downsamples the feature matrix obtained by the third activation layer through maximum pooling to obtain a feature matrix with a scale of (256, 16, 16);

[0024] The fourth convolutional layer convolves the feature matrix obtained from the third downsampling layer to obtain a feature matrix with a scale of (512, 16, 16);

[0025] The fourth activation layer uses the nonlinear ReLU to transform the output feature matrix of the fourth convolutional layer to obtain a feature matrix with a scale of (512, 16, 16);

[0026] The fourth downsampling layer downsamples the feature matrix obtained by the fourth activation layer through maximum pooling to obtain a feature matrix with a scale of (512, 8, 8);

[0027] The fifth convolutional layer convolves the feature matrix obtained by the fourth downsampling layer to obtain a feature matrix of scale (512, 8, 8);

[0028] The fifth activation layer uses the nonlinear ReLU to transform the output feature matrix of the fifth convolutional layer to obtain a feature matrix with a scale of (512, 8, 8);

[0029] The first output layer outputs the feature matrix of scale (64, 128, 128) obtained by the first activation layer;

[0030] The second output layer outputs the feature matrix of scale (128, 64, 64) obtained by the second activation layer;

[0031] The third output layer outputs the feature matrix of scale (256, 32, 32) obtained by the third activation layer;

[0032] The fourth output layer outputs the feature matrix of scale (512, 16, 16) obtained by the fourth activation layer;

[0033] The fifth output layer outputs the feature matrix of scale (512, 8, 8) obtained by the fifth activation layer.

[0034] In step (2), the decoder with attention gate short connection includes:

[0035] The second input layer is the feature matrix of scale (512, 7, 7) output by the encoder network of the pathology base model;

[0036] The third input layer is the feature matrix of scale (512, 8, 8) output by the fifth output layer of the local feature encoder;

[0037] The fourth input layer is the feature matrix of the scale (512, 16, 16) output by the fourth output layer of the local feature encoder;

[0038] The fifth input layer is the feature matrix of the scale (256, 32, 32) output by the third output layer of the local feature encoder;

[0039] The sixth input layer is the feature matrix of the scale (128, 64, 64) output by the second output layer of the local feature encoder;

[0040] The seventh input layer is the feature matrix of the scale (64, 128, 128) output by the first output layer of the local feature encoder;

[0041] The first upsampling layer upsamples the feature matrix of the second input layer to obtain a feature matrix with a scale of (512, 16, 16);

[0042] The second upsampling layer upsamples the feature matrix of the third input layer to obtain a feature matrix with a scale of (512, 16, 16);

[0043] The first attention gate operates the feature matrix obtained by the fourth input layer and the second upsampling layer through the attention gate to obtain a feature matrix of size (512, 16, 16);

[0044] The first concatenation layer concatenates the feature matrix obtained by the first upsampling layer and the feature matrix obtained by the first attention gate in the first dimension to obtain a feature matrix with a scale of (1024, 16, 16);

[0045] The sixth convolutional layer convolves the feature matrix obtained from the first concatenation layer to obtain a feature matrix with a scale of (512, 16, 16);

[0046] The sixth activation layer uses the nonlinear ReLU transformation on the output matrix of the sixth convolutional layer to obtain a feature matrix of size (512, 16, 16);

[0047] The third upsampling layer upsamples the feature matrix obtained in the sixth activation layer to obtain a feature matrix with a scale of (512, 32, 32);

[0048] The second attention gate operates the feature matrix obtained by the fifth input layer and the third upsampling layer through the attention gate to obtain a feature matrix of size (256, 32, 32);

[0049] In the second concatenation layer, the feature matrix obtained by the third upsampling layer and the feature matrix obtained by the second attention gate are concatenated in the first dimension to obtain a feature matrix with a scale of (768, 32, 32);

[0050] In the seventh convolutional layer, the feature matrix obtained in the second concatenation layer is convolved to obtain a feature matrix with a scale of (256, 32, 32);

[0051] The seventh activation layer uses the nonlinear ReLU transformation on the output matrix of the seventh convolutional layer to obtain a feature matrix of size (256, 32, 32);

[0052] The fourth upsampling layer upsamples the feature matrix obtained by the seventh activation layer to obtain a feature matrix with a scale of (256, 64, 64);

[0053] The third attention gate operates the feature matrix obtained by the sixth input layer and the fourth upsampling layer through the attention gate to obtain a feature matrix of size (128, 64, 64);

[0054] The third splicing layer concatenates the feature matrix obtained by the fourth upsampling layer and the feature matrix obtained by the third attention gate in the first dimension to obtain a feature matrix with a scale of (384, 64, 64);

[0055] The eighth convolutional layer convolves the feature matrix obtained in the third concatenation layer to obtain a feature matrix of scale (128, 64, 64);

[0056] The eighth activation layer uses the nonlinear ReLU transformation on the output matrix of the eighth convolutional layer to obtain a feature matrix of size (128, 64, 64);

[0057] The fifth upsampling layer upsamples the feature matrix obtained in the eighth activation layer to obtain a feature matrix with a scale of (128, 128, 128);

[0058] The fourth attention gate operates the feature matrix obtained by the seventh input layer and the fifth upsampling layer through the attention gate to obtain a feature matrix of size (64, 128, 128);

[0059] The fourth concatenation layer concatenates the feature matrix obtained by the fifth upsampling layer and the feature matrix obtained by the fourth attention gate in the first dimension to obtain a feature matrix with a scale of (192, 128, 128);

[0060] The ninth convolutional layer convolves the feature matrix obtained in the fourth concatenation layer to obtain a feature matrix of scale (64, 128, 128);

[0061] The ninth activation layer uses the nonlinear ReLU transformation on the output matrix of the ninth convolutional layer to obtain a feature matrix of size (64, 128, 128);

[0062] The tenth convolutional layer convolves the feature matrix obtained in the ninth activation layer to obtain a feature matrix of scale (3, 128, 128);

[0063] The tenth activation layer, the output matrix of the tenth convolutional layer is transformed into a feature matrix of size (3, 128, 128) through the nonlinear transformation ReLU;

[0064] The sixth output layer outputs the feature matrix obtained by the tenth activation layer;

[0065] The first upsampling layer is upsampled by transposed convolution with a convolution kernel size of 3×3, a stride of 2, an output padding of 1, and an output channel of 512. The second, third, fourth, and fifth upsampling layers are implemented by bilinear interpolation upsampling.

[0066] In step (2), the mIHC image discriminator network is constructed based on the PatchGAN network. An upsampling module is added before the output layer of the PatchGAN network, and a feature matrix of size (1, 128, 128) is obtained by bilinear interpolation upsampling.

[0067] Step (3) specifically includes the following steps in order:

[0068] (3a) Input the H&E images in the training set into the pathology-based model encoder to obtain the feature matrix P;

[0069] (3b) Input the H&E images in the training set into the local feature encoder network to obtain 5 output feature matrices L1, L2, L3, L4, and L5 respectively;

[0070] (3c) Input the feature matrix P and feature matrices L1, L2, L3, L4, and L5 into the decoder with attention gate short connections to obtain the feature matrix Q;

[0071] (3d) Input the mIHC images in the training set into the mIHC discriminator network to obtain the feature matrix C; input the feature matrix Q into the mIHC discriminator network to obtain the feature matrix H;

[0072] (3e) Construct the supervised loss in the total loss function based on the mIHC image and the feature matrix Q, namely the frequency loss and the downsampling loss; construct the unsupervised loss in the total loss, namely the discriminant loss, based on the feature matrix C and the feature matrix H;

[0073] The formula for the total loss function is:

[0074] L Total (G,D)=L GAN (G,D)+L FR (G)·+L Down (G)

[0075] Among them, G represents the mIHC image generator network, D represents the mIHC image discriminator network; L GAN represents the discrimination loss; L FR Indicates frequency loss; L Down represents the downsampling loss;

[0076] The expression of the discriminant loss is:

[0077] L GAN (G,D)=E x,y [logD(x,y)]+E x,n [log(1-D(x,G(x,n)))]

[0078] Where x, y, and n represent H&E images, real mIHC images, and random noise, respectively; E x,y Represents the mathematical expectation of the joint distribution of x and y; E x,n Represents the mathematical expectation of the joint distribution of x and n;

[0079] The expression for frequency loss is:

[0080] L FR (G) = w1·E x,y,n [||F High (y)-F High (G(x,n))||1]+w2·E x,y,n [||F Low (y)-F Low (G(x,n))||1]

[0081] Among them, F High (·)=IFFT[HPA[FFT(·)]],F Low (·)=IFFT[LPA[FFT(·)]], where FFT and IFFT represent Fourier transform and inverse Fourier transform, respectively, HPA and LPA represent high-pass filter and low-pass filter, respectively; w1 and w2 represent high-frequency weight and low-frequency weight, respectively, w1 is set to 1 and w2 is set to 2; ||·||1 represents the L1 norm; E x,y,n Represents the mathematical expectation of the joint distribution of x, y, and n;

[0082] The expression of downsampling loss is:

[0083]

[0084] Among them, α j Represents the loss weight after the jth downsampling, SampD j Indicates a scale transformation after j times of downsampling;

[0085] Each scale transformation includes four Gaussian filters and one downsampling using a Gaussian kernel with a mean of 0 and a variance of 1;

[0086] (3f) All parameters of the mIHC generation model are updated through the back-propagation mechanism according to the total loss function;

[0087] (3g) Repeat steps (3a) to (3f) until the training is completed and save the final model parameter weights;

[0088] (3h) Load the final model parameter weights and use the test set to test the performance of the mIHC generation model.

[0089] As can be seen from the above technical solutions, the present invention has the following beneficial effects: First, the present invention can quickly and accurately acquire mIHC images using readily available H&E images through the mIHC generation model, avoiding the traditional chemical staining process, reducing the cost of acquiring mIHC images, and providing a convenient and accurate tool for acquiring mIHC images, thereby providing pathological information such as cancer, shortening the diagnosis and analysis cycle of diseases, and facilitating timely diagnosis and treatment of patients; Second, the mIHC images generated by the present invention using the H&E images through the mIHC generation model are strictly aligned with the H&E images themselves, so the H&E images and mIHC images can be accurately combined for disease diagnosis, thereby improving the accuracy of disease diagnosis and analysis; Third, the present invention introduces a pathology foundation model for feature extraction and encoding, capable of extracting high-quality global pathology features, and constructs a fully convolutional neural network-based local pathology feature extraction. By combining global and local pathology features, high-quality pathology image encoding is extracted, thereby improving the generation quality of mIHC images. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] Figure 1 is a flow chart of the method of the present invention;

[0091] Figure 2 Schematic diagram of the mIHC generation model of the present invention;

[0092] Figure 3 for Figure 2 Schematic diagram of the mIHC image generator network;

[0093] Figure 4 for Figure 3 Schematic diagram of the structure of the encoder of the pathological basis model;

[0094] Figure 5 for Figure 3 Schematic diagram of the structure of the local feature encoder;

[0095] Figure 6 for Figure 3 Schematic diagram of the decoder with short-circuit attention gates;

[0096] Figure 7 for Figure 2 Schematic diagram of the structure of the mIHC image discriminator network;

[0097] Figure 8 This is a visualization example diagram of the present invention;

[0098] Figure 9 It is the evaluation index of the present invention in the mIHC image generation task. DETAILED DESCRIPTION

[0099] like Figure 1 As shown, a method for generating multiple immunohistochemical images comprises the following steps in order:

[0100] (1) Obtain the H&E dataset and the paired mIHC dataset to form a dataset and preprocess it, then divide the preprocessed dataset into a training set and a test set;

[0101] (2) Constructing an mIHC generation model, such as Figure 2 As shown in Figure 2, the mIHC generation model includes an mIHC image generator network and an mIHC image discriminator network; Figure 3 As shown, the mIHC image generator network includes a pathology base model encoder, a local feature encoder, and a decoder with an attention gate short connection; the first, second, third, fourth, and fifth output layers of the local feature encoder are connected to the third, fourth, fifth, sixth, and seventh input layers of the decoder with an attention gate short connection respectively;

[0102] (3) training the mIHC generation model to obtain a trained mIHC generation model;

[0103] (4) The H&E images in the test set are input into the trained mIHC generation model to obtain the generated mIHC image results, and the generated results of the mIHC images are visualized.

[0104] In step (1), the preprocessing specifically refers to: registering the WSI data of the H&E images in the H&E dataset and the WSI data of the mIHC images in the mIHC dataset using the Valis method, then downsampling the WSI data of the H&E images and the WSI data of the mIHC images by a factor of four, and then cutting them into image pairs of size 128×128.

[0105] In step (2), if Figure 4 As shown, the pathology base model encoder adds a reorganization layer and a projection layer to the PLIP pathology base model image encoder. After feature extraction and encoding through the multi-head self-attention module, the encoder enters the reorganization layer. The reorganization layer first removes the first feature representing the category code in the first dimension of the feature matrix obtained by the multi-head self-attention module, resulting in a feature matrix of size (49, 768). The 49 features in the first dimension are then reorganized by spatial position to obtain a feature matrix of size (7, 7, 768). The third dimension is then adjusted to the front of the first dimension to obtain a feature matrix of size (768, 7, 7). The encoder enters the projection layer, which obtains a feature matrix of size (512, 7, 7) through convolution. The convolution kernel size is 3×3, the stride is 1, and the output channel is 512. The output is the feature matrix obtained by the projection layer. The pathology base model encoder reorganizes the extracted features by spatial position before output and reduces the feature dimension by changing the output dimension from 768 to 512 through the convolution layer.

[0106] In step (2), if Figure 5 As shown, the local feature encoder includes:

[0107] The first input layer uses a feature matrix with a scale of (3, 128, 128).

[0108] The first convolutional layer Conv1 convolves the input feature matrix to obtain a feature matrix of scale (64, 128, 128);

[0109] The first activation layer, the output feature matrix of the first convolutional layer Conv1 is transformed into a feature matrix of scale (64, 128, 128) by nonlinear transformation ReLU, i.e. ReLU1;

[0110] The first downsampling layer Down1 downsamples the feature matrix obtained by the first activation layer through maximum pooling to obtain a feature matrix of scale (64, 64, 64);

[0111] The second convolutional layer Conv2 convolves the feature matrix obtained by the first downsampling layer Down1 to obtain a feature matrix with a scale of (128, 64, 64);

[0112] The second activation layer is to transform the output feature matrix of the second convolutional layer Conv2 through the nonlinear transformation ReLU, i.e. ReLU2, to obtain a feature matrix with a scale of (128, 64, 64);

[0113] The second downsampling layer Down2 downsamples the feature matrix obtained by the second activation layer through maximum pooling to obtain a feature matrix with a scale of (128, 32, 32);

[0114] The third convolutional layer Conv3 convolves the feature matrix obtained by the second downsampling layer Down2 to obtain a feature matrix with a scale of (256, 32, 32);

[0115] The third activation layer, the output feature matrix of the third convolutional layer Conv3 is transformed into a feature matrix of scale (256, 32, 32) by nonlinear transformation ReLU, i.e. ReLU3;

[0116] The third downsampling layer Down3 downsamples the feature matrix obtained by the third activation layer through maximum pooling to obtain a feature matrix with a scale of (256, 16, 16);

[0117] The fourth convolutional layer Conv4 convolves the feature matrix obtained by the third downsampling layer Down3 to obtain a feature matrix with a scale of (512, 16, 16);

[0118] The fourth activation layer is to transform the output feature matrix of the fourth convolutional layer Conv4 through the nonlinear transformation ReLU, i.e. ReLU4, to obtain a feature matrix with a scale of (512, 16, 16);

[0119] The fourth downsampling layer Down4 downsamples the feature matrix obtained by the fourth activation layer through maximum pooling to obtain a feature matrix with a scale of (512, 8, 8);

[0120] The fifth convolutional layer Conv5 convolves the feature matrix obtained by the fourth downsampling layer Down4 to obtain a feature matrix of scale (512, 8, 8);

[0121] The fifth activation layer, the output feature matrix of the fifth convolutional layer Conv5 is transformed into a feature matrix of scale (512, 8, 8) by nonlinear transformation ReLU, i.e. ReLU5;

[0122] The first output layer outputs the feature matrix of scale (64, 128, 128) obtained by the first activation layer;

[0123] The second output layer outputs the feature matrix of scale (128, 64, 64) obtained by the second activation layer;

[0124] The third output layer outputs the feature matrix of scale (256, 32, 32) obtained by the third activation layer;

[0125] The fourth output layer outputs the feature matrix of scale (512, 16, 16) obtained by the fourth activation layer;

[0126] The fifth output layer outputs the feature matrix of scale (512, 8, 8) obtained by the fifth activation layer.

[0127] In step (2), if Figure 6 As shown, the decoder with attention gate short connection includes:

[0128] The second input layer is the feature matrix of scale (512, 7, 7) output by the encoder network of the pathology base model;

[0129] The third input layer is the feature matrix of scale (512, 8, 8) output by the fifth output layer of the local feature encoder;

[0130] The fourth input layer is the feature matrix of the scale (512, 16, 16) output by the fourth output layer of the local feature encoder;

[0131] The fifth input layer is the feature matrix of the scale (256, 32, 32) output by the third output layer of the local feature encoder;

[0132] The sixth input layer is the feature matrix of the scale (128, 64, 64) output by the second output layer of the local feature encoder;

[0133] The seventh input layer is the feature matrix of the scale (64, 128, 128) output by the first output layer of the local feature encoder;

[0134] The first upsampling layer Up1 upsamples the feature matrix of the second input layer to obtain a feature matrix with a scale of (512, 16, 16);

[0135] The second upsampling layer Up2 upsamples the feature matrix of the third input layer to obtain a feature matrix with a scale of (512, 16, 16);

[0136] The first attention gate A1 operates the feature matrix obtained by the fourth input layer and the second upsampling layer Up2 through the attention gate to obtain a feature matrix of size (512, 16, 16);

[0137] The first concatenation layer C1 concatenates the feature matrix obtained by the first upsampling layer Up1 and the feature matrix obtained by the first attention gate A1 in the first dimension to obtain a feature matrix with a scale of (1024, 16, 16);

[0138] The sixth convolutional layer Conv6 performs a convolution operation on the feature matrix obtained by the first concatenation layer C1 to obtain a feature matrix of scale (512, 16, 16);

[0139] The sixth activation layer, the output matrix of the sixth convolutional layer Conv6 is transformed into a feature matrix of size (512, 16, 16) through the nonlinear transformation ReLU, i.e. ReLU6;

[0140] The third upsampling layer Up3 upsamples the feature matrix obtained in the sixth activation layer to obtain a feature matrix with a scale of (512, 32, 32);

[0141] The second attention gate A2 operates the feature matrix obtained by the fifth input layer and the third upsampling layer Up3 through the attention gate to obtain a feature matrix of size (256, 32, 32);

[0142] The second concatenation layer C2 concatenates the feature matrix obtained by the third upsampling layer Up3 and the feature matrix obtained by the second attention gate A2 in the first dimension to obtain a feature matrix with a scale of (768, 32, 32);

[0143] The seventh convolutional layer Conv7 performs a convolution operation on the feature matrix obtained by the second splicing layer C2 to obtain a feature matrix of scale (256, 32, 32);

[0144] The seventh activation layer, the output matrix of the seventh convolutional layer Conv7 is transformed into a feature matrix of size (256, 32, 32) by nonlinear transformation ReLU, i.e. ReLU7;

[0145] The fourth upsampling layer Up4 upsamples the feature matrix obtained by the seventh activation layer to obtain a feature matrix with a scale of (256, 64, 64);

[0146] The third attention gate A3 operates the feature matrix obtained by the sixth input layer and the fourth upsampling layer Up4 through the attention gate to obtain a feature matrix of size (128, 64, 64);

[0147] The third concatenation layer C3 concatenates the feature matrix obtained by the fourth upsampling layer Up4 and the feature matrix obtained by the third attention gate A3 in the first dimension to obtain a feature matrix with a scale of (384, 64, 64);

[0148] The eighth convolutional layer Conv8 performs a convolution operation on the feature matrix obtained by the third splicing layer C3 to obtain a feature matrix of scale (128, 64, 64);

[0149] The eighth activation layer, the output matrix of the eighth convolutional layer Conv8 is transformed into a feature matrix of size (128, 64, 64) by nonlinear transformation ReLU, i.e. ReLU8;

[0150] The fifth upsampling layer Up5 upsamples the feature matrix obtained from the eighth activation layer to obtain a feature matrix with a scale of (128, 128, 128);

[0151] The fourth attention gate A4 operates the feature matrix obtained by the seventh input layer and the fifth upsampling layer Up5 through the attention gate to obtain a feature matrix of size (64, 128, 128);

[0152] The fourth concatenation layer C4 concatenates the feature matrix obtained by the fifth upsampling layer Up5 and the feature matrix obtained by the fourth attention gate A4 in the first dimension to obtain a feature matrix with a scale of (192, 128, 128);

[0153] The ninth convolutional layer Conv9 performs a convolution operation on the feature matrix obtained by the fourth splicing layer C4 to obtain a feature matrix of scale (64, 128, 128);

[0154] The ninth activation layer, the output matrix of the ninth convolutional layer Conv9 is transformed into a feature matrix of size (64, 128, 128) by nonlinear transformation ReLU, i.e. ReLU9;

[0155] The tenth convolutional layer Conv10 convolves the feature matrix obtained by the ninth activation layer to obtain a feature matrix of scale (3, 128, 128);

[0156] The tenth activation layer, the output matrix of the tenth convolutional layer Conv10 is transformed into a feature matrix of size (3, 128, 128) through the nonlinear transformation ReLU, i.e. ReLU10;

[0157] The sixth output layer outputs the feature matrix obtained by the tenth activation layer;

[0158] The first upsampling layer Up1 is upsampled by transposed convolution with a convolution kernel size of 3×3, a stride of 2, an output padding of 1, and an output channel of 512. The second upsampling layer Up2, the third upsampling layer Up3, the fourth upsampling layer Up4, and the fifth upsampling layer Up5 are implemented by bilinear interpolation upsampling.

[0159] In step (2), if Figure 7 As shown in FIG, the mIHC image discriminator network is constructed based on the PatchGAN network. An upsampling module is added before the output layer of the PatchGAN network, and a feature matrix of size (1, 128, 128) is obtained by bilinear interpolation upsampling.

[0160] Step (3) specifically includes the following steps in order:

[0161] (3a) Input the H&E images in the training set into the pathology-based model encoder to obtain the feature matrix P;

[0162] (3b) Input the H&E images in the training set into the local feature encoder network to obtain 5 output feature matrices L1, L2, L3, L4, and L5 respectively;

[0163] (3c) Input the feature matrix P and feature matrices L1, L2, L3, L4, and L5 into the decoder with attention gate short connections to obtain the feature matrix Q;

[0164] (3d) Input the mIHC images in the training set into the mIHC discriminator network to obtain the feature matrix C; input the feature matrix Q into the mIHC discriminator network to obtain the feature matrix H;

[0165] (3e) Construct the supervised loss in the total loss function based on the mIHC image and the feature matrix Q, namely the frequency loss and the downsampling loss; construct the unsupervised loss in the total loss, namely the discriminant loss, based on the feature matrix C and the feature matrix H;

[0166] The formula for the total loss function is:

[0167] L Total (G,D)=L GAN (G,D)+L FR (G)+L Down (G)

[0168] Among them, G represents the mIHC image generator network, D represents the mIHC image discriminator network; L GAN represents the discrimination loss; L FR Indicates frequency loss; L Down represents the downsampling loss;

[0169] The expression of the discriminant loss is:

[0170] L GAN (G,D)=E x,y [logD(x,y)]+E x,n [log(1-D(x,G(x,n)))]

[0171] Where x, y, and n represent H&E images, real mIHC images, and random noise, respectively; E x,y Represents the mathematical expectation of the joint distribution of x and y; E x,n Represents the mathematical expectation of the joint distribution of x and n;

[0172] The expression for frequency loss is:

[0173] L FR (G) = w1·E x,y,n [||F High (y)-F High (G(x,n))||1]+w2·E x,y,n [||F Low (y)-F Low (G(x,n))||1]

[0174] Among them, F High (·)=IFFT[HPA[FFT(·)]],F Low (·)=IFFT[LPA[FFT(·)]], where FFT and IFFT represent Fourier transform and inverse Fourier transform, respectively, HPA and LPA represent high-pass filter and low-pass filter, respectively; w1 and w2 represent high-frequency weight and low-frequency weight, respectively, w1 is set to 1 and w2 is set to 2; ||·||1 represents the L1 norm; E x,y,n Represents the mathematical expectation of the joint distribution of x, y, and n;

[0175] The expression of downsampling loss is:

[0176]

[0177] Among them, α j Represents the loss weight after the jth downsampling, SampD j Indicates a scale transformation after j times of downsampling;

[0178] Each scale transformation includes four Gaussian filters and one downsampling using a Gaussian kernel with a mean of 0 and a variance of 1;

[0179] (3f) All parameters of the mIHC generation model are updated through the back-propagation mechanism according to the total loss function;

[0180] (3g) Repeat steps (3a) to (3f) until the training is completed and save the final model parameter weights;

[0181] (3h) Load the final model parameter weights and use the test set to test the performance of the mIHC generation model.

[0182] The mIHC image generation results are color mapped to visualize the expression of different molecular markers. The visualization example is as follows Figure 8 As shown, Figure 8 In the figure, (a) is the H&E image, (b) is the real mIHC image, and (c) is the generated mIHC image.

[0183] like Figure 9 As shown in the figure, the multi-immunohistochemistry image generation network based on the combination of a pathology-based model and a generative model can quickly and accurately generate mIHC images in the mIHC image generation task, and has high evaluation metrics. PSNR reflects the degree of loss of the original signal, SNR reflects the ratio between signal power and noise power, and SSIM comprehensively evaluates the image based on visual perception, considering brightness, contrast, and structure. All three metrics are important for image evaluation, and higher values ​​indicate better image quality.

[0184] In summary, the present invention utilizes readily available H&E images through an mIHC generation model to quickly and accurately acquire mIHC images, avoiding the traditional chemical staining process and reducing the cost of acquiring mIHC images. This provides a convenient and accurate tool for acquiring mIHC images, thereby providing pathological information such as cancer and shortening the diagnosis and analysis cycle of diseases, facilitating timely diagnosis and treatment of patients. The mIHC images generated by the present invention using the H&E image generation model are strictly registered with the H&E image, allowing accurate combination of H&E and mIHC images for disease diagnosis, thereby improving the accuracy of disease diagnosis and analysis. The present invention introduces a pathology-based model for feature extraction and encoding, capable of extracting high-quality global pathology features. A fully convolutional neural network-based approach is also constructed for extracting local pathology features. High-quality pathology image encoding is extracted by combining global and local pathology features, thereby improving the quality of mIHC image generation.

Claims

1. A method for generating multiple immunohistochemical images, characterized by: The method comprises the following steps in sequence: (1) Obtain the H&E dataset and the paired mIHC dataset to form a dataset and preprocess it, then divide the preprocessed dataset into a training set and a test set; In step (1), the preprocessing specifically refers to: registering the WSI data of the H&E images in the H&E dataset and the WSI data of the mIHC images in the mIHC dataset using the Valis method, then downsampling the WSI data of the H&E images and the WSI data of the mIHC images by a factor of four, and then cutting them into image pairs of size 128×128; (2) constructing an mIHC generative model, the mIHC generative model comprising an mIHC image generator network and an mIHC image discriminator network; the mIHC image generator network comprising a pathology basis model encoder, a local feature encoder, and a decoder with an attention gate short connection; the first, second, third, fourth, and fifth output layers of the local feature encoder are connected to the third, fourth, fifth, sixth, and seventh input layers of the decoder with an attention gate short connection, respectively; In step (2), the pathology basic model encoder adds a reorganization layer and a projection layer on the basis of the PLIP pathology basic model image encoder; after the multi-head self-attention module performs feature extraction and encoding, the reorganization layer is entered, and the reorganization layer first removes the first feature representing the category code in the first dimension of the feature matrix obtained by the multi-head self-attention module to obtain a feature matrix with a scale of (49, 768), and then reorganizes the 49 features of the first dimension according to the spatial position to obtain a feature matrix of (7, 7, 768) size, and then adjusts the third dimension to the front of the first dimension to obtain a feature matrix of (768, 7, 7) size; enters the projection layer, and the projection layer obtains a feature matrix of a scale of (512, 7, 7) size by convolution, the convolution kernel size is 3×3, the step size is 1, and the output channel is 512; the output is the feature matrix obtained by the projection layer; (3) training the mIHC generation model to obtain a trained mIHC generation model; (4) The H&E images in the test set are input into the trained mIHC generation model to obtain the generated mIHC image results, and the generated results of the mIHC images are visualized.

2. The method for generating multiple immunohistochemical images according to claim 1, wherein: In step (2), the local feature encoder includes: The first input layer uses a feature matrix with a scale of (3, 128, 128). The first convolutional layer convolves the input feature matrix to obtain a feature matrix with a scale of (64, 128, 128); The first activation layer uses the nonlinear ReLU to transform the output feature matrix of the first convolutional layer to obtain a feature matrix with a scale of (64, 128, 128); The first downsampling layer downsamples the feature matrix obtained by the first activation layer through maximum pooling to obtain a feature matrix of scale (64, 64, 64); The second convolutional layer convolves the feature matrix obtained by the first downsampling layer to obtain a feature matrix with a scale of (128, 64, 64); The second activation layer uses the nonlinear ReLU to transform the output feature matrix of the second convolutional layer to obtain a feature matrix with a scale of (128, 64, 64); The second downsampling layer downsamples the feature matrix obtained by the second activation layer through maximum pooling to obtain a feature matrix with a scale of (128, 32, 32); The third convolutional layer performs a convolution operation on the feature matrix obtained by the second downsampling layer to obtain a feature matrix with a scale of (256, 32, 32); The third activation layer uses the nonlinear ReLU to transform the output feature matrix of the third convolutional layer to obtain a feature matrix with a scale of (256, 32, 32); The third downsampling layer downsamples the feature matrix obtained by the third activation layer through maximum pooling to obtain a feature matrix with a scale of (256, 16, 16); The fourth convolutional layer convolves the feature matrix obtained from the third downsampling layer to obtain a feature matrix with a scale of (512, 16, 16); The fourth activation layer uses the nonlinear ReLU to transform the output feature matrix of the fourth convolutional layer to obtain a feature matrix with a scale of (512, 16, 16); The fourth downsampling layer downsamples the feature matrix obtained by the fourth activation layer through maximum pooling to obtain a feature matrix with a scale of (512, 8, 8); The fifth convolutional layer convolves the feature matrix obtained by the fourth downsampling layer to obtain a feature matrix of scale (512, 8, 8); The fifth activation layer uses the nonlinear ReLU to transform the output feature matrix of the fifth convolutional layer to obtain a feature matrix with a scale of (512, 8, 8); The first output layer outputs the feature matrix of scale (64, 128, 128) obtained by the first activation layer; The second output layer outputs the feature matrix of scale (128, 64, 64) obtained by the second activation layer; The third output layer outputs the feature matrix of scale (256, 32, 32) obtained by the third activation layer; The fourth output layer outputs the feature matrix of scale (512, 16, 16) obtained by the fourth activation layer; The fifth output layer outputs the feature matrix of scale (512, 8, 8) obtained by the fifth activation layer.

3. The method for generating multiple immunohistochemical images according to claim 1, wherein: In step (2), the decoder with attention gate short connection includes: The second input layer is the feature matrix of scale (512, 7, 7) output by the encoder network of the pathology base model; The third input layer is the feature matrix of scale (512, 8, 8) output by the fifth output layer of the local feature encoder; The fourth input layer is the feature matrix of the scale (512, 16, 16) output by the fourth output layer of the local feature encoder; The fifth input layer is the feature matrix of the scale (256, 32, 32) output by the third output layer of the local feature encoder; The sixth input layer is the feature matrix of the scale (128, 64, 64) output by the second output layer of the local feature encoder; The seventh input layer is the feature matrix of the scale (64, 128, 128) output by the first output layer of the local feature encoder; The first upsampling layer upsamples the feature matrix of the second input layer to obtain a feature matrix with a scale of (512, 16, 16); The second upsampling layer upsamples the feature matrix of the third input layer to obtain a feature matrix with a scale of (512, 16, 16); The first attention gate operates the feature matrix obtained by the fourth input layer and the second upsampling layer through the attention gate to obtain a feature matrix of size (512, 16, 16); The first concatenation layer concatenates the feature matrix obtained by the first upsampling layer and the feature matrix obtained by the first attention gate in the first dimension to obtain a feature matrix with a scale of (1024, 16, 16); The sixth convolutional layer convolves the feature matrix obtained from the first concatenation layer to obtain a feature matrix with a scale of (512, 16, 16); The sixth activation layer uses the nonlinear ReLU transformation on the output matrix of the sixth convolutional layer to obtain a feature matrix of size (512, 16, 16); The third upsampling layer upsamples the feature matrix obtained in the sixth activation layer to obtain a feature matrix with a scale of (512, 32, 32); The second attention gate operates the feature matrix obtained by the fifth input layer and the third upsampling layer through the attention gate to obtain a feature matrix of size (256, 32, 32); In the second concatenation layer, the feature matrix obtained by the third upsampling layer and the feature matrix obtained by the second attention gate are concatenated in the first dimension to obtain a feature matrix with a scale of (768, 32, 32); In the seventh convolutional layer, the feature matrix obtained in the second concatenation layer is convolved to obtain a feature matrix with a scale of (256, 32, 32); The seventh activation layer uses the nonlinear ReLU transformation on the output matrix of the seventh convolutional layer to obtain a feature matrix of size (256, 32, 32); The fourth upsampling layer upsamples the feature matrix obtained by the seventh activation layer to obtain a feature matrix with a scale of (256, 64, 64); The third attention gate operates the feature matrix obtained by the sixth input layer and the fourth upsampling layer through the attention gate to obtain a feature matrix of size (128, 64, 64); The third splicing layer concatenates the feature matrix obtained by the fourth upsampling layer and the feature matrix obtained by the third attention gate in the first dimension to obtain a feature matrix with a scale of (384, 64, 64); The eighth convolutional layer convolves the feature matrix obtained in the third concatenation layer to obtain a feature matrix of scale (128, 64, 64); The eighth activation layer uses the nonlinear ReLU transformation on the output matrix of the eighth convolutional layer to obtain a feature matrix of size (128, 64, 64); The fifth upsampling layer upsamples the feature matrix obtained in the eighth activation layer to obtain a feature matrix with a scale of (128, 128, 128); The fourth attention gate operates the feature matrix obtained by the seventh input layer and the fifth upsampling layer through the attention gate to obtain a feature matrix of size (64, 128, 128); The fourth concatenation layer concatenates the feature matrix obtained by the fifth upsampling layer and the feature matrix obtained by the fourth attention gate in the first dimension to obtain a feature matrix with a scale of (192, 128, 128); The ninth convolutional layer convolves the feature matrix obtained in the fourth concatenation layer to obtain a feature matrix of scale (64, 128, 128); The ninth activation layer uses the nonlinear ReLU transformation on the output matrix of the ninth convolutional layer to obtain a feature matrix of size (64, 128, 128); The tenth convolutional layer convolves the feature matrix obtained in the ninth activation layer to obtain a feature matrix of scale (3, 128, 128); The tenth activation layer, the output matrix of the tenth convolutional layer is transformed into a feature matrix of size (3, 128, 128) through the nonlinear transformation ReLU; The sixth output layer outputs the feature matrix obtained by the tenth activation layer; The first upsampling layer is upsampled by transposed convolution with a convolution kernel size of 3×3, a stride of 2, an output padding of 1, and an output channel of 512. The second, third, fourth, and fifth upsampling layers are implemented by bilinear interpolation upsampling.

4. The method for generating multiple immunohistochemical images according to claim 1, wherein: In step (2), the mIHC image discriminator network is constructed based on the PatchGAN network. An upsampling module is added before the output layer of the PatchGAN network, and a feature matrix of size (1, 128, 128) is obtained by bilinear interpolation upsampling.

5. The method for generating multiple immunohistochemical images according to claim 1, wherein: Step (3) specifically includes the following steps in order: (3a) Input the H&E images in the training set into the pathology-based model encoder to obtain the feature matrix P; (3b) Input the H&E images in the training set into the local feature encoder network to obtain 5 output feature matrices L1, L2, L3, L4, and L5 respectively; (3c) Input the feature matrix P and feature matrices L1, L2, L3, L4, and L5 into the decoder with attention gate short connections to obtain the feature matrix Q; (3d) Input the mIHC images in the training set into the mIHC discriminator network to obtain the feature matrix C; Input the feature matrix Q into the mIHC discriminator network to obtain the feature matrix H; (3e) Construct the supervised loss in the total loss function based on the mIHC image and the feature matrix Q, namely the frequency loss and the downsampling loss; construct the unsupervised loss in the total loss, namely the discriminant loss, based on the feature matrix C and the feature matrix H; The formula for the total loss function is: L Total (G,D)=L GAN (G,D)+L FR (G)+L Down (G) Among them, G represents the mIHC image generator network, D represents the mIHC image discriminator network; L GAN represents the discrimination loss; L FR Indicates frequency loss; L Down represents the downsampling loss; The expression of the discriminant loss is: L GAN (G,D)=E x,y [logD(x,y)]+E x,n [log(1-D(x,G(x,n)))] Where x, y, and n represent H&E images, real mIHC images, and random noise, respectively; E x,y Represents the mathematical expectation of the joint distribution of x and y; E x,n Represents the mathematical expectation of the joint distribution of x and n; The expression for frequency loss is: L FR (G)=w1·E x,y,n [||F High (y)-F High (G(x,n))||1]+ w2·E x,y,n [||F Low (y)-F Low (G(x,n))||1] Among them, F High (·)=IFFT[HPA[FFT(·)]],F Low (·)=IFFT[LPA[FFT(·)]], where FFT and IFFT represent Fourier transform and inverse Fourier transform, respectively, HPA and LPA represent high-pass filter and low-pass filter, respectively; w1 and w2 represent high-frequency weight and low-frequency weight, respectively, w1 is set to 1 and w2 is set to 2; ‖·‖1 represents the L1 norm; E x,y,n Represents the mathematical expectation of the joint distribution of x, y, and n; The expression of downsampling loss is: Among them, α j Represents the loss weight after the jth downsampling, SampD j Indicates a scale transformation after j times of downsampling; Each scale transformation includes four Gaussian filters and one downsampling using a Gaussian kernel with a mean of 0 and a variance of 1; (3f) All parameters of the mIHC generation model are updated through the back-propagation mechanism according to the total loss function; (3g) Repeat steps (3a) to (3f) until the training is completed and save the final model parameter weights; (3h) Load the final model parameter weights and use the test set to test the performance of the mIHC generation model.

Images