Two-stage image generation learning method for cell detection and analysis based on molecular expression prediction

Abstract

The present application relates to a two-stage image generation learning method for cell detection and analysis based on molecular expression prediction, comprising: obtaining H&E data set and paired mIHC data set to form a data set and preprocessing; constructing a cell segmentation model and training; obtaining cell binary segmentation results for visual display; constructing a molecular expression prediction model and training; obtaining molecular expression prediction results for visual display; obtaining cell semantic segmentation results for visual display. The first stage of the present application uses a generative cell segmentation model to extract cell feature information from H&E images, generates more accurate cell binary segmentation results, and improves the accuracy of the model for cell segmentation; the second stage uses a generative molecular expression prediction model to extract pathological information features and generate virtual mIHC images, and the use of the pathological basis model improves the feature extraction capability of the model, thereby improving the acquisition efficiency of the mIHC image and increasing the accuracy of the generated content.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image generation, and particularly relates to a two-stage image generation learning method for cell detection and analysis based on molecular expression prediction. BACKGROUND

[0002] WSI (Whole Slide Image, whole slide image) is a traditional pathological section scanned by a digital scanner to collect high-resolution tissue section images, which contains a large amount of pathological information and is an important tool in cancer research. H&E (Hematoxylin and Eosin, hematoxylin and eosin) staining image is a commonly used digital pathology image, which can be used to help pathologists analyze cancer at the cellular level. Analyzing WSI at the cellular level is an important research topic, which includes accurately identifying the location of cells, distinguishing cell subtypes, and analyzing the expression level of related molecules. These tasks rely on cell segmentation, cell subtype identification, and obtaining the expression level of related molecules, respectively. However, it is difficult, if not impossible, for pathologists to complete these tasks by visual observation alone. Therefore, it is necessary to use H&E images to simultaneously achieve cell binary segmentation, cell semantic segmentation, and molecular expression prediction.

[0003] Deep learning technology has achieved a series of successful applications in histopathology analysis. Through artificial neural networks, models can automatically extract and encode pathological features, and use these features to efficiently and accurately achieve pathological tasks. Unsupervised learning technology can extract effective information from data that does not rely on human annotation, and use this information to perform related pathological analysis and diagnosis tasks. Using unsupervised learning technology for cell segmentation can overcome the problem of requiring a large amount of human-annotated data in general supervised learning tasks and improve the robustness of the model. However, unsupervised learning also has certain difficulties in application, mainly in that the information that can be directly used from unannotated data is limited, and the algorithm design requirements are high. Therefore, when performing cell segmentation through unsupervised learning, it is necessary to fully utilize the prior information in the existing data, and select appropriate technical strategies according to the actual situation, fully extract and encode the effective information in the unannotated data, in order to improve the accuracy and application value of cell segmentation.

[0004] The expression level of relevant molecules in cells plays an important role in the identification of cell subtypes and other related pathological studies. Although mIHC (Multiplex Immunohistochemistry) images can be used to analyze the expression level of relevant molecules, it is generally difficult to obtain mIHC images. The preparation of mIHC images requires a large amount of time and is expensive. At the same time, since the staining process can cause damage to the tissue section, repeated staining cannot be performed on the same section, so it is generally difficult to align the obtained H&E images and mIHC images. SUMMARY

[0005] To solve the problems that it is difficult to accurately classify cell subtypes using pathological images and it is difficult to quickly obtain the expression level of cell molecular markers, the purpose of the present application is to provide a two-stage image generation learning method for cell detection and analysis based on molecular expression prediction, which increases the accuracy of generated content, classifies cell subtypes at the molecular expression level, and increases the accuracy of cell semantic segmentation.

[0006] To achieve the above-mentioned purpose, the present application adopts the following technical solutions: a two-stage image generation learning method for cell detection and analysis based on molecular expression prediction, which comprises the following steps in sequence:

[0007] (1) obtaining an H&E dataset and a paired mIHC dataset to form a dataset and performing preprocessing, and dividing the preprocessed dataset into a training set and a test set;

[0008] (2) constructing a cell segmentation model, wherein the cell segmentation model is composed of a pre-trained cell segmentation network, a cell segmentation generator network, an information integrator, and a discriminator network; the pre-trained cell segmentation network comprises a pre-trained Cellpose network, a pre-trained Hover-Net network, and a pre-trained CellViT network;

[0009] (3) training the cell segmentation model to obtain a trained cell segmentation model;

[0010] (4) inputting the H&E images in the test set into the trained cell segmentation model to obtain cell binary segmentation results, and visualizing and displaying the cell binary segmentation results;

[0011] (5) constructing a molecular expression prediction model, wherein the molecular expression prediction model is composed of a molecular expression prediction generator network and a molecular expression prediction discriminator network;

[0012] (6) training the molecular expression prediction model to obtain a trained molecular expression prediction model;

[0013] (7) inputting the H&E images in the test set into the trained molecular expression prediction model to obtain a molecular expression prediction result, and visualizing the molecular expression prediction result;

[0014] (8) performing one-to-one mapping of the cell binary segmentation result and the molecular expression prediction result at the pixel level in space to obtain a cell semantic segmentation result, and visualizing the cell semantic segmentation result.

[0015] In step (1), the preprocessing specifically refers to: using the valis method to register the WSI data of the H&E images in the H&E dataset and the WSI data of the mIHC images in the mIHC dataset, then performing four times down-sampling on the WSI data of the H&E images and the WSI data of the mIHC images, and cutting them into image pairs of 128x128 size.

[0016] In step (2), the cell segmentation generator network adopts the Attention U-Net architecture, the depth of the Attention U-Net is 5, the input feature channel and the output feature channel are both 3, and the feature channels are set to 64, 128, 256, 512 and 1024 respectively in the feature extraction process; the down-sampling is realized by maximum pooling, and the up-sampling is realized by bilinear interpolation.

[0017] In step (2), the Cellpose network is pre-trained on the Cellpose official dataset to obtain a pre-trained Cellpose network, and the pre-training weight loads the nuclei weight parameter; the Hover-Net network is pre-trained on the PanNuke dataset to obtain a pre-trained Hover-Net network, and the pre-training weight loads the hovernet_fast_pannuke weight parameter; the CellViT network is pre-trained on the PanNuke dataset to obtain a pre-trained CellViT network, and the pre-training weight loads the CellViT-SAM-H-x40 weight parameter; the nuclei weight parameter, the hovernet_fast_pannuke weight parameter and the CellViT-SAM-H-x40 weight parameter are kept frozen in the training process of the cell segmentation model.

[0018] In step (2), the information integrator and the discriminator network are constructed based on the PatchGAN network, an up-sampling module is added before the output layer of the PatchGAN network, and a feature matrix of (1, 128, 128) size is obtained by bilinear interpolation up-sampling.

[0019] Step (3) specifically includes the following steps in sequence:

[0020] (3a) inputting the H&E images in the training set into the pre-trained cell segmentation network with weight freezing to obtain a feature matrix F;

[0021] (3b) inputting the H&E images in the training set into the cell segmentation generator network to obtain a feature matrix P;

[0022] (3c) inputting the feature matrix F into the information integrator and discriminator network to obtain a feature matrix A; inputting the feature matrix P into the information integrator and discriminator network to obtain a feature matrix B;

[0023] (3d) constructing a loss function according to the feature matrix A and the feature matrix B, and the loss function is a multivariate discriminant loss, and the expression is:

[0024]

[0025] wherein G s represents the cell segmentation generator network, D s represents the information integrator and discriminator network, x represents the input H&E image, z represents random noise, y cepo , y hove and y cevi respectively represent the cell binary segmentation results obtained from the pre-trained Cellpose network with weight freezing, the pre-trained Hover-Net network and the pre-trained CellViT network;

[0026] (3e) updating all non-frozen parameters of the cell segmentation model according to the loss function through the mechanism of back propagation;

[0027] (3f) repeating steps (3a) to (3e) until the training is completed, and saving the final model parameter weight;

[0028] (3g) loading the final model parameter weight, and testing the network performance of the cell segmentation generator using the test set.

[0029] In step (5), the molecular expression prediction generator network comprises:

[0030] an input layer adopting a feature matrix with a scale size of (3, 128, 128);

[0031] a pathology-based model adopting a PLIP pathology-based model image encoder to extract and encode the features of the input layer through a multi-head self-attention module to obtain a feature matrix with a scale size of (50, 768);

[0032] The reorganization layer first removes the first feature representing the category code in the first dimension of the feature matrix obtained by the pathological basis model to obtain a feature matrix with a size of (49, 768), then reorganizes the 49 features in the first dimension according to the spatial position to obtain a feature matrix with a size of (7, 7, 768), and then adjusts the third dimension to the front of the first dimension to obtain a feature matrix with a size of (768, 7, 7);

[0033] The projection layer obtains a feature matrix with a size of (512, 16, 16) through convolution operation on the feature matrix obtained by the reorganization layer;

[0034] The eleventh activation layer obtains a feature matrix with a size of (512, 16, 16) through nonlinear transformation ReLU on the feature matrix of the projection layer;

[0035] The first convolution layer obtains a feature matrix with a size of (64, 128, 128) through the convolution layer on the feature matrix of the input layer;

[0036] The first activation layer obtains a feature matrix with a size of (64, 128, 128) through nonlinear transformation ReLU on the output feature matrix of the first convolution layer;

[0037] The first down-sampling layer obtains a feature matrix with a size of (64, 64, 64) through maximum pooling down-sampling on the feature matrix obtained by the first activation layer;

[0038] The second convolution layer obtains a feature matrix with a size of (128, 64, 64) through the convolution layer on the feature matrix obtained by the first down-sampling layer;

[0039] The second activation layer obtains a feature matrix with a size of (128, 64, 64) through nonlinear transformation ReLU on the output feature matrix of the second convolution layer;

[0040] The second down-sampling layer obtains a feature matrix with a size of (128, 32, 32) through maximum pooling down-sampling on the feature matrix obtained by the second activation layer;

[0041] The third convolution layer obtains a feature matrix with a size of (256, 32, 32) through the convolution layer on the feature matrix obtained by the second down-sampling layer;

[0042] The third activation layer obtains a feature matrix with a size of (256, 32, 32) through nonlinear transformation ReLU on the output feature matrix of the third convolution layer;

[0043] The third down-sampling layer obtains a feature matrix with a size of (256, 16, 16) through maximum pooling down-sampling on the feature matrix obtained by the third activation layer;

[0044] The fourth convolutional layer obtains a feature matrix with a size of (512, 16, 16) from the feature matrix obtained by the third down-sampling layer through a convolutional layer;

[0045] The fourth activation layer obtains a feature matrix with a size of (512, 16, 16) from the output feature matrix of the fourth convolutional layer through a nonlinear transformation ReLU;

[0046] The fourth down-sampling layer obtains a feature matrix with a size of (512, 8, 8) from the feature matrix obtained by the fourth activation layer through a maximum pooling down-sampling;

[0047] The fifth convolutional layer obtains a feature matrix with a size of (512, 8, 8) from the feature matrix obtained by the fourth down-sampling layer through a convolutional layer;

[0048] The fifth activation layer obtains a feature matrix with a size of (512, 8, 8) from the output feature matrix of the fifth convolutional layer through a nonlinear transformation ReLU;

[0049] The first up-sampling layer obtains a feature matrix with a size of (512, 16, 16) from the feature matrix obtained by the fifth activation layer through a bilinear interpolation up-sampling;

[0050] The first attention gate obtains a feature matrix (512, 16, 16) from the feature matrix obtained by the first up-sampling layer and the feature matrix obtained by the fourth activation layer through an attention gate operation;

[0051] The first splicing layer obtains a feature matrix with a size of (1024, 16, 16) from the feature matrix obtained by the first attention gate and the feature matrix obtained by the eleventh activation layer through a splicing operation in the first dimension;

[0052] The sixth convolutional layer obtains a feature matrix with a size of (512, 16, 16) from the feature matrix obtained by the first splicing layer through a convolution;

[0053] The sixth activation layer obtains a feature matrix with a size of (512, 16, 16) from the output matrix of the sixth convolutional layer through a nonlinear transformation ReLU;

[0054] The second up-sampling layer obtains a feature matrix with a size of (512, 32, 32) from the feature matrix obtained by the sixth activation layer through a bilinear interpolation up-sampling;

[0055] The second attention gate obtains a feature matrix (256, 32, 32) from the feature matrix obtained by the second up-sampling layer and the feature matrix obtained by the third activation layer through an attention gate operation;

[0056] The second concatenation layer concatenates the feature matrix obtained by the second attention gate and the feature matrix obtained by the second up-sampling layer in the first dimension to obtain a feature matrix with a scale size of (768, 32, 32);

[0057] The seventh convolutional layer convolves the feature matrix obtained by the second concatenation layer to obtain a feature matrix with a scale size of (256, 32, 32);

[0058] The seventh activation layer performs a non-linear transformation ReLU on the output matrix of the seventh convolutional layer to obtain a feature matrix with a size of (256, 32, 32);

[0059] The third up-sampling layer performs bilinear interpolation up-sampling on the feature matrix obtained by the seventh activation layer to obtain a feature matrix with a scale size of (256, 64, 64);

[0060] The third attention gate obtains a feature matrix of (128, 64, 64) by performing an attention gate operation on the feature matrix obtained by the third up-sampling layer and the feature matrix obtained by the second activation layer;

[0061] The third concatenation layer concatenates the feature matrix obtained by the third attention gate and the feature matrix obtained by the third up-sampling layer in the first dimension to obtain a feature matrix with a scale size of (384, 64, 64);

[0062] The eighth convolutional layer convolves the feature matrix obtained by the third concatenation layer to obtain a feature matrix with a scale size of (128, 64, 64);

[0063] The eighth activation layer performs a non-linear transformation ReLU on the output matrix of the eighth convolutional layer to obtain a feature matrix with a size of (128, 64, 64);

[0064] The fourth up-sampling layer performs bilinear interpolation up-sampling on the feature matrix obtained by the eighth activation layer to obtain a feature matrix with a scale size of (128, 128, 128);

[0065] The fourth attention gate obtains a feature matrix of (64, 128, 128) by performing an attention gate operation on the feature matrix obtained by the fourth up-sampling layer and the feature matrix obtained by the first activation layer;

[0066] The fourth concatenation layer concatenates the feature matrix obtained by the fourth attention gate and the feature matrix obtained by the fourth up-sampling layer in the first dimension to obtain a feature matrix with a scale size of (192, 128, 128);

[0067] The ninth convolutional layer convolves the feature matrix obtained by the fourth concatenation layer to obtain a feature matrix with a scale size of (64, 128, 128);

[0068] The ninth activation layer obtains a feature matrix with a size of (64, 128, 128) by performing a non-linear transformation ReLU on the output matrix of the ninth convolutional layer;

[0069] The tenth convolutional layer obtains a feature matrix with a size of (3, 128, 128) by performing convolution on the feature matrix obtained by the ninth activation layer;

[0070] The tenth activation layer obtains a feature matrix with a size of (3, 128, 128) by performing a non-linear transformation ReLU on the output matrix of the tenth convolutional layer;

[0071] The output layer outputs the feature matrix obtained by the tenth activation layer.

[0072] In step (5), the molecular expression prediction discriminator network is constructed based on a PatchGAN network, and an up-sampling module is added before the output layer of the PatchGAN network to obtain a feature matrix with a size of (1, 128, 128) by bilinear interpolation up-sampling.

[0073] Step (6) specifically includes the following steps in sequence:

[0074] (6a) inputting the H&E image in the training set into the molecular expression prediction generator network to obtain a feature matrix Q;

[0075] (6b) inputting the mIHC image in the training set into the molecular expression prediction discriminator network to obtain a feature matrix T; inputting the feature matrix Q into the molecular expression prediction discriminator network to obtain a feature matrix S;

[0076] (6c) constructing a total loss function according to the mIHC image, the feature matrix Q, the feature matrix T and the feature matrix S, and the formula of the total loss function is:

[0077] L Total (G,D)=L AD (G,D)+L F (G)+L Mul (G)

[0078] Wherein, G represents the molecular expression prediction generator network, D represents the molecular expression prediction discriminator network; L AD represents a discrimination loss; L F represents a frequency loss; L Mul represents a multi-scale loss;

[0079] The expression of the discrimination loss is:

[0080] L AD (G,D)=E x,y [log D(x,y)]+E x,e[log(1-D(x, G(x, e)))]

[0081] where x, y, e represent the input H&E image, the ground truth mIHC image and the random noise, respectively; E x,y denotes the mathematical expectation of the joint distribution of x, y; E x,e denotes the mathematical expectation of the joint distribution of x, e;

[0082] The expression of the frequency loss is:

[0083] L F (G) = p·E x,y,e [||F HP (y)-F HP (G(x, e))‖1]+q·E x,y,e [||F LP (y)-F L P(G(x, e))||1]

[0084] where F HP (·) = IFFT[HP[FFT(·)]] and F LP (·) = IFFT[LP[FFT(·)]], FFT and IFFT represent the Fourier transform and the inverse Fourier transform, respectively, HP and LP represent the high-pass filter and the low-pass filter, respectively, p and q represent the high-frequency weight and the low-frequency weight, respectively, p is set to 1 and q is set to 2; E x,y,e denotes the mathematical expectation of the joint distribution of x, y, e;

[0085] The expression of the multi-scale loss is:

[0086]

[0087] where λ i denotes the weight of the i-th down-sampling loss; DS i (·) represents that the real molecular expression image or the generated molecular expression prediction image has undergone i times of scale transformation, each scale transformation including 4 times of Gaussian filtering using a Gaussian kernel with a mean of 0 and a variance of 1 and 1 time of down-sampling;

[0088] (6d) updating all parameters of the molecular expression prediction model according to the total loss function through the mechanism of back propagation;

[0089] (6e) repeating steps (6a) to (6d) until the training is completed, and saving the final model parameter weight;

[0090] (6) loading the final model parameter weight, and testing the performance of the molecular expression prediction generator network using the test set.

[0091] As can be seen from the above technical solution, the present invention has the following beneficial effects: First, the present invention achieves end-to-end cell binary segmentation, cell semantic segmentation, and molecular expression level prediction through a two-stage generative adversarial learning framework. Second, in the first stage, the present invention uses a generative cell segmentation model to extract cell feature information from H&E images and integrates the cell binary segmentation results obtained from an existing pre-trained cell segmentation network as prior information, which can generate more accurate cell binary segmentation results and improve the model's accuracy in cell segmentation. Third, in the second stage, the present invention uses a generative molecular expression prediction model to extract pathological information features from H&E images and generate virtual mIHC images reflecting molecular expression levels. At the same time, the use of a pathological foundation model improves the model's feature extraction capability, thereby improving the efficiency of acquiring mIHC images and increasing the accuracy of the generated content. Fourth, by mapping the cell binary segmentation results and virtual mIHC images at the spatial pixel level, the present invention can classify cell subtypes at the molecular expression level, improving the accuracy of cell semantic segmentation, thereby providing a convenient and accurate tool for cancer diagnosis at the cellular level. BRIEF DESCRIPTION OF THE DRAWINGS

[0092] Figure 1 is a flow chart of the method of the present invention;

[0093] Figure 2 Schematic diagram of the structure of the cell segmentation model and molecular expression prediction model in the present invention;

[0094] Figure 3 for Figure 2 Schematic diagram of the structure of the cell segmentation model;

[0095] Figure 4 for Figure 3 Schematic diagram of the structure of the pre-trained cell segmentation network;

[0096] Figure 5 for Figure 3 Schematic diagram of the structure of the information integrator and discriminator network;

[0097] Figure 6 for Figure 2 Schematic diagram of the structure of the molecular expression prediction model;

[0098] Figure 7 for Figure 6 Schematic diagram of the structure of the molecular expression prediction generator network;

[0099] Figure 8 This is a flow chart of the present invention for cell detection and analysis based on molecular expression prediction;

[0100] Figure 9 This is an example diagram of the visualization of the cell binary segmentation result of the present invention;

[0101] Figure 10 Visualization example of molecular expression prediction results of the present application;

[0102] Figure 11 Visualization example of cell semantic segmentation results of the present application;

[0103] Figure 12 Evaluation index of the present application in cell binary segmentation, cell semantic segmentation and molecular expression prediction tasks. DETAILED DESCRIPTION

[0104] As shown in Figure 1 ,a two-stage image generation learning method for cell detection and analysis based on molecular expression prediction, the method comprising the following sequential steps: Figure 2 (1) Obtain H&E dataset and paired mIHC dataset to form a dataset and perform preprocessing, and divide the preprocessed dataset into a training set and a test set;

[0105] (2) Construct a cell segmentation model, as shown in

[0106] , the cell segmentation model is composed of a pre-trained cell segmentation network, a cell segmentation generator network, an information integrator and a discriminator network; the pre-trained cell segmentation network includes a pre-trained Cellpose network, a pre-trained Hover-Net network and a pre-trained CellViT network, as shown in Figure 3 ; Figure 4

[0107] (3) Train the cell segmentation model to obtain a trained cell segmentation model;

[0108] (4) Input the H&E image in the test set to the trained cell segmentation model to obtain a cell binary segmentation result, as shown in Figure 9 , visualize and display the cell binary segmentation result;

[0109] (5) Construct a molecular expression prediction model, as shown in Figure 6 , the molecular expression prediction model is composed of a molecular expression prediction generator network and a molecular expression prediction discriminator network;

[0110] (6) Train the molecular expression prediction model to obtain a trained molecular expression prediction model;

[0111] (7) Input the H&E image in the test set to the trained molecular expression prediction model to obtain a molecular expression prediction result, as shown in Figure 10 , visualize and display the molecular expression prediction result;

[0112] (8) The cell binary segmentation result and the molecular expression prediction result are one-to-one mapped at the pixel level in space to obtain a cell semantic segmentation result, as shown in Figure 11 The cell semantic segmentation result is visualized. Through the mapping of the cell binary segmentation result and the virtual mIHC image at the spatial pixel level, the cell subtypes can be classified at the molecular expression level, increasing the accuracy of cell semantic segmentation, thereby providing a convenient and accurate tool for cancer diagnosis at the cell level.

[0113] In step (1), the preprocessing specifically refers to: using the valis method to register the WSI data of the H&E images in the H&E dataset and the WSI data of the mIHC images in the mIHC dataset, then performing four times down-sampling on the WSI data of the H&E images and the WSI data of the mIHC images, and cutting them into image pairs of 128x128 size.

[0114] In step (2), the cell segmentation generator network adopts the Attention U-Net architecture, the depth of the Attention U-Net is 5, the input feature channel and the output feature channel are both 3, and the feature channels are set to 64, 128, 256, 512, and 1024 in the feature extraction process; down-sampling is achieved by maximum pooling, and up-sampling is achieved by bilinear interpolation. The cell segmentation model extracts cell feature information from H&E images and integrates the cell binary segmentation result obtained from the existing pre-trained cell segmentation network as prior information, which can generate more accurate cell binary segmentation results and improve the accuracy of cell segmentation.

[0115] In step (2), the Cellpose network is pre-trained on the Cellpose official dataset to obtain a pre-trained Cellpose network, and the pre-training weight loads the nuclei weight parameter; the Hover-Net network is pre-trained on the PanNuke dataset to obtain a pre-trained Hover-Net network, and the pre-training weight loads the hovernet_fast_pannuke weight parameter; the CellViT network is pre-trained on the PanNuke dataset to obtain a pre-trained CellViT network, and the pre-training weight loads the CellViT-SAM-H-x40 weight parameter; the nuclei weight parameter, the hovernet_fast_pannuke weight parameter, and the CellViT-SAM-H-x40 weight parameter remain frozen during the training of the cell segmentation model.

[0116] In step (2), as shown in Figure 5As shown, the information integrator and discriminator network is constructed based on a PatchGAN network, and an up-sampling module is added before the output layer of the PatchGAN network to obtain a feature matrix of size (1, 128, 128) through bilinear interpolation up-sampling.

[0117] Step (3) specifically comprises the following steps in sequence:

[0118] (3a) inputting the H&E images in the training set into the pre-trained cell segmentation network with frozen weights to obtain a feature matrix F;

[0119] (3b) inputting the H&E images in the training set into the cell segmentation generator network to obtain a feature matrix P;

[0120] (3c) inputting the feature matrix F into the information integrator and discriminator network to obtain a feature matrix A; and inputting the feature matrix P into the information integrator and discriminator network to obtain a feature matrix B;

[0121] (3d) constructing a loss function according to the feature matrix A and the feature matrix B, and the loss function is a multivariate discriminant loss, and the expression is:

[0122]

[0123] wherein, G s represents the cell segmentation generator network, D s represents the information integrator and discriminator network, x represents the input H&E image, z represents random noise, y cepo , y hove and y cevi respectively represent the cell binary segmentation results obtained from the pre-trained Cellpose network with frozen weights, the pre-trained Hover-Net network and the pre-trained CellViT network;

[0124] (3e) updating all non-frozen parameters of the cell segmentation model through the mechanism of back propagation according to the loss function;

[0125] (3f) repeating steps (3a) to (3e) until the training is completed, and saving the final model parameter weight;

[0126] (3g) loading the final model parameter weight, and testing the network performance of the cell segmentation generator using the test set.

[0127] In step (5), as shown in the formula (5), the molecular expression prediction generator network comprises: Figure 7

[0128] an input layer, which adopts a feature matrix with a scale size of (3, 128, 128); ​

[0129] The pathological basis model adopts a PLIP pathological basis model image encoder to extract and encode the features of the input layer through a multi-head self-attention module to obtain a feature matrix with a scale size of (50, 768);

[0130] The reshape layer, i.e., Reshape, first removes the first feature representing the category code in the first dimension of the feature matrix obtained by the pathological basis model to obtain a feature matrix with a scale size of (49, 768), then reorganizes the 49 features in the first dimension according to the spatial position to obtain a feature matrix with a size of (7, 7, 768), and then adjusts the third dimension to the front of the first dimension to obtain a feature matrix with a size of (768, 7, 7);

[0131] The projection layer obtains a feature matrix with a scale size of (512, 16, 16) through the eleventh convolution layer Conv11 from the feature matrix obtained by the reshape layer;

[0132] The eleventh activation layer obtains a feature matrix with a scale size of (512, 16, 16) through the nonlinear transformation ReLU, i.e., ReLU11, from the feature matrix of the projection layer;

[0133] The first convolution layer Conv1 obtains a feature matrix with a scale size of (64, 128, 128) through convolution operation from the feature matrix of the input layer;

[0134] The first activation layer obtains a feature matrix with a scale size of (64, 128, 128) through the nonlinear transformation ReLU, i.e., ReLU1, from the output feature matrix of the first convolution layer Conv1;

[0135] The first downsampling layer Down1 obtains a feature matrix with a scale size of (64, 64, 64) through maximum pooling downsampling from the feature matrix obtained by the first activation layer;

[0136] The second convolution layer Conv2 obtains a feature matrix with a scale size of (128, 64, 64) through convolution operation from the feature matrix obtained by the first downsampling layer Down1;

[0137] The second activation layer obtains a feature matrix with a scale size of (128, 64, 64) through the nonlinear transformation ReLU, i.e., ReLU2, from the output feature matrix of the second convolution layer Conv2;

[0138] The second downsampling layer Down2 obtains a feature matrix with a scale size of (128, 32, 32) through maximum pooling downsampling from the feature matrix obtained by the second activation layer;

[0139] The third convolutional layer Conv3 obtains a feature matrix with a scale size of (256, 32, 32) by performing convolutional operation on the feature matrix obtained by the second downsampling layer Down2.

[0140] The third activation layer ReLU3 obtains a feature matrix with a scale size of (256, 32, 32) by performing nonlinear transformation on the output feature matrix of the third convolutional layer Conv3.

[0141] The third downsampling layer Down3 obtains a feature matrix with a scale size of (256, 16, 16) by performing maximum pooling downsampling on the feature matrix obtained by the third activation layer.

[0142] The fourth convolutional layer Conv4 obtains a feature matrix with a scale size of (512, 16, 16) by performing convolutional operation on the feature matrix obtained by the third downsampling layer Down3.

[0143] The fourth activation layer ReLU4 obtains a feature matrix with a scale size of (512, 16, 16) by performing nonlinear transformation on the output feature matrix of the fourth convolutional layer Conv4.

[0144] The fourth downsampling layer Down4 obtains a feature matrix with a scale size of (512, 8, 8) by performing maximum pooling downsampling on the feature matrix obtained by the fourth activation layer.

[0145] The fifth convolutional layer Conv5 obtains a feature matrix with a scale size of (512, 8, 8) by performing convolutional operation on the feature matrix obtained by the fourth downsampling layer Down4.

[0146] The fifth activation layer ReLU5 obtains a feature matrix with a scale size of (512, 8, 8) by performing nonlinear transformation on the output feature matrix of the fifth convolutional layer Conv5.

[0147] The first upsampling layer Up1 obtains a feature matrix with a scale size of (512, 16, 16) by performing bilinear interpolation upsampling on the feature matrix obtained by the fifth activation layer.

[0148] The first attention gate A1 obtains a feature matrix with a scale size of (512, 16, 16) by performing attention gate operation on the feature matrix obtained by the first upsampling layer Up1 and the feature matrix obtained by the fourth activation layer.

[0149] The first concatenation layer C1 obtains a feature matrix with a scale size of (1024, 16, 16) by performing concatenation operation on the feature matrix obtained by the first attention gate A1 and the feature matrix obtained by the eleventh activation layer in the first dimension.

[0150] The sixth convolutional layer Conv6 obtains a feature matrix with a size of (512, 16, 16) by performing convolutional operation on the feature matrix obtained by the first concatenation layer C1.

[0151] The sixth activation layer obtains a feature matrix with a size of (512, 16, 16) by performing nonlinear transformation ReLU (ReLU6) on the output matrix of the sixth convolutional layer Conv6.

[0152] The second upsampling layer Up2 obtains a feature matrix with a size of (512, 32, 32) by performing bilinear interpolation upsampling on the feature matrix obtained by the sixth activation layer.

[0153] The second attention gate A2 obtains a feature matrix of (256, 32, 32) by performing attention gate operation on the feature matrix obtained by the second upsampling layer Up2 and the feature matrix obtained by the third activation layer.

[0154] The second concatenation layer C2 obtains a feature matrix with a size of (768, 32, 32) by performing concatenation operation on the feature matrix obtained by the second attention gate A2 and the feature matrix obtained by the second upsampling layer Up2 in the first dimension.

[0155] The seventh convolutional layer Conv7 obtains a feature matrix with a size of (256, 32, 32) by performing convolutional operation on the feature matrix obtained by the second concatenation layer C2.

[0156] The seventh activation layer obtains a feature matrix with a size of (256, 32, 32) by performing nonlinear transformation ReLU (ReLU7) on the output matrix of the seventh convolutional layer Conv7.

[0157] The third upsampling layer Up3 obtains a feature matrix with a size of (256, 64, 64) by performing bilinear interpolation upsampling on the feature matrix obtained by the seventh activation layer.

[0158] The third attention gate A3 obtains a feature matrix of (128, 64, 64) by performing attention gate operation on the feature matrix obtained by the third upsampling layer Up3 and the feature matrix obtained by the second activation layer.

[0159] The third concatenation layer C3 obtains a feature matrix with a size of (384, 64, 64) by performing concatenation operation on the feature matrix obtained by the third attention gate A3 and the feature matrix obtained by the third upsampling layer Up3 in the first dimension.

[0160] The eighth convolutional layer Conv8 obtains a feature matrix with a size of (128, 64, 64) by performing convolutional operation on the feature matrix obtained by the third concatenation layer C3.

[0161] The eighth activation layer obtains a feature matrix with a size of (128, 64, 64) through a non-linear transformation ReLU of the output matrix of the eighth convolutional layer Conv8, i.e., ReLU8;

[0162] The fourth up-sampling layer Up4 obtains a feature matrix with a size of (128, 128, 128) through bilinear interpolation up-sampling of the feature matrix obtained by the eighth activation layer;

[0163] The fourth attention gate A4 obtains a feature matrix of (64, 128, 128) through an attention gate operation of the feature matrix obtained by the fourth up-sampling layer Up4 and the feature matrix obtained by the first activation layer;

[0164] The fourth concatenation layer C4 obtains a feature matrix with a size of (192, 128, 128) through a concatenation operation in the first dimension of the feature matrix obtained by the fourth attention gate A4 and the feature matrix obtained by the fourth up-sampling layer Up4;

[0165] The ninth convolutional layer Conv9 obtains a feature matrix with a size of (64, 128, 128) through a convolution operation of the feature matrix obtained by the fourth concatenation layer C4;

[0166] The ninth activation layer obtains a feature matrix with a size of (64, 128, 128) through a non-linear transformation ReLU of the output matrix of the ninth convolutional layer Conv9, i.e., ReLU9;

[0167] The tenth convolutional layer Conv10 obtains a feature matrix with a size of (3, 128, 128) through a convolution operation of the feature matrix obtained by the ninth activation layer;

[0168] The tenth activation layer obtains a feature matrix with a size of (3, 128, 128) through a non-linear transformation ReLU of the output matrix of the tenth convolutional layer Conv10, i.e., ReLU10;

[0169] The output layer outputs the feature matrix obtained by the tenth activation layer.

[0170] In step (5), the molecular expression prediction discriminator network is constructed based on a PatchGAN network, and an up-sampling module is added before the output layer of the PatchGAN network to obtain a feature matrix with a size of (1, 128, 128) through bilinear interpolation up-sampling. The molecular expression prediction model extracts pathological information features from H&E images and generates virtual mIHC images reflecting molecular expression levels. Meanwhile, the use of the pathology-based model improves the feature extraction capability of the model, thereby improving the efficiency of obtaining mIHC images and increasing the accuracy of generated content.

[0171] Step (6) specifically includes the following steps in sequence:

[0172] (6a) inputting the H&E images in the training set into the molecular expression prediction generator network to obtain a feature matrix Q;

[0173] (6b) inputting the mIHC images in the training set into the molecular expression prediction discriminator network to obtain a feature matrix T; inputting the feature matrix Q into the molecular expression prediction discriminator network to obtain a feature matrix S;

[0174] (6c) constructing a total loss function according to the mIHC images, the feature matrix Q, the feature matrix T and the feature matrix S, and a formula of the total loss function is:

[0175] L Total (G,D)=L AD (G,D)+L F (G)+L Mul (G)

[0176] Wherein, G represents the molecular expression prediction generator network, and D represents the molecular expression prediction discriminator network; L AD represents a discrimination loss; L F represents a frequency loss; L Mul represents a multi-scale loss;

[0177] The expression of the discrimination loss is:

[0178] L AD (G,D)=E x,y [log D(x,y)]+E x,e [log(1-D(x,G(x,e)))]

[0179] Wherein, x, y and e respectively represent the input H&E image, the real mIHC image and random noise; E x,y represents a mathematical expectation of the joint distribution of x and y; E x,e represents a mathematical expectation of the joint distribution of x and e;

[0180] The expression of the frequency loss is:

[0181] L F (G)=p·E x,y,e [||F HP (y)-F HP (G(x,e))||1]+q·E x,y,e [||F LP (y)-F LP (G(x,e))||1]

[0182] Wherein, F HP (·)=IFFT[HP[FFT(·)]] and FLP (·) = IFFT[LP[FFT(·)]], FFT and IFFT represent Fourier transform and inverse Fourier transform respectively, HP and LP represent high-pass filter and low-pass filter respectively, p and q represent high-frequency weight and low-frequency weight respectively, p is set to 1 and q is set to 2;E x,y,e denotes the mathematical expectation of the joint distribution of x, y, e;

[0183] The expression of the multi-scale loss is:

[0184]

[0185] where λ i denotes the weight of the i-th downsampling loss; DS i (·) denotes that the real molecular expression image or the generated molecular expression prediction image has undergone i times of scale transformation, each scale transformation includes using a Gaussian kernel with a mean of 0 and a variance of 1 to perform 4 times of Gaussian filtering and 1 time of downsampling;

[0186] (6d) Update all parameters of the molecular expression prediction model by the mechanism of back propagation according to the total loss function;

[0187] (6e) Repeat steps (6a) to (6d) until the training is completed, and save the final model parameter weight;

[0188] (6) Load the final model parameter weight, and test the performance of the molecular expression prediction generator network using the test set.

[0189] As shown in Figure 8 , the process of cell detection and analysis based on molecular expression prediction is as follows:

[0190] Step 1: Obtain the H&E image;

[0191] Step 2: Input the H&E image into the trained cell segmentation generator network to perform cell binary segmentation;

[0192] Step 3: Input the H&E image into the trained molecular expression prediction generator network to perform molecular expression prediction;

[0193] Step 4: Map the cell binary segmentation result and the molecular expression prediction result at the pixel level in space to obtain the cell semantic segmentation result;

[0194] Step 5: Save the cell binary segmentation result, the molecular expression prediction result and the cell semantic segmentation result.

[0195] Visualize the cell binary segmentation result, and the visualization result is as shown in Figure 9are shown, where (a) is an H&E image, (b) is a real cell binary segmentation result, and (c) is a generated cell binary segmentation result.

[0196] The molecular expression prediction result is visualized, and the visualization result is as shown in Figure 10 are shown, where (a) is an H&E image, (b) is a real cell binary segmentation result, and (c) is a generated cell binary segmentation result.

[0197] The cell semantic segmentation result is visualized, and the visualization result is as shown in Figure 11 are shown, where (a) is an H&E image, (b) is a real cell binary segmentation result, and (c) is a generated cell binary segmentation result.

[0198] As shown in Figure 12 The network for cell detection and analysis based on molecular expression prediction can accurately segment cells and predict molecular expression in the cell binary segmentation, cell semantic segmentation, and molecular expression prediction tasks, and has high evaluation indexes. Figure 12 In the table, ACC represents the consistency between the cell segmentation result and the real result; Dice represents the ratio of the intersection area size of the segmentation result and the real result multiplied by 2 to the sum of the area size of the segmentation result and the area size of the real result; IoU represents the ratio of the intersection area size of the binary segmentation result and the real result to the union area size of the two; MIoU represents the ratio of the intersection area size of the semantic segmentation result and the real result to the union area size of the two. ACC, Dice, IoU, and MIoU are all important indexes for measuring segmentation accuracy, and the larger the index value is, the better the segmentation accuracy is. PSNR reflects the degree of loss of the original signal; SNR reflects the ratio between signal power and noise power; SSIM is based on visual perception and comprehensively evaluates the image from brightness, contrast, and structure. PSNR, SNR, and SSIM are all important indexes for measuring images, and the larger the index value is, the better the image quality is.

[0199] In conclusion, the application realizes end-to-end cell binary segmentation, cell semantic segmentation and molecular expression level prediction through a two-stage generative adversarial learning framework; the first stage of the application uses a generative cell segmentation model to extract cell feature information from H&E images, and integrates the cell binary segmentation result obtained from an existing pre-trained cell segmentation network as prior information, so that a more accurate cell binary segmentation result can be generated, and the accuracy of the model for cell segmentation is improved; the second stage of the application uses a generative molecular expression prediction model to extract pathological information features from H&E images and generate virtual mIHC images reflecting the molecular expression level, and the use of a pathology-based model improves the feature extraction capability of the model, thereby improving the acquisition efficiency of the mIHC image and increasing the accuracy of the generated content; through the mapping of the cell binary segmentation result and the virtual mIHC image at the spatial pixel level, the cell subtype can be classified at the molecular expression level, the accuracy of the cell semantic segmentation is increased, and thus a convenient and accurate tool is provided for cell-level cancer diagnosis.

Claims

1. A two-stage image generation learning method for cell detection and analysis based on molecular expression prediction, characterized by: The method comprises the following steps in sequence: (1) Obtain the H&E dataset and the paired mIHC dataset to form a dataset and preprocess it, then divide the preprocessed dataset into a training set and a test set; (2) constructing a cell segmentation model, wherein the cell segmentation model consists of a pre-trained cell segmentation network, a cell segmentation generator network, an information integrator, and a discriminator network; the pre-trained cell segmentation network includes a pre-trained Cellpose network, a pre-trained Hover-Net network, and a pre-trained CellViT network; (3) training the cell segmentation model to obtain a trained cell segmentation model; (4) Input the H&E images in the test set into the trained cell segmentation model to obtain the cell binary segmentation results, and visualize the cell binary segmentation results; (5) Constructing a molecular expression prediction model, which consists of a molecular expression prediction generator network and a molecular expression prediction discriminator network; (6) training the molecular expression prediction model to obtain a trained molecular expression prediction model; (7) Inputting the H&E images in the test set into the trained molecular expression prediction model to obtain molecular expression prediction results, and visually displaying the molecular expression prediction results; (8) Mapping the cell binary segmentation results and the molecular expression prediction results spatially at the pixel level to obtain the cell semantic segmentation results, and visually displaying the cell semantic segmentation results; Step (3) specifically includes the following steps in order: (3a) Input the H&E images in the training set into the pre-trained cell segmentation network with frozen weights to obtain the feature matrix F; (3b) Input the H&E images in the training set into the cell segmentation generator network to obtain the feature matrix P; (3c) Input the feature matrix F into the information integrator and discriminator network to obtain the feature matrix A; Input the feature matrix P into the information integrator and discriminator network to obtain the feature matrix B; (3d) A loss function is constructed based on the feature matrix A and the feature matrix B. The loss function is a multivariate discriminant loss, and its expression is: Among them, G s represents the cell segmentation generator network, D s represents the information integrator and discriminator network, x represents the input H&E image, z represents random noise, y cepo 、y hove and y cevi Respectively represent the cell binary segmentation results obtained from the weight-frozen pre-trained Cellpose network, pre-trained Hover-Net network, and pre-trained CellViT network; (3e) Based on the loss function, all non-frozen parameters of the cell segmentation model are updated through the back-propagation mechanism; (3f) Repeat steps (3a) to (3e) until the training is completed and save the final model parameter weights; (3g) Load the final model parameter weights and use the test set to test the network performance of the cell segmentation generator.

2. The two-stage image generation learning method for cell detection and analysis based on molecular expression prediction according to claim 1, characterized in that: In step (1), the preprocessing specifically refers to: registering the WSI data of the H&E images in the H&E dataset and the WSI data of the mIHC images in the mIHC dataset using the Valis method, then downsampling the WSI data of the H&E images and the WSI data of the mIHC images by a factor of four, and then cutting them into image pairs of size 128×128.

3. The two-stage image generation learning method for cell detection and analysis based on molecular expression prediction according to claim 1, characterized in that: In step (2), the cell segmentation generator network adopts the Attention U-Net architecture. The depth of Attention U-Net is 5, the input feature channel and the output feature channel are both 3, and the feature channels are set to 64, 128, 256, 512, and 1024 respectively during the feature extraction process; downsampling is achieved by maximum pooling, and upsampling is achieved by bilinear interpolation.

4. The two-stage image generation learning method for cell detection and analysis based on molecular expression prediction according to claim 1, characterized in that: In step (2), the Cellpose network is pre-trained on the Cellpose official dataset to obtain a pre-trained Cellpose network, and the pre-trained weights are loaded with the nuclei weight parameters; the Hover-Net network is pre-trained on the PanNuke dataset to obtain a pre-trained Hover-Net network, and the pre-trained weights are loaded with the hovernet_fast_pannuke weight parameters; the CellViT network is pre-trained on the PanNuke dataset to obtain a pre-trained CellViT network, and the pre-trained weights are loaded with the CellViT-SAM-H-x40 weight parameters; the nuclei weight parameters, hovernet_fast_pannuke weight parameters and CellViT-SAM-H-x40 weight parameters remain frozen during the cell segmentation model training process.

5. The two-stage image generation learning method for cell detection and analysis based on molecular expression prediction according to claim 1, characterized in that: In step (2), the information integrator and discriminator network are constructed based on the PatchGAN network. An upsampling module is added before the output layer of the PatchGAN network, and a feature matrix of size (1, 128, 128) is obtained by bilinear interpolation upsampling.

6. The two-stage image generation learning method for cell detection and analysis based on molecular expression prediction according to claim 1, characterized in that: In step (5), the molecular expression prediction generator network includes: The input layer uses a feature matrix with a scale of (3, 128, 128); The pathology basic model uses the PLIP pathology basic model image encoder to extract and encode the features of the input layer through a multi-head self-attention module to obtain a feature matrix with a scale of (50, 768); In the reorganization layer, the first feature representing the category code in the first dimension of the feature matrix obtained by the pathological basic model is first removed to obtain a feature matrix with a scale of (49, 768). The 49 features in the first dimension are then reorganized according to their spatial positions to obtain a feature matrix with a scale of (7, 7, 768). The third dimension is then adjusted to the front of the first dimension to obtain a feature matrix with a scale of (768, 7, 7). In the projection layer, the feature matrix obtained by the recombinant layer is subjected to a convolution operation to obtain a feature matrix of scale (512, 16, 16); The eleventh activation layer uses the nonlinear ReLU transformation to transform the feature matrix of the projection layer into a feature matrix with a scale of (512, 16, 16); The first convolutional layer convolves the feature matrix of the input layer to obtain a feature matrix of scale (64, 128, 128); The first activation layer uses the nonlinear ReLU to transform the output feature matrix of the first convolutional layer to obtain a feature matrix with a scale of (64, 128, 128); In the first downsampling layer, the feature matrix obtained by the first activation layer is downsampled by maximum pooling to obtain a feature matrix of scale (64, 64, 64); The second convolutional layer convolves the feature matrix obtained by the first downsampling layer to obtain a feature matrix with a scale of (128, 64, 64); The second activation layer uses the nonlinear ReLU to transform the output feature matrix of the second convolutional layer to obtain a feature matrix with a scale of (128, 64, 64); The second downsampling layer downsamples the feature matrix obtained by the second activation layer through maximum pooling to obtain a feature matrix with a scale of (128, 32, 32); The third convolutional layer performs a convolution operation on the feature matrix obtained by the second downsampling layer to obtain a feature matrix with a scale of (256, 32, 32); The third activation layer uses the nonlinear ReLU to transform the output feature matrix of the third convolutional layer to obtain a feature matrix with a scale of (256, 32, 32); The third downsampling layer downsamples the feature matrix obtained by the third activation layer through maximum pooling to obtain a feature matrix with a scale of (256, 16, 16); The fourth convolutional layer convolves the feature matrix obtained from the third downsampling layer to obtain a feature matrix with a scale of (512, 16, 16); The fourth activation layer uses the nonlinear ReLU to transform the output feature matrix of the fourth convolutional layer to obtain a feature matrix with a scale of (512, 16, 16); The fourth downsampling layer downsamples the feature matrix obtained by the fourth activation layer through maximum pooling to obtain a feature matrix with a scale of (512, 8, 8); The fifth convolutional layer convolves the feature matrix obtained by the fourth downsampling layer to obtain a feature matrix of scale (512, 8, 8); The fifth activation layer uses the nonlinear ReLU to transform the output feature matrix of the fifth convolutional layer to obtain a feature matrix with a scale of (512, 8, 8); In the first upsampling layer, the feature matrix obtained in the fifth activation layer is upsampled by bilinear interpolation to obtain a feature matrix with a scale of (512, 16, 16); The first attention gate operates the feature matrix obtained by the first upsampling layer and the feature matrix obtained by the fourth activation layer through the attention gate to obtain the feature matrix of (512, 16, 16); The first concatenation layer concatenates the feature matrix obtained by the first attention gate and the feature matrix obtained by the eleventh activation layer in the first dimension to obtain a feature matrix with a scale of (1024, 16, 16); The sixth convolutional layer convolves the feature matrix obtained from the first concatenation layer to obtain a feature matrix with a scale of (512, 16, 16); The sixth activation layer uses the nonlinear ReLU transformation on the output matrix of the sixth convolutional layer to obtain a feature matrix of size (512, 16, 16); In the second upsampling layer, the feature matrix obtained by the sixth activation layer is upsampled by bilinear interpolation to obtain a feature matrix with a scale of (512, 32, 32); The second attention gate operates the feature matrix obtained by the second upsampling layer and the feature matrix obtained by the third activation layer through the attention gate to obtain a feature matrix of (256, 32, 32); In the second concatenation layer, the feature matrix obtained by the second attention gate and the feature matrix obtained by the second upsampling layer are concatenated in the first dimension to obtain a feature matrix with a scale of (768, 32, 32); The seventh convolutional layer convolves the feature matrix obtained in the second concatenation layer to obtain a feature matrix with a scale of (256, 32, 32); The seventh activation layer uses the nonlinear transformation ReLU to transform the output matrix of the seventh convolutional layer to obtain a feature matrix of size (256, 32, 32); The third upsampling layer upsamples the feature matrix obtained by the seventh activation layer through bilinear interpolation to obtain a feature matrix with a scale of (256, 64, 64); The third attention gate operates the feature matrix obtained by the third upsampling layer and the feature matrix obtained by the second activation layer through the attention gate to obtain a feature matrix of (128, 64, 64); The third splicing layer concatenates the feature matrix obtained by the third attention gate and the feature matrix obtained by the third upsampling layer in the first dimension to obtain a feature matrix with a scale of (384, 64, 64); The eighth convolutional layer convolves the feature matrix obtained in the third concatenation layer to obtain a feature matrix of scale (128, 64, 64); The eighth activation layer uses the nonlinear transformation ReLU to transform the output matrix of the eighth convolutional layer to obtain a feature matrix of size (128, 64, 64); The fourth upsampling layer upsamples the feature matrix obtained by the eighth activation layer through bilinear interpolation to obtain a feature matrix with a scale of (128, 128, 128); The fourth attention gate operates the feature matrix obtained by the fourth upsampling layer and the feature matrix obtained by the first activation layer through the attention gate to obtain a feature matrix of (64, 128, 128); The fourth concatenation layer concatenates the feature matrix obtained by the fourth attention gate and the feature matrix obtained by the fourth upsampling layer in the first dimension to obtain a feature matrix with a scale of (192, 128, 128); The ninth convolutional layer convolves the feature matrix obtained in the fourth concatenation layer to obtain a feature matrix of scale (64, 128, 128); The ninth activation layer uses the nonlinear ReLU transformation on the output matrix of the ninth convolutional layer to obtain a feature matrix of size (64, 128, 128); The tenth convolutional layer convolves the feature matrix obtained in the ninth activation layer to obtain a feature matrix of scale (3, 128, 128); The tenth activation layer, the output matrix of the tenth convolutional layer is transformed into a feature matrix of size (3, 128, 128) through the nonlinear transformation ReLU; The output layer outputs the feature matrix obtained by the tenth activation layer.

7. The two-stage image generation learning method for cell detection and analysis based on molecular expression prediction according to claim 1, characterized in that: In step (5), the molecular expression prediction discriminator network is constructed based on the PatchGAN network, and an upsampling module is added before the output layer of the PatchGAN network. By bilinear interpolation upsampling, a feature matrix of size (1, 128, 128) is obtained.

8. The two-stage image generation learning method for cell detection and analysis based on molecular expression prediction according to claim 1, characterized in that: Step (6) specifically includes the following steps in order: (6a) Input the H&E images in the training set into the molecular expression prediction generator network to obtain the feature matrix Q; (6b) Input the mIHC images in the training set into the molecular expression prediction discriminator network to obtain the feature matrix T; Input the feature matrix Q into the molecular expression prediction discriminator network to obtain the feature matrix S; (6c) The total loss function is constructed based on the mIHC image, feature matrix Q, feature matrix T, and feature matrix S. The formula of the total loss function is: L Total (G,D)=L AD (G,D)+L F (G)+L Mul (G) Among them, G represents the molecular expression prediction generator network, D represents the molecular expression prediction discriminator network; L AD represents the discrimination loss; L F Indicates frequency loss; L Mul represents multi-scale loss; The expression of the discriminant loss is: L AD (G,D)=E x,y [logD(x,y)]+E x,e [log(1-D(x,G(x,e)))] Where x, y, and e represent the input H&E image, the real mIHC image, and random noise, respectively; E x,y Represents the mathematical expectation of the joint distribution of x and y; E x,e Represents the mathematical expectation of the joint distribution of x and e; The expression for frequency loss is: L F (G)=p·E x,y,e [||F HP (y)-F HP (G(x,e))||1]+q·E x,y,e [||F LP (y)-F LP (G(x,e))||1] Among them, F HP (·)=IFFT[HP[FFT(·)]],F LP (·)=IFFT[LP[FFT(·)]], where FFT and IFFT represent Fourier transform and inverse Fourier transform, HP and LP represent high-pass filter and low-pass filter, respectively; p and q represent high-frequency weight and low-frequency weight, respectively, with p set to 1 and q set to 2; E x,y,e Represents the mathematical expectation of the joint distribution of x, y, and e; The expression of multi-scale loss is: Among them, λ i Represents the weight of the i-th downsampling loss; DS i (·) indicates that the real molecular expression image or the generated molecular expression prediction image has undergone i scale transformations, and each scale transformation includes four Gaussian filters and one downsampling using a Gaussian kernel with a mean of 0 and a variance of 1; (6d) updating all parameters of the molecular expression prediction model through the back-propagation mechanism according to the total loss function; (6e) Repeat steps (6a) to (6d) until the training is completed, and save the final model parameter weights; (6f) Load the final model parameter weights and use the test set to test the performance of the molecular expression prediction generator network.

Images