Gene expression prediction method and system based on h&e staining pathological images

Abstract

The application provides a gene expression prediction method and system based on H&E staining pathological images. The method comprises the following steps: constructing a training set; inputting a pathological image sample set into an initial gene expression prediction model to output image fusion features and gene expression prediction values; obtaining image projection features by using a contrast learning model, projecting gene expression true values in a latent space to obtain gene expression projection features, aligning the image projection features and the gene expression projection features to construct a contrast learning loss function, comparing the gene expression prediction values with the gene expression true values to construct a regression loss function, adding the contrast learning loss function and the regression loss function to obtain a total loss function, calculating errors by using the total loss function and optimizing the model to obtain a gene expression prediction model; and inputting a pathological image set into the gene expression prediction model to obtain gene expression prediction results. The application can efficiently and accurately realize gene expression prediction.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of biomedical image processing, in particular to a gene expression prediction method and system based on H&E staining pathological images. BACKGROUND

[0002] In computer-aided diagnosis and tumor disease analysis work, generating digital pathology images from tissue H&E staining to assist doctors in diagnosis is a relatively low-cost and relatively mature method. However, the picture after H&E staining can only provide morphological heterogeneity within the tumor, and cannot clearly identify molecular differences. If combined with spatial transcriptome information, molecular-level heterogeneity such as spatially differentially expressed genes and gene gradual change patterns in paracancer tissues can be identified, which can be applied to rare tumor identification, cancer progression diagnosis, and other diagnostic scenarios. However, the spatial transcriptome technology is relatively high in cost. If gene expression data can be predicted based on H&E staining images with more accumulated data, it is expected to greatly improve the application range of spatial transcriptome technology and promote its research progress in clinical use.

[0003] The spatial transcriptome technology in the prior art is high in cost and cannot well meet the accuracy requirements of gene expression prediction. SUMMARY

[0004] Therefore, it is necessary to provide a gene expression prediction method and system based on H&E staining pathological images, a computer device and a computer readable storage medium in view of the above technical problems.

[0005] In a first aspect, the present application provides a gene expression prediction method based on H&E staining pathological images, which comprises:

[0006] Obtaining an H&E staining pathological image sample set and a corresponding gene expression data sample set, the pathological image sample set comprising pathological image samples of different scales, and constructing a training set according to the pathological image sample set and the corresponding gene expression true value; the gene expression true value is obtained from the gene expression data sample set after normalization processing;

[0007] Inputting the pathological image sample set into an initial gene expression prediction model to output image fusion features and gene expression prediction values;

[0008] projecting the image fusion feature in the latent space by using a contrast learning model to obtain an image projection feature, projecting the gene expression true value in the latent space after coding to obtain a gene expression projection feature, aligning the image projection feature with the gene expression projection feature to construct a contrast learning loss function, comparing the gene expression predicted value with the gene expression true value to construct a regression loss function, adding the contrast learning loss function and the regression loss function to obtain a total loss function, calculating an error by using the total loss function, and optimizing the initial gene expression prediction model by using a back propagation algorithm to obtain a gene expression prediction model;

[0009] inputting the H&E stained pathological image set into the trained gene expression prediction model to obtain a gene expression prediction result.

[0010] In some embodiments, the pathological image sample set includes a current image block sample with a test point position coordinate as a center point, a neighborhood image block sample centered on the current image block sample, and an overall pathological image sample.

[0011] In some embodiments, the initial gene expression prediction model includes an image encoder and a gene expression decoder.

[0012] The image encoder is configured to encode image features corresponding to pathological image samples of different scales to obtain corresponding image encoded features, and fuse the image encoded features to obtain the image fusion feature.

[0013] The gene expression decoder is configured to decode the image fusion feature to obtain a gene expression predicted value.

[0014] In some embodiments, the image encoder includes a first encoder, a second encoder, a third encoder, and a feature fusioner.

[0015] The first encoder is configured to encode image features corresponding to the overall pathological image sample to obtain first image encoded features.

[0016] The second encoder is configured to encode image features corresponding to the neighborhood image block sample to obtain second image encoded features.

[0017] The third encoder is configured to encode image features corresponding to the current image block sample to obtain third image encoded features.

[0018] The feature fusioner is configured to fuse the first image encoded features and the second image encoded features to obtain first fusion features, fuse the first image encoded features and the third image encoded features to obtain second fusion features, and fuse the first fusion features and the second fusion features to obtain the image fusion feature.

[0019] In some embodiments, the first encoder comprises a UNI network module, a first Transformer layer, a CNN layer, and a second Transformer layer;

[0020] The UNI network module extracts image features of the whole pathological image sample;

[0021] The first Transformer layer encodes image features corresponding to the whole pathological image sample to obtain first image encoding sub-features;

[0022] The first image encoding sub-features are rearranged according to the original position to obtain second image encoding sub-features;

[0023] The CNN layer respectively performs multi-layer convolution operation on the second image encoding sub-features to obtain third image encoding sub-features;

[0024] The third image encoding sub-features are rearranged to obtain fourth image encoding sub-features;

[0025] The second Transformer layer encodes the fourth image encoding sub-features to obtain the first image encoding features.

[0026] In some embodiments, the contrast learning model comprises an image projector, a gene expression encoder, a gene expression projector, and a contrast learner;

[0027] The image projector is configured to project the image fusion features in a latent space to obtain image projection features;

[0028] The gene expression encoder is configured to encode the gene expression true value to obtain gene expression encoding features;

[0029] The gene expression projector is configured to project the gene expression encoding features in a latent space to obtain gene expression projection features;

[0030] The contrast learner aligns the image projection features with the gene expression projection features to construct a contrast learning loss function, compares the gene expression predicted value with the gene expression true value to construct a regression loss function, adds the contrast learning loss function and the regression loss function to obtain a total loss function, calculates errors using the total loss function, and optimizes the initial gene expression prediction model through a backpropagation algorithm to obtain a gene expression prediction model.

[0031] In some embodiments, the method further comprises:

[0032] According to the pathological image sample set and the corresponding gene expression true value, a validation set is constructed;

[0033] The initial gene expression prediction model is verified by using the verification set until the correlation coefficient between the output gene expression prediction value and the corresponding gene expression true value no longer improves in multiple training rounds, and then the training is stopped.

[0034] In a second aspect, an embodiment of the present application provides a gene expression prediction system based on H&E stained pathological images, which comprises:

[0035] An H&E stained pathological image sample set and a corresponding gene expression data sample set are obtained, the pathological image sample set comprises pathological image samples of different scales, and a training set is constructed according to the pathological image sample set and the corresponding gene expression true value; the gene expression true value is obtained from the gene expression data sample set after normalization processing;

[0036] A model training module is configured to input the pathological image sample set into an initial gene expression prediction model, output image fusion features and gene expression prediction values, project the image fusion features in a latent space by using a contrast learning model to obtain image projection features, project the gene expression true value after encoding in the latent space to obtain gene expression projection features, align the image projection features with the gene expression projection features to construct a contrast learning loss function, compare the gene expression prediction values with the gene expression true values to construct a regression loss function, add the contrast learning loss function and the regression loss function to obtain a total loss function, calculate errors by using the total loss function, and optimize the initial gene expression prediction model by using a backpropagation algorithm to obtain a gene expression prediction model.

[0037] A gene expression prediction module is configured to input an H&E stained pathological image set into the trained gene expression prediction model to obtain a gene expression prediction result.

[0038] In a third aspect, an embodiment of the present application provides a computer device, which comprises a memory and a processor, the memory stores a computer program, and the processor executes the steps of the first aspect.

[0039] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the processor executes the steps of the first aspect when the computer program is executed.

[0040] The method, system, computer device and storage medium described above acquire a H&E stained pathological image sample set and a corresponding gene expression data sample set, the pathological image sample set includes pathological image samples of different scales, and a training set is constructed according to the pathological image sample set and a corresponding gene expression true value; the gene expression true value is obtained from a gene expression data sample set after normalization processing; the pathological image sample set is input into an initial gene expression prediction model to output image fusion features and a gene expression prediction value; a contrast learning model is used to project the image fusion features in a latent space to obtain image projection features, and the gene expression true value is encoded and projected in the latent space to obtain gene expression projection features, the image projection features are aligned with the gene expression projection features to construct a contrast learning loss function, the gene expression prediction value is compared with the gene expression true value to construct a regression loss function, the contrast learning loss function and the regression loss function are added to obtain a total loss function, the total loss function is used to calculate an error, and the initial gene expression prediction model is optimized through a back propagation algorithm to obtain a gene expression prediction model; a H&E stained pathological image set is input into the trained gene expression prediction model to obtain a gene expression prediction result. The present application can efficiently and accurately realize gene expression prediction only by using pathological images, and greatly reduce the acquisition cost of gene expression. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 A flowchart of a gene expression prediction method based on H&E stained pathological images in an embodiment;

[0042] Figure 2 A training schematic diagram of a gene expression prediction model in an embodiment;

[0043] Figure 3 A schematic diagram of an image encoder in an embodiment;

[0044] Figure 4 A schematic diagram of a first encoder in an embodiment;

[0045] Figure 5 A schematic diagram of a feature fusioner in an embodiment;

[0046] Figure 6 A schematic diagram of a gene expression decoder in an embodiment;

[0047] Figure 7 A schematic diagram of an image feature projector in an embodiment;

[0048] Figure 8 A schematic diagram of a gene expression prediction system based on H&E stained pathological images in an embodiment. DETAILED DESCRIPTION

[0049] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required to be used in the embodiments will be briefly introduced. Obviously, the drawings in the following description are only some examples or embodiments of the present application, and for those skilled in the art, the present application can also be applied to other similar scenarios without creative labor on the basis of these drawings. Unless the context clearly indicates otherwise or otherwise stated, the same reference numbers in the drawings represent the same structures or operations.

[0050] As shown in the present application and claims, unless the context clearly indicates otherwise or otherwise stated, the words "one", "a", "an", and / or "the" do not specifically refer to the singular, but can also include the plural. Generally speaking, the terms "comprise" and "include" only indicate the inclusion of the steps and elements explicitly identified, and these steps and elements do not constitute an exclusive list, and the method or device can also include other steps or elements.

[0051] Although the present application makes various references to certain modules in the device according to the embodiments of the present application, however, any number of different modules can be used and run on the computing device and / or processor. The modules are only illustrative, and different aspects of the device and method can use different modules.

[0052] It should be understood that when a unit or module is described as "connected", "coupled" to other units, modules or blocks, it can refer to direct connection or coupling, or communication with other units, modules or blocks, or there can be intermediate units, modules or blocks, unless the context clearly indicates otherwise. The term "and / or" used herein can include any and all combinations of one or more related listed items.

[0053] The inventive concept of the present application is that, in view of the high cost of existing spatial transcriptome technology, H&E staining images are widely used and easy to obtain, the present application provides a gene expression prediction method based on H&E staining pathological images, and the gene expression prediction result is obtained by predicting the H&E staining pathological picture through a gene expression prediction model, which has high prediction accuracy.

[0054] As shown in the present application and claims, unless the context clearly indicates otherwise or otherwise stated, the words "one", "a", "an", and / or "the" do not specifically refer to the singular, but can also include the plural. Generally speaking, the terms "comprise" and "include" only indicate the inclusion of the steps and elements explicitly identified, and these steps and elements do not constitute an exclusive list, and the method or device can also include other steps or elements. Figure 1 As shown in the present application and claims, unless the context clearly indicates otherwise or otherwise stated, the words "one", "a", "an", and / or "the" do not specifically refer to the singular, but can also include the plural. Generally speaking, the terms "comprise" and "include" only indicate the inclusion of the steps and elements explicitly identified, and these steps and elements do not constitute an exclusive list, and the method or device can also include other steps or elements.

[0055] S102, acquiring an H&E staining pathological image sample set and a corresponding gene expression data sample set, and constructing a training set according to the pathological image sample set and the corresponding gene expression true value;

[0056] The pathological image sample set includes pathological image samples of different scales.

[0057] In some embodiments, the pathological image sample set includes a current image block sample centered at a test point position coordinate, a neighborhood image block sample centered at the current image block sample, and an overall pathological image sample.

[0058] The neighborhood image block is, for example, 5x5 times the size of the current image block, and each current image block is, for example, 224 pixels x 224 pixels.

[0059] The gene expression ground truth is obtained from a set of normalized gene expression data samples. Specifically, the gene expression ground truth is divided by the sum of the gene expression values of each gene at the measurement points, and then normalized by The normalization process is as follows:

[0060]

[0061] wherein, represents the normalized gene expression ground truth corresponding to the current measurement point, represents the gene expression ground truth of the current measurement point, i represents the position index of the measurement point on the long side, and j represents the position index of the measurement point on the short side.

[0062] S104, inputting the pathological image sample set into an initial gene expression prediction model to output image fusion features and gene expression prediction values;

[0063] S106, projecting the image fusion features in the latent space using a contrast learning model to obtain image projection features, encoding the gene expression ground truth and projecting it in the latent space to obtain gene expression projection features, aligning the image projection features with the gene expression projection features to construct a contrast learning loss function, comparing the gene expression prediction values with the gene expression ground truth to construct a regression loss function, adding the contrast learning loss function and the regression loss function to obtain a total loss function, calculating errors using the total loss function, and optimizing the initial gene expression prediction model through a backpropagation algorithm to obtain a gene expression prediction model;

[0064] S108, inputting the H&E stained pathological image set into the trained gene expression prediction model to obtain gene expression prediction results.

[0065] In this embodiment, the pathological image sample set includes pathological image samples of different scales, the current image block sample records the feature information of the measurement points, the neighborhood image block sample records the texture information between image blocks and blocks, and the overall pathological image sample records the complete feature information of the H&E stained pathological image sample set. Therefore, the gene expression prediction model trained using the H&E stained pathological image sample set and the corresponding gene expression data sample set can more accurately obtain gene expression prediction results.

[0066] In the embodiment, the contrast learning loss function is constructed by aligning the image projection feature with the gene expression projection feature, the regression loss function is constructed by comparing the gene expression predicted value with the gene expression true value, the total loss function is obtained by adding the contrast learning loss function and the regression loss function, the lower bound of mutual information of the two modalities is improved through contrast learning between modalities, and the gene expression prediction model trained in this way can more accurately obtain the gene expression prediction result.

[0067] Figure 2 The following will describe the training of the gene expression prediction model in the embodiment of the application in detail with reference to the accompanying drawings. Figure 2 The technical solution of the application will be described in detail.

[0068] As shown in the figure, the gene expression prediction model comprises an image encoder and a gene expression decoder. The image encoder is configured to encode image features corresponding to pathological image samples of different scales to obtain corresponding image encoded features, and fuse the image encoded features to obtain the image fusion feature. The gene expression decoder is configured to decode the image fusion feature to obtain a gene expression predicted value. Figure 2 As shown in the figure, the contrast learning model comprises an image projector, a gene expression encoder, a gene expression projector and a contrast learner. The image projector is configured to project the image fusion feature in a latent space to obtain an image projection feature. The gene expression encoder is configured to encode the gene expression true value to obtain a gene expression encoded feature. The gene expression projector is configured to project the gene expression encoded feature in the latent space to obtain a gene expression projection feature. The contrast learner is configured to align the image projection feature with the gene expression projection feature to construct a contrast learning loss function, compare the gene expression predicted value with the gene expression true value to construct a regression loss function, add the contrast learning loss function and the regression loss function to obtain a total loss function, calculate an error using the total loss function, and optimize the initial gene expression prediction model through a backpropagation algorithm to obtain a gene expression prediction model.

[0069] Figure 2 The following will describe the architecture of each part of the gene expression prediction model in detail.

[0070] Specifically, as shown in the figure, the image encoder comprises a feature extractor and a feature fusion module. The feature extractor is configured to extract image features corresponding to pathological image samples of different scales. The feature fusion module is configured to fuse the image features extracted by the feature extractor to obtain the image fusion feature.

[0071] Specifically, as shown in the figure, the image encoder comprises a feature extractor and a feature fusion module. The feature extractor is configured to extract image features corresponding to pathological image samples of different scales. The feature fusion module is configured to fuse the image features extracted by the feature extractor to obtain the image fusion feature. Figure 3 ​As shown, the image encoder comprises a first encoder, a second encoder, a third encoder, and a feature fusioner. The first encoder is configured to encode image features corresponding to the whole pathological image sample to obtain first image encoding features; the second encoder is configured to encode image features corresponding to the neighborhood image block sample to obtain second image encoding features; the third encoder is configured to encode image features corresponding to the current image block sample to obtain third image encoding features; and the feature fusioner is configured to fuse the first image encoding features and the second image encoding features to obtain first fusion features, fuse the first image encoding features and the third image encoding features to obtain second fusion features, and fuse the first fusion features and the second fusion features to obtain the image fusion features.

[0072] In some example embodiments, as shown in FIG. 2, the first encoder comprises a UNI network module, a first Transformer layer, a CNN layer, and a second Transformer layer. Figure 4 As shown, the first encoder comprises a UNI network module, a first Transformer layer, a CNN layer, and a second Transformer layer; the UNI network module extracts image features of the whole pathological image sample; the first Transformer layer encodes image features corresponding to the whole pathological image sample to obtain first image encoding sub-features; the first image encoding sub-features are rearranged according to the original positions to obtain second image encoding sub-features; the CNN layer respectively processes the second image encoding sub-features through multiple convolution operations to obtain third image encoding sub-features; the third image encoding sub-features are rearranged to obtain fourth image encoding sub-features; and the second Transformer layer encodes the fourth image encoding sub-features to obtain the first image encoding features.

[0073] The UNI network module is pre-trained by a large number of pathological images and can extract features of pathological images, and the parameters of the UNI network module are unchanged in the training process of the gene expression prediction model.

[0074] The second encoder is composed of a UNI network module and a Transformer layer. Image features of 5x5 image blocks in the neighborhood extracted by the UNI network module are input into the Transformer layer after rearrangement to obtain second image encoding features of the neighborhood image blocks.

[0075] The third encoder is composed of a parameter-adjustable ResNet34 network layer and a Transformer layer. The current image block is encoded by the parameter-adjustable ResNet34 network layer to obtain a feature map with a size of 7 pixels x 7 pixels and a dimension of 512, and the feature map is input into the Transformer layer to obtain third image encoding features of the current image block.

[0076] The feature fusioner is composed of three Cross Attention fusion modules, as shown in FIG. 4. Figure 5The first fusion module is configured to fuse the first image encoding feature and the second image encoding feature to obtain a first fusion feature Fusion1; the second fusion module is configured to fuse the first image encoding feature and the third image encoding feature to obtain a second fusion feature Fusion2; and the third fusion module is configured to fuse the first fusion feature and the second fusion feature to obtain the image fusion feature Fusion3.

[0077] As shown in FIG. 6, the gene expression decoder is composed of a linear layer, a ReLU activation function and a linear layer. The image fusion feature Fusion3 obtained by the image encoder is input into the gene expression decoder to obtain a gene expression prediction value. Figure 6

[0078] The architecture of each part of the contrastive learning model is described in detail as follows.

[0079] As shown in FIG. 6, the image feature projector is composed of a linear layer, a GeLU activation function, a linear layer, a dropout layer and a normalization layer. The image fusion feature Fusion3 is input into the image projector to obtain an image projection feature projected in the latent space. Figure 7

[0080] The gene expression encoder is composed of a first Transformer layer, a CNN layer and a second Transformer layer. First, all the gene expressions corresponding to the image blocks on the whole pathological picture are arranged according to the positions of the image blocks. After the first Transformer layer, the obtained gene expression projection features are rearranged according to the original positions. The CNN layer with multiple convolutional layers is used to perform convolution operation on the rearranged gene expression projection features. Then, the convolutional gene expression projection features are arranged again. After the second Transformer layer, the gene expression encoding features of the whole tissue are obtained. The gene expression encoding features corresponding to the current image block are taken out separately.

[0081] The gene expression projector is also composed of a linear layer, a GeLU activation function, a linear layer, a dropout layer and a normalization layer. The gene expression encoding features are input into the gene expression projector to obtain a gene expression projection feature projected in the latent space.

[0082] In the contrastive learner, the similarity between the positive samples of the image projection features and the gene expression projection features in the latent space is calculated, and the contrastive loss is solved by using the cross-entropy loss function to align the image blocks and the gene expression data from the same position.

[0083] ​​Specifically, the obtained gene expression prediction value is compared with the gene expression true value using an MSE loss function to obtain a regression loss function. The comparison learning loss function is added thereto as a final total loss function for reverse gradient propagation. As shown in the following formula:

[0084]

[0085]

[0086]

[0087] wherein Y represents the gene expression true value, X represents the gene expression prediction value, i represents the index number of the gene, and n represents the total number of regression genes, represents the regression loss function, represents the comparison learning loss function, represents the final total loss function value.

[0088] In some embodiments, the pathological image sample set and the corresponding gene expression true value set are divided into a training set and a validation set according to patients, ensuring that different pathological images of each patient are located in the same data set division, and preventing the similarity of pathological images of the same patient in morphology and gene expression from affecting the training and evaluation of the model.

[0089] Then, the network parameters of the initial gene expression prediction model are optimized using an Adam optimizer, and the learning rate is In the training, the learning rate is adjusted to 0.9 times of the previous value every 50 rounds. A total of 100 rounds of training are performed at most, and when the correlation coefficient between the gene expression prediction value and the gene expression true value in the validation set does not increase in the last 20 rounds, the training is terminated. The weight of the round with the largest correlation coefficient in the training process is saved as the trained model weight.

[0090] After the gene expression prediction model is trained, a new pathological image is taken as a center point on the pathological image according to the test point position coordinates, and a current image block, a neighborhood image block with a size of 5 times the current image block, and an overall pathological image are taken as the center. The image block corresponding to the measurement point is 224 pixels x 224 pixels in size. The current image block, the neighborhood image block, and the overall pathological image are input into the trained gene expression prediction model, and the gene expression prediction result is output.

[0091] An embodiment of the present application proposes a gene expression prediction system based on H&E staining pathological images, as shown in Figure 8 The system comprises:

[0092] The data processing module 802 is configured to obtain a set of H&E-stained pathological image samples and a set of corresponding gene expression data samples, the set of pathological image samples including pathological image samples of different scales, and construct a training set according to the set of pathological image samples and corresponding gene expression true values; the gene expression true values are obtained from the set of gene expression data samples after normalization processing.

[0093] The model training module 804 is configured to input the set of pathological image samples into an initial gene expression prediction model, output image fusion features and gene expression prediction values, project the image fusion features in a latent space by using a contrast learning model to obtain image projection features, project the gene expression true values in the latent space after encoding to obtain gene expression projection features, align the image projection features with the gene expression projection features to construct a contrast learning loss function, compare the gene expression prediction values with the gene expression true values to construct a regression loss function, add the contrast learning loss function and the regression loss function to obtain a total loss function, calculate errors by using the total loss function, and optimize the initial gene expression prediction model by using a back propagation algorithm to obtain a gene expression prediction model.

[0094] The gene expression prediction module 806 is configured to input a set of H&E-stained pathological images into the trained gene expression prediction model to obtain gene expression prediction results.

[0095] It should be understood that, although each step in the above flowchart is displayed in sequence according to the direction of the arrow, these steps are not necessarily executed in sequence according to the direction of the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other sequences. Moreover, at least part of the steps in the above flowchart can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of these steps or stages is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or steps or stages in other steps.

[0096] In an embodiment, the present embodiment provides a computer device, including a memory and a processor, the memory stores a computer program, and the processor implements the steps in any one of the above-mentioned gene expression prediction methods based on H&E-stained pathological images when executing the computer program.

[0097] In an embodiment, the present embodiment provides a computer-readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps in any one of the above-mentioned gene expression prediction methods based on H&E-stained pathological images.

[0098] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiment methods. Any reference to memory, storage, database or other medium used in the embodiments provided by the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0099] Any combination of the technical features of the above embodiments can be made. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, but as long as the combination of the technical features does not exist, it should be considered as the scope of the present application.

[0100] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent. It should be pointed out that for those skilled in the art, without departing from the concept of the present application, some modifications and improvements can be made, which are all within the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. A gene expression prediction method based on H&E staining pathological images, characterized in that, The method includes: A set of pathological images with H&E staining and a corresponding set of gene expression data samples were obtained. The set of pathological images included pathological images at different scales. A training set was constructed based on the set of pathological images and the corresponding true gene expression values. The true gene expression values ​​were obtained from the normalized set of gene expression data samples. The pathological image sample set is input into the initial gene expression prediction model, and the image fusion features and gene expression prediction values ​​are output. The image fusion features are projected into the latent space using a contrastive learning model to obtain image projection features. The true gene expression values ​​are encoded and projected into the latent space to obtain gene expression projection features. The image projection features and gene expression projection features are aligned to construct a contrastive learning loss function. The predicted gene expression values ​​are compared with the true gene expression values ​​to construct a regression loss function. The contrastive learning loss function and the regression loss function are added to obtain a total loss function. The error is calculated using the total loss function, and the initial gene expression prediction model is optimized using a backpropagation algorithm to obtain a gene expression prediction model. The set of pathological images stained with H&E is input into the trained gene expression prediction model to obtain gene expression prediction results.

2. The method according to claim 1, characterized in that, The pathological image sample set includes the current image block sample centered on the test point location coordinates, the neighboring image block samples centered on the current image block sample, and the overall pathological image sample.

3. The method according to claim 2, characterized in that, The initial gene expression prediction model includes an image encoder and a gene expression decoder; The image encoder is used to encode the image features corresponding to pathological image samples of different scales to obtain the corresponding image encoded features, and then fuse them to obtain the image fusion features; The gene expression decoder is used to decode the image fusion features to obtain gene expression prediction values.

4. The method according to claim 3, characterized in that, The image encoder includes a first encoder, a second encoder, a third encoder, and a feature fusion unit; The first encoder is used to encode the image features corresponding to the overall pathological image sample to obtain the first image encoded features; The second encoder is used to encode the image features corresponding to the neighboring image block samples to obtain the second image encoded features; The third encoder is used to encode the image features corresponding to the current image block sample to obtain the third image encoded features; The feature fusion unit is used to fuse the first image coding feature and the second image coding feature to obtain a first fused feature, fuse the first image coding feature and the third image coding feature to obtain a second fused feature, and fuse the first fused feature and the second fused feature to obtain the image fused feature.

5. The method according to claim 4, characterized in that, The first encoder includes a UNI network module, a first Transformer layer, a CNN layer, and a second Transformer layer; The UNI network module extracts the image features of the overall pathological image sample; The first Transformer layer encodes the image features corresponding to the overall pathological image sample to obtain the first image encoded sub-feature; The first image coding sub-features are rearranged according to their original positions to obtain the second image coding sub-features; The CNN layer performs multiple convolution operations on the second image coding sub-features to obtain the third image coding sub-features; The third image coding sub-feature is rearranged to obtain the fourth image coding sub-feature; The second Transformer layer encodes the fourth image coding sub-feature to obtain the first image coding feature.

6. The method according to claim 1, characterized in that, The contrastive learning model includes an image projector, a gene expression encoder, a gene expression projector, and a contrastive learner. The image projector is used to project the image fusion features into the latent space to obtain image projection features; The gene expression encoder is used to encode the true gene expression value to obtain gene expression coding features; The gene expression projector is used to project the gene expression coding features into the latent space to obtain gene expression projection features; The contrastive learner aligns the image projection features with the gene expression projection features to construct a contrastive learning loss function, compares the predicted gene expression value with the true gene expression value to construct a regression loss function, adds the contrastive learning loss function and the regression loss function to obtain a total loss function, calculates the error using the total loss function, and optimizes the initial gene expression prediction model using the backpropagation algorithm to obtain a gene expression prediction model.

7. The method according to claim 1, characterized in that, The method further includes: A validation set is constructed based on the pathological image sample set and the corresponding true gene expression values; The initial gene expression prediction model is validated using the validation set until the correlation coefficient between the predicted gene expression value and the corresponding true gene expression value no longer increases over multiple training rounds, at which point training is stopped.

8. A gene expression prediction system based on H&E staining pathological images, characterized in that, The system includes: A set of pathological images with H&E staining and a corresponding set of gene expression data samples were obtained. The set of pathological images included pathological images at different scales. A training set was constructed based on the set of pathological images and the corresponding true gene expression values. The true gene expression values ​​were obtained from the normalized set of gene expression data samples. The model training module is used to input the pathological image sample set into the initial gene expression prediction model and output image fusion features and gene expression prediction values. Using a contrastive learning model, the image fusion features are projected into the latent space to obtain image projection features, and the encoded gene expression values ​​are projected into the latent space to obtain gene expression projection features. The image projection features and gene expression projection features are aligned to construct a contrastive learning loss function. The predicted gene expression values ​​are compared with the actual gene expression values ​​to construct a regression loss function. The contrastive learning loss function and the regression loss function are added to obtain a total loss function. The error is calculated using the total loss function, and the initial gene expression prediction model is optimized using a backpropagation algorithm to obtain the gene expression prediction model. The gene expression prediction module is used to input the set of H&E-stained pathological images into the trained gene expression prediction model to obtain gene expression prediction results.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images