Gene expression prediction method and system based on multi-modal comparative learning and guidance mechanism

Abstract

The invention discloses the technical field of pathology and space transcriptomics, and particularly relates to a gene expression prediction method and system based on multi-modal comparative learning and a guidance mechanism. Cutting the histological slice image into image blocks according to space coordinates; according to the method, a local convolution branch and a global Transform branch are combined to extract image features, the image features are mapped to a shared potential space through projection, soft contrast, hard contrast and global consistency constraints are introduced into the space, and cross-modal alignment of an image modal and a gene expression modal is realized; an expression prediction head is introduced in the training stage, representation learning is directly guided by a regression signal, and the relation between feature learning and gene expression prediction is broken through; in the inference stage, k-nearest neighbor retrieval and a multi-distance weighted aggregation strategy are combined to infer a gene expression profile of an unknown position. According to the method, the accuracy and robustness of space gene expression prediction can be effectively improved, the tissue space heterogeneity structure is kept, and the method has high clinical application and scientific research and popularization value.

Description

Technical Field

[0001] This invention relates to the fields of pathology and spatial transcriptomics, specifically to a gene expression prediction method and system based on multimodal contrastive learning and guidance mechanisms. Background Technology

[0002] In clinical diagnosis and pathological research, histological imaging techniques such as H&E staining are widely used. They can clearly present tissue morphology and cellular characteristics, providing core morphological evidence for disease diagnosis. However, they can only reflect morphological information and cannot reveal the molecular regulatory mechanism of gene expression, making it difficult to link to the molecular pathological basis of diseases.

[0003] The emergence of spatial transcriptomics has solved this problem, as it can measure local gene expression while preserving tissue spatial location information, providing new tools for elucidating the molecular mechanisms of diseases and discovering diagnostic biomarkers, and showing significant potential in tumor microenvironment and neurodegenerative disease research. However, this technology has complex experimental procedures, high requirements for equipment and operation, and high reagent and instrument costs, making it difficult to popularize in large-scale clinical screening and multi-center studies, and failing to meet the demand for large-scale acquisition of gene expression information.

[0004] To address this, researchers have attempted to use deep learning methods to infer spatially resolved gene expression from H&E images, hoping to indirectly obtain gene expression information using low-cost and readily available images. However, existing deep learning methods have significant drawbacks: First, they simplify prediction to a single regression problem, focusing only on minimizing error and failing to establish a synergistic relationship between representation learning and prediction modeling, resulting in a disconnect between the two and an inability of feature learning to effectively support prediction performance. Second, they are insufficient in characterizing the long-range dependencies and spatial heterogeneity of complex tissues, relying heavily on local convolutions to extract features, making it difficult to model spatial relationships between different regions and failing to fully capture gene expression heterogeneity, thus failing to accurately reflect differences in the spatial distribution of gene expression. Third, their generalization ability is limited, with training overly relying on the distribution of specific datasets and failing to consider sample morphology and experimental batch differences, leading to decreased accuracy when applied to new samples. Furthermore, some models lose key spatial pattern information in pursuit of smooth results, further reducing their practical value. Summary of the Invention

[0005] The purpose of this invention is to propose a gene expression prediction method and system based on multimodal contrastive learning and guidance mechanisms. In the training phase, an expression prediction head is introduced as a guidance signal, and joint modeling of local convolutional features and global Transformer features is combined. In the inference phase, robust cross-sample prediction is achieved through k-nearest neighbor retrieval and multi-distance weighted aggregation.

[0006] According to a first aspect of the embodiments of this disclosure, a gene expression prediction method based on multimodal contrastive learning and guidance mechanisms is provided, comprising the following steps: Acquire histological images and corresponding spatial gene expression data, and preprocess the histological images and spatial gene expression data respectively to obtain training data; Local convolutional neural networks and lightweight visual Transformers are used to extract fine-grained image features and global semantic features from histological images in the training data, respectively. At the same time, multi-layer fully connected layers are used to extract spatial gene expression data features. The fine-grained image features, global semantic features, and gene expression data features are projected into a shared latent space; An expression prediction head is introduced into the shared latent space to perform regression prediction on histological images, thereby obtaining a supervision signal. The supervision signal is a regression supervision signal, which is used to guide the optimization process of contrastive learning and bridge the gap between representation learning and gene expression modeling. Global consistency constraints are obtained through global semantic features. The supervision signal and global consistency constraints are jointly modeled to guide the comparative learning of fine-grained image features and spatial gene expression data. During the inference phase, k-nearest neighbor retrieval is performed on the target image embedding, and then combined with a multi-distance weighted aggregation mechanism, the gene expression value at the target location is inferred from the true expression profiles of neighboring spots; Output a spatially resolved predicted gene expression matrix for disease partitioning or functional enrichment analysis.

[0007] In one embodiment, the local convolutional neural network is ResNet50, which extracts fine-grained image features to capture cellular-level textures and local area features, and the lightweight visual Transformer is MobileViT, which extracts global semantic features to model cross-regional dependencies and long-range relationships.

[0008] In one embodiment, the contrast loss includes soft contrast loss and hard contrast loss; the soft contrast loss uses KL divergence to constrain the alignment of cross-modal distributions, and the hard contrast loss is used to maximize the probability of correct matching between image patches and gene loci.

[0009] In one embodiment, the global consistency constraint is implemented by comparing global semantic features with gene expression data features, and is used to guide the comparative learning between supervised local features and gene expression data features.

[0010] In one embodiment, the expression prediction head includes multiple fully connected layers, each employing the GELU activation function and LayerNormalization.

[0011] In one embodiment, the multi-distance weighted aggregation mechanism uses distances including cosine distance, Euclidean distance, and Manhattan distance, and uses the reciprocal of each distance as a weight to perform a weighted average of the true representation spectra of adjacent spots.

[0012] In one embodiment, the k-nearest neighbor retrieval searches for the most similar training sample embeddings in a shared latent space to determine neighboring spots.

[0013] According to a second aspect of the present disclosure, a gene expression prediction system based on multimodal contrastive learning and guidance mechanisms is provided, comprising: The preprocessing module acquires histological images and corresponding spatial gene expression data, and preprocesses the histological images and spatial gene expression data respectively to obtain training data. The multimodal feature extraction module uses a local convolutional neural network and a lightweight visual Transformer to extract fine-grained image features and global semantic features of histological images in the training data, respectively, while using multi-layer fully connected layers to extract spatial gene expression data features. The feature space mapping module projects the fine-grained image features, global semantic features, and gene expression data features into a shared latent space; The supervision signal generation module introduces an expression prediction head into the shared latent space, performs regression prediction on the histological image, and obtains a supervision signal; the supervision signal is a regression supervision signal, which is used to guide the optimization process of contrastive learning and bridge the gap between representation learning and gene expression modeling. The contrastive learning guidance module obtains global consistency constraints through global semantic features, jointly models the supervision signal and global consistency constraints, and guides the contrastive learning of fine-grained image features and spatial gene expression data. In the inference and prediction module, during the inference phase, k-nearest neighbor retrieval is performed on the target image embedding, and then combined with a multi-distance weighted aggregation mechanism, the gene expression value at the target location is inferred from the true expression profile of the neighboring spots. The results output and application module outputs a spatially resolved predicted gene expression matrix for disease partitioning or functional enrichment analysis.

[0014] According to a third aspect of the present disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and running on the memory, wherein the processor executes the program to implement the gene expression prediction method based on multimodal contrastive learning and guidance mechanism.

[0015] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the gene expression prediction method based on multimodal contrastive learning and guidance mechanism.

[0016] The advantages of the above technical solutions adopted in this invention compared with the prior art are as follows: 1. This invention accurately captures the deep correlation between histological images and gene expression through multimodal feature joint extraction, cross-modal feature alignment, and contrastive learning-guided mechanisms, effectively improving the accuracy of spatial gene expression prediction. It is particularly outstanding in predicting marker genes, highly expressed genes, and highly variable genes, providing reliable data support for subsequent disease-related analyses.

[0017] 2. In the inference stage, k-nearest neighbor retrieval and multi-distance weighted aggregation mechanism are adopted to make full use of the real expression information in the training samples, reduce the bias caused by a single distance metric, improve the model's generalization ability on unknown new samples and samples from different experimental batches, and avoid the problem of large fluctuations in prediction results due to sample differences.

[0018] 3. By jointly modeling with local convolution and global Transformer, we can take into account both the fine-grained features of local tissues and long-range dependencies across regions. At the same time, we can avoid excessive smoothing of features by using global consistency constraints, thus fully preserving the spatial heterogeneity of tissues and the spatial distribution pattern of gene expression, and accurately restoring key information such as the gene expression boundary between diseased and normal areas.

[0019] 4. It effectively solves the core problems in existing methods, such as the separation of representation learning and prediction modeling, insufficient characterization of long-range dependencies, and limited generalization performance. It constructs a complete technical link of "feature extraction-spatial mapping-supervised guidance-inference prediction", providing a better technical solution for spatial gene expression prediction.

[0020] 5. Without relying on complex and expensive spatial transcriptomics experiments, spatially resolved gene expression inference can be achieved using only conventional H&E staining images, significantly reducing the cost and threshold for obtaining gene expression information. This facilitates widespread application in scenarios such as large-scale clinical screening and multi-center pathological research. At the same time, it provides a powerful tool for research on disease molecular mechanisms, disease regional identification, and functional enrichment analysis, and has broad prospects for promotion. Attached Figure Description

[0021] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments of this application and their descriptions are used to explain this application and do not constitute an undue limitation of this application.

[0022] Figure 1This is a block diagram illustrating the principle of a gene expression prediction method based on multimodal contrastive learning and guidance mechanisms. Detailed Implementation

[0023] The present disclosure will be further described below with reference to the accompanying drawings and embodiments.

[0024] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0025] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of methods and systems according to various embodiments of this disclosure. It should be noted that each block in a flowchart or block diagram may represent a module, segment, or portion of code, which may include one or more executable instructions for implementing the logical functions specified in the various embodiments. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than that shown in the drawings. For example, two consecutively represented blocks may actually be executed substantially in parallel, or they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, may be implemented using a dedicated hardware-based system that performs the specified functions or operations, or using a combination of dedicated hardware and computer instructions.

[0026] Example 1: This embodiment provides a gene expression prediction method based on multimodal contrastive learning and guidance mechanisms, including the following steps: S1. Acquire histological images and corresponding spatial gene expression data, and preprocess the histological images and spatial gene expression data respectively to obtain training data; Specifically, the preprocessing includes: cropping and data augmentation of the histological image, and normalization and feature screening of the gene expression matrix corresponding to the spatial gene expression data.

[0027] In this embodiment, three slices (numbered 1, 2, and 4) from the human liver 10xVisium dataset were selected. Each image was divided into 224×224 blocks and randomly rotated. ), horizontal flipping, and color normalization. After normalization, 3,000 highly variable genes were selected from approximately 20,000 genes.

[0028] S2. Local convolutional neural networks and lightweight visual Transformers are used to extract fine-grained image features and global semantic features of histological images in the training data, respectively. At the same time, multi-layer fully connected layers are used to extract spatial gene expression data features. In this embodiment, the input image block is first processed by a local convolutional neural network ResNet50 to obtain a local feature vector with a dimension of 2048; at the same time, the image block is processed by a lightweight visual TransformerMobileViT to obtain a global feature vector with a dimension of 512.

[0029] S3. Project the fine-grained image features, global semantic features, and gene expression data features into a shared latent space; In this embodiment, image features and gene expression features are concatenated and projected into a 128-dimensional latent space. L2 regularization is used to ensure that the vector lengths of different modal features remain consistent, facilitating subsequent similarity calculations.

[0030] S4. An expression prediction head is introduced into the shared latent space to perform regression prediction on the histological image to obtain a supervision signal; the supervision signal is a regression supervision signal, which is used to guide the optimization process of contrastive learning and bridge the gap between representation learning and gene expression modeling. In this embodiment, the prediction head includes three fully connected layers with dimensions of 512→256→128 respectively. Each layer employs GELU activation and Layer Normalization. The final output dimension is 3000, corresponding to the selected gene set.

[0031] S5. Global consistency constraints are obtained through global semantic features. The supervision signal and global consistency constraints are jointly modeled to guide the comparative learning of fine-grained image features and spatial gene expression data. In this embodiment, for each image patch, its cosine similarity with all gene loci is calculated, and a probability distribution is obtained through softmax. This distribution is compared with the soft target distribution based on the real spatial location to calculate the KL divergence, which serves as the soft contrast loss. Simultaneously, cross-entropy loss is applied to the closest real gene loci as the hard contrast loss. Then, consistency constraints are constructed by comparing the features aggregated from local patches with the global MobileViT features. The loss function uses mean squared error (MSE) to minimize the difference between the predicted expression and the real expression.

[0032] S6. In the inference stage, k-nearest neighbor retrieval is performed on the target image embedding, and then combined with the multi-distance weighted aggregation mechanism, the gene expression value of the target position is inferred from the true expression profile of the neighboring spots; In this embodiment, k is set to 5. For each test spot, the cosine distance, Euclidean distance, and Manhattan distance between it and all training spots are calculated. Using the reciprocal of each distance as a weight, a weighted average of the true representations of the five adjacent spots is taken to obtain the final prediction result.

[0033] S7. Output a spatially resolved predicted gene expression matrix for disease partitioning or functional enrichment analysis.

[0034] As shown in Table 1, based on the human liver 10xVisium dataset, the MViTGene of this invention improves the prediction relevance of marker genes (MG), highly expressed genes (HEG), and highly variable genes (HVG) by approximately 0.20, 0.33, and 0.12, respectively, which is a significant advantage over existing methods.

[0035] Table 1 shows the average correlation of predicted expression. Table 1 shows the average predicted expression correlation of the 8 marker genes (MG), the predicted expression correlation of the top 50 highly expressed genes (HEG), and the predicted expression correlation of the top 50 highly variable genes (HVG). All of these correlations are obtained by comparing the predicted expression of each gene with the actual expression on the preserved dataset.

[0036] Example 2: This embodiment provides a gene expression prediction system based on multimodal contrastive learning and guidance mechanisms, including: The preprocessing module acquires histological images and corresponding spatial gene expression data, and preprocesses the histological images and spatial gene expression data respectively to obtain training data. The multimodal feature extraction module uses a local convolutional neural network and a lightweight visual Transformer to extract fine-grained image features and global semantic features of histological images in the training data, respectively, while using multi-layer fully connected layers to extract spatial gene expression data features. The feature space mapping module projects the fine-grained image features, global semantic features, and gene expression data features into a shared latent space; The supervision signal generation module introduces an expression prediction head into the shared latent space, performs regression prediction on the histological image, and obtains a supervision signal; the supervision signal is a regression supervision signal, which is used to guide the optimization process of contrastive learning and bridge the gap between representation learning and gene expression modeling. The contrastive learning guidance module obtains global consistency constraints through global semantic features, jointly models the supervision signal and global consistency constraints, and guides the contrastive learning of fine-grained image features and spatial gene expression data. In the inference and prediction module, during the inference phase, k-nearest neighbor retrieval is performed on the target image embedding, and then combined with a multi-distance weighted aggregation mechanism, the gene expression value at the target location is inferred from the true expression profile of the neighboring spots. The results output and application module outputs a spatially resolved predicted gene expression matrix for disease partitioning or functional enrichment analysis.

[0037] The above modules can be deployed on the same device or distributed devices; the division of modules is only a functional logic description and does not limit the specific physical boundaries or implementation order.

[0038] Example 3: An electronic device is provided for running the aforementioned "gene expression prediction method based on multimodal contrastive learning and guidance mechanism". The electronic device includes: a processor, a memory, and optional communication interface / display device / input device, etc.; the memory stores a computer program that can run on the processor, and when the processor executes the program, it implements steps S1 to S7 of the method described in Embodiment 1, specifically including but not limited to: S1. Acquire histological images and corresponding spatial gene expression data, and preprocess the histological images and spatial gene expression data respectively to obtain training data; S2. Local convolutional neural networks and lightweight visual Transformers are used to extract fine-grained image features and global semantic features of histological images in the training data, respectively. At the same time, multi-layer fully connected layers are used to extract spatial gene expression data features. S3. Project the fine-grained image features, global semantic features, and gene expression data features into a shared latent space; S4. An expression prediction head is introduced into the shared latent space to perform regression prediction on the histological image to obtain a supervision signal; the supervision signal is a regression supervision signal, which is used to guide the optimization process of contrastive learning and bridge the gap between representation learning and gene expression modeling. S5. Global consistency constraints are obtained through global semantic features. The supervision signal and global consistency constraints are jointly modeled to guide the comparative learning of fine-grained image features and spatial gene expression data. S6. In the inference stage, k-nearest neighbor retrieval is performed on the target image embedding, and then combined with the multi-distance weighted aggregation mechanism, the gene expression value of the target position is inferred from the true expression profile of the neighboring spots; S7. Output a spatially resolved predicted gene expression matrix for disease partitioning or functional enrichment analysis.

[0039] The electronic device hardware can be one of a server, personal computer, workstation, industrial controller, edge computing device, or mobile terminal; the processor can be a general-purpose CPU, GPU, NPU, FPGA, or a combination thereof; the memory can be RAM, ROM, flash memory, or disk array. The device can interact with local / remote data storage (acquiring observation data and outputting inversion results) through a communication interface. The above hardware configuration does not constitute a limitation of the present invention.

[0040] Example 4: A computer-readable storage medium storing a computer program, which, when run on a processor of an electronic device, causes the program to execute the method steps S1 to S7 described in Embodiment 1; the storage medium may be a disk, optical disk, flash memory, solid-state drive, read-only memory, random access memory, or any combination of the above media.

[0041] Those skilled in the art will understand that the modules or steps described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, which can then be stored in a storage device for execution by a computer device. Alternatively, they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. This disclosure is not limited to any particular combination of hardware and software.

[0042] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

[0043] While the specific embodiments of this disclosure have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of this disclosure. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of this disclosure are still within the scope of protection of this disclosure.

Claims

1. A gene expression prediction method based on multi-modal contrastive learning and guidance mechanism, characterized in that, The method comprises the following steps: histological images and corresponding spatial gene expression data are obtained, and the histological images and the spatial gene expression data are preprocessed respectively to obtain training data; local convolutional neural networks and lightweight visual Transformers are used to extract fine-grained image features and global semantic features of the histological images in the training data, and multi-layer fully connected layers are used to extract features of the spatial gene expression data; the fine-grained image features, the global semantic features and the gene expression data features are projected into a shared latent space; an expression prediction head is introduced in the shared latent space to regress and predict the histological images to obtain a supervision signal; the supervision signal is a regression supervision signal, which is used to guide the optimization process of contrast learning and bridge the gap between representation learning and gene expression modeling; global consistency constraints are obtained from the global semantic features, and the supervision signal and the global consistency constraints are jointly modeled to guide the contrast learning of the fine-grained image features and the spatial gene expression data; in the inference stage, k-nearest neighbor retrieval is performed on target image embedding, and then a multi-distance weighted aggregation mechanism is combined to infer gene expression values of target positions from real expression profiles of adjacent spots; a spatially resolved predicted gene expression matrix is output, which is used for disease partition identification or functional enrichment analysis.

2. The method of claim 1, wherein, The local convolutional neural network is ResNet50, which extracts fine-grained image features to capture cell-level texture and local range region features. The lightweight visual Transformer is MobileViT, which extracts global semantic features to model cross-region dependencies and long-range relationships.

3. The method of claim 1, wherein the method is based on a multi-modal contrastive learning and a guidance mechanism. The contrast loss includes soft contrast loss and hard contrast loss; the soft contrast loss uses KL divergence to constrain the alignment of cross-modal distributions, and the hard contrast loss is used to maximize the correct matching probability of image blocks and gene sites.

4. The method of claim 1, wherein, The global consistency constraint is realized by comparing the global semantic features with the gene expression data features, and is used to guide the contrast learning between the supervised local features and the gene expression data features.

5. The method of claim 1, wherein, The expression prediction head comprises multi-layer fully connected layers, each layer of which adopts a GELU activation function and a Layer Normalization.

6. The method of claim 1, wherein, The multi-distance weighted aggregation mechanism adopts distances including cosine distance, Euclidean distance and Manhattan distance, and inverses of the distances are used as weights to perform weighted averaging on real expression profiles of adjacent spots.

7. The method of claim 1, wherein, The k-nearest neighbor retrieval searches for the most similar training sample embedding in the shared latent space to determine adjacent spots.

8. A gene expression prediction system based on multi-modal contrastive learning and guiding mechanism, characterized in that, The method comprises the following steps: a preprocessing module is used to obtain histological images and corresponding spatial gene expression data, and the histological images and the spatial gene expression data are preprocessed respectively to obtain training data; a multi-modal feature extraction module is used to extract fine-grained image features and global semantic features of the histological images in the training data by using local convolutional neural networks and lightweight visual Transformers, and features of the spatial gene expression data are extracted by using multi-layer fully connected layers; The feature space mapping module projects the fine-grained image features, global semantic features and gene expression data features into a shared latent space; The supervised signal generation module introduces an expression prediction head in the shared latent space to perform regression prediction on the histology image to obtain a supervised signal; the supervised signal is a regression supervised signal, used to guide the optimization process of the contrast learning and bridge the gap between representation learning and gene expression modeling; The contrast learning guidance module obtains a global consistency constraint through the global semantic features, jointly models the supervised signal and the global consistency constraint, and guides the contrast learning of the fine-grained image features and the spatial gene expression data; The inference prediction module, in the inference stage, performs k-neighbor retrieval on the target image embedding, and then combines a multi-distance weighted aggregation mechanism to infer the gene expression value of the target position from the real expression profile of the adjacent spot. The result output and application module outputs a spatially resolved predicted gene expression matrix for disease partition identification or functional enrichment analysis.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory to run on the processor, characterized in that, The processor executes the program to implement the gene expression prediction method based on multi-modal contrast learning and guidance mechanism of any one of claims 1-7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the gene expression prediction method based on multi-modal contrast learning and guidance mechanism of any one of claims 1-7.

Images