Cell type and cell abundance identification method and system based on cross-modal training

Abstract

The present invention proposes a method and system for identifying cell types and cell abundance based on cross-modal training, which relates to the field of image processing and addresses the following problem: existing recognition technologies mostly rely on non-standard factors such as manually labeled position annotation information, and fail to fully utilize the rich morphological patterns in tissue pathology images, affecting recognition reliability and accuracy. The method obtains spatial transcriptomics data of pathology image-gene expression matching, preprocesses it, and obtains highly expressed gene expression data, local image blocks, cell types, and abundance labels; constructs a cross-modal joint representation learning model, inputs highly expressed gene expression data and local image blocks into the model for training; and predicts the histological image to be predicted based on the trained model. The present invention solves the problems existing in the prior art, improves the ability to predict cell abundance from histological images, and fully reveals the spatial distribution of fine-grained cell types.

Description

Technical Field

[0001] The present invention belongs to the field of image processing, and in particular relates to a method and system for identifying cell types and cell abundance based on cross-modal training. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Cells are the fundamental units of life, forming the structural and functional basis of organisms' tissues and organs. Tissue cytoarchitecture, which refers to the spatial arrangement and morphological characteristics of cells, provides key insights into how cellular interactions contribute to biological behavior, including tissue development, disease progression, and therapeutic response. In cancer research, cellular spatial architecture can elucidate key features of the tumor microenvironment, heterogeneity, and tumor lymphocyte infiltration, and their impact on patient prognosis and personalized treatment strategies.

[0004] Current methods for cell type identification can be broadly divided into two groups.

[0005] (1) Methods based on computational pathology use deep learning to achieve cell nucleus instance segmentation and classification. These methods can identify cell types in tissue pathology images, but they are limited to identifying coarse-grained cell categories, usually no more than five major cell types. This limits the exploration of more refined cell type subtypes. Moreover, these cell type recognition methods all rely on manually annotated position annotation information, which introduces non-standard factors and affects the reliability of the final inference recognition results.

[0006] (2) Spatial transcriptomics (ST)-based methods, such as Seurat, RCTD, and Cell2location, typically use cell type deconvolution algorithms to estimate the cell types and proportions within each spatial transcriptome grid using gene expression data. These methods integrate single-cell RNA sequencing (scRNA-seq) reference transcriptome data to achieve fine-grained cell type analysis based on spatial transcriptome profiles. However, these methods mainly rely on gene expression and fail to utilize the rich morphological patterns present in tissue pathology images. Summary of the Invention

[0007] To overcome the shortcomings of the above-mentioned existing technologies, the present invention provides a method and system for cell type and cell abundance identification based on cross-modal training. Through a cross-modal joint representation learning framework, it integrates image morphology and molecular gene expression patterns, enhances the interaction between different modalities, and realizes the identification of fine-grained cell types and the abundance information of corresponding cell types from only tissue pathology images.

[0008] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0009] A first aspect of the present invention provides a method for identifying cell types and cell abundance based on cross-modal training, comprising:

[0010] Acquire spatial transcriptomics data matching pathological images and gene expression;

[0011] Preprocessing the spatial transcriptomics data of the pathological image-gene expression matching to obtain a preprocessed comprehensive data set, which includes highly expressed gene expression data, local image blocks, cell types, and cell abundance labels;

[0012] Construct a cross-modal joint representation learning model, input the highly expressed gene expression data and local image blocks into the cross-modal joint representation learning model for training, and obtain a trained cross-modal joint representation learning model;

[0013] Based on the trained cross-modal joint representation learning model, cell type and cell abundance prediction are performed on the histological images to be identified.

[0014] As an embodiment, the spatial transcriptomics data of the pathological image-gene expression matching is preprocessed, and the specific process is as follows:

[0015] For the spatial transcriptomics data of the pathological image-gene expression matching, a cell type deconvolution method is used to generate fine-grained cell types and corresponding cell abundance labels;

[0016] Extracting local image blocks and highly expressed gene expression data from the spatial transcriptomics data of the pathological image-gene expression matching;

[0017] A comprehensive dataset is constructed by combining fine-grained cell types, corresponding cell abundance labels, local image patches, and highly expressed gene expression data.

[0018] As an implementation method, a cross-modal joint representation learning model is constructed, wherein the cross-modal joint representation learning model includes a morphological modality representation module, a molecular modality representation module and a cross-modal embedding alignment module.

[0019] As an implementation method, the highly expressed gene expression data and local image patches are input into a cross-modal joint representation learning model for training. The specific process is as follows:

[0020] Input the local image block into the morphological modality representation module to obtain image feature embedding;

[0021] The expression data of highly expressed genes are input into the molecular modality representation module to obtain molecular expression embeddings;

[0022] The image feature embedding and molecular expression embedding are input into the cross-modal embedding alignment module to obtain the predicted abundance value and original gene expression pattern of the cell type, respectively.

[0023] As an implementation method, the local image block is input into a morphological modality representation module, wherein the morphological modality representation module includes a feature backbone module and a transformation layer. The specific process is as follows:

[0024] The morphological features of local image blocks are extracted through the feature backbone module;

[0025] The morphological features of the local image block are input into the transformation layer for nonlinear transformation to obtain the image block feature embedding.

[0026] As an embodiment, the expression data of highly expressed genes are input into a molecular modality representation module, wherein the molecular modality representation module includes a self-normalization network module and a transformation layer. The specific process is as follows:

[0027] Through the self-normalization network module, the features of the highly expressed gene expression data are enhanced to obtain the enhanced highly expressed gene expression data;

[0028] The enhanced highly expressed gene expression data are mapped to molecular expression embeddings through the transformation layer.

[0029] As an implementation method, the image feature embedding and the molecular expression embedding are input into a cross-modal embedding alignment module, which includes a multi-layer perceptron and a decoder. The specific process is as follows:

[0030] Align image patch feature embeddings with molecular expression embeddings;

[0031] The aligned image patch features are embedded and input into a multilayer perceptron to obtain the predicted abundance value of the cell type;

[0032] The aligned molecular expression embeddings are input to the decoder to reconstruct the original gene expression patterns.

[0033] As an implementation method, the model is optimized by the overall loss, and the overall loss formula is:

[0034]

[0035] in and Represent the cross-modal consistency comparison loss Cell abundance predicts loss and gene expression reconstruction loss The balance weight of θ morph ,θ molec 、 and yes and The parameter in argmin(·) aims to find the function that minimizes the overall loss Optimization parameters

[0036] A second aspect of the present invention provides a cell type and cell abundance identification system based on cross-modal training, comprising:

[0037] Data acquisition module, used to obtain spatial transcriptomics data for pathological image-gene expression matching;

[0038] a data processing module for preprocessing the spatial transcriptomics data of the pathological image-gene expression matching to obtain a preprocessed comprehensive data set, which includes highly expressed gene expression data, local image blocks, cell type and cell abundance labels;

[0039] The model construction and training module is used to build a cross-modal joint representation learning model. The highly expressed gene expression data and local image blocks are input into the cross-modal joint representation learning model for training to obtain a trained cross-modal joint representation learning model.

[0040] The model recognition module is used to predict the cell type and cell abundance of the histological images to be identified based on the trained cross-modal joint representation learning model.

[0041] A third aspect of the present invention provides a computer device comprising a memory, a processor, and a program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the method described in the first aspect of the present invention are implemented.

[0042] One or more of the above technical solutions have the following beneficial effects:

[0043] In this embodiment, by constructing a cross-modal joint representation learning model, cross-modal training for identifying fine-grained cell types and cell abundance from pathological images is achieved. This model not only improves the ability to predict cell abundance from histological images, but also reveals the spatial distribution of fine-grained cell types, thereby enabling the prediction of fine-grained cell types and the abundance information of corresponding cell types from only tissue pathology images.

[0044] In this example, the morphological modality representation module is used to learn the morphological patterns present in local image patches. Simultaneously, a molecular modality representation module is introduced to extract key molecular features inherent in gene expression data through gene expression reconstruction tasks, fully leveraging the rich morphological patterns present in tissue pathology images.

[0045] In this embodiment, cross-modal embedding is used to align the embedding spaces of morphological and molecular modalities during training, which can integrate image morphology and molecular gene expression patterns, resulting in better fine-grained cell type recognition effects, which reveals that molecular gene expression information enhances the morphological pattern features in the image.

[0046] In this example, the constructed cross-modal joint representation learning model exhibits superior capabilities in the inference application stage. It can parse fine-grained cellular spatial distribution, reveal co-localization interaction patterns between cell types, and provide valuable insights into the intercellular spatial representation within the tumor ecosystem.

[0047] Advantages of additional aspects of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0049] Figure 1 This is a flow chart of a method for identifying cell types and cell abundance based on cross-modal training in the first embodiment of the present invention;

[0050] Figure 2 This is a flowchart of spatial transcriptome data preprocessing in Example 1. DETAILED DESCRIPTION

[0051] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0052] It should be noted that the terms used herein are for describing particular embodiments only and are not intended to limit the exemplary embodiments according to the present invention.

[0053] In the absence of conflict, the embodiments of the present invention and the features thereof may be combined with each other.

[0054] Example 1

[0055] This embodiment discloses a method for identifying cell types and cell abundance based on cross-modal training.

[0056] To more clearly illustrate this embodiment, a process for implementing cell type and cell abundance recognition based on cross-modal training can be specifically described as follows:

[0057] like Figure 1As shown, a cell type and cell abundance identification method based on cross-modal training includes:

[0058] S1. Acquire spatial transcriptomics data for pathological image-gene expression matching;

[0059] S2. Preprocessing the spatial transcriptomics data of the pathological image-gene expression matching to obtain a preprocessed comprehensive data set, which includes highly expressed gene expression data, local image blocks, cell types, and cell abundance labels;

[0060] S3. Construct a cross-modal joint representation learning model, input the highly expressed gene expression data and the local image blocks into the cross-modal joint representation learning model for training, and obtain a trained cross-modal joint representation learning model;

[0061] S4. Based on the trained cross-modal joint representation learning model, cell type and cell abundance prediction is performed on the histological images to be identified.

[0062] In step S1, spatial transcriptomics data of pathological image-gene expression matching is acquired.

[0063] In this embodiment, the data used are complete spatial transcriptomics data with matching of morphological and pathological images and gene expression.

[0064] This data is spatial transcriptomics data, which includes basic information, gene expression information at different sequencing locations, and pathological slide image regions at corresponding locations. Spatial transcriptomics data is directly downloaded from sequencing data from existing studies.

[0065] In this example, single-cell transcriptomic maps were simultaneously downloaded and obtained from sequencing data from existing studies.

[0066] After the above steps, the original spatial transcriptomics data is downloaded, providing a comprehensive data foundation for subsequent operations and model training, obtaining the required data set, ensuring the integrity and accuracy of the data, and providing support for subsequent analysis.

[0067] like Figure 2 As shown, in step S2, the spatial transcriptomics data of the pathological image-gene expression matching is preprocessed to obtain a preprocessed comprehensive data set, which includes highly expressed gene expression data, local image blocks, cell types and cell abundance labels.

[0068] In this example, the raw spatial transcriptomics data is preprocessed to generate fine-grained cell types and corresponding cell abundances using a cell type deconvolution method.

[0069] The spatial transcriptomics data of pathological image-gene expression matching were preprocessed. The specific process was as follows:

[0070] (1) For the spatial transcriptomics data of pathological image-gene expression matching, the cell type deconvolution method is used to generate fine-grained cell types and corresponding cell abundance labels.

[0071] Based on this single-cell transcriptomic profile as a reference, the cellular deconvolution method D(·) was used to estimate cell abundance from spatial transcriptomic data.

[0072] Generate cell abundance estimates, i.e., cell abundance signatures, based on gene expression data from single-cell transcriptomic profiles and spatial transcriptomic data grids.

[0073] In this example, we use the gene expression data of the reference single-cell transcriptomic map and the spatial transcriptomic data grid to generate cell abundance estimates as a basis for the subsequent model. The true labels for training.

[0074] (2) Extract local image blocks and highly expressed gene expression data from the spatial transcriptomics data of pathological image-gene expression matching.

[0075] In this embodiment, the specific process is: 1) extracting the highly expressed genes in the preceding sequence.

[0076] Specifically, first, the raw data is standardized using TPM (Transcripts Per Million) or FPKM (Fragments Per Kilobase Million) normalization method to eliminate the effects of sequencing depth and gene length.

[0077] Then, low-expression genes were filtered, aiming to remove genes whose expression levels in the cohort samples were below a certain threshold.

[0078] Furthermore, the gene matrix was log2 transformed to reduce the impact of skewed data distribution.

[0079] Finally, all genes were sorted in descending order according to the selected expression level index (average expression), and the gene expression results corresponding to the top 250 genes were extracted.

[0080] 2) Extract local image blocks.

[0081] A local image block of 224×224 pixels was extracted from the histological pathology image at a magnification of 20×.

[0082] Specifically, first, the Otsu threshold method is used to distinguish the tissue area from the background and extract the effective tissue area; then, for the extracted tissue area, a fixed step size (such as 224 pixels here) non-overlapping sliding window image sampling method is used to extract a 224×224 pixel image block.

[0083] (3) A comprehensive dataset is constructed by combining fine-grained cell types, corresponding cell abundance labels, local image patches, and highly expressed gene expression data.

[0084] In this embodiment, a comprehensive data set X is finally generated. i , including expression data of highly expressed genes Local image patch and labeling of cell types and abundance

[0085] The synthetic dataset is represented as:

[0086]

[0087] After the above steps, the original spatial transcriptomics data is preprocessed to obtain label information of cell types and cell abundance for subsequent model training.

[0088] like Figure 1 As shown, in step S3, a cross-modal joint representation learning model is constructed, and the highly expressed gene expression data and the local image blocks are input into the cross-modal joint representation learning model for training to obtain a trained cross-modal joint representation learning model.

[0089] S3-1. Build a cross-modal joint representation learning model.

[0090] In this embodiment, the cross-modal joint representation learning model includes three key components: a morphological modality representation module for local image patches in histological pathology images for spatial transcriptomics, a molecular modality representation module based on highly expressed gene expression data, and a cross-modal embedding alignment module for integrating tissue morphology and molecular gene patterns during the training phase.

[0091] (1) Morphological modality representation module.

[0092] In this embodiment, the morphological modality representation module includes a feature backbone module and a transformation layer, and the morphological modality representation module learns the morphological patterns presented in the local image blocks.

[0093] (2) Molecular modal representation module.

[0094] In this embodiment, the molecular modality representation module includes a self-normalization network module and a transformation layer. The molecular modality representation module is introduced to extract key molecular features inherent in gene expression data through the gene expression reconstruction task.

[0095] (3) Cross-modal embedding alignment module.

[0096] In this embodiment, the cross-modal embedding alignment module includes a multi-layer perceptron and a decoder.

[0097] A cross-modal embedding alignment module is used to align the embedding spaces of morphological and molecular modalities during training, integrate image morphology and molecular gene expression patterns, and enhance the interaction between different modalities.

[0098] S3-2. Input the highly expressed gene expression data and the local image blocks into the cross-modal joint representation learning model for training to obtain a trained cross-modal joint representation learning model.

[0099] The highly expressed gene expression data and local image patches are input into the cross-modal joint representation learning model for training. The specific process is as follows:

[0100] S3-2-1. Input the local image block into the morphological modality representation module to obtain image feature embedding.

[0101] In this embodiment, the morphological modality representation module is represented as It aims to extract morphological features directly from local image patches of spatial transcriptomics data.

[0102] The local image block is input into the morphological modality representation module. The specific process is as follows:

[0103] (1) The morphological features of local image blocks are extracted through the feature backbone module.

[0104] In this embodiment, the existing pathology foundation model (PFM) is used as the feature backbone structure It is fine-tuned using spatial transcriptomics data and local image patch data to capture cellular and histological morphological patterns.

[0105] The PFM pathology model was chosen due to its powerful image representation capabilities. Considering the large number of parameters in the PFM model, the LoRA fine-tuning method was used. LoRA is an efficient fine-tuning method that significantly reduces the number of parameters required for training while maintaining model performance by introducing low-rank decomposition into the weight matrix of the pre-trained model. Its core concept is to approximate weight updates using the product of two low-rank matrices.

[0106] Specifically, the pre-trained pathology foundation model (PFM) is first loaded, most of the parameters of the pre-trained model are frozen, and only the low-rank matrices A and B introduced by LoRA are trained, while the original weight W remains unchanged.

[0107] During inference, the updates to the LoRA module are merged into the original weights:

[0108] W′=W+A·B(2)

[0109] Therefore, the core formula of LoRA is the weight update of low-rank decomposition: adopting this fine-tuning strategy can significantly reduce the number of parameters that need to be trained while maintaining model performance.

[0110] Specifically, the local image block data with a size of 224×224 pixels As input to this branch, the fine-tuned PFM model is then used Extract relevant morphological features.

[0111] (2) The morphological features of the local image block are input into the transformation layer for nonlinear transformation to obtain the image block feature embedding.

[0112] In this embodiment, the transformation layer performs nonlinear transformation on the features, which can ensure that the morphological feature embedding suitable for this task is learned.

[0113] The morphological features obtained are in the transformation layer and projected through the head The nonlinear transformation is performed to produce a a Dimensionality-modality embedding That is, image block feature embedding, the formula is:

[0114]

[0115] After the above steps, the morphological modality representation module can learn to represent the morphological patterns presented in local image blocks of H&E-stained pathological images in spatial transcriptomics data. At the same time, it provides one of the key data for the subsequent integration of image morphology and molecular gene expression patterns.

[0116] S3-2-2. Input the expression data of highly expressed genes into the molecular modality representation module to obtain molecular expression embedding.

[0117] In this embodiment, the molecular mode representation module is represented as For learning gene-level molecular patterns from gene expression data corresponding to local image patches in spatial transcriptomics data.

[0118] The expression data of highly expressed genes are input into the molecular modality representation module. The specific process is as follows:

[0119] (1) The self-normalization network module is used to enhance the features of the highly expressed gene expression data to obtain the enhanced highly expressed gene expression data.

[0120] In this example, the highly expressed gene data are processed by the optimized self-normalization network module SNN to obtain enhanced highly expressed gene expression data, which is expressed as

[0121] The expressive power of molecular feature representation is enhanced by adopting self-normalizing network for molecular representation learning.

[0122] Among them, SNN is an existing technology, and it can also be improved and utilized by single-cell basic models to further enhance the feature representation ability of gene expression data.

[0123] (2) Through the transformation layer, the enhanced highly expressed gene expression data is mapped into molecular expression embeddings.

[0124] In this embodiment, in the transformation layer, the projection head Mapping features into embeddings of molecular modalities. These embeddings are called gene expression embeddings That is, the molecular expression embedding, whose dimension is also d a

[0125]

[0126] After the above steps, the gene expression reconstruction task is used to learn and extract the key molecular features inherent in the gene expression data, which are used to align features with morphological pattern features.

[0127] S3-2-3. Input the image feature embedding and molecular expression embedding into the cross-modal representation learning module to obtain the predicted abundance value and original gene expression pattern of the cell type, respectively.

[0128] In this embodiment, based on morphological modality representation and molecular modal feature representation Proposed a cross-modal embedding alignment design To combine gene expression patterns with histological features.

[0129] The image feature embedding and molecular expression embedding are input into the cross-modal representation learning module. The specific process is as follows:

[0130] (1) Align image patch feature embeddings and molecular expression embeddings.

[0131] In this example, in order to unify the features of morphological histology and molecular expression, a cross-modal consistency comparison loss is constructed. to align the embedding spaces of the two modalities.

[0132] Specifically, for a sample set with a batch size of B Paired morphological modality image embedding (image patch feature embedding) and molecular modality gene expression embedding (molecular expression embedding) Learning objectives are defined as:

[0133]

[0134] where ||·||2 represents the L2 norm, which aims to minimize the distance between paired morphological and molecular modality embeddings from the same sample.

[0135] (2) The aligned image patch features are embedded and input into a multi-layer perceptron to obtain the predicted abundance value of the cell type.

[0136] In this example, in order to establish an end-to-end training framework for cross-modal embedding alignment and cell type prediction, the cell abundance estimation marker y m As a supervisory signal, the histological image p m Learning cell type patterns.

[0137] Specifically, aligned embeddings of morphological modalities is fed into a multilayer perceptron This process embeds the aligned morphological modality images (image patch feature embedding) Linear transformation to predicted abundance values ​​of cell types Loss Function Defined as:

[0138]

[0139] Given the cell abundance label y m It is continuous and adopts a regression strategy and uses the root mean square error loss as the objective function to optimize the model for predicting cell abundance profiles.

[0140] (3) The aligned molecular expression embeddings are input into the decoder to reconstruct the original gene expression pattern.

[0141] In this embodiment, in order to enhance the model to extract the inherent molecular information s in gene expression data m The ability to propose gene expression reconstruction tasks. Aligned molecular modality gene expression embedding (Molecular Expression Embedding) is input to the decoder block In this paper, we use the learned cross-modal molecular representation The original gene expression pattern was reconstructed using the formula:

[0142]

[0143] in, represents the reconstructed gene expression value, represents the gene expression reconstruction loss function.

[0144] This approach effectively captures molecular information to enrich histological representations, thereby facilitating fine-grained cell type identification from histological images.

[0145] After the above steps, using cross-modal embedding to align the embedding spaces of morphological and molecular modalities during training can integrate image morphology and molecular gene expression patterns, presenting better fine-grained cell type recognition effects, which reveals that molecular gene expression information enhances the morphological pattern characteristics in the image.

[0146] S3-3. Construct overall loss optimization.

[0147] In this embodiment, the model is optimized by the overall loss, and the overall loss target is It is expressed as the weighted sum of three loss objective terms, and the formula is:

[0148]

[0149] in, and Represent the cross-modal consistency comparison loss Cell abundance predicts loss and gene expression reconstruction loss The balance weight of θ morph ,θ molec 、 and yes and The parameter in argmin(·) aims to find the function that minimizes the overall loss Optimization parameters

[0150] like Figure 1 As shown, in step S4, based on the trained cross-modal joint representation learning model, the cell type and cell abundance of the histological image to be identified are predicted.

[0151] In this example, histological images are directly fed into a fine-tuned PFMs encoder and projection head for processing to extract cross-modal embedding features that capture morphological patterns enhanced with molecular gene expression information. The MLP regression head then converts the resulting embeddings into predicted fine-grained cell types and corresponding cell abundance values.

[0152] In this embodiment, the rich morphological patterns present in tissue pathology images are integrated and utilized to propose a pathological morphology-molecular gene cross-modal feature alignment training method, which ultimately achieves the automatic prediction of fine-grained cell types and corresponding cell type abundance information in the inference stage. Compared with methods based on computational pathology, the method of this embodiment can identify more and finer-grained cell types and corresponding cell richness; compared with methods based on spatial transcriptomics (ST), the method of this embodiment combines the rich morphological patterns present in tissue pathology images to predict fine-grained cell types and corresponding cell type abundance information from tissue pathology images alone, further reflecting the spatial distribution of each cell type and providing interaction patterns between different cell types, with the potential to reveal key insights into how cell communication promotes biological behavior.

[0153] Example 2

[0154] The purpose of this embodiment is to provide a cell type and cell abundance identification system based on cross-modal training, including:

[0155] Data acquisition module, used to obtain spatial transcriptomics data of pathological image-gene expression matching;

[0156] a data processing module for preprocessing the spatial transcriptomics data of the pathological image-gene expression matching to obtain a preprocessed comprehensive data set, which includes highly expressed gene expression data, local image blocks, cell type and cell abundance labels;

[0157] The model construction and training module is used to build a cross-modal joint representation learning model. The highly expressed gene expression data and local image blocks are input into the cross-modal joint representation learning model for training to obtain a trained cross-modal joint representation learning model.

[0158] The model recognition module is used to predict the cell type and cell abundance of the histological images to be identified based on the trained cross-modal joint representation learning model.

[0159] Based on providing a cell type and cell abundance identification system based on cross-modal training, the method steps in Example 1 are implemented.

[0160] Example 3

[0161] The purpose of this embodiment is to provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the program.

[0162] Example 4

[0163] The purpose of this embodiment is to provide a computer-readable storage medium.

[0164] A computer-readable storage medium stores a computer program, which, when executed by a processor, performs the steps of the above method.

[0165] Example 5

[0166] The purpose of this embodiment is to provide a computer program product containing instructions, which, when running on a computer, enables the computer to execute the methods and functions involved in any of the above embodiments.

[0167] The steps involved in the apparatus of the above embodiment correspond to those of the method embodiment 1. For detailed implementation, please refer to the relevant description of embodiment 1. The term "computer-readable storage medium" should be understood to mean a single medium or multiple media containing one or more instruction sets; it should also be understood to include any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and causing the processor to perform any method of the present invention.

[0168] Those skilled in the art will appreciate that the modules or steps of the present invention described above can be implemented using a general-purpose computer device. Alternatively, they can be implemented using program code executable by a computing device, which can then be stored in a storage device and executed by the computing device. Alternatively, they can be fabricated into separate integrated circuit modules, or multiple modules or steps can be fabricated into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0169] Although the above describes the specific embodiments of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without any creative work are still within the scope of protection of the present invention.

Claims

1. A method for identifying cell types and cell abundance based on cross-modal training, characterized in that: include: Acquire spatial transcriptomics data matching pathological images and gene expression; Preprocessing the spatial transcriptomics data of the pathological image-gene expression matching to obtain a preprocessed comprehensive data set, which includes highly expressed gene expression data, local image blocks, cell types, and cell abundance labels; Construct a cross-modal joint representation learning model, input the highly expressed gene expression data and local image blocks into the cross-modal joint representation learning model for training, and obtain a trained cross-modal joint representation learning model; Based on the trained cross-modal joint representation learning model, the cell type and cell abundance of the predicted histological image are predicted; The highly expressed gene expression data and local image patches are input into the cross-modal joint representation learning model for training. The specific process is as follows: The local image block is input into the morphological modality representation module to obtain image feature embedding. The morphological modality representation module includes a feature backbone module and a transformation layer. The specific process is as follows: The morphological features of local image blocks are extracted through the feature backbone module; The morphological features of the local image block are input into the transformation layer for nonlinear transformation to obtain the image block feature embedding; The expression data of highly expressed genes are input into the molecular modality representation module to obtain molecular expression embedding. The molecular modality representation module includes a self-normalization network module and a transformation layer. The specific process is as follows: Through the self-normalization network module, the features of the highly expressed gene expression data are enhanced to obtain the enhanced highly expressed gene expression data; Through the transformation layer, the enhanced highly expressed gene expression data is mapped into molecular expression embeddings; The image feature embedding and the molecular expression embedding are input into the cross-modal embedding alignment module to obtain the predicted abundance value of the cell type and the original gene expression pattern, respectively. This includes a multi-layer perceptron and a decoder. The specific process is as follows: Align image patch feature embeddings with molecular expression embeddings; The aligned image patch features are embedded and input into a multilayer perceptron to obtain the predicted abundance value of the cell type; The aligned molecular expression embeddings are input to the decoder to reconstruct the original gene expression patterns.

2. A method for identifying cell types and cell abundance based on cross-modal training according to claim 1, characterized in that: The spatial transcriptomics data of the pathological image-gene expression matching is preprocessed, and the specific process is as follows: For the spatial transcriptomics data of the pathological image-gene expression matching, a cell type deconvolution method is used to generate fine-grained cell types and corresponding cell abundance labels; Extracting local image blocks and highly expressed gene expression data from the spatial transcriptomics data of the pathological image-gene expression matching; A comprehensive dataset is constructed by combining fine-grained cell types, corresponding cell abundance labels, local image patches, and highly expressed gene expression data.

3. The method for identifying cell types and cell abundance based on cross-modal training according to claim 1, wherein: A cross-modal joint representation learning model is constructed, wherein the cross-modal joint representation learning model includes a morphological modality representation module, a molecular modality representation module and a cross-modal embedding alignment module.

4. The method for identifying cell types and cell abundance based on cross-modal training according to claim 1, wherein: The model is optimized by the overall loss, and the overall loss formula is: ； in , and Represent the cross-modal consistency comparison loss , cell abundance prediction loss and gene expression reconstruction loss The balance weight of 、 、 and yes 、 、 and Parameters in The goal of this term is to find the function that minimizes the overall loss Optimization parameters , 、 、 and They represent the morphological modality representation module, the molecular modality representation module, the multi-layer perceptron part for predicting cell type abundance, and the decoder part for reconstructing the original gene expression pattern.

5. A cell type and cell abundance recognition system based on cross-modal training, characterized in that: Implementing a method for identifying cell types and cell abundances based on cross-modal training according to any one of claims 1 to 4, comprising: Data acquisition module, used to obtain spatial transcriptomics data of pathological image-gene expression matching; a data processing module for preprocessing the spatial transcriptomics data of the pathological image-gene expression matching to obtain a preprocessed comprehensive data set, which includes highly expressed gene expression data, local image blocks, cell type and cell abundance labels; The model construction and training module is used to build a cross-modal joint representation learning model. The highly expressed gene expression data and local image blocks are input into the cross-modal joint representation learning model for training to obtain a trained cross-modal joint representation learning model. The model recognition module is used to predict the cell type and cell abundance of the histological images to be identified based on the trained cross-modal joint representation learning model.

6. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the method according to any one of claims 1 to 4 are implemented.

Images