Method, device and equipment for predicting spatial gene expression level based on histological images

Abstract

The present invention relates to the field of neural network prediction technology, and discloses a method, device and equipment for predicting spatial gene expression levels based on histological images. The method comprises: obtaining visual features of image blocks through a DenseNet‑121 model, extracting fusion features of sequencing points through a sequencing point feature extraction model based on a multi-head attention mechanism; using a contrastive learning method to optimize and enhance the prediction ability of the model, thereby accurately predicting gene expression levels in histological images at different spatial resolutions. The present invention uses a multimodal fusion method, comprehensively considering histological image information, gene expression information of spatial transcriptomics sequencing points, and location information of sequencing points, and can obtain more accurate prediction results; the present invention improves the effect of predicting gene expression levels, reduces the misjudgment rate, and can accurately identify cancer-specific genes.

Description

Technical Field

[0001] The present invention relates to the field of neural network prediction technology, and in particular to a method, device and equipment for predicting spatial gene expression levels based on histological images. Background Art

[0002] Histological pathological images and spatial transcriptomics data are two key types of biomedical data. Histological pathological images are biomedical images obtained through tissue sections and microscope imaging, which can provide detailed information on cell and tissue structures. Traditional pathological image analysis mainly relies on visual observation and interpretation by professionals, which has the problems of subjectivity and high time cost. On the other hand, spatial transcriptomics data provides location information of gene expression in cells. Through spatial transcriptomics methods, the expression levels of a large number of genes in each cell in tissue sections can be determined, providing a detailed understanding of cell function and state. These pathological images and spatial transcriptomics data are crucial for disease diagnosis, formulation of treatment strategies, and understanding of the mechanism of disease development. However, spatial transcriptomics data usually have high dimensions and complex structures, and the cost of generating spatial transcriptomics data is very expensive, so it cannot be applied on a large scale at present.

[0003] Therefore, the prior art still needs to be improved and developed. Summary of the invention

[0004] In order to overcome the deficiencies of the prior art, the purpose of the present invention is to provide a spatial gene expression level prediction, device and equipment based on histological images, aiming to solve the problem that the prior art cannot obtain accurate spatial transcriptome data based on histological images.

[0005] The first aspect of the present invention provides a method for predicting spatial gene expression levels based on histological images, comprising the steps of: obtaining a pathological data set of a cancer patient, the pathological data set comprising a histological image, spatial transcriptome gene expression data corresponding to the histological image, and position coordinates of sequencing points; segmenting the histological image into image blocks of a predetermined pixel size centered on each sequencing point according to the position coordinates of each sequencing point, and pairing the image blocks with the spatial transcriptome gene expression data of the corresponding sequencing point; selecting common genes in all tissue sections and the top 1000 highly variable genes in each tissue section from the spatial transcriptome gene expression data, normalizing the gene expression values ​​in each sequencing point, and obtaining normalized gene expression data of the sequencing point; segmenting the histological image into image blocks and inputting them into a pre-trained DenseNet-121 model to output visual features; inputting the normalized gene expression data of the sequencing point and the position coordinates of the sequencing point into a sequencing point feature extraction model based on a multi-head attention mechanism; In the model, fusion features are output; the visual features and fusion features are projected into the feature space through the projection layer respectively to construct a multimodal embedding space; in the multimodal embedding space, the sequencing points and image blocks located at the same position are defined to form positive sample pairs, and the sequencing points and image blocks located at different positions are defined to form negative sample pairs, and the distance between the positive sample pairs is reduced by the contrast learning method, while the distance between the negative sample pairs is increased, so as to obtain an optimized multimodal embedding space; in the optimized multimodal embedding space, the cosine similarity between the visual features of the histological image to be tested and the fusion features of all sequencing points is calculated, and the fusion features of the first K sequencing points with the highest similarity score are found; in the optimized multimodal embedding space, the Euclidean distance between the visual features of the histological image to be tested and the fusion features of the first K sequencing points is calculated, and the corresponding weight is designed according to the size of the Euclidean distance, and the weight is correspondingly multiplied by the real gene expression value of the first K sequencing points and the sum is taken as the spatial gene expression level of the histological image to be tested.

[0006] Optionally, in a first implementation of the first aspect of the present invention, the step of normalizing the gene expression value in each sequencing point to obtain normalized gene expression data of the sequencing point includes:

[0007] Normalization is performed by dividing the UMI count of each gene by the total UMI count of all genes in that sequencing point, and then multiplying by a scaling factor of 1,000,000. The normalization formula is: Among them, N ij represents the UMI count of the i-th gene at the j-th sequencing point, T j Represents the total UMI count of all genes at the jth sequencing point; the normalized value is converted to a natural logarithmic scale, and the expression formula is:ij =log 10 (1+Normalized ij ).

[0008] Optionally, in a second implementation of the first aspect of the present invention, the normalized gene expression data of the sequencing points and the position coordinates of the sequencing points are input into a sequencing point feature extraction model based on a multi-head attention mechanism, and the step of outputting fusion features includes: encoding the x-axis coordinates and the y-axis coordinates of the sequencing points, and converting them into a one-hot encoding matrix P of size N×1, respectively. x and P y ; Use a learnable linear layer to encode the one-hot encoding matrix P x and P y Perform a linear transformation to obtain the corresponding x-coordinate encoding matrix S x and the y-coordinate encoding matrix S y ; Normalize the gene expression data of the sequencing points and the x-coordinate encoding matrix S x and the y-coordinate encoding matrix S y After merging, they are input into the sequencing point feature extraction model based on the multi-head attention mechanism to output the fused features.

[0009] Optionally, in a third implementation of the first aspect of the present invention, in a multimodal embedding space, sequencing points and image blocks located at the same position are defined to form positive sample pairs, and sequencing points and image blocks located at different positions are defined to form negative sample pairs, and a contrastive learning method is used to reduce the distance between the positive sample pairs and increase the distance between the negative sample pairs, thereby obtaining an optimized multimodal embedding space, comprising the steps of: defining sequencing points and image blocks located at the same position to form positive sample pairs, and sequencing points and image blocks located at different positions to form negative sample pairs, and designing a label matrix in which diagonal elements represent positive sample pairs and non-diagonal elements represent negative sample pairs; and defining a cosine similarity function cos_sim between image blocks and sequencing point embeddings as follows: The cosine similarity function between the sequencing points and the image patch embedding is defined as follows: Among them, h spot is the fusion feature of the extracted sequencing points, is the fusion feature h spot The transpose of h patch is the visual feature of the extracted image patch, is the visual feature h patch The transpose of ; using the cross entropy loss function to calculate two independent loss components loss image and loss spot , where Loss image =cross_entropy_loss(cos_sim(hpatch ,h spot ),label),Loss spot =cross_entropy_loss(cos_sim(h spot ,h patch ), label); the overall loss of the DenseNet-121 model and the sequencing point feature extraction model is calculated as the average of these two losses: Loss = (Loss image +Loss spot ) / 2.0.

[0010] Optionally, in a fourth implementation of the first aspect of the present invention, in the optimized multimodal embedding space, by formula The Euclidean distance between the visual feature of the histological image to be tested and the fusion feature of the first K sequencing points is calculated, where u is the visual feature, v is the fusion feature of the first K sequencing points, and n is the dimension of the feature space.

[0011] Optionally, in a fifth implementation of the first aspect of the present invention, in the step of designing corresponding weights according to the size of the Euclidean distance, the weights are defined as: The formula for the spatial gene expression level of the histological image to be tested is: in, Indicates sequencing point u j The actual gene expression level.

[0012] The second aspect of the present invention provides a spatial gene expression level prediction device based on histological images, comprising: an acquisition module, used to acquire a pathological data set of a cancer patient, the pathological data set comprising a histological image, spatial transcriptome gene expression data corresponding to the histological image, and the position coordinates of a sequencing point; a segmentation module, used to segment the histological image into image blocks of a predetermined pixel size centered on each sequencing point according to the position coordinates of each sequencing point, and pair the image blocks with the spatial transcriptome gene expression data of the corresponding sequencing point; a data processing module, used to select common genes in all tissue sections and the top 1000 highly variable genes in each tissue section from the spatial transcriptome gene expression data, normalize the gene expression values ​​in each sequencing point, and obtain normalized gene expression data of the sequencing point; a visual feature extraction module, used to segment the histological image into image blocks and input them into a pre-trained DenseNet-121 model to output visual features; a fusion feature extraction module, used to combine the normalized gene expression data of the sequencing point with the image blocks; The position coordinates of the sequencing points are input into the sequencing point feature extraction model based on the multi-head attention mechanism, and the fusion features are output; a space construction module is used to project the visual features and the fusion features into the feature space through the projection layer respectively, and construct a multimodal embedding space; an optimization module is used to define the sequencing points and image blocks located at the same position to form positive sample pairs, and the sequencing points and image blocks located at different positions to form negative sample pairs in the multimodal embedding space, and use the contrast learning method to reduce the distance between the positive sample pairs and increase the distance between the negative sample pairs to obtain an optimized multimodal embedding space; a calculation module is used to calculate the cosine similarity between the visual features of the histological image to be tested and the features of all sequencing points in the optimized multimodal embedding space, and find the first K sequencing point features with the highest similarity score; a prediction module is used to calculate the Euclidean distance between the visual features of the histological image to be tested and the features of the first K sequencing points in the optimized multimodal embedding space, and use the weighted distance of the gene expression values ​​of the first K sequencing points to predict the spatial gene expression level of the sequencing point.

[0013] The third aspect of the present invention provides a spatial gene expression level prediction device based on histological images, comprising: a memory and at least one processor, wherein the memory stores computer-readable instructions, and the memory and the at least one processor are interconnected through lines; the at least one processor calls the computer-readable instructions in the memory so that the spatial gene expression level prediction device based on histological images performs the various steps of the spatial gene expression level prediction method based on histological images as described above.

[0014] A fourth aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, which, when executed on a computer, enable the computer to execute the various steps of the spatial gene expression level prediction method based on histological images as described above.

[0015] Beneficial effects: The present invention provides a method for predicting spatial gene expression levels based on histological images. The present invention uses a multimodal fusion method, comprehensively considering the histological image information, the gene expression information of spatial transcriptomics sequencing points, and the location information of the sequencing points, thereby obtaining a more accurate prediction result. The present invention improves the effect of predicting gene expression levels, reduces the misjudgment rate, and can accurately identify cancer-specific genes. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A flowchart of a method for predicting spatial gene expression levels based on histological images provided in an embodiment of the present invention.

[0017] Figure 2 This is a diagram showing the architecture composition of the mclSTExp model of the present invention.

[0018] Figure 3 Figure 3. Gene expression prediction data of different methods evaluated on HER2+ dataset by PCCs.

[0019] Figure 4 Figure 2. Gene expression prediction data of different methods evaluated on cSCC dataset by PCCs.

[0020] Figure 5 Figure 3. Gene expression prediction data on the Alex+10x dataset evaluated by PCCs for different methods.

[0021] Figure 6 Visualization of the top 7 predictive gene tissue fractions in the HER2+ dataset based on all calculated highest mean -log10(p-values).

[0022] Figure 7 A schematic diagram of the structure of a device for predicting spatial gene expression levels based on histological images provided by an embodiment of the present invention;

[0023] Figure 8 A schematic diagram of the structure of a device for predicting spatial gene expression levels based on histological images provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0024] Embodiments of the present invention provide a method, apparatus, device and storage medium for predicting spatial gene expression levels based on histological images. The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0025] See also Figure 1 , Figure 1 A flowchart of a method for predicting spatial gene expression levels based on histological images provided by the present invention is shown in the figure, which includes the following steps:

[0026] S10, obtaining a pathological data set of a cancer patient, wherein the pathological data set includes a histological image, spatial transcriptome gene expression data corresponding to the histological image, and position coordinates of sequencing points;

[0027] Specifically, spatial transcriptomics is a discipline that studies the spatial distribution and regulatory mechanism of gene transcription in organisms. It reveals the spatial expression pattern of genes and their association with cell function, tissue structure and disease development by analyzing the gene expression in biological tissue sections. In this process, histological images play an important role. The histological images are pictures of biological tissue sections or cell structures observed and photographed under a microscope. Histological images are an important data source for spatial transcriptomics research. Histological images observed and photographed under a microscope can show the microstructure and cell type of biological tissues, providing rich information for spatial transcriptomics. Based on these histological images, researchers can combine high-throughput sequencing technology and bioinformatics analysis methods to quantitatively analyze the gene expression in tissue sections, and then study the spatial expression pattern of genes. Secondly, by comparing the histological images and gene expression data of different tissue sections or different cell types, researchers can verify the accuracy and reliability of the results of spatial transcriptomics analysis. In addition, histological images can also help researchers explain the association between gene expression patterns and tissue structure and cell function, and further reveal the biological processes in organisms. There is a close correlation between histological images and spatial transcriptomics. Spatial transcriptomic data usually have high dimensions and complex structures, and the cost of generating spatial transcriptomic data is very expensive, and it is currently not possible to apply it on a large scale. The present invention is to use histological images to predict the cellular gene expression levels in spatial transcriptomic data from histological images by artificial intelligence. First, it is necessary to obtain a pathological data set of a cancer patient as training data, and the pathological data set includes a histological image, spatial transcriptomic gene expression data corresponding to the histological image, and the location coordinates of the sequencing points. The pathological data set can be downloaded from a public database, for example, two human breast cancer data sets and one human skin squamous cell carcinoma data set.

[0028] In the present embodiment, the spatial transcriptome gene expression data describes the expression level of each gene on each sequencing point (spot). These data are obtained by sequencing technology, and the expression pattern and distribution law of genes in tissue sections can be revealed. The accuracy of gene expression data is crucial for understanding tissue function and cell-cell interaction. Secondly, sequencing point (spot) is the region sequenced on tissue sections, and its position coordinates are used to determine the spatial distribution of gene expression, and these coordinates are usually in pixels, obtained by the digital image of tissue sections, and the sequencing point position coordinates are key information in spatial transcriptome technology, and these coordinates indicate the specific position of each sequencing point on tissue sections, so that researchers can correspond gene expression data with actual tissue structure. Finally, organized images are another important component of spatial transcriptome technology, and these images are obtained by microscope or other imaging techniques, and the morphological structure and cell distribution of tissue sections can be displayed, and organized images provide intuitive visual information for researchers, helping to understand the practical significance of sequencing point position coordinates and gene expression data in tissues.

[0029] S20, segmenting the histological image into image blocks of a predetermined pixel size centered on each sequencing point according to the position coordinates of each sequencing point, and pairing the image blocks with the spatial transcriptome gene expression data of the corresponding sequencing point;

[0030] Specifically, determine the exact position of each sequencing point on the tissue section, which is usually given in the form of pixel coordinates (x, y) or (x, y, z), depending on the dimension of the histological image (two-dimensional or three-dimensional); then, set a predetermined pixel size, which will determine the range of histological information contained in each image block. This size is usually determined according to the purpose of the study, the resolution of the tissue, and the density of the sequencing points, such as 224*224; then, use image processing software or programming tools to segment image blocks of a predetermined pixel size centered on each sequencing point from the histological image based on the position coordinates of each sequencing point. This process is similar to drawing small boxes centered on the sequencing point on the image and extracting the image content in these small boxes. Finally, match each image block with the gene expression data of the corresponding sequencing point. In this way, each image block corresponds to the gene expression pattern at a sequencing point, thereby realizing the spatial correspondence between the histological image and the gene expression data.

[0031] S30, selecting common genes in all tissue sections and the top 1000 highly variable genes in each tissue section from the spatial transcriptome gene expression data, normalizing the gene expression values ​​in each sequencing point, and obtaining normalized gene expression data of the sequencing point;

[0032] Specifically, for spot gene expression data, we first determine the genes common to all tissue sections in the training spatial transcriptome (ST) data, and then select the top 1,000 highly variable genes (highly variable genes) in each tissue section; then normalize the gene expression values ​​in each spot by dividing the UMI count of each gene by the total UMI count of all genes in the sequencing point, and then multiplying by the scale factor 1,000,000. The normalization formula is: Among them, N ij represents the UMI count of the i-th gene at the j-th sequencing point, T j Represents the total UMI count of all genes at the jth sequencing point; finally, the normalized value is converted to the natural logarithmic scale, and its expression formula is: ij =log 10 (1+Normalized ij ), recorded as spot i .

[0033] S40, segmenting the histological image into image blocks and inputting them into the pre-trained DenseNet-121 model to output visual features;

[0034] Specifically, firstly, image blocks of 224*224 pixels are segmented from the histological image according to the location of the sequencing points. For each extracted image block, i , using the pre-trained DenseNet-121 to extract its visual features: Different from the skip connections in ResNet, the dense connection mechanism in DenseNet-121 enhances the reusability of features and helps the neural network capture image features more effectively, thereby enhancing the expressiveness of the model.

[0035] S50, inputting the normalized gene expression data of the sequencing points and the position coordinates of the sequencing points into a sequencing point feature extraction model based on a multi-head attention mechanism, and outputting fusion features;

[0036] Specifically, the position coordinates of the test points are first processed, that is, the x-axis coordinates and y-axis coordinates of the sequencing points are encoded and converted into a one-hot encoding matrix P of size N×1. x and P y , where N is the maximum x-coordinate of all tissue slices. For all datasets, N = 65536; a learnable linear layer is used to encode the one-hot encoding matrix P x and P y Perform a linear transformation to obtain the corresponding x-coordinate encoding matrix S x =N×hvg num and the y-coordinate encoding matrix Sy =N×hvg num ;

[0037] This example considers each sequencing point in the spatial transcriptome data as a "word", and considers the sequence of these sequencing points as a "sentence" containing multiple "words". Then, the self-attention mechanism is used to extract features from these "words", and the position information of these "words" is integrated in combination with learnable position encoding. The definition of the attention mechanism is as follows:

[0038]

[0039] in, is the weight matrix, It is called an attention map, and V is the value of the self-attention mechanism, where V=Q=K.

[0040] Multi-head attention is an extension of the attention mechanism. By learning multiple sets of independent attention weights simultaneously, it enhances the model's ability to capture complex patterns and global information in the input sequence:

[0041] MultiHead(Q,K,V)=[head1,...,head n ]×W0, where W0 represents the weight matrix for aggregating attention heads and n represents the number of attention heads. In addition, Q, K, and V correspond to Query, Key, and Value, respectively. Based on this, this embodiment can normalize the gene expression data of the sequencing point spot i , x-coordinate encoding matrix S x and the y-coordinate encoding matrix S y After merging, it is input into the sequencing point feature extraction model based on the multi-head attention mechanism, and the fusion feature is output, which is expressed as

[0042] S60, projecting the visual features and the fusion features into the feature space through the projection layer respectively, to construct a multimodal embedding space;

[0043] Specifically, the visual features of all histological images extracted are projected through the projection layer. Projected into the feature space, it can be expressed as: The fusion features of all the sequencing points extracted are projected through the projection layer Projected into the feature space, it can be expressed as: All visual features projected into the feature space and their corresponding fused features constitute the multimodal embedding space.

[0044] S70, in the multimodal embedding space, defining sequencing points and image blocks located at the same position to form positive sample pairs, and sequencing points and image blocks located at different positions to form negative sample pairs, and using a contrastive learning method to reduce the distance between the positive sample pairs and increase the distance between the negative sample pairs, thereby obtaining an optimized multimodal embedding space;

[0045] The present invention utilizes a self-attention mechanism to fuse the gene expression features and spatial position features of each sequencing point, which not only fuses the key information of gene expression, but also takes into account the specific spatial position of each sequencing point in the tissue image. On the other hand, the segmentation of histological image blocks is based on the position of the sequencing points. Therefore, sequencing points and image blocks located at the same position essentially form positive sample pairs, while those located at different positions constitute negative sample pairs. Therefore, in order to fuse image features, the present invention adopts a contrastive learning method to reduce the distance between positive sample pairs while increasing the distance between negative sample pairs. Specifically, in each batch, N pairs (image blocks, sequencing points) are included, and then the image encoder (DenseNet-121 model) and the point encoder (sequencing point feature extraction model) are trained simultaneously to obtain an optimized multimodal embedding space. The optimization goal of this space is to maximize the cosine similarity of N positive sample pairs while minimizing N 2 - Cosine similarity of N negative sample pairs.

[0046] To achieve this goal, a label matrix (label) is defined, where the diagonal elements represent positive sample pairs (labeled as 1) and the off-diagonal elements represent negative sample pairs (labeled as 0). Subsequently, the cross entropy loss function is used to achieve effective positive sample classification. In order to show the overall loss of the model, the cosine similarity function cos_sim between the image patch and the sequencing point embedding is first defined as follows: The cosine similarity function between the sequencing points and the image patch embedding is defined as follows: Among them, h spot is the fusion feature of the extracted sequencing points, is the fusion feature h spot The transpose of h patch is the visual feature of the extracted image patch, is the visual feature h patch The transpose of .

[0047] Then, the cross entropy loss function is used to calculate two independent loss components: image and loss spot , where Loss image =cross_entropy_loss(cos_sim(h patch ,h spot ),label),Loss spot=cross_entropy_loss(cos_sim(h spot ,h patch ),label), where the label matrix is:

[0048]

[0049] Finally, the overall loss of the DenseNet-121 model and the sequencing point feature extraction model is calculated as the average of these two losses: Loss = (Loss image +Loss spot ) / 2.0.

[0050] S80, calculating the cosine similarity between the visual features of the histological image to be tested and the fusion features of all sequencing points in the optimized multimodal embedding space, and finding the fusion features of the first K sequencing points with the highest similarity scores;

[0051] Specifically, in the prediction stage of the model, the histological image data to be tested is input into the pre-trained DenseNet-121 model to obtain the extracted visual features; then, in the optimized shared embedding space, the cosine similarity between the visual features of the histological image to be tested and the fusion features of all sequencing points is calculated. Subsequently, the fusion features of the first K sequencing points with the highest similarity scores are found.

[0052] S90. In the optimized multimodal embedding space, the Euclidean distance between the visual features of the histological image to be tested and the fusion features of the first K sequencing points is calculated, and corresponding weights are designed according to the size of the Euclidean distance. The weights are multiplied by the real gene expression values ​​of the first K sequencing points and the sum is taken as the spatial gene expression level of the histological image to be tested.

[0053] Specifically, in the optimized multimodal embedding space, the Euclidean distance between the visual features of the histological image to be tested and the fusion features of the first K sequencing points is as follows: Among them, u is the visual feature, v is the fusion feature of the first K sequencing points, and n is the dimension of the feature space. Next, the corresponding weight is designed according to the size of the Euclidean distance. The weight is defined as: The formula for the spatial gene expression level of the histological image to be tested is: in, Indicates sequencing point u j The actual gene expression level.

[0054] Compared with the prior art, the method for predicting cell gene expression levels based on histological images provided by the present invention uses a multimodal fusion method, comprehensively considers histological image information, gene expression information of spatial transcriptomics sequencing points, and location information of sequencing points, thereby obtaining more accurate prediction results. The present invention improves the effect of predicting gene expression levels, reduces the misjudgment rate, and can accurately identify cancer-specific genes, providing reference opinions for personalized medicine.

[0055] The following is a further explanation of a method for predicting spatial gene expression levels based on histological images of the present invention through specific examples:

[0056] Example 1

[0057] This example collects and organizes two human breast cancer datasets and one human skin squamous cell carcinoma dataset from public databases, each of which includes the following contents:

[0058] HER2+ dataset: contains 36 tissue sections from eight patients. We selected 32 sections from seven patients to ensure that each section contains at least 180 spots, and the diameter of each spot is 100 μm.

[0059] cSCC dataset: contains 12 tissue sections from four patients, each patient contributes three sections, and the diameter of each spot is 100μm.

[0060] Alex+10x dataset: Contains 9 breast cancer tissue samples, including 3 samples from 10xGenomics (2 fresh frozen samples and 1 formalin-fixed paraffin-embedded sample, FFPE), and 6 samples from Swarbrick Laboratory. The diameter of each spot is 55μm.

[0061] For histology (H&E) images, we divided an area with a width and height of 224 pixels into an image block according to the position coordinates of each sequencing point.

[0062] For spatial gene expression data, we first identified common genes in all tissue sections of the training ST data. Subsequently, we selected the top 1,000 highly variable genes (HVGs) in each tissue section and excluded genes that were expressed in less than 1,000 spots across all tissue sections. The counts for each spot were then normalized by dividing the UMI count of each gene by the total UMI count of all genes in that sequencing spot and then multiplying by a scaling factor of 1,000,000. Finally, the normalized values ​​were converted to a natural logarithmic scale.

[0063] After preprocessing the ST dataset, HER2+ retained 11,548 spots and 785 genes, cSCC retained 8,671 spots and 171 genes, and Alex+10x retained 25,914 spots and 685 genes. We paired the image patches with the spots, generating N^2 samples of (image patches, spot) pairs. Since the image patches are divided based on the position coordinates of the spots, the spots and image patches with the same position coordinates naturally form positive sample pairs. Among these samples, there are N positive samples, representing correctly matched (image patches, spot) pairs, and the remaining N^2-N samples are negative samples, representing incorrectly matched (image patches, spot) pairs. To evaluate the prediction accuracy of gene expression data, we used a leave-one-out cross-validation training method, in which one tissue slice was retained as a test set and the remaining slices were used as a training set.

[0064] Parameter setting: All learning rates in the present invention are set to 0.0001, weight decay is set to 0.0001, and batch size is set to 128.

[0065] Implementation steps:

[0066] 1. Obtain the histological images provided in the dataset, the cell gene expression matrix of the spatial transcriptomics sequencing points, and the position coordinate matrix of the sequencing points.

[0067] 2. The histological images provided in the dataset were divided into 224*224 image blocks according to the location of the sequencing points for model input. Highly variable genes were screened out in the cell gene expression matrix and normalized.

[0068] 3. Use the image encoder (DenseNet-121 model) to extract histological image features, and use the gene expression encoder (sequencing point feature extraction model) to extract gene expression features. The extracted features are embedded into the feature space, and then the multimodal model is trained using the contrastive learning method. The model parameters are updated using the back-propagation algorithm to minimize the loss function.

[0069] 4. After the model training is completed, the model is used to predict the cellular gene expression levels in the histological images, and the accuracy of the model is calculated and analyzed using an evaluation algorithm. The performance of the model in predicting cellular gene expression levels is evaluated by comparison with the actual gene expression data, including the Pearson correlation coefficient (PCC), mean square error (MSE), and mean absolute error (MAE) indicators. Among them, PCC measures the average correlation of each gene type, taking into account the relationship between the predicted value and the actual value in all histological images. MSE and MAE measure the sample deviation between the predicted value and the actual value in each histological image. For PCC, higher values ​​indicate better performance. Conversely, for MSE and MAE, lower values ​​indicate better performance.

[0070]

[0071] Among them, Cov(·) represents covariance, Var(·) represents variance, X true and X pred represent the true value and the predicted value respectively.

[0072] To evaluate the performance of the method mclSTExp of the present invention, we analyzed a HER2+ breast cancer dataset consisting of 32 tissue sections, a cSCC dataset consisting of 12 tissue sections, and an Alex+10x dataset consisting of 9 sections. To evaluate the accuracy of gene expression prediction, we performed leave-one-out cross-validation. Specifically, for each dataset, we used one slice as a test set and the remaining slices as a training set. For each tissue section, we calculated the PCC of all highly variable genes (ACG) and the top 50 highly expressed genes (HEG), as well as the MSE and MAE of all highly variable genes. Subsequently, the average values ​​of PCC(ACG), PCC(HEG), MSE, and MAE of all tissue sections were calculated to evaluate the overall model performance.

[0073] We compared the mclSTExp method with five other recently developed advanced methods for predicting spatial gene expression (STnet, HisToGene, His2ST, THItoGene, BLEEP). Considering that the gene expression prediction task emphasizes capturing relative changes, we prefer evaluation metrics related to PCC. As shown in Table 1, mclSTExp achieves the highest average PCC (ACG) and PCC (HEG) on these three datasets.

[0074] Table 1 Comparison data of prediction levels of the present invention and other methods

[0075]

[0076]

[0077]

[0078] Specifically, mclSTExp’s PCC(ACG) is 23.01%, 32.09%, and 25.57% higher than the second-place method BLEEP, while PCC(HEG) is 32.89%, 36.48%, and 27.82% higher, respectively. To examine the results for each slice, we visualized the PCC between the gene expression predicted by mclSTExp and the gene expression observed on each slice. Figure 3 As shown in Figure 2, mclSTExp achieved the highest PCC in 32 slices of the HER2+ dataset and performed best on 26 of them. However, for slices E1-F3, the PCC values ​​of all methods were relatively low, which suggests that ST technology may have problems in terms of gene detection sensitivity or specificity. It is worth noting that mclSTExp outperformed the second-place method Bleep on all slices. In addition, as Figure 4 As shown in Figure 3, mclSTExp showed the highest PCC among all tissue sections in the cSCC dataset. It is noteworthy that it showed a significant improvement in predicting gene expression correlations, especially for the P10_ST_rep3 section. Figure 5 In the Alex+10x dataset, mclSTExp showed the highest PCC on 7 out of 9 slices. However, for slice 1142243F, all methods had significantly lower PCC scores. This lower score may be attributed to various factors, including the weak correlation between the expression of specific genes and morphological features, the poor detection of some genes by the Visium platform resulting in difficulty in predicting their expression, and the potential impact of abiotic variations introduced during the experiment, which is not related to the image itself.

[0079] To further evaluate the gene expression predicted by mclSTExp, we explored whether the gene expression predicted by mclSTExp accurately reflects the actual status of tumor-related genes. For the HER2+ dataset, we visualized the top seven genes: GANS, FN1, FASN, HLA-B, SCD, IGKC, and HLA-DRA. Figure 6As shown in the figure, the PCCs of these genes predicted by mclSTExp are 0.840, 0.815, 0.780, 0.844, 0.808, 0.629, and 0.833, respectively, which are 11.1%, 18.6%, 18.5%, 4.7%, 28.4%, 16.2%, and 5.3% higher than those predicted by the second-place method Bleep, respectively. In particular, for the gene IGKC, its correlation coefficient with HisToGene is -0.234, while its correlation coefficient with STnet is -0.058.

[0080] In addition, it is noteworthy that all the top seven genes predicted by mclSTExp are closely related to breast cancer and play an important role in its pathogenesis and development. High expression of GANS can activate the PI3K / AKT / Snail1 / E-cadherin pathway, thereby promoting the proliferation, migration and invasion of breast cancer cells. FN1 is considered a potential therapeutic target or clinical prognostic marker for breast cancer because its high expression is closely related to the metastasis and malignant process of breast cancer. FASN is highly expressed in cancer stem cells, and its inhibition effectively inhibits the proliferation and survival of breast cancer cells. SCD is closely related to the proliferation, survival and invasion of breast cancer cells, highlighting its potential therapeutic target in breast cancer treatment strategies. IGKC has a significant value as a prognostic marker in breast cancer patients to predict disease progression and survival outcomes. In particular, among all the compared methods, mclSTExp is the first method to predict the genes HLA-B and HLA-DRA. On the one hand, human leukocyte antigen B (HLA-B) belongs to the major histocompatibility complex (MHC) class I molecule, which is mainly responsible for the presentation of endogenous peptides. One study indicated that HLA-B expression was associated with survival and recurrence in breast cancer patients. On the other hand, HLA-DRA is a typical MHC class II molecule expressed on professional antigen-presenting cells. Studies have shown that HLA-DRA plays an important role as a predictor of breast cancer prognosis. The expression level of HLA-DRA may represent a way to enhance high-level treatment of breast cancer and improve overall survival. Another study highlighted how cancer cells exploit various immune system functions to promote their growth. In conclusion, the mclSTExp method not only sheds light on cancer-specific overexpressed genes, but also identifies immune-related genes, providing valuable insights into cancer treatment.

[0081] The above describes the spatial gene expression level prediction method based on histological images in the embodiment of the present invention. The following describes the spatial gene expression level prediction device based on histological images in the embodiment of the present invention. Figure 7 In one embodiment of the present invention, a device for predicting spatial gene expression levels based on histological images includes:

[0082] An acquisition module 10 is used to acquire a pathological data set of a cancer patient, wherein the pathological data set includes a histological image, spatial transcriptome gene expression data corresponding to the histological image, and position coordinates of sequencing points;

[0083] A segmentation module 20, for segmenting the histological image into image blocks of a predetermined pixel size centered on each sequencing point according to the position coordinates of each sequencing point, and pairing the image blocks with the spatial transcriptome gene expression data of the corresponding sequencing point;

[0084] A data processing module 30 is used to select the common genes in all tissue sections and the top 1000 highly variable genes in each tissue section from the spatial transcriptome gene expression data, and normalize the gene expression values ​​in each sequencing point to obtain normalized gene expression data of the sequencing point;

[0085] A visual feature extraction module 40 is used to segment the histological image into image blocks and then input them into a pre-trained DenseNet-121 model to output visual features;

[0086] A fusion feature extraction module 50 is used to input the normalized gene expression data of the sequencing points and the position coordinates of the sequencing points into a sequencing point feature extraction model based on a multi-head attention mechanism, and output fusion features;

[0087] A space construction module 60 projects the visual features and the fusion features into the feature space through a projection layer to construct a multimodal embedding space;

[0088] The optimization module 70 is used to define, in the multimodal embedding space, that the sequencing points and image blocks at the same position form positive sample pairs, and that the sequencing points and image blocks at different positions form negative sample pairs, and to use a contrastive learning method to reduce the distance between the positive sample pairs and increase the distance between the negative sample pairs, thereby obtaining an optimized multimodal embedding space;

[0089] A calculation module 80 is used to calculate the cosine similarity between the visual features of the histological image to be tested and all the sequencing point features in the optimized multimodal embedding space, and find the top K sequencing point features with the highest similarity score;

[0090] The prediction module 90 is used to calculate the Euclidean distance between the visual features of the histological image to be tested and the features of the first K sequencing points in the optimized multimodal embedding space, and use the weighted distance of the gene expression values ​​of the first K sequencing points to predict the spatial gene expression level of the sequencing points.

[0091] above Figure 7The spatial gene expression level prediction device based on histological images in the embodiment of the present invention is described in detail from the perspective of modular functional entities. The spatial gene expression level prediction device based on histological images in the embodiment of the present invention is described in detail from the perspective of hardware processing.

[0092] Figure 8 1 is a schematic diagram of the structure of a spatial gene expression level prediction device based on histological images provided by an embodiment of the present invention. The spatial gene expression level prediction device 100 based on histological images may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 11 (for example, one or more processors) and a memory 12, and one or more storage media 13 (for example, one or more massive storage devices) storing application programs 133 or data 132. Among them, the memory 12 and the storage medium 13 can be short-term storage or permanent storage. The program stored in the storage medium 13 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the spatial gene expression level prediction device 100 based on histological images. Furthermore, the processor 11 can be configured to communicate with the storage medium 13 to execute a series of instruction operations in the storage medium 13 on the spatial gene expression level prediction device 100 based on histological images.

[0093] The spatial gene expression level prediction device 100 based on histological images may also include one or more power supplies 14, one or more wired or wireless network interfaces 15, one or more input and output interfaces 16, and / or one or more operating systems 131, such as Windows Serve, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will appreciate that Figure 8 The device structure shown does not constitute a limitation of the device 100 for predicting spatial gene expression levels based on histological images, and may include more or less components than those shown in the figure, or combine certain components, or arrange the components differently.

[0094] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. The computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the steps of a method for predicting spatial gene expression levels based on histological images.

[0095] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described system, device, or unit can refer to the corresponding process in the aforementioned method embodiment and will not be repeated here.

[0096] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program codes.

[0097] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for predicting spatial gene expression levels based on histological images, characterized in that: Includes steps: Acquire a pathological data set of a cancer patient, wherein the pathological data set includes a histological image, spatial transcriptome gene expression data corresponding to the histological image, and position coordinates of sequencing points; Segmenting the histological image into image blocks of a predetermined pixel size centered on each sequencing point according to the position coordinates of each sequencing point, and pairing the image blocks with the spatial transcriptome gene expression data of the corresponding sequencing point; Selecting common genes in all tissue sections and the top 1000 highly variable genes in each tissue section from the spatial transcriptome gene expression data, normalizing the gene expression values ​​in each sequencing point, and obtaining normalized gene expression data of the sequencing point; The histological images are segmented into image patches and then input into the pre-trained DenseNet-121 model to output visual features; The normalized gene expression data of the sequencing points and the location coordinates of the sequencing points are input into the sequencing point feature extraction model based on the multi-head attention mechanism, and the fusion features are output; Projecting the visual features and the fusion features into the feature space through the projection layer respectively to construct a multimodal embedding space; In the multimodal embedding space, the sequencing points and image blocks at the same position are defined to form positive sample pairs, and the sequencing points and image blocks at different positions are defined to form negative sample pairs. The contrastive learning method is used to reduce the distance between the positive sample pairs and increase the distance between the negative sample pairs, thereby obtaining an optimized multimodal embedding space. In the optimized multimodal embedding space, the cosine similarity between the visual features of the histological image to be tested and the fusion features of all sequencing points is calculated, and the fusion features of the first K sequencing points with the highest similarity scores are found; In the optimized multimodal embedding space, the Euclidean distance between the visual features of the histological image to be tested and the fusion features of the first K sequencing points is calculated, and a corresponding weight is designed according to the size of the Euclidean distance. The weight is multiplied by the real gene expression value of the first K sequencing points and the sum is taken as the spatial gene expression level of the histological image to be tested.

2. The method for predicting spatial gene expression levels based on histological images according to claim 1, characterized in that: The steps of normalizing the gene expression value in each sequencing point to obtain the normalized gene expression data of the sequencing point include: Normalization is performed by dividing the UMI count of each gene by the total UMI count of all genes in that sequencing point, and then multiplying by a scaling factor of 1,000,000. The normalization formula is: Among them, N ij represents the UMI count of the i-th gene at the j-th sequencing point, T j Represents the total UMI counts of all genes at the jth sequencing point; The normalized value is converted to the natural logarithmic scale, and the expression formula is: ij =log 10 (1+Normalized ij ).

3. The method for predicting spatial gene expression levels based on histological images according to claim 1, characterized in that: The normalized gene expression data of the sequencing points and the location coordinates of the sequencing points are input into the sequencing point feature extraction model based on the multi-head attention mechanism. The steps of outputting the fusion features include: The x-axis coordinates and y-axis coordinates of the sequencing points are encoded and converted into a one-hot encoding matrix P of size N×1. x and P y ; Use a learnable linear layer to encode the one-hot encoding matrix P x and P y Perform a linear transformation to obtain the corresponding x-coordinate encoding matrix S x and the y-coordinate encoding matrix S y ; The gene expression normalization data of the sequencing points and the x-coordinate encoding matrix S x and the y-coordinate encoding matrix S y After merging, they are input into the sequencing point feature extraction model based on the multi-head attention mechanism to output the fused features.

4. The method for predicting spatial gene expression levels based on histological images according to claim 1, characterized in that: In a multimodal embedding space, sequencing points and image blocks at the same position are defined to form positive sample pairs, and sequencing points and image blocks at different positions are defined to form negative sample pairs. A contrastive learning method is used to reduce the distance between the positive sample pairs and increase the distance between the negative sample pairs, thereby obtaining an optimized multimodal embedding space, including the following steps: It is defined that the sequencing points and image blocks at the same position form a positive sample pair, and the sequencing points and image blocks at different positions form a negative sample pair. A label matrix is ​​designed, in which the diagonal elements represent positive sample pairs and the non-diagonal elements represent negative sample pairs; The cosine similarity function cos_sim between the image patch and the sequencing point embedding is defined as follows: The cosine similarity function between the sequencing points and the image patch embedding is defined as follows: Among them, h spot is the fusion feature of the extracted sequencing points, is the fusion feature h spot The transpose of h patch is the visual feature of the extracted image patch, is the visual feature h patch The transpose of Use the cross entropy loss function to calculate two independent loss components loss image and loss spot ,in, Loss image =cross_entropy_loss(cos_sim(h patch ,h spot ),label), Loss spot =cross_entropy_loss(cos_sim(h spot ,h patch ),label) The overall loss of the DenseNet-121 model and the sequencing point feature extraction model is calculated as the average of these two losses: Loss=(Loss image +Loss spot ) / 2.0。 5. The method for predicting spatial gene expression levels based on histological images according to claim 1, characterized in that: In the optimized multimodal embedding space, by formula The Euclidean distance between the visual features of the histological image to be tested and the fusion features of the first K sequencing points is calculated, where u is the visual feature, v is the fusion feature of the first K sequencing points, and n is the dimension of the feature space.

6. The method for predicting spatial gene expression levels based on histological images according to claim 5, characterized in that: In the step of designing the corresponding weight according to the size of the Euclidean distance, the weight is defined as: The formula for the spatial gene expression level of the histological image to be tested is: in, Indicates sequencing point u j The actual gene expression level.

7. A spatial gene expression level prediction device based on histological images, characterized in that: include: An acquisition module, used to acquire a pathological data set of a cancer patient, wherein the pathological data set includes a histological image, spatial transcriptome gene expression data corresponding to the histological image, and position coordinates of sequencing points; A segmentation module, for segmenting the histological image into image blocks of a predetermined pixel size centered on each sequencing point according to the position coordinates of each sequencing point, and pairing the image blocks with the spatial transcriptome gene expression data of the corresponding sequencing point; A data processing module, used to select the common genes in all tissue sections and the top 1000 highly variable genes in each tissue section from the spatial transcriptome gene expression data, normalize the gene expression values ​​in each sequencing point, and obtain the normalized gene expression data of the sequencing point; The visual feature extraction module is used to segment the histological image into image blocks and then input them into the pre-trained DenseNet-121 model to output visual features; A fusion feature extraction module is used to input the normalized gene expression data of the sequencing points and the location coordinates of the sequencing points into the sequencing point feature extraction model based on the multi-head attention mechanism, and output the fusion features; A space construction module projects the visual features and the fusion features into the feature space through the projection layer to construct a multimodal embedding space; An optimization module is used to define, in a multimodal embedding space, sequencing points and image blocks at the same position to form positive sample pairs, and sequencing points and image blocks at different positions to form negative sample pairs, and to use a contrastive learning method to reduce the distance between the positive sample pairs and increase the distance between the negative sample pairs, thereby obtaining an optimized multimodal embedding space; A calculation module, used for calculating the cosine similarity between the visual features of the histological image to be tested and all the sequencing point features in the optimized multimodal embedding space, and finding the top K sequencing point features with the highest similarity scores; The prediction module is used to calculate the Euclidean distance between the visual features of the histological image to be tested and the features of the first K sequencing points in the optimized multimodal embedding space, and use the weighted distance of the gene expression values ​​of the first K sequencing points to predict the spatial gene expression level of the sequencing points.

8. A spatial gene expression level prediction device based on histological images, characterized in that: comprising a memory and at least one processor, wherein the memory has computer-readable instructions stored therein; The at least one processor calls the computer-readable instructions in the memory to execute the various steps of the method for predicting spatial gene expression levels based on histological images as claimed in any one of claims 1 to 6.

9. A computer-readable storage medium having computer-readable instructions stored thereon, characterized in that: When the computer-readable instructions are executed by a processor, the steps of the method for predicting spatial gene expression levels based on histological images as described in any one of claims 1 to 6 are implemented.

Images