Multi-modal feature fusion and model training method, electronic device, and storage medium

Abstract

This application provides a method, electronic device, and storage medium for multimodal feature fusion and model training. The method includes: inputting acquired transcriptomic information and image information into a preset multimodal fusion model to obtain multimodal fusion features, including: inputting transcriptomic information into a first unimodal feature extractor to obtain transcriptomic features, and inputting image information into a second unimodal feature extractor to obtain image features; fusing the transcriptomic features and image features using an attention mechanism to form a feature vector; and encoding the feature vector and spatial location map into a first latent variable using an autoencoder of a graph generation model. This application can improve the fusion performance of multimodal features.

Description

Multimodal feature fusion and model training method, electronic device and storage medium Technical Field

[0001] The present application relates to the field of biomedical technology, and in particular to a multimodal feature fusion and model training method, electronic device, and storage medium. Background Art

[0002] Spatial transcriptomics is a technology used to study gene expression in tissues. Through spatial transcriptomics, transcriptome information, imaging information and spatial location can be obtained, which has aroused people's interest in using machine learning to conduct fusion research with multimodal data. In related technologies, on the one hand, directly using a multi-layer perceptron to splice multiple modal data and then directly perform modal fusion leads to problems such as large amount of calculation, data redundancy, and data noise, which affects the calculation time and has poor accuracy after fusion. On the other hand, the method of appending or dot multiplication after feature extraction lacks the ability to learn the complex nonlinear relationship between features and has poor fusion ability. At present, the methods of modal fusion using autoencoders or graph convolutional networks do not extract single modal information well or do not fully fuse multiple modal information, resulting in unsatisfactory multimodal fusion effects.

[0003] Summary of the Invention

[0004] The embodiments of the present application disclose a multimodal feature fusion and model training method, an electronic device, and a storage medium, which solve the technical problem of poor multimodal fusion effect in related technologies.

[0005] The present application provides a method for fusing multimodal features, the method comprising: obtaining transcriptome information, imagery information, and a spatial position map corresponding to a medical sample; inputting the transcriptome information and the imagery information into a preset multimodal fusion model to obtain a first latent variable corresponding to the medical sample, comprising: inputting the transcriptome information into a first unimodal feature extractor to obtain corresponding transcriptome features, and inputting the imagery information into a second unimodal feature extractor to obtain corresponding imagery features; performing an attention mechanism information fusion on the transcriptome features and the imagery features and then splicing them into a feature vector; and encoding the feature vector and the spatial position map into a first latent variable according to an autoencoder of a graph generation model.

[0006] In some optional embodiments of the present application, the preset multimodal fusion model is iteratively trained, including: obtaining a biological sample for training, the biological sample including transcriptome training data, image training data, and spatial position map training data; inputting the transcriptome training data into a first initial unimodal feature extractor to obtain a first feature corresponding to the transcriptome data; inputting the image training data into a second initial unimodal feature extractor to obtain a second feature corresponding to the image training data; performing attention mechanism information fusion on the first feature and the second feature and splicing them into a third feature; performing dimensionality reduction on the third feature and the spatial position map training data according to the autoencoder of the initial graph generation model, and calculating the mean vector and variance vector of the reduced-dimensional feature; calculating a second latent variable in the latent space based on the mean vector and the variance vector, decoding the second latent variable through the autodecoder of the initial graph generation model to obtain a reconstruction vector; calculating a reconstruction error between the third feature and the reconstruction vector, and a first KL loss of the second latent variable distributed in a Gaussian distribution; performing iterative training based on the reconstruction error and the first KL loss to complete the training of the preset multimodal fusion model.

[0007] In some optional embodiments of the present application, the iterative training based on the reconstruction error and the KL loss is performed to complete the training of the preset multimodal fusion model, including: if the number of iterations of the iterative training of the reconstruction error and the first KL loss reaches a preset iteration parameter, the preset multimodal fusion model is completed.

[0008] In some optional embodiments of the present application, after completing the training of the preset multimodal fusion model, the method further includes: verifying the prediction accuracy of the trained multimodal fusion model, including: obtaining a verification sample for verification and a verification label corresponding to the verification sample; inputting the verification sample into the multimodal fusion model to generate a prediction value; calculating a clustering index based on the prediction value and the verification label; if the clustering index is greater than or equal to the preset index, determining that the multimodal fusion model has achieved the prediction accuracy; if the clustering index is less than the preset index, adjusting the hyperparameters of the multimodal fusion model, and iteratively training the preset multimodal fusion model again.

[0009] In some optional embodiments of the present application, the method also includes: using a preset clustering algorithm to calculate the cluster center of the second latent variable; calculating the similarity between the cluster center and the second latent variable; obtaining the vector distribution between the cluster center and the second latent variable based on the similarity; calculating the second KL loss of the second latent variable based on the similarity and the vector distribution; performing iterative training based on the reconstruction error, the first KL loss, and the second KL loss to complete the training of the preset multimodal fusion model.

[0010] In some optional embodiments of the present application, the iterative training based on the reconstruction error, the first KL loss, and the second KL loss is performed to complete the training of the preset multimodal fusion model, including: if the number of iterations of the iterative training of the reconstruction error, the first KL loss and the second KL loss reaches a preset iteration parameter, the preset multimodal fusion model is completed.

[0011] In some optional embodiments of the present application, obtaining the transcriptome information, imaging information and spatial location map corresponding to the medical sample includes: processing the medical sample using spatiotemporal transcriptome technology to obtain the transcriptome information, the imaging information and the spatial location information; generating the spatial location map based on a preset radius threshold and the spatial location information; or generating the spatial location map based on the radius threshold, a preset number of adjacent points and the spatial location information.

[0012] In some optional embodiments of the present application, the method further includes: obtaining multiple detection areas corresponding to the medical sample; determining the spatial position corresponding to each detection area based on the spatial position information; and calculating the spatial distance between each detection area and other detection areas based on the center point of each detection area and its spatial position.

[0013] In some optional embodiments of the present application, the spatial position map is generated based on a preset radius threshold and the spatial position information, including: taking the detection area corresponding to the spatial distance less than or equal to the radius threshold as the first adjacent detection area; and generating the spatial position map based on the first adjacent detection area.

[0014] In some optional embodiments of the present application, the spatial position map is generated based on the radius threshold, the preset number of adjacent points and the spatial position information, including: based on the radius threshold, sorting the spatial distances between the detection areas in order from small to large; based on the sorting, selecting the detection area with the preset number of adjacent points as the second adjacent detection area; based on the second adjacent detection area, generating the spatial position map.

[0015] In some optional embodiments of the present application, the method further includes: using the first latent variable to perform at least one task of clustering and cell trajectory analysis.

[0016] The present application also provides a model training method, which includes: obtaining a biological sample for training, wherein the biological sample includes transcriptome training data, image training data, and spatial position map training data; inputting the transcriptome training data into a first initial unimodal feature extractor to obtain a first feature corresponding to the transcriptome data; inputting the image training data into a second initial unimodal feature extractor to obtain a second feature corresponding to the image training data; performing attention mechanism information fusion on the first feature and the second feature and splicing them into a third feature; according to the autoencoder of the initial graph generation model, reducing the dimension of the third feature and the spatial position map training data, and calculating the mean vector and variance vector of the reduced-dimensional feature; calculating the second latent variable of the latent space based on the mean vector and the variance vector, decoding the second latent variable through the autodecoder of the initial graph generation model to obtain a reconstruction vector; calculating the reconstruction error between the third feature and the reconstruction vector, and the first KL loss of the second latent variable distributed in a Gaussian distribution; performing iterative training based on the reconstruction error and the first KL loss to complete the training of the multimodal fusion model.

[0017] In some optional embodiments of the present application, the method further includes: obtaining transcriptome information, imaging information, and a spatial location map corresponding to the medical sample; inputting the transcriptome information and the imaging information into the multimodal fusion model to obtain a first latent variable corresponding to the medical sample.

[0018] The present application also provides an electronic device, which includes a processor and a memory, and the processor is used to implement the multimodal feature fusion method and model training method when executing the computer program stored in the memory.

[0019] The present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the multimodal feature fusion method and model training method.

[0020] In the multimodal feature fusion method of the present application, in the feature processing stage, transcriptome features are obtained by a first unimodal feature extractor and image features by a second unimodal feature extractor, which can improve the accuracy of the input features and reduce data noise. The transcriptome features and image features are fused with attention mechanism information and spliced ​​into a feature vector. The feature vector is used as the node feature of the graph generation model, and the spatial position graph is used as the adjacency matrix of the graph generation model. The multimodal fusion features output by the graph generation model contain spatial position information, fully integrating the multimodal features, improving the accuracy of multimodal feature fusion, and to a certain extent, improving the robustness of downstream tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] FIG1 is a schematic structural diagram of an electronic device provided in an embodiment of the present application.

[0022] FIG2 is a flowchart of a multimodal feature fusion method provided in an embodiment of the present application.

[0023] FIG3 is a schematic diagram of adjacent detection areas provided in an embodiment of the present application.

[0024] FIG4 is a schematic diagram of clustering of image features provided in an embodiment of the present application.

[0025] FIG5 is a schematic diagram of clustering of transcriptome features provided in an embodiment of the present application.

[0026] Figure 6 is a schematic diagram of the attention mechanism information fusion provided in an embodiment of the present application.

[0027] FIG7 is a schematic diagram of an application of a multimodal fusion model provided in another embodiment of the present application.

[0028] FIG8 is a flowchart of the training of the multimodal fusion model provided in an embodiment of the present application.

[0029] FIG9 is a schematic diagram of a loss function curve provided in an embodiment of the present application.

[0030] FIG10 is a schematic diagram of the training of the multimodal fusion model provided in an embodiment of the present application.

[0031] FIG11 is a schematic diagram of training a multimodal fusion model provided in another embodiment of the present application.

[0032] FIG12 is a schematic diagram of the predicted values ​​provided in an embodiment of the present application.

[0033] FIG13 is a schematic diagram of a verification label provided in an embodiment of the present application.

[0034] FIG14 is a schematic diagram of the results of the multimodal fusion model and the single modal model provided in an embodiment of the present application.

[0035] FIG15 is a schematic diagram of the overall process of the multimodal feature fusion method provided in an embodiment of the present application.

[0036] FIG16 is an overall flow chart of the multimodal feature fusion method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0037] To facilitate understanding, some illustrations of concepts related to the embodiments of the present application are given for reference.

[0038] It should be noted that in this application, "at least one" means one or more, and "more than one" means two or more than two. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The terms "first", "second", "third", "fourth", etc. (if any) in the specification, claims and drawings of this application are used to distinguish similar objects, rather than to describe a specific order or sequence.

[0039] Spatial transcriptomics is a technology used to study gene expression in tissues. It enables high-throughput transcriptome analysis of organ tissue sections in situ at high resolution and subcellular resolution. It exhibits high sensitivity and uniform capture rates, and the analysis area can be expanded to the centimeter level, providing a panoramic field of view. Furthermore, spatial transcriptomics can generate H&E-stained images of corresponding tissue sections, providing a richer dimension of information beyond transcriptome data and spatial location. Spatial transcriptomics can provide transcriptome information, imaging information, and spatial location, which has sparked interest in using machine learning to integrate multimodal data. By integrating and analyzing this complementary information, it provides opportunities for more precise patient stratification and clinical decision support.

[0040] In related technologies, on the one hand, when processing multimodal data, feature extraction is not performed. Instead, multi-layer perceptrons are directly used to splice multiple modal data and then perform modal fusion. This leads to problems such as large amount of calculation, data redundancy, and high data noise, which affects the calculation time and has poor accuracy after fusion. On the other hand, the method of appending or dot multiplication after feature extraction can achieve a preliminary modal fusion effect on de-redundant data, but this type of method lacks the ability to learn complex nonlinear relationships between features and has poor fusion capabilities. At present, methods that use autoencoders or graph convolutional networks for modal fusion do not extract single modal information well or do not fully fuse multiple modal information, resulting in unsatisfactory multimodal fusion effects.

[0041] In order to solve the technical problem in the related art that the complex nonlinear relationship between features is not learned, resulting in poor multimodal fusion effect, the embodiments of the present application provide a multimodal feature fusion and model training method, electronic device and storage medium. The structure of the electronic device to which the multimodal feature fusion method of the present application is applied is first described below.

[0042] Figure 1 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The multimodal feature fusion method provided in an embodiment of the present application is applied to an electronic device 10, which can be a mobile phone, tablet computer, laptop computer, netbook, server, or other device. The embodiment of the present application does not impose any restrictions on the specific type of electronic device.

[0043] As shown in Figure 1, the electronic device 10 may include a communication module 101, a memory 102, a processor 103, an input / output (I / O) interface 104, and a bus 105. The processor 103 is coupled to the communication interface 101, the memory 102, and the I / O interface 104 via the bus 105.

[0044] The communication module 101 may include a wired communication module and / or a wireless communication module. The wired communication module may provide one or more wired communication solutions such as Universal Serial Bus (USB) and Controller Area Network (CAN). The wireless communication module may provide one or more wireless communication solutions such as Wireless Fidelity (Wi-Fi), Bluetooth (BT), mobile communication network, Frequency Modulation (FM), Near Field Communication (NFC), and Infrared (IR).

[0045] The memory 102 may include one or more random access memories (RAMs) and one or more non-volatile memories (NVMs). The RAM can be directly read and written by the processor 103 and can be used to store executable programs (e.g., machine instructions) of the operating system or other running programs, as well as user and application data. The RAM may include static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), etc.

[0046] The non-volatile memory can also store executable programs and user and application data, etc., which can be pre-loaded into the random access memory for direct reading and writing by the processor 103. The non-volatile memory can include disk storage devices and flash memory.

[0047] The memory 102 is configured to store one or more computer programs. The one or more computer programs are configured to be executed by the processor 103. The one or more computer programs include multiple instructions. When the multiple instructions are executed by the processor 103, the multimodal feature fusion method executed on the electronic device 10 can be implemented.

[0048] In other embodiments, the electronic device 10 further includes an external memory interface for connecting to an external memory to expand the storage capacity of the electronic device 10 .

[0049] The processor 103 may include one or more processing units. For example, the processor 103 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processor (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0050] The processor 103 provides computing and control capabilities. For example, the processor 103 is used to execute the computer program stored in the memory 102 to implement the above-mentioned multimodal feature fusion method.

[0051] The I / O interface 104 is used to provide a channel for user input or output. For example, the I / O interface 104 can be used to connect various input and output devices, such as a mouse, keyboard, touch device, display screen, etc., so that the user can enter information or visualize information.

[0052] The bus 105 is at least used to provide a channel for mutual communication among the communication module 101 , the memory 102 , the processor 103 , and the I / O interface 104 in the electronic device 10 .

[0053] It should be understood that the structures illustrated in the embodiments of the present application do not constitute a specific limitation on the electronic device 10. In other embodiments of the present application, the electronic device 10 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0054] Please refer to Figure 2, which is a flowchart of a multimodal feature fusion method provided in an embodiment of the present application, which is applied in an electronic device (e.g., electronic device 10 of Figure 1). Depending on different requirements, the order of the steps in the flowchart can be changed, and some steps can be omitted.

[0055] Step S201: Acquire transcriptome information, image information, and spatial location map corresponding to a medical sample.

[0056] In some embodiments of the present application, the medical sample can be human organ tissue, whole blood, plasma, serum, biological fluid or processed biological sample. By studying the medical samples, medical practitioners or researchers can effectively discover cancerous cells that are potentially or have already become cancerous, thereby promoting the development of medicine.

[0057] In one example, lung cancer is a malignant tumor. The clinical manifestations of lung cancer are complex and diverse, and the manifestations of early lung cancer are hidden and diverse. The clinical characteristics and signs are not obvious, and only manifest as non-specific symptoms such as cough, sputum, chest pain, fever, and physical decline. If only observing the early symptoms, the cause cannot be accurately determined. Therefore, serum biomarkers (biological samples) can be extracted to detect the cause and the direction of disease development, and the composition of cells can be accurately analyzed. Biological samples have the advantages of less traumatic development, simple operation, wide screening range and high detection efficiency.

[0058] In some embodiments of the present application, spatial transcriptomics technology is a technology used to study gene expression in tissues, which can detect medical samples, thereby obtaining transcriptome information, imaging information and spatial position information generated by spatial transcriptomics technology, wherein the imaging information can be H&E image data.

[0059] Transcriptome information includes information such as gene expression levels, transcript types, and abundance. This transcriptome information can be used to analyze differences and dynamic changes in gene expression under different temporal and spatial conditions. Imagery information refers to spatial transcriptomics technology that combines microscopic imaging and sequencing technologies to obtain spatial distribution information of medical samples. Imagery information can be used to study the position and morphology of cells in space, as well as the interactions between cells. Spatial location information includes the location information of each observation point, which allows researchers to understand the distribution of different cells in medical samples and the relative positions of cells.

[0060] In some embodiments of the present application, spatial transcriptomics technology typically uses a detection region (bin) as the basic unit of data analysis. Each detection region represents a fixed-sized spatial region, and the expression levels of DNA nanospheres (DNBs) in this spatial region are accumulated to obtain gene expression information for that region. The detection region is displayed as pixels on the gene expression heat map.

[0061] In some embodiments of the present application, H&E image data and transcriptome gene data corresponding to the biological sample are obtained, wherein the H&E image data is an image obtained by staining the chromatin in the cell nucleus and the nucleic acid in the cytoplasm of the biological sample. Based on bin as the unit for analysis, the H&E image data can be divided into multiple sub-images as image information based on physical space, and the transcriptome gene data is divided into multiple transcriptome gene sub-data as transcriptome information based on spatial position information. The gene expression levels of approximately 10,000 genes that can be obtained by sequencing data (reads) are then a bin corresponding to a 10,000-dimensional vector, corresponding to the gene expression levels of 10,000 genes. A bin corresponds to a transcriptome information and image information. Wherein, the transcriptome information corresponding to a bin can be a 10,000-dimensional vector, which corresponds to the gene expression levels of 10,000 genes. The transcriptome aggregates the reads within a bin range by the bin center point, and the gene expression levels of a bin in each gene can be obtained. The transcriptome information corresponding to multiple bins can form a matrix, where the rows of the matrix correspond to the bins and the columns correspond to the genes (types), and the elements of the matrix are the gene expression levels of a gene corresponding to a bin.

[0062] In some embodiments of the present application, after the detection areas are determined, the spatial position corresponding to each detection area can be determined based on the spatial position information, and the spatial distance between each detection area and the other detection areas can be calculated based on the center point of each detection area and the spatial position of each detection area. The detection areas corresponding to the spatial distances less than or equal to the preset radius threshold are regarded as the first adjacent detection areas, and the number of all connected detection areas and the average number of connections per detection area are counted to obtain the connection data of the first adjacent detection areas, thereby generating a spatial position map based on the first adjacent detection areas. The spatial position map can be a K-Nearest Neighbor (KNN) map.

[0063] In other embodiments of the present application, the method for generating the spatial location map may further include: sorting the spatial distances between the detection areas from smallest to largest based on a radius threshold, selecting a detection area with a preset number of adjacent points as a second adjacent detection area based on the sorting, and generating the spatial location map based on the second adjacent detection area.

[0064] In one example, the number of adjacent points is preset to 2. Assuming that the number of first adjacent detection areas determined based on the radius threshold is 4, the four detection areas are sorted in ascending order based on spatial distance. Based on the sorting, the first two detection areas are selected and used as the second adjacent detection areas. In this way, a spatial position map can be generated based on the second adjacent detection areas.

[0065] Figure 3 is a schematic diagram of adjacent detection areas provided in an embodiment of the present application. As shown in Figure 3, the horizontal axis represents the number of adjacent detection areas of each detection area, and the vertical axis represents the percentage of the number of adjacent detection areas of each detection area. As shown in Figure 3, according to statistics, 97% of the detection areas have 4 adjacent detection areas, and 2% of the detection areas have 2 adjacent detection areas. For example, the adjacent detection areas of 26,732 detection areas are counted, and the number of edges formed by the detection areas with connection relationships is 105,642, and the mean Mean is equal to 3.95, that is, each detection area has approximately 4 adjacent detection areas (Number of Neighbors).

[0066] Step S202: input the transcriptome information and the image information into a preset multimodal fusion model to obtain a first latent variable corresponding to the medical sample.

[0067] In some embodiments of the present application, the multimodal fusion model includes two unimodal feature extractors (for example, a first unimodal feature extractor and a second unimodal feature extractor), an attention mechanism, and a graph generation model, wherein the graph generation model includes an autoencoder and a decoder (for training), and the graph generation model can be any one of the graph neural networks including Graph Attention Networks (GAT), Graph Convolutional Neural Networks (GCN), and Graph Sampling Neural Networks (GraphSAGE).

[0068] In some embodiments of the present application, the transcriptome information is input into the first unimodal feature extractor, and the transcriptome highly variable genes, the genes with the largest variance, the differential genes in the transcriptome information can be extracted, and principal component analysis (PCA) extracts low-dimensional and main features as transcriptome features. For example, a 1000-dimensional bin (containing transcriptome information) is input into the first unimodal feature extractor to obtain a 2048-dimensional transcriptome feature.

[0069] In some embodiments of the present application, the image information includes multiple blocks (patches) after image preprocessing, which can also be called sub-regions, that is, H&E is cut into multiple patches according to the center point of the bin to obtain the sub-region corresponding to the bin. Each patch contains a certain number of pixels, and the purpose of using the patch is to extract local features from the image. The image information is input into the second unimodal feature extractor, that is, each patch is input into the second unimodal feature extractor, and feature extraction is performed on each patch through a convolutional neural network (CNN) or a Vision Transformer (VIT) to obtain image features. In one example, the patch corresponding to the bin (containing image information) is input into the second unimodal feature extractor to perform dimensionality reduction of the gene expression vector to obtain image features of 2048 dimensions. VIT is a Transformer-based model for processing image data.

[0070] FIG4 is a schematic diagram of clustering the image features provided in the embodiment of the present application, as shown in FIG4 , and the clustering results of the image features can be displayed in the spatial in situ map. FIG5 is a schematic diagram of clustering the transcriptome features provided in the embodiment of the present application, as shown in FIG5 , and the clustering results of the transcriptome features can be displayed in the spatial in situ map.

[0071] In some embodiments of the present application, the transcriptome feature extracted belongs to the same dimension (such as, dimension is 2048) with the iconography feature, and the transcriptome feature and iconography feature are carried out attention mechanism information fusion, and the number of features with the same output dimension is output.Wherein, when carrying out attention mechanism fusion, transcriptome feature and iconography feature can be referenced to each other and perform the mutual attention (cross-attention) in the attention mechanism, it is also possible to select any one feature as guidance from transcriptome feature and iconography feature, perform the collaborative attention (co-attention) in the attention mechanism, it is also possible to perform the self-attention (self-attention) in the attention mechanism to transcriptome feature and iconography feature.The application is not limited to the mode that transcriptome feature and iconography feature carry out attention mechanism information fusion, and can be a kind of in mutual attention, collaborative attention and self-attention.The transcriptome feature and iconography feature after attention mechanism information fusion are spliced ​​to obtain feature vector.

[0072] In one example, mutual attention is used as an example for explanation. Figure 6 is a schematic diagram of the attention mechanism information fusion provided by an embodiment of the present application. As shown in Figure 6, the principle of mutual attention is: the Q matrix of one modality (modality 1) and the K matrix of another modality (modality 2) are calculated as the inner product to represent the similarity matrix, and then the similarity matrix of one modality and the V matrix of the other modality are inner-producted to obtain the fusion matrices corresponding to modality 1 and modality 2 respectively.

[0073] In this way, based on the principle of the mutual attention mechanism. Based on the query parameters (Wq1), key parameters (Wk1), and value parameters (Wv1) of the linear layer, the transcriptome features are linearly transformed to obtain the linearly transformed matrices Q1, K1, and V1. Based on the query parameters (Wq2), key parameters (Wk2), and value parameters (Wv2) of the linear layer, the image features are linearly transformed to obtain the linearly transformed matrices Q2, K2, and V2. The inner product of Q1 and K2 is calculated to obtain the first similarity matrix, and the inner product of the first similarity matrix and V2 is calculated to obtain the fused image features. The inner product of Q2 and K1 is calculated to obtain the second similarity matrix, and the inner product of the second similarity matrix and V1 is calculated to obtain the fused transcriptome features. The fused transcriptome features and the fused image features are spliced ​​together to obtain the feature vector.

[0074] In some embodiments of the present application, the autoencoder of the graph generative model can be a variational autoencoder. The feature vectors generated by feature concatenation are input into the graph generative model as node features of the graph generative model, and the spatial position graph is input into the graph generative model as the adjacency matrix of the graph generative model. The graph generative model uses the encoder to map the feature vectors and the spatial position graph to a latent space to obtain a first latent variable, which is a multimodal fusion feature.

[0075] In other embodiments of the present application, multimodal fusion features are used to perform at least one of clustering and cell trajectory analysis. Multimodal fusion features contain a fused feature representation of spatial location information and can be used to perform a variety of downstream tasks, such as clustering and cell trajectory analysis. Using multimodal fusion features as input to downstream tasks can improve the accuracy of downstream tasks.

[0076] In an embodiment of the present application, a multimodal feature fusion model is constructed by using a first unimodal feature extractor, a second unimodal feature extractor, an attention mechanism, and a graph generation model. This can solve the problems of unimodal feature noise and insufficient feature information, improve the fusion effect of multimodal features and the accuracy of downstream tasks, thereby improving the accuracy of tissue differentiation in medicine and accurately stratifying diseases. Among them, by preprocessing the features by the first unimodal feature extractor and the second unimodal feature extractor, it is possible to solve the technical problem in the related art that the multimodal fusion method without feature extraction directly causes a large amount of network calculation and data redundancy and noise, which affects the calculation time and has poor accuracy. The multimodal feature fusion model can solve the technical problem in the related art that the method of appending or dot multiplication after feature extraction can achieve a preliminary modal fusion effect on de-redundant data, but cannot learn the complex nonlinear relationship between features, resulting in poor fusion ability.

[0077] Figure 7 is an application diagram of a multimodal fusion model provided by another embodiment of the present application. As shown in Figure 7, after obtaining the transcriptome features (obtained through transcriptomics analysis) and the image features, they are fused through the attention mechanism (Attention module), and then spliced ​​to obtain a feature vector. The feature vector and the spatial position map (SNG) are input into the graph generation model (AGAE) to generate a first latent variable, which is a multimodal fusion feature.

[0078] FIG8 is a flowchart of the training of the multimodal fusion model provided in an embodiment of the present application, which includes the following steps:

[0079] Step S801: Obtain biological samples for training.

[0080] In some embodiments of the present application, biological samples used for training can be extracted from a historical database, such as lung cancer data. Before training, initial hyperparameters can be set for the model, such as an iteration parameter, which can be the number of iterations, which can be set to 100, a learning rate of 0.001, an optimizer of Adam, an activation function of ReLU, and three hidden layers, with the number of hidden layers being 2048, 128, and 50, respectively.

[0081] Step S802: input the transcriptome training data into a first initial unimodal feature extractor to obtain a first feature corresponding to the transcriptome data.

[0082] Step S803: Input the image training data into a second initial unimodal feature extractor to obtain a second feature corresponding to the image training data.

[0083] In step S804, the first feature and the second feature are fused using the attention mechanism information and then spliced ​​into a third feature.

[0084] In some embodiments of the present application, steps S801 to S804 can refer to the specific description of the embodiment shown in Figure 2. In order to distinguish the application process and the training process of the model, in the training process of steps S801 to S804, the first unimodal feature extractor shown in Figure 2 is used as the first initial unimodal feature extractor, and the second unimodal feature extractor is used as the second initial unimodal feature extractor.

[0085] Step S805 , based on the autoencoder of the initial graph generation model, the third feature and the spatial position graph training data are reduced in dimension, and the mean vector and variance vector of the features obtained after the dimension reduction are calculated.

[0086] Step S806 , calculating a second latent variable of the latent space based on the mean vector and the variance vector, decoding the second latent variable through the self-decoder of the initial graph generation model to obtain a reconstructed vector.

[0087] Step S807 , calculating a reconstruction error between the third feature and the reconstruction vector, and a first KL loss of the second latent variable distributed in a preset distribution.

[0088] The preset distribution may be a Gaussian distribution.

[0089] Step S808: Perform iterative training based on the reconstruction error and the first KL loss to complete the training of the preset multimodal fusion model.

[0090] In some embodiments of the present application, before model training begins, the iteration parameters required for training can be pre-set, for example, the number of iterations can be set to 100. When the number of iterative training reaches 100, training is stopped. The loss function value calculated during the training process can be the reconstruction loss of the graph structure and the first KL loss of the encoder.

[0091] In one example, FIG9 is a schematic diagram of a loss function curve provided in an embodiment of the present application. As shown in FIG9 , the horizontal axis represents the number of iterations, and the vertical axis represents the loss function value. The loss function value of each iteration in 100 iterations is calculated to form a loss function curve. If the loss function curve is smooth, it is determined that the trained multimodal fusion model is approaching a convergence state, which means that the training of the multimodal fusion model is completed. If the loss function curve is not smooth, the hyperparameters of the trained multimodal fusion model can be adjusted and retrained, wherein the adjustable hyperparameters may include, but are not limited to: iteration parameters, learning rate, optimizer, activation function, number of hidden layers, and number of neurons in the hidden layer.

[0092] Figure 10 is a training diagram of a multimodal fusion model provided in an embodiment of the present application, and Figure 11 is a training diagram of a multimodal fusion model provided in another embodiment of the present application. As shown in Figure 10, the transcriptome training data is processed using the first initial unimodal feature extractor to obtain the first feature, and the second initial unimodal feature extractor is used to obtain the second feature. After the first and second features pass through the attention mechanism module, they are spliced ​​into the third feature. The third feature and the spatial position graph training data (SNG) are input into the encoder to perform dimensionality reduction calculations, and the mean (Zmean) and variance (Zvar) as shown in Figure 11 are generated. Based on the mean vector and the variance vector, the second latent variable of the latent space (Latent representation) is calculated based on the mean vector and the variance vector. The second latent variable is decoded by the self-decoder of the initial graph generation model to obtain a reconstruction vector. The reconstruction error (Lreconstect) of the third feature (Xi) and the reconstruction vector is calculated, and the first KL loss (Llatent_kl) of the second latent variable distributed in the Gaussian distribution is obtained. The reconstruction error is reversely corrected W1, W2, W3 and W4. If the reconstruction error and the first KL loss meet a preset iteration stopping condition, for example, if the number of iterations of the reconstruction error and the first KL loss reaches a preset iteration parameter, training of the preset multimodal fusion model is determined to be complete. The DEC module in Figure 11 is optional; see the following description of adding DEC based on the embodiment shown in Figure 8.

[0093] In some embodiments of the present application, after determining that the training of the preset multimodal fusion model is completed, the prediction accuracy of the trained multimodal fusion model can be verified to ensure the fusion effect of the features output by the multimodal fusion model. A verification sample for verification and a sample label corresponding to the verification sample are obtained. The sample label can be labeled data processed by professionals. The verification sample is input into the multimodal fusion model to generate a predicted value. Based on the predicted value and the verification label, a clustering index is calculated. The clustering index can be ARI (Adjusted Rand Index), NMI (Normalized Mutual Information), etc.

[0094] In some embodiments of the present application, if the clustering index is greater than or equal to the preset index, it is determined that the multimodal fusion model has achieved the prediction accuracy. If the clustering index is less than the preset index, the hyperparameters of the multimodal fusion model are adjusted, and the multimodal fusion model is iteratively trained again to improve the accuracy of the multimodal fusion model. Figure 12 is a schematic diagram of the prediction value provided in the embodiment of the present application, and Figure 13 is a schematic diagram of the verification label provided in the embodiment of the present application. Figure 12 shows the result of tissue differentiation by the second prediction value, and Figure 13 shows the result of tissue differentiation by the verification label. By comparing Figures 12 and 13, it can be seen that the effectiveness of the multimodal fusion model can provide more effective analysis data for the disease.

[0095] Figure 14 is a schematic diagram of the results of the multimodal fusion model and the single modal model provided in an embodiment of the present application. As shown in Figure 14, the vertical axis represents the clustering index ARI, and the horizontal axis represents the multimodal fusion model (stereoMM) and the single modal model (single Img). The result output by the multimodal fusion model is better than the single modality in tissue differentiation, which illustrates the effectiveness of multimodal fusion.

[0096] In other embodiments of the present application, in order to improve the accuracy of the multimodal fusion model, the multimodal fusion model can also add a clustering algorithm for optimization training during the training process. For example, the clustering algorithm can be a deep embedded clustering algorithm (DEC). Based on the embodiment shown in Figure 8, DEC is added (specifically, the DEC module in Figure 11 is an optional module), specifically: DEC is used to calculate the cluster center of the second latent variable, the similarity between the cluster center and the second latent variable is calculated, and the vector distribution between the cluster center and the second latent variable is obtained based on the similarity. According to the similarity and the vector distribution, the second KL loss of the second latent variable is calculated. Iterative training is performed based on the reconstruction error, the first KL loss, and the second KL loss to complete the training of the preset multimodal fusion model. Among them, if the number of iterations of iterative training of the reconstruction error, the first KL loss, and the second KL loss reaches the preset iteration parameter, the preset multimodal fusion model is completed. Adding DEC can solve the problem of multi-slice joint analysis and achieve the effect of removing batch effects.

[0097] Figure 15 is the overall flow diagram of the fusion method of the multimodal feature that the present application embodiment provides, as shown in Figure 15, transcriptome information and iconography information are respectively input into unimodal feature extractor, obtain transcriptome feature and iconography feature, wherein, unimodal feature extractor includes the first unimodal feature extractor and the second unimodal feature extractor.After transcriptome feature and iconography feature are carried out attention mechanism information fusion, feature splicing is performed to obtain the feature vector after splicing. The feature vector after splicing and spatial position diagram are all input into graph generation model, extract latent variable feature (such as, the first latent variable). During training, latent variable feature can be the second latent variable and reconstruction vector, and optional DEC module can also be increased in training process to optimize feature representation, thus updating feature variable. The result of multimodal fusion model output can be verified by gold standard (clustering index) ARI, MNI etc., thus assessing the model effect of multimodal fusion model, such as, differential gene, cell communication, cell composition and copy number change etc.

[0098] FIG16 is an overall flow chart of the multimodal feature fusion method provided in an embodiment of the present application. As shown in FIG16 , the method includes the following steps:

[0099] Step S1601: Obtain biological samples for training.

[0100] Step S1602: input the transcriptome training data into a first initial unimodal feature extractor to obtain a first feature corresponding to the transcriptome data.

[0101] Step S1603: Input the image training data into a second initial unimodal feature extractor to obtain a second feature corresponding to the image training data.

[0102] In step S1604, the first feature and the second feature are fused using the attention mechanism information and then spliced ​​into a third feature.

[0103] Step S1605 , generating the autoencoder of the model based on the initial graph, performing dimensionality reduction on the third feature and the spatial position graph training data, and calculating the mean vector and variance vector of the features after dimensionality reduction.

[0104] Step S1606: Calculate a second latent variable in the latent space based on the mean vector and the variance vector, and decode the second latent variable through the self-decoder of the initial graph generation model to obtain a reconstructed vector.

[0105] Step S1607 , calculating the reconstruction error between the third feature and the reconstruction vector, and the first KL loss of the second latent variable distributed in the Gaussian distribution.

[0106] Step S1608: Perform iterative training based on the reconstruction error and the first KL loss to complete the training of the multimodal fusion model.

[0107] Step S1609: Obtain transcriptome information, image information, and spatial location map corresponding to the medical sample.

[0108] Step S1610: Input the transcriptome information and the image information into the trained multimodal fusion model to obtain the first latent variable corresponding to the medical sample.

[0109] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. The computer program includes program instructions. The method implemented when the program instructions are executed can refer to the methods in the above-mentioned embodiments of the present application.

[0110] The computer-readable storage medium may be an internal memory of the electronic device described in the above embodiment, such as a hard disk or memory of the electronic device. The computer-readable storage medium may also be an external storage device of the electronic device, such as a plug-in hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash memory card, etc. equipped on the electronic device.

[0111] In some embodiments, the computer-readable storage medium may include a program storage area and a data storage area, wherein the program storage area may store an operating system, applications required for at least one function, etc.; the data storage area may store data created according to the use of the electronic device, etc.

[0112] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0113] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0114] In the embodiments provided in this application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are merely illustrative. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0115] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0116] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A method for fusing multimodal features, characterized in that The method includes: Obtaining the transcriptome information, imaging information, and spatial location map corresponding to a medical sample; Inputting the transcriptome information and the imaging information into a preset multimodal fusion model to obtain a first latent variable corresponding to the medical sample, including: Inputting the transcriptome information into a first single-modal feature extractor to obtain corresponding transcriptome features, and inputting the imaging information into a second single-modal feature extractor to obtain corresponding imaging features; Performing attention mechanism information fusion on the transcriptome features and the imaging features and then splicing them into a feature vector; Encoding the feature vector and the spatial location map into a first latent variable according to the autoencoder of the graph generation model.

2. The fusion method of multi-modal features according to claim 1, characterized in that The method further includes: iteratively training the preset multimodal fusion model, including: Obtaining a biological sample for training, where the biological sample includes transcriptome training data, imaging training data, and spatial location map training data; Inputting the transcriptome training data into a first initial single-modal feature extractor to obtain a first feature corresponding to the transcriptome data; Inputting the imaging training data into a second initial single-modal feature extractor to obtain a second feature corresponding to the imaging training data; Performing attention mechanism information fusion on the first feature and the second feature and then splicing them into a third feature; Reducing the dimension of the third feature and the spatial location map training data according to the autoencoder of the initial graph generation model, and calculating the mean vector and variance vector of the feature after dimension reduction; Calculating a second latent variable in the latent space based on the mean vector and the variance vector, and decoding the second latent variable through the auto-decoder of the initial graph generation model to obtain a reconstructed vector; Calculating the reconstruction error between the third feature and the reconstructed vector, and the first KL loss of the distribution of the second latent variable in a preset distribution; Performing iterative training based on the reconstruction error and the first KL loss to complete the training of the preset multimodal fusion model.

3. The fusion method of multimodal features according to claim 2, wherein The performing iterative training based on the reconstruction error and the KL loss to complete the training of the preset multimodal fusion model includes: If the number of iterations of the iterative training of the reconstruction error and the first KL loss reaches a preset iteration parameter, completing the training of the preset multimodal fusion model.

4. The fusion method of multi-modal features according to claim 2, characterized in that After completing the training of the preset multimodal fusion model, the method further includes: verifying the prediction accuracy of the trained multimodal fusion model, including: Obtaining a verification sample for verification and a verification label corresponding to the verification sample; Inputting the verification sample into the multimodal fusion model to generate a predicted value; Calculating a clustering index according to the predicted value and the verification label; If the clustering index is greater than or equal to a preset index, determining that the multimodal fusion model reaches the prediction accuracy; If the clustering index is less than the preset index, adjusting the hyperparameters of the multimodal fusion model and performing iterative training on the preset multimodal fusion model again.

5. The fusion method of multi-modal features according to claim 2, wherein The method further includes: Calculating the clustering center of the second latent variable using a preset clustering algorithm; Calculating the similarity between the clustering center and the second latent variable; Obtain the vector distribution between the clustering center and the second latent variable based on the similarity; Calculate the second KL loss of the second latent variable according to the similarity and the vector distribution; Perform iterative training based on the reconstruction error, the first KL loss, and the second KL loss to complete the training of the preset multimodal fusion model.

6. The fusion method of multimodal features according to claim 5, wherein The performing iterative training based on the reconstruction error, the first KL loss, and the second KL loss to complete the training of the preset multimodal fusion model includes: If the number of iterations of the iterative training of the reconstruction error, the first KL loss, and the second KL loss reaches the preset iteration parameter, complete the preset multimodal fusion model.

7. The fusion method of multimodal features according to claim 1, wherein The obtaining the transcriptome information, the imaging information, and the spatial location map corresponding to the medical sample includes: Process the medical sample using spatio-temporal transcriptome technology to obtain the transcriptome information, the imaging information, and the spatial location information; Generate the spatial location map based on a preset radius threshold and the spatial location information; or, Generate the spatial location map based on the radius threshold, a preset number of adjacent points, and the spatial location information.

8. The fusion method of multimodal features according to claim 7, wherein The method further includes: Obtain a plurality of detection regions corresponding to the medical sample; Determine the spatial location corresponding to each detection region based on the spatial location information; Calculate the spatial distance between each detection region and other detection regions based on the center point of each detection region and the spatial location where it is located.

9. The fusion method of multi-modal features according to claim 7, characterized in that, The generating the spatial location map based on a preset radius threshold and the spatial location information includes: Take the detection regions corresponding to the spatial distances less than or equal to the radius threshold as the first adjacent detection regions; Generate the spatial location map based on the first adjacent detection regions.

10. The fusion method of multi-modal features according to claim 7, wherein The generating the spatial location map based on the radius threshold, a preset number of adjacent points, and the spatial location information includes: Sort the spatial distances between the detection regions from small to large based on the radius threshold; Select the detection regions with the preset number of adjacent points as the second adjacent detection regions based on the sorting; Generate the spatial location map based on the second adjacent detection regions.

11. The multimodal feature fusion method according to claim 1, wherein The method further includes: Perform at least one of the tasks of clustering and cell trajectory analysis using the first latent variable.

12. A model training method, characterized in that, The method includes: Obtain a biological sample for training, where the biological sample includes transcriptome training data, imaging training data, and spatial location map training data; Input the transcriptome training data into a first initial unimodal feature extractor to obtain the first feature corresponding to the transcriptome data; Input the imaging training data into a second initial unimodal feature extractor to obtain the corresponding second feature; Perform attention mechanism information fusion on the first feature and the second feature and then splice them into a third feature; According to the autoencoder of the initial graph generation model, perform dimensionality reduction on the third feature and the spatial location map training data, and calculate the mean vector and variance vector of the dimensionality-reduced features; Calculate the second latent variable of the latent space based on the mean vector and the variance vector, and decode the second latent variable through the auto-decoder of the initial graph generation model to obtain a reconstructed vector; Calculate the reconstruction error between the third feature and the reconstructed vector, and the first KL loss of the distribution of the second latent variable in the Gaussian distribution; Perform iterative training based on the reconstruction error and the first KL loss to complete the training of the multi-modal fusion model.

13. The model training method according to claim 12, wherein The method further includes: Obtain the transcriptome information, imaging information, and spatial location map corresponding to the medical sample; Input the transcriptome information and the imaging information into the multi-modal fusion model to obtain the first latent variable corresponding to the medical sample.

14. An electronic device, characterized in that, The electronic device includes a processor and a memory, and the processor is configured to execute a computer program stored in the memory to implement the multi-modal feature fusion method according to any one of claims 1 to 11 and the model training method according to any one of claims 12 to 13.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one instruction, and when the at least one instruction is executed by a processor, it implements the multi-modal feature fusion method according to any one of claims 1 to 11 and the model training method according to any one of claims 12 to 13.

Images