A pathological image and virtual molecule information fusion analysis method based on bidirectional selection

Abstract

The application discloses a pathological image and virtual molecular information fusion analysis method based on bidirectional selection, and steps are as follows: 1, dividing whole slice pathological images into tissue blocks, and extracting spatially paired pathological features and virtual molecular features; 2, constructing a pathological concept reference library and a molecular concept reference library according to soft allocation weights of tissue phenotype concepts; 3, taking the pathological features as a query, obtaining a H2M molecular credibility selection score from pathology to molecules, and adjusting the injection intensity of virtual molecular residuals to the pathological features according to the score; 4, taking the virtual molecular features as a query, obtaining a M2H morphological support selection score from molecules to pathology, and taking the score as a multi-instance learning aggregated attention prior; 5, training a model by combining a classification task loss and a concept-level cross-modal alignment loss, aggregating and fusing block features, and outputting a class prediction result of the whole slice pathological image. The application takes the pathological features as a fusion anchor point, controls the injection intensity and action area of virtual molecular information through bidirectional selection, reduces the influence of virtual molecular prediction noise on pathological image features and slice-level aggregation, and realizes accurate classification of pathological images under the condition of lacking actual measured spatial transcriptome data.

Description

Technical Field

[0001] This invention relates to a method for fusing and analyzing pathological images and virtual molecular information based on bidirectional selection, belonging to the fields of computer vision and medical image processing. It mainly involves whole-slice pathological image processing, cross-modal feature fusion, and multi-instance learning classification technology, which can be applied to scenarios such as tumor pathology auxiliary diagnosis. Background Technology

[0002] Whole-section pathological images record cell morphology, tissue structure, and spatial heterogeneity, making them an important data source for computational pathology analysis. Due to their billion-pixel resolution, current computational methods typically divide the whole-section pathological image into multiple tissue patches, then use a pre-trained pathology encoder to obtain patch features, and complete the slice-level classification through multi-instance learning. While models can utilize directly observable tissue morphology when using only pathological images, they struggle to explicitly utilize gene expression and pathway information related to tumor phenotype and clinical outcomes.

[0003] Current multimodal pathological analysis methods typically fuse whole-section pathological images with patient-level batch genomic or transcriptomic data. Patient-level batch molecular data compresses spatially heterogeneous tissue samples into single molecular vectors, lacking spatial correspondence with patch-level tissue regions in whole-section pathological images. Furthermore, acquiring additional molecular detection data increases cost and processing time, and routine clinical pathological images often lack paired molecular detection results.

[0004] Virtual spatial transcriptome prediction models can infer gene expression profiles corresponding to specific areas based on pathological images, thus providing patch-level molecular information without the need for additional molecular detection. However, virtual molecular prediction is susceptible to factors such as tissue morphology ambiguity, data distribution differences, and the lack of observable pathological morphological manifestations for some gene expression, leading to varying reliability of molecular predictions across different tissue regions. Directly splicing virtual molecular representations with pathological representations or using a symmetrical fusion method may cause prediction errors to propagate to the pathological morphology-based representation. Furthermore, existing methods typically do not distinguish between the injection intensity of virtual molecular information into patch-level representations and its impact on slice-level aggregation positions, making it difficult to simultaneously control the amount and area of ​​influence of molecular information.

[0005] In summary, there is an urgent need for a pathological image analysis method that can primarily use pathological morphological information, selectively fuse virtual molecular information based on the degree of local cross-modal consistency, and guide slice-level aggregation based on the morphological support of molecular information. This would improve the reliability of pathological image analysis and provide technical support for clinical pathological diagnosis and precision medicine. Summary of the Invention

[0006] This invention proposes a method for fusing pathological images and virtual molecular information based on bidirectional selection, aiming to solve the problems that the reliability of virtual spatial transcriptome prediction varies with tissue region and that direct fusion of virtual molecular information easily introduces noise and interferes with pathological characterization.

[0007] The specific steps of this invention are as follows:

[0008] S1, the whole-slice pathological image to be analyzed is divided into multiple tissue blocks; pathological features are extracted for each tissue block, and the virtual space transcriptome expression profile is predicted; virtual molecular features corresponding one-to-one with the pathological feature space are obtained based on the virtual space transcriptome expression profile.

[0009] S2, obtain the soft assignment weights of each tissue patch belonging to multiple tissue phenotypic concepts; according to the soft assignment weights, perform weighted aggregation on pathological representations and virtual molecular representations respectively, and construct a pathological concept reference library and a molecular concept reference library.

[0010] S3. Using the pathological representation of each tissue patch as the query, calculate the local compatibility between it and the spatially paired virtual molecular representation, and the phenotypic condition reference compatibility between it and the molecular concept reference library. Based on the difference between the two, obtain the pathology-to-molecular (H2M) molecular credibility selection score, and use the molecular credibility selection score to adjust the injection intensity of virtual molecular residuals into the pathological representation to obtain the fused patch representation.

[0011] S4. Using the virtual molecular representation of each tissue patch as the query, calculate the local morphological compatibility between it and the spatially paired pathological representation, and the phenotypic conditional reference compatibility between it and the pathological concept reference library. Based on the difference between the two, obtain the molecular-to-histopathology (M2H) morphological support selection score, and use the morphological support selection score as the attention prior for multi-instance learning aggregation.

[0012] S5. Determine the normalized attention weights of each tissue patch based on task-driven attention and the attention prior; use the normalized attention weights to perform weighted aggregation on the fused patch representation to obtain a slice-level fused representation; input the slice-level fused representation into the task classification head and output the category prediction result of the whole slice pathological image.

[0013] Specifically, step S1 further includes: extracting pathological features of each tissue patch using a pre-trained pathological encoder; predicting the gene expression profile of the same tissue patch using a virtual spatial transcriptome prediction model; converting the gene expression profile into gene-level characterization and pathway activity characterization respectively, and fusing them to obtain a virtual molecular characterization.

[0014] Step S2 further includes: calculating the similarity between tissue patches and the textual prompts for each tissue phenotype concept using a pre-trained pathological image-text encoder, and performing Softmax normalization along the tissue phenotype concept dimension to obtain soft assignment weights; the soft assignment weights represent the relative degree to which a tissue patch belongs to each tissue phenotype concept, taking values ​​from 0 to 1, and the sum of the soft assignment weights for each tissue phenotype concept corresponding to the same tissue patch is 1. The soft assignment weights are then used to perform weighted averaging on the pathological representations and virtual molecular representations within the current whole-slice pathological image to obtain the pathological concept reference state and molecular concept reference state for the corresponding tissue phenotype concept; all the pathological concept reference states constitute a pathological concept reference library, and all the molecular concept reference states constitute a molecular concept reference library.

[0015] Step S3 further includes: mapping pathological representations to pathological queries through a selector projection network, mapping spatially paired virtual molecular representations to molecular bonds, and mapping molecular concept reference states to molecular reference bonds; the selector space is a low-dimensional feature space used to calculate cross-modal compatibility. The local compatibility between pathological queries and spatially paired molecular bonds, and the concept reference compatibility between pathological queries and each molecular reference bond are calculated; phenotypic conditional reference retrieval weights are determined based on concept reference compatibility, selection temperature, and the soft-assignment weights of the corresponding tissue tiles, and the molecular reference compatibility is obtained by weighted summation of each concept reference compatibility. The H2M molecular credibility selection score is determined based on the difference between local compatibility and molecular reference compatibility; the pathological representation and virtual molecular representation are fused to obtain candidate multimodal representations, and the difference between the candidate multimodal representation and the pathological representation is used as the candidate virtual molecular residual; using the pathological representation as an anchor point, the injection intensity of the candidate virtual molecular residual is adjusted according to the H2M molecular credibility selection score to obtain the fused tile representation.

[0016] Step S4 further includes: mapping virtual molecular representations to molecular queries, spatially paired pathological representations to pathological keys, and pathological concept reference states to pathological reference keys via a selector projection network; calculating the local morphological compatibility between molecular queries and spatially paired pathological keys, and the morphological reference compatibility between molecular queries and each pathological reference key; determining the phenotypic conditional reference retrieval weight based on the morphological reference compatibility, selection temperature, and the soft assignment weight of the corresponding tissue patch, and weighting and summing the morphological reference compatibility to obtain the pathological reference compatibility; and determining the M2H morphological support selection score based on the difference between the local morphological compatibility and the pathological reference compatibility.

[0017] Step S5 further includes: calculating the task-driven attention log value supervised by the classification task based on the fused patch representation using a gated attention network; multiplying the logarithm of the M2H morphological support selection score by a preset attention prior strength coefficient and adding it to the task-driven attention log value to obtain the corrected attention log value; and performing a normalized exponential operation on the corrected attention log values ​​of each tissue patch within the same whole-slice pathological image to obtain the normalized attention weight.

[0018] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0019] (1) This invention uses pathological characterization as the fusion anchor point and adaptively adjusts the injection intensity of virtual molecular residuals based on the H2M molecule confidence score selection. Compared with the method of directly splicing or symmetrically fusing the two types of characterization, it can reduce the interference of regional virtual molecular prediction errors on pathological characterization.

[0020] (2) This invention uses M2H morphological support selection score to correct the multi-instance learning attention, so that the virtual molecular information that has obtained local morphological support participates in slice-level aggregation, while retaining the task-driven attention directly supervised by the classification task.

[0021] (3) This invention constructs a bimodal concept reference library that adapts to the current whole-slice pathological image based on soft assignment of tissue phenotype concepts, so that each tissue patch only needs to be associated with the current whole-slice pathological image. Each concept references the state interaction. When the number of organizational tiles is... and At that time, the computational complexity of reference retrieval can be reduced by pairwise matching of all map tiles. Reduce to .

[0022] (4) In the training phase, this invention employs concept-level cross-modal alignment to minimize the cosine distance between the pathological concept reference state and the molecular concept reference state of the same tissue phenotypic concept. Under the condition that the tile-level pathological representation and the virtual molecular representation are strictly equal one by one without the requirement of prediction noise, cross-modal consistency constraints of the same tissue phenotypic concept are achieved. Attached Figure Description

[0023] Figure 1 This is a flowchart of a method for fusing and analyzing pathological images and virtual molecular information based on bidirectional selection, according to the present invention.

[0024] Figure 2 This is a schematic diagram of the overall structure of the bidirectional selection frame of the present invention.

[0025] Figure 3 This is a schematic diagram illustrating the construction of the phenotypic condition concept reference library of this invention.

[0026] Figure 4This is a schematic diagram of the structure for selecting H2M molecule credibility and M2H morphology support in this invention.

[0027] Figure 5 This is a schematic diagram illustrating the training objectives and inference output of the model in this invention. Detailed Implementation

[0028] To make the technical solution of the present invention clearer and more explicit, the following description is provided in conjunction with the accompanying drawings and specific embodiments. The embodiments described are merely illustrative of the implementation of the present invention and do not constitute a limitation on the scope of protection of the present invention. Equivalent substitutions made to the encoder, projection network, number of tissue phenotypic concepts, and parameter values ​​without departing from the technical concept of the present invention should all fall within the scope of protection of the present invention.

[0029] Example 1: As Figure 1 and Figure 2 As shown, this embodiment uses whole-section pathological images to be analyzed. Input: Whole slide pathology image. Divided into A tile containing an organizational region ,in This is the tile index, with values ​​ranging from 1 to... Each tile Simultaneously, pathological representation and virtual molecular representation branches are input, with both branches processing the same patch coordinates to obtain spatially paired pathological and virtual molecular representations. After obtaining the bimodal representations of each tissue patch, a phenotypic conditional concept reference library is constructed based on the soft-assigned weights of tissue phenotypic concepts. Subsequently, H2M molecular credibility selection, residual fusion anchored by pathological representation, M2H morphological support selection, and multi-instance learning aggregation are performed sequentially to obtain slice-level fused representations. Finally, the slice-level fused representations are input into the task classification head, outputting the category prediction results of the whole-slice pathological image.

[0030] Example 2: In step S1, the pre-trained pathology encoder Receive the Organizational diagram and output pathological features: In this embodiment, the pre-trained pathological encoder uses the UNI model. Virtual spatial transcriptome prediction model. Receive the same tissue plot It also outputs a tile-level gene expression profile: In this embodiment, the virtual spatial transcriptome prediction model adopts the STPath model. This is because the pathological characterization branch and the virtual molecular characterization branch process the same tissue patch. Pathological features Plot-level gene expression prediction profile Having the same spatial location. Map-level gene expression profiles. Input pre-trained molecular encoder To obtain gene-level characterization: In this embodiment, the pre-trained molecular encoder uses the scGPT model. Simultaneously, a patch-level gene expression prediction profile is generated based on the MSigDB gene set and the KEGG pathway gene set. Gene set activity scoring was performed to obtain pathway activity characterization. The gene set activity score can be achieved using an enrichment scoring method based on preset gene membership relationships.

[0031] Pathological features Gene-level characterization and pathway activity characterization Mapped to modal-specific projection networks respectively Dimensional shared characterization space yields pathological characterization. Gene-level characterization and pathway activity characterization In this embodiment, Set it to 256. This is achieved by adjusting the parameters used to determine the mixing weights. and Perform Softmax normalization to obtain a mixed weighting of gene-level characterization and pathway activity characterization. and : ,in, , This indicates the Softmax normalization operation. and This represents the parameters used to determine the mixing weights. In this embodiment, and All values ​​are set to 0.5. Virtual molecular characterization is represented as... ,in, and They represent the first Gene-level characterization and pathway activity characterization of individual tissue tiles within a shared characterization space. The layer normalization operation represents the branching of molecular characterization. Indicates the first Virtual molecular characterization of tissue blocks.

[0032] Example 3: In step S2, as follows Figure 3As shown, a pre-trained pathological image-text encoder is used to calculate the similarity between each tissue patch and the corresponding cue text for the tissue phenotype concept. This embodiment uses the QuiltNet model and sets six tissue phenotype concepts: tumor, stroma, immune, necrosis, normal tissue, and background. For each tissue phenotype concept, one or more cue texts describing that tissue phenotype concept can be set, and the corresponding text embeddings are integrated. Let the... The first organizational block and the first The similarity between the corresponding prompt text embeddings of each organizational phenotypic concept The similarity of organizational phenotypic concepts is determined along the concept index. Perform Softmax normalization to obtain soft-assigned weights: ,in , Indicates the number of organizational phenotypic concepts. Indicates the first The organizational block corresponds to the first Soft-assigned weights for organizational phenotypic concepts Indicates along the concept index Perform a Softmax normalization operation. The soft weighting allows the same tissue patch to have multiple tissue phenotypic components simultaneously.

[0033] Calculate the conceptual quality of each organizational phenotypic concept based on soft-assigned weights: The pathological characterization and virtual molecular characterization were weighted and aggregated separately to obtain a pathological concept reference state. and molecular concept reference state .in, This indicates the number of tissue patches in the current whole-section pathological image. Indicates the first Conceptual quality of organizational phenotypic concepts and They represent the first Pathological characterization and virtual molecular characterization of individual tissue plots and They represent the first Pathological and molecular reference states for individual tissue phenotypic concepts. All pathological concept reference states. This constitutes a complete reference library of pathological concepts. A molecular concept reference library is constructed. Both the pathological concept reference library and the molecular concept reference library are dynamically built based on tissue patches within the current whole-section pathological image to adapt to the tissue composition of the current section. Each patch only needs to be associated with... The concept reference states are interacted with, and the number of operations for reference retrieval is... .

[0034] Example 4: In step S3, as Figure 4 As shown above, through trainable linear projection The first Pathological features of individual tissue patches Project to The selector space is defined, and L2 normalization is performed to obtain the pathological query. Through trainable linear projection Virtual molecular characterization was processed separately. and molecular concept reference state And perform L2 normalization to obtain spatially paired molecular bonds. and the Molecular reference bond of an organization's phenotypic concept In this embodiment, Set to 64. Pathological queries, molecular bonds, and molecular reference bonds are represented as follows: , , ,in, This represents the L2 norm normalization operation.

[0035] Calculate the local compatibility between pathological queries and spatially paired molecular bonds, and the conceptual reference compatibility between pathological queries and reference bonds of each molecule: , ,in, Indicates local compatibility. Indicates the first The first organizational block and the first Conceptual reference compatibility between individual molecular conceptual reference states. This involves considering conceptual reference compatibility and the selected temperature. and soft allocation weights Used for phenotypic condition retrieval, to obtain the phenotypic condition retrieval weight in the H2M direction: The concept reference compatibility is weighted according to the phenotypic conditions to obtain the molecular reference compatibility: Further calculations were performed to determine the difference between the local compatibility and the molecular reference compatibility: And the confidence selection score of H2M molecules is obtained through the Sigmoid function: ,in, This represents the Sigmoid function. Indicates the first Molecular credibility selection score for each tissue patch. A value >0.5 indicates that the virtual molecular representation of spatial pairing has higher cross-modal compatibility compared to the phenotypic conditional molecular concept reference state.

[0036] Pathological features With virtual molecular characterization Vector concatenation is performed, followed by linear mapping and layer normalization to construct candidate multimodal representations: And obtain the candidate virtual molecular residuals relative to the pathological characterization: The semicolon indicates vector concatenation. and These represent the weights and biases of the fused linear mapping, respectively. This represents the layer normalization operation for merging branches. This represents the candidate virtual molecule residual. The average confidence selection score for H2M molecules within the current whole-slice pathological image is calculated. Adjusting the residuals of candidate virtual molecules around the aforementioned average value yields a fused patch representation: ,in Representing the fused patch representation, This represents the residual intensity coefficient, used to control the overall injection intensity of candidate virtual molecule residuals. This represents the patch contrast coefficient, used to control the magnitude of residual variation in the confidence level of different tissue patches relative to the average confidence level of the current slice. In this embodiment, Set it to 0.5. Set it to 0.1.

[0037] Example 5: In step S4, as Figure 4 As shown in the lower section, virtual molecular representation is used as the query direction. This is achieved through trainable linear projection. The first Virtual molecular characterization of tissue plots Project onto the selector space and perform L2 normalization to obtain the molecular query: Through trainable linear projection Pathological features of spatial pairing were treated separately. and pathological concept reference status And perform L2 normalization to obtain pathological keys and pathological reference keys: , ,in, This indicates a normalized molecule query. Normalized pathological bonds representing spatial pairings, Indicates the first Normalized pathological reference keys for the concept of tissue phenotype.

[0038] Calculate the local morphological compatibility between molecular queries and spatially paired pathological bonds, and the morphological reference compatibility between molecular queries and each pathological reference bond: , Combining soft-assignment weights and select temperature Obtain the phenotypic retrieval weights in the M2H direction: The morphological reference compatibility is weighted according to the phenotypic conditions to obtain the pathological reference compatibility: Further calculations were performed to determine the difference between local morphological compatibility and pathological reference compatibility. The M2H morphological support selection score is obtained through the Sigmoid function. ,in This represents the Sigmoid function. Indicates the first The morphological support score for each tissue patch is selected. A higher score indicates that the spatially paired pathological patches provide stronger local morphological support for predicting molecular states relative to the phenotypic conditional pathological reference state. M2H does not modify the fused patch representation but instead provides an attention prior to subsequent multi-instance learning aggregation.

[0039] In step S5, the gated attention network performs the fusion patch representation. Computational task-driven attention logarithm: ,in, , , and Indicates the parameters of the gated attention network. and This represents the attention output mapping parameters. This represents element-wise multiplication. This represents the logarithmic value of task-driven attention. The logarithm of the M2H morphological support selection score is added as a soft prior to the logarithmic value of task-driven attention to obtain the corrected logarithmic value of attention: ,in, This represents the prior attention intensity coefficient. In this embodiment, Set to 0.1. Perform Softmax normalization on the logarithmic values ​​of the corrected attention for each tissue patch within the same whole-slice pathological image to obtain the normalized attention weights: The fused patch representations are weighted and aggregated based on normalized attention weights to obtain slice-level fused representations: ,in, Indicates the first Normalized attention weights for each organizational block. This represents the slice-level fusion characterization.

[0040] Finally, the slice-level fusion representation Input the task category header to get Logarithmic values ​​for each category: ,in, This represents a task classification head composed of a multi-layered sensing mechanism. Represents the logarithmic values ​​of each category. This indicates the number of categories. The category corresponding to the largest logarithmic value is taken as the category prediction result for the whole-slice pathological image. ,in, This indicates the first item output by the task classification header. Log-like values, This indicates the category prediction results for whole-section pathological images.

[0041] Example 6: As Figure 5 As shown, during the model training phase, the pathological concept reference state and the molecular concept reference state are constructed using the same soft-assigned weights. For each tissue phenotype concept, the cosine distance between the pathological concept reference state and the molecular concept reference state is calculated, and the corresponding concept weights are determined according to concept quality. The concept-level cross-modal alignment loss is expressed as: Among them, concept weight Represented as: , This represents the conceptual-level cross-modal alignment loss. and They represent the first Pathological and molecular reference states for tissue phenotypic concepts. Indicates the first Conceptual quality of organizational phenotypic concepts Indicates the first Quality weights of organizational phenotypic concepts Indicates the number of organizational phenotypic concepts. This represents the L2 norm.

[0042] The classification task uses cross-entropy loss, expressed as: ,in, This indicates the number of whole-slice pathological images in the batch. Indicates the number of categories. Indicates the first The whole-section pathological image belongs to the first The true label of the class, The model predicts the first... The whole-section pathological image belongs to the first The probability of the class. The overall training objective consists of the cross-entropy loss from the classification task and the concept-level cross-modal alignment loss: ,in, Indicates the overall training objective. This represents the concept-level cross-modal alignment loss weights. In this embodiment, Set it to 0.1.

[0043] The base model features of the pre-trained pathology encoder, virtual spatial transcriptome prediction model, pre-trained molecular encoder, and pre-trained pathology image-text encoder are frozen during downstream training. Modality-specific projection networks, linear projection networks in H2M and M2H selectors, fusion networks, gated attention networks, and task classification heads are involved in training.

[0044] One specific training setup is as follows: train for 15 epochs using the AdamW optimizer, with a base learning rate of 2× -4 Weight decay is set to -4 The batch size is set to one whole-slice pathological image, the upper limit of the gradient norm is set to 5, and the learning rate is reduced to a minimum using a cosine annealing strategy. -6 The above parameter settings are used to illustrate one possible implementation of the present invention and do not constitute a limitation on the scope of protection of the present invention.

[0045] Example 7: Selecting Temperature Used to adjust the concept retrieval weight, and the sensitivity to the difference between local compatibility and concept reference compatibility. When When the score is reduced, the selection score becomes more sensitive to differences in compatibility. As the size increases, the selection results become smoother. In this embodiment, Set to 0.2. Residual strength coefficient. Used to control the overall intensity of injecting candidate virtual molecular residuals into pathological characterization. Patch contrast factor This is used to control the magnitude of residual variation in the confidence level of different tissue patches relative to the average confidence level of the current whole-slice pathological image. Attention prior strength coefficient. This is used to control the degree of influence of the M2H morphological support selection score on task-driven attention. In this embodiment, , and All of these are fixed hyperparameters that do not participate in gradient updates, and are set to 0.5, 0.1, and 0.1 respectively.

[0046] The pre-trained pathological encoder is not limited to the UNI model; it only needs to be able to map pathological tissue patches to pathological features. The virtual spatial transcriptome prediction model is not limited to the STPath model; it only needs to be able to predict gene expression based on the spatial correspondence of the pathological tissue patches. The pre-trained molecular encoder is not limited to the scGPT model, and the pathological image-text encoder is not limited to the QuiltNet model. The alternative models used should maintain the spatial correspondence between pathological representations and virtual molecular representations, the bidirectional selection relationship between H2M and M2H, and the residual fusion relationship anchored by pathological representations in steps S1 to S5. The category prediction results can be pathological stage categories, molecular subtype categories, or other discrete categories that can be determined based on whole-slice pathological images. For different analysis tasks, the network structure and number of categories of the task classification head can be adjusted. However, this does not alter the collaborative processing between the phenotypic condition conceptual reference library, H2M molecule credibility selection, and M2H morphological support selection.

[0047] The above embodiments are only used to illustrate the technical solutions of the present invention and do not constitute a limitation on the scope of protection of the present invention. Equivalent substitutions, combinations, or modifications made by those skilled in the art without departing from the technical concept of the present invention should all be included within the scope of protection of the present invention.

Claims

1. A method for fusing and analyzing pathological images and virtual molecular information based on bidirectional selection, characterized in that, Includes the following steps: S1, the whole-slice pathological image to be analyzed is divided into multiple tissue blocks, pathological features are extracted for each tissue block and virtual space transcriptome expression profile is predicted, and virtual molecular features corresponding to the pathological feature space are obtained based on the virtual space transcriptome expression profile. S2, obtain soft-assignment weights for multiple tissue phenotypic concepts for each tissue patch, and perform weighted aggregation of pathological representation and virtual molecular representation according to the soft-assignment weights to construct a pathological concept reference library and a molecular concept reference library. S3, using the pathological representation of each tissue patch as the query, calculate the local compatibility between it and the spatially paired virtual molecular representation, and the phenotypic conditional reference compatibility between it and the molecular concept reference library. Based on the local compatibility and the phenotypic conditional reference compatibility, obtain the H2M molecular credibility selection score from pathology to molecule. Adjust the injection intensity of virtual molecular residuals into the pathological representation based on the molecular credibility selection score to obtain the fused patch representation. S4. Using the virtual molecular representation of each tissue block as the query, calculate the local morphological compatibility between it and the spatially paired pathological representation, and the phenotypic conditional reference compatibility between it and the pathological concept reference library. Based on the local morphological compatibility and the phenotypic conditional reference compatibility, obtain the M2H morphological support selection score from molecular to pathology. The aforementioned morphological support selects a score as an attention prior for multi-instance learning aggregation; S5. Determine the normalized attention weight of each tissue patch based on the task-driven attention and the attention prior, perform weighted aggregation on the fused patch representation to obtain the slice-level fused representation, and output the category prediction result of the whole slice pathological image based on the slice-level fused representation.

2. The method according to claim 1, characterized in that, Step S1 includes: encoding tissue patches to obtain pathological features using a pre-trained pathological encoder; predicting patch-level gene expression profiles based on the same tissue patch using a virtual spatial transcriptome prediction model; encoding the patch-level gene expression profiles to obtain gene-level representations using a pre-trained molecular encoder, and calculating pathway activity representations based on a preset gene set; projecting the pathological features, gene-level representations, and pathway activity representations to a shared representation space of the same dimension using a modality-specific projection network; determining the mixed weights of gene-level representations and pathway activity representations using a normalized exponential function, and performing weighted fusion and layer normalization on the two to obtain the virtual molecular representations.

3. The method according to claim 2, characterized in that, The virtual molecular characterization satisfies: ,in, and They represent the first Gene-level characterization and pathway activity characterization of individual tissue patches Indicates the first Virtual molecular characterization of tissue patches Presentation layer normalization operation, and This represents the corresponding mixed weights, and satisfies: , ,in, and This represents the parameters used to determine the mixing weights. This indicates the Softmax normalization operation.

4. The method according to claim 1, characterized in that, Step S2 includes: calculating the similarity between tissue patches and the text embeddings of each tissue phenotype concept cue using a pre-trained pathological image-text encoder, and performing a normalized exponential operation on the similarity along the tissue phenotype concept dimension to obtain soft-assigned weights; The organizational block belongs to the first The soft-assigned weights for each organizational phenotypic concept are: ,in, , Indicates the first The organizational block corresponds to the first Soft-assigned weights for organizational phenotypic concepts Indicates the similarity, Indicates the number of organizational phenotypic concepts. Indicates along the concept index Perform Softmax normalization; the tissue phenotype concept includes at least two of the following: tumor, stroma, immune, necrosis, normal tissue, and background.

5. The method according to claim 1 or 4, characterized in that, In step S2, the first The conceptual quality, pathological conceptual reference status, and molecular conceptual reference status of each tissue phenotype concept are as follows: , , ,in, This indicates the number of tissue patches in the current whole-section pathological image. Indicates the first Conceptual quality of organizational phenotypic concepts and They represent the first Pathological characterization and virtual molecular characterization of individual tissue plots and They represent the first Pathological and molecular reference states for individual tissue phenotypic concepts. All pathological concept reference states. This constitutes a complete reference library of pathological concepts. This constitutes a reference library of molecular concepts.

6. The method according to claim 1, characterized in that, Step S3 includes: projecting and normalizing pathological representations into pathological queries; projecting and normalizing spatially paired virtual molecular representations into molecular bonds; projecting and normalizing each molecular concept reference state into molecular reference bonds; calculating the local compatibility between the pathological query and the spatially paired molecular bonds, and calculating the conceptual reference compatibility between the pathological query and each molecular reference bond; determining the phenotypic condition retrieval weight by combining the conceptual reference compatibility and the soft allocation weight; weighting the conceptual reference compatibility according to the phenotypic condition retrieval weight to obtain the molecular reference compatibility; and calculating the H2M molecule credibility selection score based on the difference between the local compatibility and the molecular reference compatibility and the selection temperature, respectively: , , , , , , , ,in, This represents the L2 norm normalization operation; and The projection matrix for pathological queries and the projection matrix for molecular bonds are not represented separately. Indicates the first Pathological query for individual tissue plots Molecular bonds representing spatial pairing, Indicates the first Molecular reference bonds for the concept of an organization's phenotype; Indicates local compatibility. Indicates conceptual reference compatibility. This indicates the weight of the retrieval criteria. Indicates molecular reference compatibility. Indicates the selected temperature. This represents the Sigmoid function. The confidence score for H2M molecules is represented by the selection score.

7. The method according to claim 1 or 6, characterized in that, In step S3, the pathological characterization and the virtual molecular characterization are concatenated, and candidate multimodal characterizations are obtained through linear mapping and layer normalization: The semicolon indicates vector concatenation. and These represent the weights and biases of the fusion linear mapping, respectively. The difference between the candidate multimodal representation and the pathological representation is used as the candidate virtual molecular residual. The average score was selected based on the confidence level of H2M molecules within the whole-section pathological images. Adjusting the residuals of the candidate virtual molecules yields the fused patch representation: ,in, Represents the residual strength coefficient. This represents the contrast coefficient of the map tiles.

8. The method according to claim 1, characterized in that, Step S4 includes: projecting and normalizing virtual molecular representations into molecular queries; projecting and normalizing spatially paired pathological representations into pathological keys; and projecting and normalizing each pathological concept reference state into pathological reference keys; calculating the local morphological compatibility between the molecular queries and spatially paired pathological keys, and calculating the morphological reference compatibility between the molecular queries and each pathological reference key; determining phenotypic condition retrieval weights by combining the morphological reference compatibility and soft allocation weights; weighting the morphological reference compatibility according to the phenotypic condition retrieval weights to obtain the pathological reference compatibility; and calculating the M2H morphological support selection score based on the difference between the local morphological compatibility and the pathological reference compatibility and the selection temperature, as follows: , , , , , , , ,in, This represents the L2 norm normalization operation; and The molecular query projection matrix and the pathological bond projection matrix are represented separately. Indicates the first Molecular queries for tissue blocks Pathological bonds representing spatial pairings Indicates the first Pathological reference key for a tissue phenotypic concept; Indicates local morphological compatibility. Indicates morphological reference compatibility. This indicates the weight of the retrieval criteria. Indicates pathological reference compatibility. Indicates the selected temperature. This represents the Sigmoid function. This indicates that the M2H morphology supports the selection of fractions.

9. The method according to claim 1 or 8, characterized in that, Step S5 includes: calculating the task-driven attention log value based on the fused patch representation using a gated attention network. ,in, , , and Indicates the parameters of the gated attention network. and This represents the attention output mapping parameters. This represents element-wise multiplication. The task-driven attention logarithm is represented by the logarithm of the morphological support selection score, which is then added as an attention prior to the task-driven attention logarithm to obtain the corrected attention logarithm: ,in, This represents the prior intensity coefficient of attention; normalized exponential operations are performed on the corrected attention logarithmic values ​​of each tissue patch within the same whole-slice pathological image to obtain the normalized attention weights: The fused patch representations are weighted and aggregated based on normalized attention weights to obtain slice-level fused representations: ,in, Indicates the first Normalized attention weights for each organizational block. This represents the slice-level fusion characterization.

10. The method according to claim 1 or 9, characterized in that, In step S5, the slice-level fusion characterization is performed. Input the task category header to get Logarithmic values ​​for each category: ,in, This represents a task classification head composed of a multi-layered sensing mechanism. Represents the logarithmic values ​​of each category. This indicates the number of categories. The category corresponding to the largest logarithmic value is taken as the category prediction result for the whole-slice pathological image. ,in, This indicates the first item output by the task classification header. Log-value pairs, This indicates the category prediction results for whole-section pathological images.

11. The method according to claim 1, characterized in that, When training the bidirectional selection framework, a concept-level cross-modal alignment loss is constructed based on the pathological concept reference state and the molecular concept reference state of the same tissue phenotype concept: Among them, concept weight Represented as: , This represents the conceptual-level cross-modal alignment loss. and They represent the first Pathological and molecular reference states for tissue phenotypic concepts. Indicates the first Conceptual quality of organizational phenotypic concepts Indicates the first Quality weights of organizational phenotypic concepts Indicates the number of organizational phenotypic concepts. Let L2 represent the norm. The classification task uses cross-entropy loss, expressed as: ,in, This indicates the number of whole-slice pathological images in the batch. Indicates the number of categories. Indicates the first The whole-section pathological image belongs to the first The true label of the class, The model predicts the first... The whole-section pathological image belongs to the first The probability of the class. The overall training objective consists of the cross-entropy loss from the classification task and the concept-level cross-modal alignment loss: ,in, Indicates the overall training objective. This represents the concept-level cross-modal alignment loss weights.

Images